Can I read ATTENTION2D: Communication Efficient Distributed Self-Attention Mechanism on EtoBox?

ATTENTION2D: Communication Efficient Distributed Self-Attention Mechanism by Elango, Venmugil is a scholarly article available to read on EtoBox.

What is ATTENTION2D: Communication Efficient Distributed Self-Attention Mechanism about?

Transformer-based models have emerged as a leading architecture for natural language processing, natural language generation, and image generation tasks. A fundamental element of the transformer architecture is self-attention, which allows the model to capture intricate dependencies within the data. However, the self-attention mechanism also incurs significant computational and memory costs, particularly for long sequences. In this paper, we introduce ATTENTION2D, a novel approach that exploits parallelism along two dimensions - query and key/value - of the self-attention operation. This method enables efficient distribution and parallelization of computations across multiple devices. Our approach facilitates asymptotically faster training and inference phases compared to previous methods, without relying on approximations or incurring additional computational or memory overheads. Furthermore, unlike existing techniques that struggle to scale with an increasing number of processing units, our approach effectively scales with additional processing units. Our experimental results confirm the effectiveness of our method in improving communication efficiency and scalability. Compared t

Author

Elango, Venmugil

Published

2025

Language

EN