Can I read Online Softmax for FlashAttention on EtoBox?

Online Softmax for FlashAttention by Thiyagarajan Palaniyappan is a document available to read on EtoBox.

What is Online Softmax for FlashAttention about?

The document discusses the implementation of Online Softmax in FlashAttention, focusing on improving numerical stability in the softmax computation. It introduces a parallelizable approach to compute the necessary components without storing intermediate results, allowing for efficient GPU execution. The method involves constructing a surrogate variable, δ, that can be computed inductively while adhering to specific properties for parallel processing.

Author

Thiyagarajan Palaniyappan

Language

EN