Can I read A2SF: Accumulative Attention Scoring with Forgetting Factor for Token Pruning in Transformer Decoder on EtoBox?

A2SF: Accumulative Attention Scoring with Forgetting Factor for Token Pruning in Transformer Decoder by Jo, Hyun-rae; Shin, Dongkun is a scholarly article available to read on EtoBox.

What is A2SF: Accumulative Attention Scoring with Forgetting Factor for Token Pruning in Transformer Decoder about?

Recently, large language models (LLM) based on transformers are facing memory bottleneck issues due to KV cache, especially in long sequence handling. Previous researches proposed KV cache compression techniques that identify insignificant tokens based on Accumulative Attention Scores and removes their items from KV cache, noting that only few tokens play an important role in attention operations. However, we have observed that the existing Accumulative Attention Score is not suitable for the transformer decoder structure. In the decoder model, the number of times the Attention Score accumulates varies depending on the order of token appearance due to the effect of masking, causing an uneven comparison between tokens. To solve this, we propose Accumulative Attention Score with Forgetting Factor (A2SF) technique, which introduces a Forgetting Factor in the Attention Score accumulation process. A2SF applies a penalty to the past Attention Score generated from old tokens by repeatedly multiplying the Forgetting Factor to the Attention Score over time. Therefore, older tokens receive a larger penalty, providing fairness among different ages of tokens. Through the fair comparison among

Author

Jo, Hyun-rae; Shin, Dongkun

Published

2024

Language

EN