Can I read KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches on EtoBox?

KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches by Yuan, Jiayi; Liu, Hongyi; Zhong, Shaochen; Chuang, Yu-Neng; Li, Songchen; Wang, Guanchu; Le, Duy; Jin, Hongye; Chaudhary, Vipin; Xu, Zhaozhuo; Liu, Zirui; Hu, Xia is a scholarly article available to read on EtoBox.

What is KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approaches about?

Long context capability is a crucial competency for large language models (LLMs) as it mitigates the human struggle to digest long-form texts. This capability enables complex task-solving scenarios such as book summarization, code assistance, and many more tasks that are traditionally manpower-intensive. However, transformer-based LLMs face significant challenges with long context input due to the growing size of the KV cache and the intrinsic complexity of attending to extended inputs; where multiple schools of efficiency-driven approaches - such as KV cache quantization, token dropping, prompt compression, linear-time sequence models, and hybrid architectures - have been proposed to produce efficient yet long context-capable models. Despite these advancements, no existing work has comprehensively benchmarked these methods in a reasonably aligned environment. In this work, we fill this gap by providing a taxonomy of current methods and evaluating 10+ state-of-the-art approaches across seven categories of long context tasks. Our work reveals numerous previously unknown phenomena and offers insights - as well as a friendly workbench - for the future development of long context-capab

Author

Yuan, Jiayi; Liu, Hongyi; Zhong, Shaochen; Chuang, Yu-Neng; Li, Songchen; Wang, Guanchu; Le, Duy; Jin, Hongye; Chaudhary, Vipin; Xu, Zhaozhuo; Liu, Zirui; Hu, Xia

Published

2024

Language

EN