Can I read Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selection on EtoBox?

Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selection by Zhu, Yun; Gu, Jia-Chen; Sikora, Caitlin; Ko, Ho; Liu, Yinxiao; Lin, Chu-Cheng; Shu, Lei; Luo, Liangchen; Meng, Lei; Liu, Bang; Chen, Jindong is a scholarly article available to read on EtoBox.

What is Accelerating Inference of Retrieval-Augmented Generation via Sparse Context Selection about?

Large language models (LLMs) augmented with retrieval exhibit robust performance and extensive versatility by incorporating external contexts. However, the input length grows linearly in the number of retrieved documents, causing a dramatic increase in latency. In this paper, we propose a novel paradigm named Sparse RAG, which seeks to cut computation costs through sparsity. Specifically, Sparse RAG encodes retrieved documents in parallel, which eliminates latency introduced by long-range attention of retrieved documents. Then, LLMs selectively decode the output by only attending to highly relevant caches auto-regressively, which are chosen via prompting LLMs with special control tokens. It is notable that Sparse RAG combines the assessment of each individual document and the generation of the response into a single process. The designed sparse mechanism in a RAG system can facilitate the reduction of the number of documents loaded during decoding for accelerating the inference of the RAG system. Additionally, filtering out undesirable contexts enhances the model's focus on relevant context, inherently improving its generation quality. Evaluation results of two datasets show that S

Author

Zhu, Yun; Gu, Jia-Chen; Sikora, Caitlin; Ko, Ho; Liu, Yinxiao; Lin, Chu-Cheng; Shu, Lei; Luo, Liangchen; Meng, Lei; Liu, Bang; Chen, Jindong

Published

2024

Language

EN