Can I read Making Large Language Models A Better Foundation For Dense Retrieval on EtoBox?

Making Large Language Models A Better Foundation For Dense Retrieval by Li, Chaofan; Liu, Zheng; Xiao, Shitao; Shao, Yingxia is a scholarly article available to read on EtoBox.

What is Making Large Language Models A Better Foundation For Dense Retrieval about?

Dense retrieval needs to learn discriminative text embeddings to represent the semantic relationship between query and document. It may benefit from the using of large language models (LLMs), given LLMs' strong capability on semantic understanding. However, the LLMs are pre-trained by text generation tasks, whose working pattern is completely different from representing texts as embeddings. As a result, it is imperative to study how to adapt LLMs properly so that they can be effectively initialized as the backbone encoder for dense retrieval. In this paper, we propose a novel approach, called LLaRA (LLM adapted for dense RetrievAl), which works as a post-hoc adaptation of LLM for the dense retrieval application. LLaRA consists of two pretext tasks: EBAE (Embedding-Based Auto-Encoding) and EBAR (Embedding-Based Auto-Regression), where the text embeddings from LLM are used to reconstruct the tokens for the input sentence and predict the tokens for the next sentence, respectively. LLaRA turns out to be simple, lightweight, and highly effective. It is applied to adapt LLaMA-2-7B (base) on the Wikipedia corpus, where it substantially improves the model's fine-tuned performances on a var

Author

Li, Chaofan; Liu, Zheng; Xiao, Shitao; Shao, Yingxia

Published

2023

Language

EN