Can I read Scaling Laws For Dense Retrieval on EtoBox?

Scaling Laws For Dense Retrieval by Fang, Yan; Zhan, Jingtao; Ai, Qingyao; Mao, Jiaxin; Su, Weihang; Chen, Jia; Liu, Yiqun is a scholarly article available to read on EtoBox.

What is Scaling Laws For Dense Retrieval about?

Scaling up neural models has yielded significant advancements in a wide array of tasks, particularly in language generation. Previous studies have found that the performance of neural models frequently adheres to predictable scaling laws, correlated with factors such as training set size and model size. This insight is invaluable, especially as large-scale experiments grow increasingly resource-intensive. Yet, such scaling law has not been fully explored in dense retrieval due to the discrete nature of retrieval metrics and complex relationships between training data and model sizes in retrieval tasks. In this study, we investigate whether the performance of dense retrieval models follows the scaling law as other neural models. We propose to use contrastive log-likelihood as the evaluation metric and conduct extensive experiments with dense retrieval models implemented with different numbers of parameters and trained with different amounts of annotated data. Results indicate that, under our settings, the performance of dense retrieval models follows a precise power-law scaling related to the model size and the number of annotations. Additionally, we examine scaling with prevalent d

Author

Fang, Yan; Zhan, Jingtao; Ai, Qingyao; Mao, Jiaxin; Su, Weihang; Chen, Jia; Liu, Yiqun

Published

2024

Language

EN