Can I read LLM-Driven Usefulness Labeling for IR Evaluation on EtoBox?

LLM-Driven Usefulness Labeling for IR Evaluation by Dewan, Mouly; Liu, Jiqun; Shah, Chirag is a scholarly article available to read on EtoBox.

What is LLM-Driven Usefulness Labeling for IR Evaluation about?

In the information retrieval (IR) domain, evaluation plays a crucial role in optimizing search experiences and supporting diverse user intents. In the recent LLM era, research has been conducted to automate document relevance labels, as these labels have traditionally been assigned by crowd-sourced workers - a process that is both time and consuming and costly. This study focuses on LLM-generated usefulness labels, a crucial evaluation metric that considers the user's search intents and task objectives, an aspect where relevance falls short. Our experiment utilizes task-level, query-level, and document-level features along with user search behavior signals, which are essential in defining the usefulness of a document. Our research finds that (i) pre-trained LLMs can generate moderate usefulness labels by understanding the comprehensive search task session, (ii) pre-trained LLMs perform better judgement in short search sessions when provided with search session contexts. Additionally, we investigated whether LLMs can capture the unique divergence between relevance and usefulness, along with conducting an ablation study to identify the most critical metrics for accurate usefulness la

Author

Dewan, Mouly; Liu, Jiqun; Shah, Chirag

Published

2025

Language

EN