Can I read Preference Leakage in LLM Evaluation on EtoBox?

Preference Leakage in LLM Evaluation by Gourab Kundu is a document available to read on EtoBox.

What is Preference Leakage in LLM Evaluation about?

This document discusses preference leakage, a contamination issue in the use of Large Language Models (LLMs) as judges in data annotation, where biases arise from the relatedness between data generators and evaluators. The authors identify three types of relatedness—same model, inheritance relationship, and same model family—and demonstrate through experiments that preference leakage leads to systematic biases in evaluation scores. The findings suggest that preference leakage is a pervasive and challenging

Author

Gourab Kundu

Language

EN