Can I read Split and Merge: Aligning Position Biases in LLM-based Evaluators on EtoBox?

Split and Merge: Aligning Position Biases in LLM-based Evaluators by Li, Zongjie; Wang, Chaozheng; Ma, Pingchuan; Wu, Daoyuan; Wang, Shuai; Gao, Cuiyun; Liu, Yang is a scholarly article available to read on EtoBox.

What is Split and Merge: Aligning Position Biases in LLM-based Evaluators about?

Large language models (LLMs) have shown promise as automated evaluators for assessing the quality of answers generated by AI systems. However, these LLM-based evaluators exhibit position bias, or inconsistency, when used to evaluate candidate answers in pairwise comparisons, favoring either the first or second answer regardless of content. To address this limitation, we propose PORTIA, an alignment-based system designed to mimic human comparison strategies to calibrate position bias in a lightweight yet effective manner. Specifically, PORTIA splits the answers into multiple segments, aligns similar content across candidate answers, and then merges them back into a single prompt for evaluation by LLMs. We conducted extensive experiments with six diverse LLMs to evaluate 11,520 answer pairs. Our results show that PORTIA markedly enhances the consistency rates for all the models and comparison forms tested, achieving an average relative improvement of 47.46%. Remarkably, PORTIA enables less advanced GPT models to achieve 88% agreement with the state-of-the-art GPT-4 model at just 10% of the cost. Furthermore, it rectifies around 80% of the position bias instances within the GPT-4 mode

Author

Li, Zongjie; Wang, Chaozheng; Ma, Pingchuan; Wu, Daoyuan; Wang, Shuai; Gao, Cuiyun; Liu, Yang

Published

2023

Language

EN