Can I read Benchmarking LLMs' Judgments with No Gold Standard on EtoBox?

Benchmarking LLMs' Judgments with No Gold Standard by Xu, Shengwei; Lu, Yuxuan; Schoenebeck, Grant; Kong, Yuqing is a scholarly article available to read on EtoBox.

What is Benchmarking LLMs' Judgments with No Gold Standard about?

We introduce the GEM (Generative Estimator for Mutual Information), an evaluation metric for assessing language generation by Large Language Models (LLMs), particularly in generating informative judgments, without the need for a gold standard reference. GEM broadens the scenarios where we can benchmark LLM generation performance-from traditional ones, like machine translation and summarization, where gold standard references are readily available, to subjective tasks without clear gold standards, such as academic peer review. GEM uses a generative model to estimate mutual information between candidate and reference responses, without requiring the reference to be a gold standard. In experiments on a human-annotated dataset, GEM demonstrates competitive correlations with human scores compared to the state-of-the-art GPT-4o Examiner, and outperforms all other baselines. Additionally, GEM is more robust against strategic manipulations, such as rephrasing or elongation, which can artificially inflate scores under a GPT-4o Examiner. We also present GRE-bench (Generating Review Evaluation Benchmark) which evaluates LLMs based on how well they can generate high-quality peer reviews for ac

Author

Xu, Shengwei; Lu, Yuxuan; Schoenebeck, Grant; Kong, Yuqing

Published

2024

Language

EN