Can I read Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluation on EtoBox?

Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluation by Ramprasad, Sanjana; Wallace, Byron C. is a scholarly article available to read on EtoBox.

What is Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluation about?

Modern LLMs can now produce highly readable abstractive summaries, to the point where traditional automated metrics for evaluating summary quality, such as ROUGE, have become saturated. However, LLMs still sometimes introduce unwanted content into summaries, i.e., information inconsistent with or unsupported by their source. Measuring the occurrence of these often subtle ``hallucinations'' automatically has proved to be challenging. This in turn has motivated development of a variety of metrics intended to measure the factual consistency of generated summaries against their source. But are these approaches measuring what they purport to do? In this work, we stress-test automatic factuality metrics. Specifically, we investigate whether and to what degree superficial attributes of summary texts suffice to predict ``factuality'', finding that a (supervised) model using only such shallow features is reasonably competitive with SOTA factuality scoring methods. We then evaluate how factuality metrics respond to factual corrections in inconsistent summaries and find that only a few show meaningful improvements. In contrast, some metrics are more sensitive to benign, non-factual edits. Mot

Author

Ramprasad, Sanjana; Wallace, Byron C.

Published

2024

Language

EN