Can I read Automated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluations on EtoBox?

Automated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluations by Wang, Lucy Lu; Otmakhova, Yulia; DeYoung, Jay; Truong, Thinh Hung; Kuehl, Bailey E.; Bransom, Erin; Wallace, Byron C. is a scholarly article available to read on EtoBox.

What is Automated Metrics for Medical Multi-Document Summarization Disagree with Human Evaluations about?

Evaluating multi-document summarization (MDS) quality is difficult. This is especially true in the case of MDS for biomedical literature reviews, where models must synthesize contradicting evidence reported across different documents. Prior work has shown that rather than performing the task, models may exploit shortcuts that are difficult to detect using standard n-gram similarity metrics such as ROUGE. Better automated evaluation metrics are needed, but few resources exist to assess metrics when they are proposed. Therefore, we introduce a dataset of human-assessed summary quality facets and pairwise preferences to encourage and support the development of better automated evaluation methods for literature review MDS. We take advantage of community submissions to the Multi-document Summarization for Literature Review (MSLR) shared task to compile a diverse and representative sample of generated summaries. We analyze how automated summarization evaluation metrics correlate with lexical features of generated summaries, to other automated metrics including several we propose in this work, and to aspects of human-assessed summary quality. We find that not only do automated metrics fai

Author

Wang, Lucy Lu; Otmakhova, Yulia; DeYoung, Jay; Truong, Thinh Hung; Kuehl, Bailey E.; Bransom, Erin; Wallace, Byron C.

Published

2023

Language

EN