Can I read LLM Confidence Evaluation Measures in Zero-Shot CSS Classification on EtoBox?

LLM Confidence Evaluation Measures in Zero-Shot CSS Classification by Farr, David; Cruickshank, Iain; Manzonelli, Nico; Clark, Nicholas; Starbird, Kate; West, Jevin is a scholarly article available to read on EtoBox.

What is LLM Confidence Evaluation Measures in Zero-Shot CSS Classification about?

Assessing classification confidence is critical for leveraging large language models (LLMs) in automated labeling tasks, especially in the sensitive domains presented by Computational Social Science (CSS) tasks. In this paper, we make three key contributions: (1) we propose an uncertainty quantification (UQ) performance measure tailored for data annotation tasks, (2) we compare, for the first time, five different UQ strategies across three distinct LLMs and CSS data annotation tasks, (3) we introduce a novel UQ aggregation strategy that effectively identifies low-confidence LLM annotations and disproportionately uncovers data incorrectly labeled by the LLMs. Our results demonstrate that our proposed UQ aggregation strategy improves upon existing methods andcan be used to significantly improve human-in-the-loop data annotation processes.

Author

Farr, David; Cruickshank, Iain; Manzonelli, Nico; Clark, Nicholas; Starbird, Kate; West, Jevin

Published

2024

Language

EN