Can I read Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI on EtoBox?

Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI by Pangakis, Nicholas; Wolken, Samuel is a scholarly article available to read on EtoBox.

What is Keeping Humans in the Loop: Human-Centered Automated Annotation with Generative AI about?

Automated text annotation is a compelling use case for generative large language models (LLMs) in social media research. Recent work suggests that LLMs can achieve strong performance on annotation tasks; however, these studies evaluate LLMs on a small number of tasks and likely suffer from contamination due to a reliance on public benchmark datasets. Here, we test a human-centered framework for responsibly evaluating artificial intelligence tools used in automated annotation. We use GPT-4 to replicate 27 annotation tasks across 11 password-protected datasets from recently published computational social science articles in high-impact journals. For each task, we compare GPT-4 annotations against human-annotated ground-truth labels and against annotations from separate supervised classification models fine-tuned on human-generated labels. Although the quality of LLM labels is generally high, we find significant variation in LLM performance across tasks, even within datasets. Our findings underscore the importance of a human-centered workflow and careful evaluation standards: Automated annotations significantly diverge from human judgment in numerous scenarios, despite various optimiz

Author

Pangakis, Nicholas; Wolken, Samuel

Published

2024

Language

EN