Can I read Which Prompts Make The Difference? Data Prioritization For Efficient Human LLM Evaluation on EtoBox?

Which Prompts Make The Difference? Data Prioritization For Efficient Human LLM Evaluation by Boubdir, Meriem; Kim, Edward; Ermis, Beyza; Fadaee, Marzieh; Hooker, Sara is a scholarly article available to read on EtoBox.

What is Which Prompts Make The Difference? Data Prioritization For Efficient Human LLM Evaluation about?

Human evaluation is increasingly critical for assessing large language models, capturing linguistic nuances, and reflecting user preferences more accurately than traditional automated metrics. However, the resource-intensive nature of this type of annotation process poses significant challenges. The key question driving our work: "is it feasible to minimize human-in-the-loop feedback by prioritizing data instances which most effectively distinguish between models?" We evaluate several metric-based methods and find that these metrics enhance the efficiency of human evaluations by minimizing the number of required annotations, thus saving time and cost, while ensuring a robust performance evaluation. We show that our method is effective across widely used model families, reducing instances of indecisive (or "tie") outcomes by up to 54% compared to a random sample when focusing on the top-20 percentile of prioritized instances. This potential reduction in required human effort positions our approach as a valuable strategy in future large language model evaluations.

Author

Boubdir, Meriem; Kim, Edward; Ermis, Beyza; Fadaee, Marzieh; Hooker, Sara

Published

2023

Language

EN