Can I read UserSumBench: A Benchmark Framework for Evaluating User Summarization Approaches on EtoBox?

UserSumBench: A Benchmark Framework for Evaluating User Summarization Approaches by Wang, Chao; Wu, Neo; Ning, Lin; Wu, Jiaxing; Liu, Luyang; Xie, Jun; O'Banion, Shawn; Green, Bradley is a scholarly article available to read on EtoBox.

What is UserSumBench: A Benchmark Framework for Evaluating User Summarization Approaches about?

Large language models (LLMs) have shown remarkable capabilities in generating user summaries from a long list of raw user activity data. These summaries capture essential user information such as preferences and interests, and therefore are invaluable for LLM-based personalization applications, such as explainable recommender systems. However, the development of new summarization techniques is hindered by the lack of ground-truth labels, the inherent subjectivity of user summaries, and human evaluation which is often costly and time-consuming. To address these challenges, we introduce \UserSumBench, a benchmark framework designed to facilitate iterative development of LLM-based summarization approaches. This framework offers two key components: (1) A reference-free summary quality metric. We show that this metric is effective and aligned with human preferences across three diverse datasets (MovieLens, Yelp and Amazon Review). (2) A novel robust summarization method that leverages time-hierarchical summarizer and self-critique verifier to produce high-quality summaries while eliminating hallucination. This method serves as a strong baseline for further innovation in summarization te

Author

Wang, Chao; Wu, Neo; Ning, Lin; Wu, Jiaxing; Liu, Luyang; Xie, Jun; O'Banion, Shawn; Green, Bradley

Published

2024

Language

EN