Can I read MADial-Bench: Towards Real-world Evaluation of Memory-Augmented Dialogue Generation on EtoBox?

MADial-Bench: Towards Real-world Evaluation of Memory-Augmented Dialogue Generation by He, Junqing; Zhu, Liang; Wang, Rui; Wang, Xi; Haffari, Reza; Zhang, Jiaxing is a scholarly article available to read on EtoBox.

What is MADial-Bench: Towards Real-world Evaluation of Memory-Augmented Dialogue Generation about?

Long-term memory is important for chatbots and dialogue systems (DS) to create consistent and human-like conversations, evidenced by numerous developed memory-augmented DS (MADS). To evaluate the effectiveness of such MADS, existing commonly used evaluation metrics, like retrieval accuracy and perplexity (PPL), mainly focus on query-oriented factualness and language quality assessment. However, these metrics often lack practical value. Moreover, the evaluation dimensions are insufficient for human-like assessment in DS. Regarding memory-recalling paradigms, current evaluation schemes only consider passive memory retrieval while ignoring diverse memory recall with rich triggering factors, e.g., emotions and surroundings, which can be essential in emotional support scenarios. To bridge the gap, we construct a novel Memory-Augmented Dialogue Benchmark (MADail-Bench) covering various memory-recalling paradigms based on cognitive science and psychology theories. The benchmark assesses two tasks separately: memory retrieval and memory recognition with the incorporation of both passive and proactive memory recall data. We introduce new scoring criteria to the evaluation, including memory

Author

He, Junqing; Zhu, Liang; Wang, Rui; Wang, Xi; Haffari, Reza; Zhang, Jiaxing

Published

2024

Language

EN