Can I read Annotation Inconsistency and Entity Bias in MultiWOZ on EtoBox?

Annotation Inconsistency and Entity Bias in MultiWOZ by Qian, Kun; Beirami, Ahmad; Lin, Zhouhan; De, Ankita; Geramifard, Alborz; Yu, Zhou; Sankar, Chinnadhurai is a scholarly article available to read on EtoBox.

What is Annotation Inconsistency and Entity Bias in MultiWOZ about?

MultiWOZ is one of the most popular multi-domain task-oriented dialog datasets, containing 10K+ annotated dialogs covering eight domains. It has been widely accepted as a benchmark for various dialog tasks, e.g., dialog state tracking (DST), natural language generation (NLG), and end-to-end (E2E) dialog modeling. In this work, we identify an overlooked issue with dialog state annotation inconsistencies in the dataset, where a slot type is tagged inconsistently across similar dialogs leading to confusion for DST modeling. We propose an automated correction for this issue, which is present in a whopping 70% of the dialogs. Additionally, we notice that there is significant entity bias in the dataset (e.g., "cambridge" appears in 50% of the destination cities in the train domain). The entity bias can potentially lead to named entity memorization in generative models, which may go unnoticed as the test set suffers from a similar entity bias as well. We release a new test set with all entities replaced with unseen entities. Finally, we benchmark joint goal accuracy (JGA) of the state-of-the-art DST baselines on these modified versions of the data. Our experiments show that the annotation

Author

Qian, Kun; Beirami, Ahmad; Lin, Zhouhan; De, Ankita; Geramifard, Alborz; Yu, Zhou; Sankar, Chinnadhurai

Published

2021

Language

EN