Can I read Corrected Evaluation Results of the NTCIR WWW-2, WWW-3, and WWW-4 English Subtasks on EtoBox?

Corrected Evaluation Results of the NTCIR WWW-2, WWW-3, and WWW-4 English Subtasks by Sakai, Tetsuya; Tao, Sijie; Maistro, Maria; Chu, Zhumin; Li, Yujing; Chen, Nuo; Ferro, Nicola; Wang, Junjie; Soboroff, Ian; Liu, Yiqun is a scholarly article available to read on EtoBox.

What is Corrected Evaluation Results of the NTCIR WWW-2, WWW-3, and WWW-4 English Subtasks about?

Unfortunately, the official English (sub)task results reported in the NTCIR-14 WWW-2, NTCIR-15 WWW-3, and NTCIR-16 WWW-4 overview papers are incorrect due to noise in the official qrels files; this paper reports results based on the corrected qrels files. The noise is due to a fatal bug in the backend of our relevance assessment interface. More specifically, at WWW-2, WWW-3, and WWW-4, two versions of pool files were created for each English topic: a PRI ("prioritised") file, which uses the NTCIRPOOL script to prioritise likely relevant documents, and a RND ("randomised") file, which randomises the pooled documents. This was done for the purpose of studying the effect of document ordering for relevance assessors. However, the programmer who wrote the interface backend assumed that a combination of a topic ID and a document rank in the pool file uniquely determines a document ID; this is obviously incorrect as we have two versions of pool files. The outcome is that all the PRI-based relevance labels for the WWW-2 test collection are incorrect (while all the RND-based relevance labels are correct), and all the RND-based relevance labels for the WWW-3 and WWW-4 test collections are in

Author

Sakai, Tetsuya; Tao, Sijie; Maistro, Maria; Chu, Zhumin; Li, Yujing; Chen, Nuo; Ferro, Nicola; Wang, Junjie; Soboroff, Ian; Liu, Yiqun

Published

2022

Language

EN