Can I read Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '05 - When will information retrieval be "good enough"? on EtoBox?

Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '05 - When will information retrieval be "good enough"? by Allan, James; Carterette, Ben; Lewis, Joshua is a scholarly article available to read on EtoBox.

What is Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval - SIGIR '05 - When will information retrieval be "good enough"? about?

We describe a user study that examined the relationship between the quality of an Information Retrieval system and the effectiveness of its users in performing a task. The task involves finding answer facets of questions pertaining to a collection of newswire documents over a six month period. We artificially created sets of ranked lists at increasing levels of quality by blending the output of a state-of-the-art retrieval system with truth data created by annotators. Subjects performed the task by using these ranked lists to guide their labeling of answer passages in the retrieved articles. We found that as system accuracy improves, subject time on task and error rate decrease, and the rate of finding new correct answers increases. There is a large intermediary region in which the utility difference is not significant; our results suggest that there is some threshold of accuracy for this task beyond which user utility improves rapidly, but more experiments are needed to examine the area around that threshold closely.

Author

Allan, James; Carterette, Ben; Lewis, Joshua

Publisher

ACM Press

Published

2005

Language

EN