Conversational AI from an Information Retrieval Perspective: Remaining Challenges and a Case for User Simulation
Peer reviewed, Journal article
Published version
View/ Open
Date
2021-09Metadata
Show full item recordCollections
Original version
Balog, K. (2021) Conversational AI from an Information Retrieval Perspective: Remaining Challenges and a Case for User Simulation. DESIRES 2021 – 2nd International Conference on Design of Experimental Search & Information REtrieval Systems, September 15–18, 2021, Padua, Italy. CEUR Workshop Proceedings, 2950, 80-90.Abstract
Conversational AI is an emerging field of computer science that engages multiple research communities, from information retrieval to natural language processing to dialogue systems. Within this vast space, we focus on conversational informa tion access, a problem that is uniquely suited to be addressed by the information retrieval community. We argue that despite the significant research activity in this area, progress is mostly limited to component-level improvements. There remains a disconnect between current efforts and truly conversational information access systems. Apart from the inherently chal lenging nature of the problem, the lack of progress, in large part, can be attributed to the shortage of appropriate evaluation methodology and resources. This paper highlights challenges that render both offline and online evaluation methodologies unsuitable for this problem, and discusses the use of user simulation as a viable solution.