| dc.description.abstract | Conversational Information Access systems are systems that help satisfy the information needs of their users via multi-turn conversations. These systems are different from traditional information access systems as they can ask clarification questions that can help narrow down the result set.
Traditionally, end-to-end evaluation of such systems has been online, i.e., hiring workers to use their system and provide feedback. However, this approach suffers from key issues: it is expensive, time-consuming, and does not scale to large amount of users. Consequently, the idea of simulating users has received considerable attention. User simulators can be used for evaluation, however, their performance is dependent on sophisticated user modeling.
In this thesis, we continue on previous work [1] and build a user simulator that incorporates more advanced user modeling by including additional contexts. The simulated users are represented via an interaction model, a preference model, time context, and group setting.
We evaluate our user simulator based on metrics in the literature. Furthermore, we capture subjective measures by computing user satisfaction for each conversation. Our work illustrates that the included contexts have an impact on the observed dialogs in terms of these metrics. Lastly, we release DialogueKit on PyPi, a toolkit for developing conversational agents. | |
