A User Model for Dialog System Evaluation Based on Activation of Subgoals

2013 ◽  
pp. 363-374 ◽  
Author(s):  
Klaus-Peter Engelbrecht
Keyword(s):  
User Model ◽  
System Evaluation ◽  
Dialog System

Analysis of a new simulation approach to dialog system evaluation

2009 ◽  
Vol 51 (12) ◽  
pp. 1234-1252 ◽  
Author(s):  
Klaus-Peter Engelbrecht ◽  
Michael Quade ◽  
Sebastian Möller
Keyword(s):  
System Evaluation ◽  
Simulation Approach ◽  
Dialog System

scholarly journals Estimating a User's Internal State before the First Input Utterance

2012 ◽  
Vol 2012 ◽  
pp. 1-10 ◽  
Author(s):  
Yuya Chiba ◽  
Akinori Ito
Keyword(s):  
User Modeling ◽  
Internal State ◽  
User Model ◽  
Discrimination Experiment ◽  
Wizard Of Oz ◽  
Class Discrimination ◽  
Dialog System ◽  
Spoken Dialog System ◽  
Open Test ◽  
Combined Features

This paper describes a method for estimating the internal state of a user of a spoken dialog system before his/her first input utterance. When actually using a dialog-based system, the user is often perplexed by the prompt. A typical system provides more detailed information to a user who is taking time to make an input utterance, but such assistance is nuisance if the user is merely considering how to answer the prompt. To respond appropriately, the spoken dialog system should be able to consider the user’s internal state before the user’s input. Conventional studies on user modeling have focused on the linguistic information of the utterance for estimating the user’s internal state, but this approach cannot estimate the user’s state until the end of the user’s first utterance. Therefore, we focused on the user’s nonverbal output such as fillers, silence, or head-moving until the beginning of the input utterance. The experimental data was collected on a Wizard of Oz basis, and the labels were decided by five evaluators. Finally, we conducted a discrimination experiment with the trained user model using combined features. As a three-class discrimination result, we obtained about 85% accuracy in an open test.

Dialog Processing Theory

1995 ◽  
Author(s):  
Ronnie W. Smith ◽  
D. Richard Hipp
Keyword(s):  
Light Emitting Diode ◽  
User Model ◽  
Light Emitting ◽  
Processing Theory ◽  
Dialog System ◽  
The Status ◽  
Experimental Interaction ◽  
Task Processing ◽  
Interactive Task ◽  
Dialog Processing

Consider the following dialog situation. The computer is providing a human user with assistance in fixing an electronic circuit that causes a Light Emitting Diode (LED) to display in a certain way. The current focus of the task and dialog is to determine the status of a wire between labeled connectors 84 and 99, a wire needed for the circuit that is absent. Figures 3.1 and 3.2 show two possible dialog interactions that could occur at this point. In figure 3.1, the computer has total dialog control, and a total of 29 utterances are needed to guide the user through the rest of the dialog. In figure 3.2, the human user has overall dialog control, but the computer is allowed to provide direct assistance as needed (i.e. in helping add the wire). Only 11 utterances are needed for the experienced user to complete the dialog. These samples are from interactions with a working spoken natural language dialog system. To engage in such dialog interactions, a system must exhibit the behaviors mentioned at the beginning of chapter 1: (1) problem solving for providing task assistance, (2) conducting subdialogs to achieve appropriate subgoals, (3) exploiting user model to enable useful interactions, (4) exploiting context dependent expectations when interpreting user inputs, and (5) engaging in variable initiative dialogs. Achieving these behaviors while facilitating the measurement of system performance via experimental interaction requires a theory of dialog processing that integrates the following subtheories. • An abstract model of interactive task processing. • A theory about the purpose of language within the interactive task processing environment. • A theory of user model usage. • A theory of contextual interpretation. • A theory of variable initiative dialog. This chapter presents such a theory of dialog processing. Frequent reference to the dialog examples in figures 3.1 and 3.2 will guide the discussion. The first section discusses the overall system architecture that facilitates integrated dialog processing. The remainder of the chapter addresses each subtheory in turn, emphasizing how each fits into the overall architecture. The chapter concludes with a summary description of the integrated model.

scholarly journals A Strategy for Information Presentation in Spoken Dialog Systems

2011 ◽  
Vol 37 (3) ◽  
pp. 489-539 ◽  
Author(s):  
Vera Demberg ◽  
Andi Winterboer ◽  
Johanna D. Moore
Keyword(s):  
User Satisfaction ◽  
Selection Process ◽  
User Model ◽  
Information Presentation ◽  
Spoken Dialog Systems ◽  
Dialog Systems ◽  
Content Selection ◽  
Trade Offs ◽  
Dialog System ◽  
Efficiency And Effectiveness

In spoken dialog systems, information must be presented sequentially, making it difficult to quickly browse through a large number of options. Recent studies have shown that user satisfaction is negatively correlated with dialog duration, suggesting that systems should be designed to maximize the efficiency of the interactions. Analysis of the logs of 2,000 dialogs between users and nine different dialog systems reveals that a large percentage of the time is spent on the information presentation phase, thus there is potentially a large pay-off to be gained from optimizing information presentation in spoken dialog systems. This article proposes a method that improves the efficiency of coping with large numbers of diverse options by selecting options and then structuring them based on a model of the user's preferences. This enables the dialog system to automatically determine trade-offs between alternative options that are relevant to the user and present these trade-offs explicitly. Multiple attractive options are thereby structured such that the user can gradually refine her request to find the optimal trade-off. To evaluate and challenge our approach, we conducted a series of experiments that test the effectiveness of the proposed strategy. Experimental results show that basing the content structuring and content selection process on a user model increases the efficiency and effectiveness of the user's interaction. Users complete their tasks more successfully and more quickly. Furthermore, user surveys revealed that participants found that the user-model based system presents complex trade-offs understandably and increases overall user satisfaction. The experiments also indicate that presenting users with a brief overview of options that do not fit their requirements significantly improves the user's overview of available options, also making them feel more confident in having been presented with all relevant options.

Crowdsourcing for Spoken Dialog System Evaluation

2013 ◽  
pp. 217-240
Author(s):  
Zhaojun Yang ◽  
Gina-Anne Levow ◽  
Helen Meng
Keyword(s):  
System Evaluation ◽  
Dialog System ◽  
Spoken Dialog System
Sign in / Sign up

Export Citation Format

Share Document