scholarly journals More to diverse: Generating diversified responses in a task oriented multimodal dialog system

PLoS ONE ◽  
2020 ◽  
Vol 15 (11) ◽  
pp. e0241271
Author(s):  
Mauajama Firdaus ◽  
Arunav Pratap Shandeelya ◽  
Asif Ekbal
Keyword(s):  
Large Scale ◽  
Contextual Information ◽  
Empirical Evaluation ◽  
Dialogue Systems ◽  
Beam Search ◽  
Dialogue System ◽  
Text And Image ◽  
Dialog System ◽  
Multimodal Dialogue ◽  
Task Oriented

Multimodal dialogue system, due to its many-fold applications, has gained much attention to the researchers and developers in recent times. With the release of large-scale multimodal dialog dataset Saha et al. 2018 on the fashion domain, it has been possible to investigate the dialogue systems having both textual and visual modalities. Response generation is an essential aspect of every dialogue system, and making the responses diverse is an important problem. For any goal-oriented conversational agent, the system’s responses must be informative, diverse and polite, that may lead to better user experiences. In this paper, we propose an end-to-end neural framework for generating varied responses in a multimodal dialogue setup capturing information from both the text and image. Multimodal encoder with co-attention between the text and image is used for focusing on the different modalities to obtain better contextual information. For effective information sharing across the modalities, we combine the information of text and images using the BLOCK fusion technique that helps in learning an improved multimodal representation. We employ stochastic beam search with Gumble Top K-tricks to achieve diversified responses while preserving the content and politeness in the responses. Experimental results show that our proposed approach performs significantly better compared to the existing and baseline methods in terms of distinct metrics, and thereby generates more diverse responses that are informative, interesting and polite without any loss of information. Empirical evaluation also reveals that images, while used along with the text, improve the efficiency of the model in generating diversified responses.

Aspect-Aware Response Generation for Multimodal Dialogue System

2021 ◽  
Vol 12 (2) ◽  
pp. 1-33
Author(s):  
Mauajama Firdaus ◽  
Nidhi Thakur ◽  
Asif Ekbal
Keyword(s):  
User Satisfaction ◽  
Dialogue Systems ◽  
Conversational Agents ◽  
Dialogue System ◽  
Multimodal Systems ◽  
Dialog System ◽  
Language And Vision ◽  
Memory Network ◽  
Multimodal Dialogue ◽  
Task Oriented

Multimodality in dialogue systems has opened up new frontiers for the creation of robust conversational agents. Any multimodal system aims at bridging the gap between language and vision by leveraging diverse and often complementary information from image, audio, and video, as well as text. For every task-oriented dialog system, different aspects of the product or service are crucial for satisfying the user’s demands. Based upon the aspect, the user decides upon selecting the product or service. The ability to generate responses with the specified aspects in a goal-oriented dialogue setup facilitates user satisfaction by fulfilling the user’s goals. Therefore, in our current work, we propose the task of aspect controlled response generation in a multimodal task-oriented dialog system. We employ a multimodal hierarchical memory network for generating responses that utilize information from both text and images. As there was no readily available data for building such multimodal systems, we create a Multi-Domain Multi-Modal Dialog (MDMMD++) dataset. The dataset comprises the conversations having both text and images belonging to the four different domains, such as hotels, restaurants, electronics, and furniture. Quantitative and qualitative analysis on the newly created MDMMD++ dataset shows that the proposed methodology outperforms the baseline models for the proposed task of aspect controlled response generation.

scholarly journals An Evaluation of Chinese Human-Computer Dialogue Technology

2019 ◽  
Vol 1 (2) ◽  
pp. 187-200
Author(s):  
Zhengyu Zhao ◽  
Weinan Zhang ◽  
Wanxiang Che ◽  
Zhigang Chen ◽  
Yibo Zhang
Keyword(s):  
Artificial Intelligence ◽  
Large Scale ◽  
Data Sets ◽  
Dialogue Systems ◽  
Online Testing ◽  
User Intent ◽  
Existing Problems ◽  
Intelligent Processing ◽  
Task Oriented ◽  
Important Branch

The human-computer dialogue has recently attracted extensive attention from both academia and industry as an important branch in the field of artificial intelligence (AI). However, there are few studies on the evaluation of large-scale Chinese human-computer dialogue systems. In this paper, we introduce the Second Evaluation of Chinese Human-Computer Dialogue Technology, which focuses on the identification of a user's intents and intelligent processing of intent words. The Evaluation consists of user intent classification (Task 1) and online testing of task-oriented dialogues (Task 2), the data sets of which are provided by iFLYTEK Corporation. The evaluation tasks and data sets are introduced in detail, and meanwhile, the evaluation results and the existing problems in the evaluation are discussed.

scholarly journals Affirmative Cue Words in Task-Oriented Dialogue

2012 ◽  
Vol 38 (1) ◽  
pp. 1-39 ◽  
Author(s):  
Agustín Gravano ◽  
Julia Hirschberg ◽  
Štefan Beňuš
Keyword(s):  
Computational Methods ◽  
Predictive Power ◽  
American English ◽  
Contextual Information ◽  
Spoken Language ◽  
Dialogue Systems ◽  
Spoken Dialogue Systems ◽  
Standard American English ◽  
Task Oriented ◽  
Cue Words

We present a series of studies of affirmative cue words—a family of cue words such as “okay” or “alright” that speakers use frequently in conversation. These words pose a challenge for spoken dialogue systems because of their ambiguity: They may be used for agreeing with what the interlocutor has said, indicating continued attention, or for cueing the start of a new topic, among other meanings. We describe differences in the acoustic/prosodic realization of such functions in a corpus of spontaneous, task-oriented dialogues in Standard American English. These results are important both for interpretation and for production in spoken language applications. We also assess the predictive power of computational methods for the automatic disambiguation of these words. We find that contextual information and final intonation figure as the most salient cues to automatic disambiguation.

scholarly journals UniMF: A Unified Framework to Incorporate Multimodal Knowledge Bases intoEnd-to-End Task-Oriented Dialogue Systems

2021 ◽  
Author(s):  
Shiquan Yang ◽  
Rui Zhang ◽  
Sarah M. Erfani ◽  
Jey Han Lau
Keyword(s):  
Large Scale ◽  
Knowledge Bases ◽  
Language Models ◽  
Dialogue Systems ◽  
Unified Framework ◽  
Proposed Model ◽  
Multimodal Information ◽  
Task Oriented ◽  
Novel Model ◽  
Single Modality

Knowledge bases (KBs) are usually essential for building practical dialogue systems. Recently we have seen rapidly growing interest in integrating knowledge bases into dialogue systems. However, existing approaches mostly deal with knowledge bases of a single modality, typically textual information. As today's knowledge bases become abundant with multimodal information such as images, audios and videos, the limitation of existing approaches greatly hinders the development of dialogue systems. In this paper, we focus on task-oriented dialogue systems and address this limitation by proposing a novel model that integrates external multimodal KB reasoning with pre-trained language models. We further enhance the model via a novel multi-granularity fusion mechanism to capture multi-grained semantics in the dialogue history. To validate the effectiveness of the proposed model, we collect a new large-scale (14K) dialogue dataset MMDialKB, built upon multimodal KB. Both automatic and human evaluation results on MMDialKB demonstrate the superiority of our proposed framework over strong baselines.

scholarly journals CrossWOZ: A Large-Scale Chinese Cross-Domain Task-Oriented Dialogue Dataset

2020 ◽  
Vol 8 ◽  
pp. 281-295
Author(s):  
Qi Zhu ◽  
Kaili Huang ◽  
Zheng Zhang ◽  
Xiaoyan Zhu ◽  
Minlie Huang
Keyword(s):  
Large Scale ◽  
Dialogue Systems ◽  
Wizard Of Oz ◽  
Cross Domain ◽  
Large Size ◽  
User Simulation ◽  
Dialogue Acts ◽  
Dialogue Modeling ◽  
State Tracking ◽  
Task Oriented

To advance multi-domain (cross-domain) dialogue modeling as well as alleviate the shortage of Chinese task-oriented datasets, we propose CrossWOZ, the first large-scale Chinese Cross-Domain Wizard-of-Oz task-oriented dataset. It contains 6K dialogue sessions and 102K utterances for 5 domains, including hotel, restaurant, attraction, metro, and taxi. Moreover, the corpus contains rich annotation of dialogue states and dialogue acts on both user and system sides. About 60% of the dialogues have cross-domain user goals that favor inter-domain dependency and encourage natural transition across domains in conversation. We also provide a user simulator and several benchmark models for pipelined task-oriented dialogue systems, which will facilitate researchers to compare and evaluate their models on this corpus. The large size and rich annotation of CrossWOZ make it suitable to investigate a variety of tasks in cross-domain dialogue modeling, such as dialogue state tracking, policy learning, user simulation, etc.

Design and Implementation Issues for Convincing Conversational Agents

2011 ◽  
pp. 156-176 ◽  
Author(s):  
Markus Löckelt
Keyword(s):  
Interactive Narrative ◽  
Dialogue Systems ◽  
Conversational Agents ◽  
Comprehensive Overview ◽  
Design And Implementation ◽  
Implementation Level ◽  
Multimodal Dialogue ◽  
Task Oriented ◽  
Future Work ◽  
New System

This chapter describes a selection of experiences from designing and implementing virtual conversational characters for multimodal dialogue systems. It uses examples from the large interactive narrative VirtualHuman and some related systems of the task-oriented variety. The idea is not to give a comprehensive overview of any one system, but rather to identify and describe some issues that might also be relevant for the designer of a new system, to show how they can be addressed, and what problems still remain unresolved for future work. Besides giving an overview of how characters for interactive narrative systems can be built in the implementation level, the focus is on what should be in the knowledge base for virtual characters, and how it should be organized to be able to provide a convincing interaction with one or multiple characters.

scholarly journals CERG: Chinese Emotional Response Generator with Retrieval Method

Research ◽  
2020 ◽  
Vol 2020 ◽  
pp. 1-8
Author(s):  
Yangyang Zhou ◽  
Fuji Ren
Keyword(s):  
Emotional Response ◽  
Dialogue Systems ◽  
Inference Process ◽  
Dialogue System ◽  
Data Set ◽  
Retrieval Method ◽  
Input Text ◽  
Proposed Model ◽  
Semantic Relevance ◽  
Task Oriented

The dialogue system has always been one of the important topics in the domain of artificial intelligence. So far, most of the mature dialogue systems are task-oriented based, while non-task-oriented dialogue systems still have a lot of room for improvement. We propose a data-driven non-task-oriented dialogue generator “CERG” based on neural networks. This model has the emotion recognition capability and can generate corresponding responses. The data set we adopt comes from the NTCIR-14 STC-3 CECG subtask, which contains more than 1.7 million Chinese Weibo post-response pairs and 6 emotion categories. We try to concatenate the post and the response with the emotion, then mask the response part of the input text character by character to emulate the encoder-decoder framework. We use the improved transformer blocks as the core to build the model and add regularization methods to alleviate the problems of overcorrection and exposure bias. We introduce the retrieval method to the inference process to improve the semantic relevance of generated responses. The results of the manual evaluation show that our proposed model can make different responses to different emotions to improve the human-computer interaction experience. This model can be applied to lots of domains, such as automatic reply robots of social application.

Study on Development of Humor Discriminator for Dialogue System

2020 ◽  
Vol 24 (3) ◽  
pp. 422-435
Author(s):  
Tomohiro Yoshikawa ◽  
◽  
Ryosuke Iwakura
Keyword(s):  
Human Subjects ◽  
The Other ◽  
P Value ◽  
Dialogue Systems ◽  
Generation System ◽  
Dialogue System ◽  
Evaluation Scores ◽  
Significant Difference ◽  
Computer Based ◽  
Task Oriented

Studies on automatic dialogue systems, which allow people and computers to communicate with each other using natural language, have been attracting attention. In particular, the main objective of a non-task-oriented dialogue system is not to achieve a specific task but to amuse users through chat and free dialogue. For this type of dialogue system, continuity of the dialogue is important because users can easily get tired if the dialogue is monotonous. On the other hand, preceding studies have shown that speech with humorous expressions is effective in improving the continuity of a dialogue. In this study, we developed a computer-based humor discriminator to perform user- or situation-independent objective discrimination of humor. Using the humor discriminator, we also developed an automatic humor generation system and conducted an evaluation experiment with human subjects to test the generated jokes. A t-test on the evaluation scores revealed a significant difference (P value: 3.5×10-5) between the proposed and existing methods of joke generation.

scholarly journals Dialogue state tracking accuracy improvement by distinguishing slot-value pairs and dialogue behaviour

2021 ◽  
Vol 21 (2) ◽  
pp. 1057
Author(s):  
Khaldoon H. Alhussayni ◽  
Alexander Zamyatin ◽  
S. Eman Alshamery
Keyword(s):  
Critical Role ◽  
Historical Context ◽  
Dialogue Systems ◽  
Tracking Accuracy ◽  
Dialogue System ◽  
Dialogue Acts ◽  
Proposed Model ◽  
State Tracking ◽  
Task Oriented ◽  
Previous User

<div><p>Dialog state tracking (DST) plays a critical role in cycle life of a task-oriented dialogue system. DST represents the goals of the consumer at each step by dialogue and describes such objectives as a conceptual structure comprising slot-value pairs and dialogue actions that specifically improve the performance and effectiveness of dialogue systems. DST faces several challenges: diversity of linguistics, dynamic social context and the dissemination of the state of dialogue over candidate values both in slot values and in dialogue acts determined in ontology. In many turns during the dialogue, users indirectly refer to the previous utterances, and that produce a challenge to distinguishing and use of related dialogue history, Recent methods used and popular for that are ineffective. In this paper, we propose a dialogue historical context self-Attention framework for DST that recognizes relevant historical context by including previous user utterance beside current user utterances and previous system actions where specific slot-value piers variations and uses that together with weighted system utterance to outperform existing models by recognizing the related context and the relevance of a system utterance. For the evaluation of the proposed model the WoZ dataset was used. The implementation was attempted with the prior user utterance as a dialogue encoder and second by the additional score combined with all the candidate slot-value pairs in the context of previous user utterances and current utterances. The proposed model obtained 0.8 per cent better results than all state-of-the-art methods in the combined precision of the target, but this is not the turnaround challenge for the submission.</p></div>

scholarly journals Source-Side Discontinuous Phrases for Machine Translation: A Comparative Study on Phrase Extraction and Search

2013 ◽  
Vol 99 (1) ◽  
pp. 17-38
Author(s):  
Matthias Huck ◽  
Erik Scharwächter ◽  
Hermann Ney
Keyword(s):  
Machine Translation ◽  
Large Scale ◽  
Search Algorithm ◽  
Statistical Machine Translation ◽  
Empirical Evaluation ◽  
Training Data ◽  
Beam Search ◽  
Phrase Extraction ◽  
System Configurations ◽  
Translation Systems

Abstract Standard phrase-based statistical machine translation systems generate translations based on an inventory of continuous bilingual phrases. In this work, we extend a phrase-based decoder with the ability to make use of phrases that are discontinuous in the source part. Our dynamic programming beam search algorithm supports separate pruning of coverage hypotheses per cardinality and of lexical hypotheses per coverage, as well as coverage constraints that impose restrictions on the possible reorderings. In addition to investigating these aspects, which are related to the decoding procedure, we also concentrate our attention on the question of how to obtain source-side discontinuous phrases from parallel training data. Two approaches (hierarchical and discontinuous extraction) are presented and compared. On a large-scale Chinese!English translation task, we conduct a thorough empirical evaluation in order to study a number of system configurations with source-side discontinuous phrases, and to compare them to setups which employ continuous phrases only.

Sign in / Sign up

Export Citation Format

Share Document