Focus on Interaction: A Novel Dynamic Graph Model for Joint Multiple Intent Detection and Slot Filling

Intent detection and slot filling are two main tasks for building a spoken language understanding (SLU) system. Since the two tasks are closely related, the joint models for the two tasks always outperform the pipeline models in SLU. However, most joint models directly incorporate multiple intent information for each token, which introduces intent noise into the sentence semantics, causing a decrease in the performance of the joint model. In this paper, we propose a Dynamic Graph Model (DGM) for joint multiple intent detection and slot filling, in which we adopt a sentence-level intent-slot interactive graph to model the correlation between the intents and slot. Besides, we design a novel method of constructing the graph, which can dynamically update the interactive graph and further alleviate the error propagation. Experimental results on several multi-intent and single-intent datasets show that our model not only achieves the state-of-the-art (SOTA) performance but also boosts the speed by three to six times over the SOTA model.

Download Full-text

Using Deep Time Delay Neural Network for Slot Filling in Spoken Language Understanding

Symmetry ◽

10.3390/sym12060993 ◽

2020 ◽

Vol 12 (6) ◽

pp. 993

Author(s):

Zhen Zhang ◽

Hao Huang ◽

Kai Wang

Keyword(s):

Neural Network ◽

Time Delay ◽

Target Word ◽

State Of The Art ◽

Contextual Information ◽

Spoken Language ◽

Continuous Speech Recognition ◽

Language Understanding ◽

Spoken Language Understanding ◽

Slot Filling

Modeling the context of a target word is of fundamental importance in predicting the semantic label for slot filling task in Spoken Language Understanding (SLU). Although Recurrent Neural Network (RNN) has shown to successfully achieve the state-of-the-art results for SLU, and Bidirectional RNN is capable of obtaining further improvement by modeling information not only from the past, but also from the future, they only consider limited contextual information of the target word. In order to make the network deeper and hence obtain longer contextual information, we propose to use a multi-layer Time Delay Neural Network (TDNN), which is prevalent in current large vocabulary continuous speech recognition tasks. In particular, we use a TDNN with symmetric time delay offset. To make the stacked TDNN easily trained, residual structures and skip concatenation are adopted. In addition, we further improve the model by introducing ResTDNN-BiLSTM, which combines the advantages of both the residual TDNN and BiLSTM. Experiments on slot filling tasks on the Air Travel Information System (ATIS) and Snips benchmark datasets show the proposed SC-TDNN-C achieves state-of-the-art results without any additional knowledge and data resources. Finally, we review and compare slot filling results by using a variety of existing models and methods.

Download Full-text

Joint Model-Based Attention for Spoken Language Understanding Task

International Journal of Digital Crime and Forensics ◽

10.4018/ijdcf.2020100103 ◽

2020 ◽

Vol 12 (4) ◽

pp. 32-43

Author(s):

Xin Liu ◽

RuiHua Qi ◽

Lin Shao

Keyword(s):

Contextual Information ◽

Spoken Language ◽

Joint Model ◽

Shared Task ◽

Global Features ◽

Language Understanding ◽

Spoken Language Understanding ◽

Model Based ◽

Sentence Level ◽

Slot Filling

Intent determination (ID) and slot filling (SF) are two critical steps in the spoken language understanding (SLU) task. Conventionally, most previous work has been done for each subtask respectively. To exploit the dependencies between intent label and slot sequence, as well as deal with both tasks simultaneously, this paper proposes a joint model (ABLCJ), which is trained by a united loss function. In order to utilize both past and future input features efficiently, a joint model based Bi-LSTM with contextual information is employed to learn the representation of each step, which are shared by two tasks and the model. This paper also uses sentence-level tag information learned from a CRF layer to predict the tag of each slot. Meanwhile, a submodule-based attention is employed to capture global features of a sentence for intent classification. The experimental results demonstrate that ABLCJ achieves competitive performance in the Shared Task 4 of NLPCC 2018.

Download Full-text

Joint intent detection and slot filling with wheel-graph attention networks

Journal of Intelligent & Fuzzy Systems ◽

10.3233/jifs-211674 ◽

2021 ◽

pp. 1-12

Author(s):

Pengfei Wei ◽

Bi Zeng ◽

Wenxiong Liao

Keyword(s):

Deep Neural Networks ◽

Semantic Information ◽

Graph Structure ◽

Joint Models ◽

Language Understanding ◽

Attention Networks ◽

Spoken Language Understanding ◽

Field Of Vision ◽

Wheel Graph ◽

Slot Filling

Intent detection and slot filling are recognized as two very important tasks in a spoken language understanding (SLU) system. In order to model these two tasks at the same time, many joint models based on deep neural networks have been proposed recently and archived excellent results. In addition, graph neural network has made good achievements in the field of vision. Therefore, we combine these two advantages and propose a new joint model with a wheel-graph attention network (Wheel-GAT), which is able to model interrelated connections directly for single intent detection and slot filling. To construct a graph structure for utterances, we create intent nodes, slot nodes, and directed edges. Intent nodes can provide utterance-level semantic information for slot filling, while slot nodes can also provide local keyword information for intent detection. The two tasks promote each other and carry out end-to-end training at the same time. Experiments show that our proposed approach is superior to multiple baselines on ATIS and SNIPS datasets. Besides, we also demonstrate that using bi-directional encoder representation from transformer (BERT) model further boosts the performance of the SLU task.

Download Full-text

Joint Slot Filling and Intent Detection in Spoken Language Understanding by Hybrid CNN-LSTM Model

2020 International Conference on Control, Robotics and Intelligent System ◽

10.1145/3437802.3437822 ◽

2020 ◽

Author(s):

Moath Al Ali ◽

Bassel Zaity ◽

Pavel Drobintsev ◽

Hazem Wannous ◽

Igor Chernoruckiy ◽

...

Keyword(s):

Spoken Language ◽

Language Understanding ◽

Spoken Language Understanding ◽

Slot Filling

Download Full-text

Deep Cascade Multi-Task Learning for Slot Filling in Online Shopping Assistant

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v33i01.33016465 ◽

2019 ◽

Vol 33 ◽

pp. 6465-6472 ◽

Cited By ~ 3

Author(s):

Yu Gong ◽

Xusheng Luo ◽

Yu Zhu ◽

Wenwu Ou ◽

Zhao Li ◽

...

Keyword(s):

Natural Language ◽

Knowledge Base ◽

Online Shopping ◽

State Of The Art ◽

Language Understanding ◽

Dialog Systems ◽

Named Entity ◽

Online Test ◽

Benchmark Datasets ◽

Slot Filling

Slot filling is a critical task in natural language understanding (NLU) for dialog systems. State-of-the-art approaches treat it as a sequence labeling problem and adopt such models as BiLSTM-CRF. While these models work relatively well on standard benchmark datasets, they face challenges in the context of E-commerce where the slot labels are more informative and carry richer expressions. In this work, inspired by the unique structure of E-commerce knowledge base, we propose a novel multi-task model with cascade and residual connections, which jointly learns segment tagging, named entity tagging and slot filling. Experiments show the effectiveness of the proposed cascade and residual structures. Our model has a 14.6% advantage in F1 score over the strong baseline methods on a new Chinese E-commerce shopping assistant dataset, while achieving competitive accuracies on a standard dataset. Furthermore, online test deployed on such dominant E-commerce platform shows 130% improvement on accuracy of understanding user utterances. Our model has already gone into production in the E-commerce platform.

Download Full-text

Graph LSTM with Context-Gated Mechanism for Spoken Language Understanding

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v34i05.6499 ◽

2020 ◽

Vol 34 (05) ◽

pp. 9539-9546

Author(s):

Linhao Zhang ◽

Dehong Ma ◽

Xiaodong Zhang ◽

Xiaohui Yan ◽

Houfeng Wang

Keyword(s):

Message Passing ◽

Spoken Language ◽

Language Understanding ◽

Semantic Correlation ◽

Spoken Language Understanding ◽

Extensive Evaluation ◽

Proposed Model ◽

Sequential Models ◽

Almost All ◽

Slot Filling

Much research in recent years has focused on spoken language understanding (SLU), which usually involves two tasks: intent detection and slot filling. Since Yao et al.(2013), almost all SLU systems are RNN-based, which have been shown to suffer various limitations due to their sequential nature. In this paper, we propose to tackle this task with Graph LSTM, which first converts text into a graph and then utilizes the message passing mechanism to learn the node representation. Not only the Graph LSTM addresses the limitations of sequential models, but it can also help to utilize the semantic correlation between slot and intent. We further propose a context-gated mechanism to make better use of context information for slot filling. Our extensive evaluation shows that the proposed model outperforms the state-of-the-art results by a large margin.

Download Full-text