scholarly journals MRD-Net: Multi-Modal Residual Knowledge Distillation for Spoken Question Answering

2021 ◽  
Author(s):  
Chenyu You ◽  
Nuo Chen ◽  
Yuexian Zou
Keyword(s):  
Network Performance ◽  
Question Answering ◽  
State Of The Art ◽  
Speech Community ◽  
Benchmark Datasets ◽  
Knowledge Distillation ◽  
Text Features ◽  
The Common ◽  
The Given ◽  
Residual Errors

Spoken question answering (SQA) has recently drawn considerable attention in the speech community. It requires systems to find correct answers from the given spoken passages simultaneously. The common SQA systems consist of the automatic speech recognition (ASR) module and text-based question answering module. However, previous methods suffer from severe performance degradation due to ASR errors. To alleviate this problem, this work proposes a novel multi-modal residual knowledge distillation method (MRD-Net), which further distills knowledge at the acoustic level from the audio-assistant (Audio-A). Specifically, we utilize the teacher (T) trained on manual transcriptions to guide the training of the student (S) on ASR transcriptions. We also show that introducing an Audio-A helps this procedure by learning residual errors between T and S. Moreover, we propose a simple yet effective attention mechanism to adaptively leverage audio-text features as the new deep attention knowledge to boost the network performance. Extensive experiments demonstrate that the proposed MRD-Net achieves superior results compared with state-of-the-art methods on three spoken question answering benchmark datasets.

scholarly journals Question-aware memory network for multi-hop question answering in human–robot interaction

2021 ◽  
Author(s):  
Xinmeng Li ◽  
Mamoun Alazab ◽  
Qian Li ◽  
Keping Yu ◽  
Quanjun Yin
Keyword(s):  
Question Answering ◽  
State Of The Art ◽  
Human Robot Interaction ◽  
Knowledge Graph ◽  
Robot Interaction ◽  
Natural Language Question ◽  
Memory Network ◽  
The Given ◽  
Fine Tune ◽  
Language Question

AbstractKnowledge graph question answering is an important technology in intelligent human–robot interaction, which aims at automatically giving answer to human natural language question with the given knowledge graph. For the multi-relation question with higher variety and complexity, the tokens of the question have different priority for the triples selection in the reasoning steps. Most existing models take the question as a whole and ignore the priority information in it. To solve this problem, we propose question-aware memory network for multi-hop question answering, named QA2MN, to update the attention on question timely in the reasoning process. In addition, we incorporate graph context information into knowledge graph embedding model to increase the ability to represent entities and relations. We use it to initialize the QA2MN model and fine-tune it in the training process. We evaluate QA2MN on PathQuestion and WorldCup2014, two representative datasets for complex multi-hop question answering. The result demonstrates that QA2MN achieves state-of-the-art Hits@1 accuracy on the two datasets, which validates the effectiveness of our model.

scholarly journals DCPNet: A Densely Connected Pyramid Network for Monocular Depth Estimation

Sensors ◽  
2021 ◽  
Vol 21 (20) ◽  
pp. 6780
Author(s):  
Zhitong Lai ◽  
Rui Tian ◽  
Zhiguo Wu ◽  
Nannan Ding ◽  
Linjian Sun ◽  
...  
Keyword(s):  
Multiple Scales ◽  
Feature Fusion ◽  
State Of The Art ◽  
Depth Estimation ◽  
Multi Scale ◽  
Pyramid Structure ◽  
Benchmark Datasets ◽  
The Common ◽  
Monocular Depth ◽  
Multiple Stages

Pyramid architecture is a useful strategy to fuse multi-scale features in deep monocular depth estimation approaches. However, most pyramid networks fuse features only within the adjacent stages in a pyramid structure. To take full advantage of the pyramid structure, inspired by the success of DenseNet, this paper presents DCPNet, a densely connected pyramid network that fuses multi-scale features from multiple stages of the pyramid structure. DCPNet not only performs feature fusion between the adjacent stages, but also non-adjacent stages. To fuse these features, we design a simple and effective dense connection module (DCM). In addition, we offer a new consideration of the common upscale operation in our approach. We believe DCPNet offers a more efficient way to fuse features from multiple scales in a pyramid-like network. We perform extensive experiments using both outdoor and indoor benchmark datasets (i.e., the KITTI and the NYU Depth V2 datasets) and DCPNet achieves the state-of-the-art results.

scholarly journals DiagnosisQA: A semi-automated pipeline for developing clinician validated diagnosis specific QA datasets.

2021 ◽  
Author(s):  
Shreya Mishra ◽  
Raghav Awasthi ◽  
Frank Papay ◽  
Kamal Maheshawari ◽  
Jacek B Cywinski ◽  
...  
Keyword(s):  
Question Answering ◽  
State Of The Art ◽  
Healthcare Providers ◽  
Unstructured Data ◽  
Significant Progress ◽  
Improve Performance ◽  
Healthcare Data ◽  
Research Areas ◽  
Benchmark Datasets ◽  
Automated Pipeline

Question answering (QA) is one of the oldest research areas of AI and Compu- national Linguistics. QA has seen significant progress with the development of state-of-the-art models and benchmark datasets over the last few years. However, pre-trained QA models perform poorly for clinical QA tasks, presumably due to the complexity of electronic healthcare data. With the digitization of healthcare data and the increasing volume of unstructured data, it is extremely important for healthcare providers to have a mechanism to query the data to find appropriate answers. Since diagnosis is central to any decision-making for the clinicians and patients, we have created a pipeline to develop diagnosis-specific QA datasets and curated a QA database for the Cerebrovascular Accident (CVA). CVA, also commonly known as Stroke, is an important and commonly occurring diagnosis amongst critically ill patients. Our method when compared to clinician validation achieved an accuracy of 0.90(with 90% CI [0.82,0.99]). Using our method, we hope to overcome the key challenges of building and validating a highly accurate QA dataset in a semiautomated manner which can help improve performance of QA models.

scholarly journals Unified Vision-Language Pre-Training for Image Captioning and VQA

2020 ◽  
Vol 34 (07) ◽  
pp. 13041-13049 ◽  
Author(s):  
Luowei Zhou ◽  
Hamid Palangi ◽  
Lei Zhang ◽  
Houdong Hu ◽  
Jason Corso ◽  
...  
Keyword(s):  
Unsupervised Learning ◽  
Question Answering ◽  
State Of The Art ◽  
Learning Objectives ◽  
Image Captioning ◽  
Language Generation ◽  
Visual Question Answering ◽  
Benchmark Datasets

This paper presents a unified Vision-Language Pre-training (VLP) model. The model is unified in that (1) it can be fine-tuned for either vision-language generation (e.g., image captioning) or understanding (e.g., visual question answering) tasks, and (2) it uses a shared multi-layer transformer network for both encoding and decoding, which differs from many existing methods where the encoder and decoder are implemented using separate models. The unified VLP model is pre-trained on a large amount of image-text pairs using the unsupervised learning objectives of two tasks: bidirectional and sequence-to-sequence (seq2seq) masked vision-language prediction. The two tasks differ solely in what context the prediction conditions on. This is controlled by utilizing specific self-attention masks for the shared transformer network. To the best of our knowledge, VLP is the first reported model that achieves state-of-the-art results on both vision-language generation and understanding tasks, as disparate as image captioning and visual question answering, across three challenging benchmark datasets: COCO Captions, Flickr30k Captions, and VQA 2.0. The code and the pre-trained models are available at https://github.com/LuoweiZhou/VLP.

scholarly journals Multistructure-Based Collaborative Online Distillation

Entropy ◽  
2019 ◽  
Vol 21 (4) ◽  
pp. 357
Author(s):  
Liang Gao ◽  
Xu Lan ◽  
Haibo Mi ◽  
Dawei Feng ◽  
Kele Xu ◽  
...  
Keyword(s):  
Performance Improvement ◽  
Network Performance ◽  
State Of The Art ◽  
Autonomous Driving ◽  
Supplementary Information ◽  
Training Method ◽  
Machine Learning Methods ◽  
Information Diversity ◽  
Knowledge Distillation ◽  
Memory Resources

Recently, deep learning has achieved state-of-the-art performance in more aspects than traditional shallow architecture-based machine-learning methods. However, in order to achieve higher accuracy, it is usually necessary to extend the network depth or ensemble the results of different neural networks. Increasing network depth or ensembling different networks increases the demand for memory resources and computing resources. This leads to difficulties in deploying depth-learning models in resource-constrained scenarios such as drones, mobile phones, and autonomous driving. Improving network performance without expanding the network scale has become a hot topic for research. In this paper, we propose a cross-architecture online-distillation approach to solve this problem by transmitting supplementary information on different networks. We use the ensemble method to aggregate networks of different structures, thus forming better teachers than traditional distillation methods. In addition, discontinuous distillation with progressively enhanced constraints is used to replace fixed distillation in order to reduce loss of information diversity in the distillation process. Our training method improves the distillation effect and achieves strong network-performance improvement. We used some popular models to validate the results. On the CIFAR100 dataset, AlexNet’s accuracy was improved by 5.94%, VGG by 2.88%, ResNet by 5.07%, and DenseNet by 1.28%. Extensive experiments were conducted to demonstrate the effectiveness of the proposed method. On the CIFAR10, CIFAR100, and ImageNet datasets, we observed significant improvements over traditional knowledge distillation.

scholarly journals Beyond RNNs: Positional Self-Attention with Co-Attention for Video Question Answering

2019 ◽  
Vol 33 ◽  
pp. 8658-8665 ◽  
Author(s):  
Xiangpeng Li ◽  
Jingkuan Song ◽  
Lianli Gao ◽  
Xianglong Liu ◽  
Wenbing Huang ◽  
...  
Keyword(s):  
Question Answering ◽  
State Of The Art ◽  
Computation Time ◽  
Comparable Result ◽  
Video Encoding ◽  
Visual Question Answering ◽  
Proposed Model ◽  
Ablation Study ◽  
The Given ◽  
Video Question Answering

Most of the recent progresses on visual question answering are based on recurrent neural networks (RNNs) with attention. Despite the success, these models are often timeconsuming and having difficulties in modeling long range dependencies due to the sequential nature of RNNs. We propose a new architecture, Positional Self-Attention with Coattention (PSAC), which does not require RNNs for video question answering. Specifically, inspired by the success of self-attention in machine translation task, we propose a Positional Self-Attention to calculate the response at each position by attending to all positions within the same sequence, and then add representations of absolute positions. Therefore, PSAC can exploit the global dependencies of question and temporal information in the video, and make the process of question and video encoding executed in parallel. Furthermore, in addition to attending to the video features relevant to the given questions (i.e., video attention), we utilize the co-attention mechanism by simultaneously modeling “what words to listen to” (question attention). To the best of our knowledge, this is the first work of replacing RNNs with selfattention for the task of visual question answering. Experimental results of four tasks on the benchmark dataset show that our model significantly outperforms the state-of-the-art on three tasks and attains comparable result on the Count task. Our model requires less computation time and achieves better performance compared with the RNNs-based methods. Additional ablation study demonstrates the effect of each component of our proposed model.

scholarly journals Few-Shot Image and Sentence Matching via Gated Visual-Semantic Embedding

2019 ◽  
Vol 33 ◽  
pp. 8489-8496 ◽  
Author(s):  
Yan Huang ◽  
Yang Long ◽  
Liang Wang
Keyword(s):  
Performance Improvement ◽  
State Of The Art ◽  
Similarity Metrics ◽  
Relative Importance ◽  
Challenging Problem ◽  
Semantic Embedding ◽  
Proposed Model ◽  
Benchmark Datasets ◽  
The Common ◽  
Sentence Matching

Although image and sentence matching has been widely studied, its intrinsic few-shot problem is commonly ignored, which has become a bottleneck for further performance improvement. In this work, we focus on this challenging problem of few-shot image and sentence matching, and propose a Gated Visual-Semantic Embedding (GVSE) model to deal with it. The model consists of three corporative modules in terms of uncommon VSE, common VSE, and gated metric fusion. The uncommon VSE exploits external auxiliary resources to extract generic features for representing uncommon instances and words in images and sentences, and then integrates them by modeling their semantic relation to obtain global representations for association analysis. To better model other common instances and words in rest content of images and sentences, the common VSE learns their discriminative representations directly from scratch. After obtaining two similarity metrics from the two VSE modules with different advantages, the gated metric fusion module adaptively fuses them by automatically balancing their relative importance. Based on the fused metric, we perform extensive experiments in terms of few-shot and conventional image and sentence matching, and demonstrate the effectiveness of the proposed model by achieving the state-of-the-art results on two public benchmark datasets.

Health assistant: answering your questions anytime from biomedical literature

2019 ◽  
Vol 35 (20) ◽  
pp. 4129-4139 ◽  
Author(s):  
Zan-Xia Jin ◽  
Bo-Wen Zhang ◽  
Fan Fang ◽  
Le-Le Zhang ◽  
Xu-Cheng Yin
Keyword(s):  
Query Processing ◽  
Large Scale ◽  
Question Answering ◽  
State Of The Art ◽  
Document Retrieval ◽  
Biomedical Literature ◽  
Search Process ◽  
Short Text ◽  
Benchmark Datasets ◽  
Health Assistant

Abstract Motivation With the abundant medical resources, especially literature available online, it is possible for people to understand their own health status and relevant problems autonomously. However, how to obtain the most appropriate answer from the increasingly large-scale database, remains a great challenge. Here, we present a biomedical question answering framework and implement a system, Health Assistant, to enable the search process. Methods In Health Assistant, a search engine is firstly designed to rank biomedical documents based on contents. Then various query processing and search techniques are utilized to find the relevant documents. Afterwards, the titles and abstracts of top-N documents are extracted to generate candidate snippets. Finally, our own designed query processing and retrieval approaches for short text are applied to locate the relevant snippets to answer the questions. Results Our system is evaluated on the BioASQ benchmark datasets, and experimental results demonstrate the effectiveness and robustness of our system, compared to BioASQ participant systems and some state-of-the-art methods on both document retrieval and snippet retrieval tasks. Availability and implementation A demo of our system is available at https://github.com/jinzanxia/biomedical-QA.

scholarly journals Hierarchical Knowledge Squeezed Adversarial Network Compression

2020 ◽  
Vol 34 (07) ◽  
pp. 11370-11377
Author(s):  
Peng Li ◽  
Chang Shu ◽  
Yuan Xie ◽  
Yan Qu ◽  
Hui Kong
Keyword(s):  
State Of The Art ◽  
Teacher Student ◽  
Adversarial Network ◽  
Benchmark Datasets ◽  
Knowledge Distillation ◽  
Adversarial Training ◽  
Rich Information ◽  
Process Oriented ◽  
Transfer Method ◽  
Network Compression

Deep network compression has been achieved notable progress via knowledge distillation, where a teacher-student learning manner is adopted by using predetermined loss. Recently, more focuses have been transferred to employ the adversarial training to minimize the discrepancy between distributions of output from two networks. However, they always emphasize on result-oriented learning while neglecting the scheme of process-oriented learning, leading to the loss of rich information contained in the whole network pipeline. Whereas in other (non GAN-based) process-oriented methods, the knowledge have usually been transferred in a redundant manner. Observing that, the small network can not perfectly mimic a large one due to the huge gap of network scale, we propose a knowledge transfer method, involving effective intermediate supervision, under the adversarial training framework to learn the student network. Different from the other intermediate supervision methods, we design the knowledge representation in a compact form by introducing a task-driven attention mechanism. Meanwhile, to improve the representation capability of the attention-based method, a hierarchical structure is utilized so that powerful but highly squeezed knowledge is realized and the knowledge from teacher network could accommodate the size of student network. Extensive experimental results on three typical benchmark datasets, i.e., CIFAR-10, CIFAR-100, and ImageNet, demonstrate that our method achieves highly superior performances against state-of-the-art methods.

scholarly journals Semantic Sentence Matching with Densely-Connected Recurrent and Co-Attentive Information

2019 ◽  
Vol 33 ◽  
pp. 6586-6593 ◽  
Author(s):  
Seonhoon Kim ◽  
Inho Kang ◽  
Nojun Kwak
Keyword(s):  
Neural Network ◽  
Natural Language ◽  
Question Answering ◽  
State Of The Art ◽  
Attention Mechanism ◽  
Semantic Relationship ◽  
Convolutional Network ◽  
Benchmark Datasets ◽  
Feature Information ◽  
Sentence Matching

Sentence matching is widely used in various natural language tasks such as natural language inference, paraphrase identification, and question answering. For these tasks, understanding logical and semantic relationship between two sentences is required but it is yet challenging. Although attention mechanism is useful to capture the semantic relationship and to properly align the elements of two sentences, previous methods of attention mechanism simply use a summation operation which does not retain original features enough. Inspired by DenseNet, a densely connected convolutional network, we propose a densely-connected co-attentive recurrent neural network, each layer of which uses concatenated information of attentive features as well as hidden features of all the preceding recurrent layers. It enables preserving the original and the co-attentive feature information from the bottommost word embedding layer to the uppermost recurrent layer. To alleviate the problem of an ever-increasing size of feature vectors due to dense concatenation operations, we also propose to use an autoencoder after dense concatenation. We evaluate our proposed architecture on highly competitive benchmark datasets related to sentence matching. Experimental results show that our architecture, which retains recurrent and attentive features, achieves state-of-the-art performances for most of the tasks.

Sign in / Sign up

Export Citation Format

Share Document