scholarly journals BLOCK: Bilinear Superdiagonal Fusion for Visual Question Answering and Visual Relationship Detection

2019 ◽  
Vol 33 ◽  
pp. 8102-8109 ◽  
Author(s):  
Hedi Ben-younes ◽  
Remi Cadene ◽  
Nicolas Thome ◽  
Matthieu Cord
Keyword(s):  
Deep Learning ◽  
Learning Community ◽  
Question Answering ◽  
Practical Interest ◽  
Tensor Decomposition ◽  
Representation Learning ◽  
Multimodal Fusion ◽  
Practical Implementation ◽  
Fusion Model ◽  
Visual Question Answering

Multimodal representation learning is gaining more and more interest within the deep learning community. While bilinear models provide an interesting framework to find subtle combination of modalities, their number of parameters grows quadratically with the input dimensions, making their practical implementation within classical deep learning pipelines challenging. In this paper, we introduce BLOCK, a new multimodal fusion based on the block-superdiagonal tensor decomposition. It leverages the notion of block-term ranks, which generalizes both concepts of rank and mode ranks for tensors, already used for multimodal fusion. It allows to define new ways for optimizing the tradeoff between the expressiveness and complexity of the fusion model, and is able to represent very fine interactions between modalities while maintaining powerful mono-modal representations. We demonstrate the practical interest of our fusion model by using BLOCK for two challenging tasks: Visual Question Answering (VQA) and Visual Relationship Detection (VRD), where we design end-to-end learnable architectures for representing relevant interactions between modalities. Through extensive experiments, we show that BLOCK compares favorably with respect to state-of-the-art multimodal fusion models for both VQA and VRD tasks. Our code is available at https://github.com/Cadene/block.bootstrap.pytorch.

DecomVQANet: Decomposing visual question answering deep network via tensor decomposition and regression

2021 ◽  
Vol 110 ◽  
pp. 107538 ◽  
Author(s):  
Zongwen Bai ◽  
Ying Li ◽  
Marcin Woźniak ◽  
Meili Zhou ◽  
Di Li
Keyword(s):  
Question Answering ◽  
Tensor Decomposition ◽  
Deep Network ◽  
Visual Question Answering

scholarly journals Variational Approach for Learning Community Structures

Complexity ◽  
2018 ◽  
Vol 2018 ◽  
pp. 1-13 ◽  
Author(s):  
Jun Jin Choong ◽  
Xin Liu ◽  
Tsuyoshi Murata
Keyword(s):  
Deep Learning ◽  
Community Structure ◽  
Learning Community ◽  
Community Detection ◽  
Representation Learning ◽  
Semisupervised Learning ◽  
Detection Methods ◽  
Community Structures ◽  
Learning Tasks ◽  
Proposed Model

Discovering and modeling community structure exist to be a fundamentally challenging task. In domains such as biology, chemistry, and physics, researchers often rely on community detection algorithms to uncover community structures from complex systems yet no unified definition of community structure exists. Furthermore, existing models tend to be oversimplified leading to a neglect of richer information such as nodal features. Coupled with the surge of user generated information on social networks, a demand for newer techniques beyond traditional approaches is inevitable. Deep learning techniques such as network representation learning have shown tremendous promise. More specifically, supervised and semisupervised learning tasks such as link prediction and node classification have achieved remarkable results. However, unsupervised learning tasks such as community detection remain widely unexplored. In this paper, a novel deep generative model for community detection is proposed. Extensive experiments show that the proposed model, empowered with Bayesian deep learning, can provide insights in terms of uncertainty and exploit nonlinearities which result in better performance in comparison to state-of-the-art community detection methods. Additionally, unlike traditional methods, the proposed model is community structure definition agnostic. Leveraging on low-dimensional embeddings of both network topology and feature similarity, it automatically learns the best model configuration for describing similarities in a community.

scholarly journals MMFT-BERT: Multimodal Fusion Transformer with BERT Encodings for Visual Question Answering

2020 ◽  
Author(s):  
Aisha Urooj ◽  
Amir Mazaheri ◽  
Niels Da vitoria lobo ◽  
Mubarak Shah
Keyword(s):  
Question Answering ◽  
Multimodal Fusion ◽  
Visual Question Answering

scholarly journals Integrating Knowledge and Reasoning in Image Understanding

2019 ◽  
Author(s):  
Somak Aditya ◽  
Yezhou Yang ◽  
Chitta Baral
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Object Recognition ◽  
Knowledge Integration ◽  
Question Answering ◽  
Semantic Segmentation ◽  
Image Understanding ◽  
Data Driven ◽  
Integration Methods ◽  
Visual Question Answering

Deep learning based data-driven approaches have been successfully applied in various image understanding applications ranging from object recognition, semantic segmentation to visual question answering. However, the lack of knowledge integration as well as higher-level reasoning capabilities with the methods still pose a hindrance. In this work, we present a brief survey of a few representative reasoning mechanisms, knowledge integration methods and their corresponding image understanding applications developed by various groups of researchers, approaching the problem from a variety of angles. Furthermore, we discuss upon key efforts on integrating external knowledge with neural networks. Taking cues from these efforts, we conclude by discussing potential pathways to improve reasoning capabilities.

scholarly journals Comprehensive Retrospection of Literature Reported Works of Community Question Answering Systems

2020 ◽  
Vol 9 (3) ◽  
pp. 1904-1917
Keyword(s):  
Machine Learning ◽  
Deep Learning ◽  
Learning Community ◽  
Question Answering ◽  
Hybrid Approach ◽  
Open Educational Resources ◽  
Stack Overflow ◽  
Community Question Answering ◽  
Question Answering Systems ◽  
Answer Quality

Community question answering CQA) systems are rapidly gaining attention in the society. Several researchers have actively engaged in improving the theories associated with question answering (QA) systems. This paper reviews the literature reported works on question answering QA systems. In this paper, we discuss on the early contributions on QA systems along with their present and future scope. We have categorized the literature reported works into 20 subgroups according to their significance and relevance. The works in each group will be brought out along with their inter-relevance. Finding the question and answer quality is the prime challenge almost addressed by many researchers. Modeling similar questions, identifying experts in prior and understanding seeker satisfaction also considered as potential challenges. Researchers at the most have done experimentations on popular CQAs like Yahoo! Answers, Wiki Answers, Baidu Knows, Brianly, Quora, Pubmed and Stack Overflow respectively. Machine learning, probabilistic modeling, deep learning and hybrid approach of solving show profound significance in addressing various challenges encounter with QA systems. Today the paradigm of CQA systems took the shift by serving as Open Educational Resources to learning community

A Survey on Representation Learning in Visual Question Answering

2021 ◽  
pp. 326-336
Author(s):  
Manish Sahani ◽  
Priyadarshan Singh ◽  
Sachin Jangpangi ◽  
Shailender Kumar
Keyword(s):  
Question Answering ◽  
Representation Learning ◽  
Visual Question Answering
Sign in / Sign up

Export Citation Format

Share Document