scholarly journals Hierarchical Object-oriented Spatio-Temporal Reasoning for Video Question Answering

2021 ◽  
Author(s):  
Long Hoang Dang ◽  
Thao Minh Le ◽  
Vuong Le ◽  
Truyen Tran
Keyword(s):  
Question Answering ◽  
Object Oriented ◽  
Neural Model ◽  
Temporal Reasoning ◽  
General Purpose ◽  
The Arts ◽  
Spatio Temporal ◽  
Event Flow ◽  
High Level ◽  
Video Question Answering

Video Question Answering (Video QA) is a powerful testbed to develop new AI capabilities. This task necessitates learning to reason about objects, relations, and events across visual and linguistic domains in space-time. High-level reasoning demands lifting from associative visual pattern recognition to symbol like manipulation over objects, their behavior and interactions. Toward reaching this goal we propose an object-oriented reasoning approach in that video is abstracted as a dynamic stream of interacting objects. At each stage of the video event flow, these objects interact with each other, and their interactions are reasoned about with respect to the query and under the overall context of a video. This mechanism is materialized into a family of general-purpose neural units and their multi-level architecture called Hierarchical Object-oriented Spatio-Temporal Reasoning (HOSTR) networks. This neural model maintains the objects' consistent lifelines in the form of a hierarchically nested spatio-temporal graph. Within this graph, the dynamic interactive object-oriented representations are built up along the video sequence, hierarchically abstracted in a bottom-up manner, and converge toward the key information for the correct answer. The method is evaluated on multiple major Video QA datasets and establishes new state-of-the-arts in these tasks. Analysis into the model's behavior indicates that object-oriented reasoning is a reliable, interpretable and efficient approach to Video QA.

Video Question Answering with Spatio-Temporal Reasoning

2019 ◽  
Vol 127 (10) ◽  
pp. 1385-1412
Author(s):  
Yunseok Jang ◽  
Yale Song ◽  
Chris Dongjoo Kim ◽  
Youngjae Yu ◽  
Youngjin Kim ◽  
...  
Keyword(s):  
Question Answering ◽  
Temporal Reasoning ◽  
Spatio Temporal ◽  
Video Question Answering

scholarly journals Video Question Answering via Hierarchical Spatio-Temporal Attention Networks

2017 ◽  
Author(s):  
Zhou Zhao ◽  
Qifan Yang ◽  
Deng Cai ◽  
Xiaofei He ◽  
Yueting Zhuang
Keyword(s):  
Visual Information ◽  
Large Scale ◽  
Question Answering ◽  
Temporal Dynamics ◽  
Temporal Attention ◽  
Attention Networks ◽  
Learning Framework ◽  
Spatio Temporal ◽  
The Given ◽  
Video Question Answering

Open-ended video question answering is a challenging problem in visual information retrieval, which automatically generates the natural language answer from the referenced video content according to the question. However, the existing visual question answering works only focus on the static image, which may be ineffectively applied to video question answering due to the temporal dynamics of video contents. In this paper, we consider the problem of open-ended video question answering from the viewpoint of spatio-temporal attentional encoder-decoder learning framework. We propose the hierarchical spatio-temporal attention network for learning the joint representation of the dynamic video contents according to the given question. We then develop the encoder-decoder learning method with reasoning recurrent neural networks for open-ended video question answering. We construct a large-scale video question answering dataset. The extensive experiments show the effectiveness of our method.

scholarly journals Divide and Conquer: Question-Guided Spatio-Temporal Contextual Attention for Video Question Answering

2020 ◽  
Vol 34 (07) ◽  
pp. 11101-11108
Author(s):  
Jianwen Jiang ◽  
Ziqiang Chen ◽  
Haojie Lin ◽  
Xibin Zhao ◽  
Yue Gao
Keyword(s):  
Question Answering ◽  
Divide And Conquer ◽  
Superior Performance ◽  
Temporal Part ◽  
Semantic Features ◽  
Temporal Dimension ◽  
Spatial Part ◽  
Spatio Temporal ◽  
Temporal Dimensions ◽  
Video Question Answering

Understanding questions and finding clues for answers are the key for video question answering. Compared with image question answering, video question answering (Video QA) requires to find the clues accurately on both spatial and temporal dimension simultaneously, and thus is more challenging. However, the relationship between spatio-temporal information and question still has not been well utilized in most existing methods for Video QA. To tackle this problem, we propose a Question-Guided Spatio-Temporal Contextual Attention Network (QueST) method. In QueST, we divide the semantic features generated from question into two separate parts: the spatial part and the temporal part, respectively guiding the process of constructing the contextual attention on spatial and temporal dimension. Under the guidance of the corresponding contextual attention, visual features can be better exploited on both spatial and temporal dimensions. To evaluate the effectiveness of the proposed method, experiments are conducted on TGIF-QA dataset, MSRVTT-QA dataset and MSVD-QA dataset. Experimental results and comparisons with the state-of-the-art methods have shown that our method can achieve superior performance.

scholarly journals Multi-Turn Video Question Answering via Multi-Stream Hierarchical Attention Context Network

2018 ◽  
Author(s):  
Zhou Zhao ◽  
Xinghua Jiang ◽  
Deng Cai ◽  
Jun Xiao ◽  
Xiaofei He ◽  
...  
Keyword(s):  
Visual Information ◽  
Large Scale ◽  
Question Answering ◽  
Context Aware ◽  
Visual Information Retrieval ◽  
Network Learning ◽  
Single Turn ◽  
Spatio Temporal ◽  
Joint Representation ◽  
Video Question Answering

Conversational video question answering is a challenging task in visual information retrieval, which generates the accurate answer from the referenced video contents according to the visual conversation context and given question. However, the existing visual question answering methods mainly tackle the problem of single-turn video question answering, which may be ineffectively applied for multi-turn video question answering directly, due to the insufficiency of modeling the sequential conversation context. In this paper, we study the problem of multi-turn video question answering from the viewpoint of multi-step hierarchical attention context network learning. We first propose the hierarchical attention context network for context-aware question understanding by modeling the hierarchically sequential conversation context structure. We then develop the multi-stream spatio-temporal attention network for learning the joint representation of the dynamic video contents and context-aware question embedding. We next devise the hierarchical attention context network learning method with multi-step reasoning process for multi-turn video question answering. We construct two large-scale multi-turn video question answering datasets. The extensive experiments show the effectiveness of our method.

scholarly journals Feature Augmented Memory with Global Attention Network for VideoQA

2020 ◽  
Author(s):  
Jiayin Cai ◽  
Chun Yuan ◽  
Cheng Shi ◽  
Lei Li ◽  
Yangyang Cheng ◽  
...  
Keyword(s):  
Question Answering ◽  
Temporal Order ◽  
State Of The Art ◽  
Memory Capacity ◽  
Coarse Grain ◽  
Fine Grained ◽  
Feature Representations ◽  
Benchmark Datasets ◽  
High Level ◽  
Video Question Answering

Recently, Recurrent Neural Network (RNN) based methods and Self-Attention (SA) based methods have achieved promising performance in Video Question Answering (VideoQA). Despite the success of these works, RNN-based methods tend to forget the global semantic contents due to the inherent drawbacks of the recurrent units themselves, while SA-based methods cannot precisely capture the dependencies of the local neighborhood, leading to insufficient modeling for temporal order. To tackle these problems, we propose a novel VideoQA framework which progressively refines the representations of videos and questions from fine to coarse grain in a sequence-sensitive manner. Specifically, our model improves the feature representations via the following two steps: (1) introducing two fine-grained feature-augmented memories to strengthen the information augmentation of video and text which can improve memory capacity by memorizing more relevant and targeted information. (2) appending the self-attention and co-attention module to the memory output thus the module is able to capture global interaction between high-level semantic informations. Experimental results show that our approach achieves state-of-the-art performance on VideoQA benchmark datasets.

Sign in / Sign up

Export Citation Format

Share Document