Spatio-Temporal Context Networks for Video Question Answering

2018 ◽  
pp. 108-118
Author(s):  
Kun Gao ◽  
Yahong Han
Keyword(s):  
Question Answering ◽  
Temporal Context ◽  
Spatio Temporal ◽  
Video Question Answering

scholarly journals TVQA+: Spatio-Temporal Grounding for Video Question Answering

2020 ◽  
Author(s):  
Jie Lei ◽  
Licheng Yu ◽  
Tamara Berg ◽  
Mohit Bansal
Keyword(s):  
Question Answering ◽  
Spatio Temporal ◽  
Video Question Answering

scholarly journals Video Question Answering via Hierarchical Spatio-Temporal Attention Networks

2017 ◽  
Author(s):  
Zhou Zhao ◽  
Qifan Yang ◽  
Deng Cai ◽  
Xiaofei He ◽  
Yueting Zhuang
Keyword(s):  
Visual Information ◽  
Large Scale ◽  
Question Answering ◽  
Temporal Dynamics ◽  
Temporal Attention ◽  
Attention Networks ◽  
Learning Framework ◽  
Spatio Temporal ◽  
The Given ◽  
Video Question Answering

Open-ended video question answering is a challenging problem in visual information retrieval, which automatically generates the natural language answer from the referenced video content according to the question. However, the existing visual question answering works only focus on the static image, which may be ineffectively applied to video question answering due to the temporal dynamics of video contents. In this paper, we consider the problem of open-ended video question answering from the viewpoint of spatio-temporal attentional encoder-decoder learning framework. We propose the hierarchical spatio-temporal attention network for learning the joint representation of the dynamic video contents according to the given question. We then develop the encoder-decoder learning method with reasoning recurrent neural networks for open-ended video question answering. We construct a large-scale video question answering dataset. The extensive experiments show the effectiveness of our method.

scholarly journals Divide and Conquer: Question-Guided Spatio-Temporal Contextual Attention for Video Question Answering

2020 ◽  
Vol 34 (07) ◽  
pp. 11101-11108
Author(s):  
Jianwen Jiang ◽  
Ziqiang Chen ◽  
Haojie Lin ◽  
Xibin Zhao ◽  
Yue Gao
Keyword(s):  
Question Answering ◽  
Divide And Conquer ◽  
Superior Performance ◽  
Temporal Part ◽  
Semantic Features ◽  
Temporal Dimension ◽  
Spatial Part ◽  
Spatio Temporal ◽  
Temporal Dimensions ◽  
Video Question Answering

Understanding questions and finding clues for answers are the key for video question answering. Compared with image question answering, video question answering (Video QA) requires to find the clues accurately on both spatial and temporal dimension simultaneously, and thus is more challenging. However, the relationship between spatio-temporal information and question still has not been well utilized in most existing methods for Video QA. To tackle this problem, we propose a Question-Guided Spatio-Temporal Contextual Attention Network (QueST) method. In QueST, we divide the semantic features generated from question into two separate parts: the spatial part and the temporal part, respectively guiding the process of constructing the contextual attention on spatial and temporal dimension. Under the guidance of the corresponding contextual attention, visual features can be better exploited on both spatial and temporal dimensions. To evaluate the effectiveness of the proposed method, experiments are conducted on TGIF-QA dataset, MSRVTT-QA dataset and MSVD-QA dataset. Experimental results and comparisons with the state-of-the-art methods have shown that our method can achieve superior performance.

scholarly journals Uncovering the Temporal Context for Video Question Answering

2017 ◽  
Vol 124 (3) ◽  
pp. 409-421 ◽  
Author(s):  
Linchao Zhu ◽  
Zhongwen Xu ◽  
Yi Yang ◽  
Alexander G. Hauptmann
Keyword(s):  
Question Answering ◽  
Temporal Context ◽  
Video Question Answering

scholarly journals Multi-Turn Video Question Answering via Multi-Stream Hierarchical Attention Context Network

2018 ◽  
Author(s):  
Zhou Zhao ◽  
Xinghua Jiang ◽  
Deng Cai ◽  
Jun Xiao ◽  
Xiaofei He ◽  
...  
Keyword(s):  
Visual Information ◽  
Large Scale ◽  
Question Answering ◽  
Context Aware ◽  
Visual Information Retrieval ◽  
Network Learning ◽  
Single Turn ◽  
Spatio Temporal ◽  
Joint Representation ◽  
Video Question Answering

Conversational video question answering is a challenging task in visual information retrieval, which generates the accurate answer from the referenced video contents according to the visual conversation context and given question. However, the existing visual question answering methods mainly tackle the problem of single-turn video question answering, which may be ineffectively applied for multi-turn video question answering directly, due to the insufficiency of modeling the sequential conversation context. In this paper, we study the problem of multi-turn video question answering from the viewpoint of multi-step hierarchical attention context network learning. We first propose the hierarchical attention context network for context-aware question understanding by modeling the hierarchically sequential conversation context structure. We then develop the multi-stream spatio-temporal attention network for learning the joint representation of the dynamic video contents and context-aware question embedding. We next devise the hierarchical attention context network learning method with multi-step reasoning process for multi-turn video question answering. We construct two large-scale multi-turn video question answering datasets. The extensive experiments show the effectiveness of our method.

Video Question Answering with Spatio-Temporal Reasoning

2019 ◽  
Vol 127 (10) ◽  
pp. 1385-1412
Author(s):  
Yunseok Jang ◽  
Yale Song ◽  
Chris Dongjoo Kim ◽  
Youngjae Yu ◽  
Youngjin Kim ◽  
...  
Keyword(s):  
Question Answering ◽  
Temporal Reasoning ◽  
Spatio Temporal ◽  
Video Question Answering

scholarly journals Hierarchical Object-oriented Spatio-Temporal Reasoning for Video Question Answering

2021 ◽  
Author(s):  
Long Hoang Dang ◽  
Thao Minh Le ◽  
Vuong Le ◽  
Truyen Tran
Keyword(s):  
Question Answering ◽  
Object Oriented ◽  
Neural Model ◽  
Temporal Reasoning ◽  
General Purpose ◽  
The Arts ◽  
Spatio Temporal ◽  
Event Flow ◽  
High Level ◽  
Video Question Answering

Video Question Answering (Video QA) is a powerful testbed to develop new AI capabilities. This task necessitates learning to reason about objects, relations, and events across visual and linguistic domains in space-time. High-level reasoning demands lifting from associative visual pattern recognition to symbol like manipulation over objects, their behavior and interactions. Toward reaching this goal we propose an object-oriented reasoning approach in that video is abstracted as a dynamic stream of interacting objects. At each stage of the video event flow, these objects interact with each other, and their interactions are reasoned about with respect to the query and under the overall context of a video. This mechanism is materialized into a family of general-purpose neural units and their multi-level architecture called Hierarchical Object-oriented Spatio-Temporal Reasoning (HOSTR) networks. This neural model maintains the objects' consistent lifelines in the form of a hierarchically nested spatio-temporal graph. Within this graph, the dynamic interactive object-oriented representations are built up along the video sequence, hierarchically abstracted in a bottom-up manner, and converge toward the key information for the correct answer. The method is evaluated on multiple major Video QA datasets and establishes new state-of-the-arts in these tasks. Analysis into the model's behavior indicates that object-oriented reasoning is a reliable, interpretable and efficient approach to Video QA.

Integrating Video Retrieval and Moment Detection in a Unified Corpus for Video Question Answering

2019 ◽  
Author(s):  
Hongyin Luo ◽  
Mitra Mohtarami ◽  
James Glass ◽  
Karthik Krishnamurthy ◽  
Brigitte Richardson
Keyword(s):  
Question Answering ◽  
Video Retrieval ◽  
Video Question Answering

scholarly journals Spatio-Temporal Context, Correlation Filter and Measurement Estimation Collaboration Based Visual Object Tracking

Sensors ◽  
2021 ◽  
Vol 21 (8) ◽  
pp. 2841
Author(s):  
Khizer Mehmood ◽  
Abdul Jalil ◽  
Ahmad Ali ◽  
Baber Khan ◽  
Maria Murad ◽  
...  
Keyword(s):  
Kalman Filter ◽  
Object Tracking ◽  
Environmental Changes ◽  
Correlation Filter ◽  
Temporal Context ◽  
Visual Object ◽  
Target Model ◽  
Change Of Scale ◽  
Tracking Model ◽  
Spatio Temporal

Despite eminent progress in recent years, various challenges associated with object tracking algorithms such as scale variations, partial or full occlusions, background clutters, illumination variations are still required to be resolved with improved estimation for real-time applications. This paper proposes a robust and fast algorithm for object tracking based on spatio-temporal context (STC). A pyramid representation-based scale correlation filter is incorporated to overcome the STC’s inability on the rapid change of scale of target. It learns appearance induced by variations in the target scale sampled at a different set of scales. During occlusion, most correlation filter trackers start drifting due to the wrong update of samples. To prevent the target model from drift, an occlusion detection and handling mechanism are incorporated. Occlusion is detected from the peak correlation score of the response map. It continuously predicts target location during occlusion and passes it to the STC tracking model. After the successful detection of occlusion, an extended Kalman filter is used for occlusion handling. This decreases the chance of tracking failure as the Kalman filter continuously updates itself and the tracking model. Further improvement to the model is provided by fusion with average peak to correlation energy (APCE) criteria, which automatically update the target model to deal with environmental changes. Extensive calculations on the benchmark datasets indicate the efficacy of the proposed tracking method with state of the art in terms of performance analysis.

Sign in / Sign up

Export Citation Format

Share Document