Graph-based multi-space semantic correlation propagation for video retrieval

Bailan Feng; Juan Cao; Xiuguo Bao; Lei Bao; Yongdong Zhang; Shouxun Lin; Xiaochun Yun

doi:10.1007/s00371-010-0510-6

Graph-based multi-space semantic correlation propagation for video retrieval

The Visual Computer ◽

10.1007/s00371-010-0510-6 ◽

2010 ◽

Vol 27 (1) ◽

pp. 21-34 ◽

Cited By ~ 8

Author(s):

Bailan Feng ◽

Juan Cao ◽

Xiuguo Bao ◽

Lei Bao ◽

Yongdong Zhang ◽

...

Keyword(s):

Video Retrieval ◽

Semantic Correlation ◽

Space Semantic

Download Full-text

Cross-Modal Learning Based on Semantic Correlation and Multi-Task Learning for Text-Video Retrieval

Electronics ◽

10.3390/electronics9122125 ◽

2020 ◽

Vol 9 (12) ◽

pp. 2125

Author(s):

Xiaoyu Wu ◽

Tiantian Wang ◽

Shengjin Wang

Keyword(s):

Video Retrieval ◽

Feature Space ◽

Retrieval Algorithm ◽

Learning Networks ◽

Semantic Association ◽

Semantic Consistency ◽

Semantic Correlation ◽

Task Learning ◽

Feature Subspace ◽

Semantic Similarity Measurement

Text-video retrieval tasks face a great challenge in the semantic gap between cross modal information. Some existing methods transform the text or video into the same subspace to measure their similarity. However, this kind of method does not consider adding a semantic consistency constraint when associating the two modalities of semantic encoding, and the associated result is poor. In this paper, we propose a multi-modal retrieval algorithm based on semantic association and multi-task learning. Firstly, the multi-level features of video or text are extracted based on multiple deep learning networks, so that the information of the two modalities can be fully encoded. Then, in the public feature space where the two modalities information are mapped together, we propose a semantic similarity measurement and semantic consistency classification based on text-video features for a multi-task learning framework. With the semantic consistency classification task, the learning of semantic association task is restrained. So multi-task learning guides the better feature mapping of two modalities and optimizes the construction of unified feature subspace. Finally, the experimental results of our proposed algorithm on the Microsoft Video Description dataset (MSVD) and MSR-Video to Text (MSR-VTT) are better than the existing research, which prove that our algorithm can improve the performance of cross-modal retrieval.

Download Full-text

Exploiting Visual Semantic Reasoning for Video-Text Retrieval

Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence ◽

10.24963/ijcai.2020/140 ◽

2020 ◽

Author(s):

Zerun Feng ◽

Zhimin Zeng ◽

Caili Guo ◽

Zheng Li

Keyword(s):

Video Retrieval ◽

Semantic Relations ◽

Semantic Reasoning ◽

Semantic Correlation ◽

Convolutional Networks ◽

Correlation Graph ◽

Benchmark Datasets ◽

Language Areas ◽

Fully Connected ◽

The Impact

Video retrieval is a challenging research topic bridging the vision and language areas and has attracted broad attention in recent years. Previous works have been devoted to representing videos by directly encoding from frame-level features. In fact, videos consist of various and abundant semantic relations to which existing methods pay less attention. To address this issue, we propose a Visual Semantic Enhanced Reasoning Network (ViSERN) to exploit reasoning between frame regions. Speciﬁcally, we consider frame regions as vertices and construct a fully-connected semantic correlation graph. Then, we perform reasoning by novel random walk rule-based graph convolutional networks to generate region features involved with semantic relations. With the beneﬁt of reasoning, semantic interactions between regions are considered, while the impact of redundancy is suppressed. Finally, the region features are aggregated to form frame-level features for further encoding to measure video-text similarity. Extensive experiments on two public benchmark datasets validate the effectiveness of our method by achieving state-of-the-art performance due to the powerful semantic reasoning.

Download Full-text