Hierarchical Attention-Based Video Captioning Using Key Frames

Author(s):  
Munusamy Hemalatha ◽  
P. Karthik
Keyword(s):  
2021 ◽  
pp. 1-13
Author(s):  
Tiancheng Qian ◽  
Xue Mei ◽  
Pengxiang Xu ◽  
Kangqi Ge ◽  
Zhelei Qi

Recently many methods use encoder-decoder framework for video captioning, aiming to translate short videos into natural language. These methods usually use equal interval frame sampling. However, lacking a good efficiency in sampling, it has a high temporal and spatial redundancy, resulting in unnecessary computation cost. In addition, the existing approaches simply splice different visual features on the fully connection layer. Therefore, features cannot be effectively utilized. In order to solve the defects, we proposed filtration network (FN) to select key frames, which is trained by deep reinforcement learning algorithm actor-double-critic. According to behavior psychology, the core idea of actor-double-critic is that the behavior of agent is determined by both the external environment and the internal personality. It avoids the phenomenon of unclear reward and sparse feedback in training because it gives steady feedback after each action. The key frames are sent to combine codec network (CCN) to generate sentences. The operation of feature combination in CCN make fusion of visual features by complex number representation to make good semantic modeling. Experiments and comparisons with other methods on two datasets (MSVD/MSR-VTT) show that our approach achieves better performance in terms of four metrics, BLEU-4, METEOR, ROUGE-L and CIDEr.


2019 ◽  
Vol 23 (1) ◽  
pp. 147-159
Author(s):  
Shagan Sah ◽  
Thang Nguyen ◽  
Ray Ptucha

Author(s):  
Alok Singh ◽  
Thoudam Doren Singh ◽  
Sivaji Bandyopadhyay
Keyword(s):  

Author(s):  
Jincan Deng ◽  
Liang Li ◽  
Beichen Zhang ◽  
Shuhui Wang ◽  
Zhengjun Zha ◽  
...  

2021 ◽  
Vol 7 (2) ◽  
pp. 12
Author(s):  
Yousef I. Mohamad ◽  
Samah S. Baraheem ◽  
Tam V. Nguyen

Automatic event recognition in sports photos is both an interesting and valuable research topic in the field of computer vision and deep learning. With the rapid increase and the explosive spread of data, which is being captured momentarily, the need for fast and precise access to the right information has become a challenging task with considerable importance for multiple practical applications, i.e., sports image and video search, sport data analysis, healthcare monitoring applications, monitoring and surveillance systems for indoor and outdoor activities, and video captioning. In this paper, we evaluate different deep learning models in recognizing and interpreting the sport events in the Olympic Games. To this end, we collect a dataset dubbed Olympic Games Event Image Dataset (OGED) including 10 different sport events scheduled for the Olympic Games Tokyo 2020. Then, the transfer learning is applied on three popular deep convolutional neural network architectures, namely, AlexNet, VGG-16 and ResNet-50 along with various data augmentation methods. Extensive experiments show that ResNet-50 with the proposed photobombing guided data augmentation achieves 90% in terms of accuracy.


2018 ◽  
Vol 273 ◽  
pp. 611-621 ◽  
Author(s):  
Sungeun Hong ◽  
Jongbin Ryu ◽  
Woobin Im ◽  
Hyun S. Yang

Sign in / Sign up

Export Citation Format

Share Document