An extensive evaluation of deep featuresof convolutional neural networks for saliency prediction of human visual attention

A few-shot personalized saliency prediction based on adaptive image selection considering object and visual attention is presented in this paper. Since general methods predicting personalized saliency maps (PSMs) need a large number of training images, the establishment of a theory using a small number of training images is needed. To tackle this problem, although finding persons who have visual attention similar to that of a target person is effective, all persons have to commonly gaze at many images. Thus, it becomes difficult and unrealistic when considering their burden. On the other hand, this paper introduces a novel adaptive image selection (AIS) scheme that focuses on the relationship between human visual attention and objects in images. AIS focuses on both a diversity of objects in images and a variance of PSMs for the objects. Specifically, AIS selects images so that selected images have various kinds of objects to maintain their diversity. Moreover, AIS guarantees the high variance of PSMs for persons since it represents the regions that many persons commonly gaze at or do not gaze at. The proposed method enables selecting similar users from a small number of images by selecting images that have high diversities and variances. This is the technical contribution of this paper. Experimental results show the effectiveness of our personalized saliency prediction including the new image selection scheme.

Download Full-text

Dilated Convolutional Neural Networks for Panoramic Image Saliency Prediction

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) ◽

10.1109/icassp40776.2020.9053888 ◽

2020 ◽

Author(s):

Feng Dai ◽

Youqiang Zhang ◽

Yike Ma ◽

Hongliang Li ◽

Qiang Zhao

Keyword(s):

Neural Networks ◽

Convolutional Neural Networks ◽

Panoramic Image ◽

Image Saliency ◽

Saliency Prediction

Download Full-text

Beyond saliency: Understanding convolutional neural networks from saliency prediction on layer-wise relevance propagation

Image and Vision Computing ◽

10.1016/j.imavis.2019.02.005 ◽

2019 ◽

Vol 83-84 ◽

pp. 70-86 ◽

Cited By ~ 2

Author(s):

Heyi Li ◽

Yunke Tian ◽

Klaus Mueller ◽

Xin Chen

Keyword(s):

Neural Networks ◽

Convolutional Neural Networks ◽

Saliency Prediction

Download Full-text

SalSAC: A Video Saliency Prediction Model with Shuffled Attentions and Correlation-Based ConvLSTM

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v34i07.6927 ◽

2020 ◽

Vol 34 (07) ◽

pp. 12410-12417 ◽

Cited By ~ 2

Author(s):

Xinyi Wu ◽

Zhenyao Wu ◽

Jinglin Zhang ◽

Lili Ju ◽

Song Wang

Keyword(s):

Neural Network ◽

Neural Networks ◽

Prediction Model ◽

Convolutional Neural Networks ◽

State Of The Art ◽

Dynamic Aspect ◽

Static Information ◽

Saliency Prediction ◽

Multi Level ◽

Video Saliency

The performance of predicting human fixations in videos has been much enhanced with the help of development of the convolutional neural networks (CNN). In this paper, we propose a novel end-to-end neural network “SalSAC” for video saliency prediction, which uses the CNN-LSTM-Attention as the basic architecture and utilizes the information from both static and dynamic aspects. To better represent the static information of each frame, we first extract multi-level features of same size from different layers of the encoder CNN and calculate the corresponding multi-level attentions, then we randomly shuffle these attention maps among levels and multiply them to the extracted multi-level features respectively. Through this way, we leverage the attention consistency across different layers to improve the robustness of the network. On the dynamic aspect, we propose a correlation-based ConvLSTM to appropriately balance the influence of the current and preceding frames to the prediction. Experimental results on the DHF1K, Hollywood2 and UCF-sports datasets show that SalSAC outperforms many existing state-of-the-art methods.

Download Full-text