HCMSL: Hybrid Cross-modal Similarity Learning for Cross-modal Retrieval

2021 ◽  
Vol 17 (1s) ◽  
pp. 1-22
Author(s):  
Chengyuan Zhang ◽  
Jiayu Song ◽  
Xiaofeng Zhu ◽  
Lei Zhu ◽  
Shichao Zhang
Keyword(s):  
Semantic Information ◽  
State Of The Art ◽  
Similarity Learning ◽  
Retrieval Method ◽  
Feature Representations ◽  
High Level ◽  
Fully Connected ◽  
The Relationship ◽  
Hybrid Cross ◽  
Fully Connected Networks

The purpose of cross-modal retrieval is to find the relationship between different modal samples and to retrieve other modal samples with similar semantics by using a certain modal sample. As the data of different modalities presents heterogeneous low-level feature and semantic-related high-level features, the main problem of cross-modal retrieval is how to measure the similarity between different modalities. In this article, we present a novel cross-modal retrieval method, named Hybrid Cross-Modal Similarity Learning model (HCMSL for short). It aims to capture sufficient semantic information from both labeled and unlabeled cross-modal pairs and intra-modal pairs with same classification label. Specifically, a coupled deep fully connected networks are used to map cross-modal feature representations into a common subspace. Weight-sharing strategy is utilized between two branches of networks to diminish cross-modal heterogeneity. Furthermore, two Siamese CNN models are employed to learn intra-modal similarity from samples of same modality. Comprehensive experiments on real datasets clearly demonstrate that our proposed technique achieves substantial improvements over the state-of-the-art cross-modal retrieval techniques.

Masking Models and Watermarking

2011 ◽  
pp. 93-116
Author(s):  
Mirko Luca Lobina ◽  
Luigi Atzori ◽  
Davide Mula
Keyword(s):  
Signal Processing ◽  
Intellectual Property ◽  
State Of The Art ◽  
The State ◽  
Audio Watermarking ◽  
Digital Right Management ◽  
The Future ◽  
Psychological Models ◽  
High Level ◽  
The Relationship

Many audio watermarking techniques presented in the last years make use of masking and psychological models derived from signal processing. Such a basic idea is winning because it guarantees a high level of robustness and bandwidth of the watermark as well as fidelity of the watermarked signal. This chapter first describes the relationship between digital right management, intellectual property, and use of watermarking techniques. Then, the crossing use of watermarking and masking models is detailed, providing schemes, examples, and references. Finally, the authors present two strategies that make use of a masking model, applied to a classic watermarking technique. The joint use of classic frameworks and masking models seems to be one of the trends for the future of research in watermarking. Several tests on the proposed strategies with the state of the art are also offered to give an idea of how to assess the effectiveness of a watermarking technique.

scholarly journals SeDAR: Reading Floorplans Like a Human—Using Deep Learning to Enable Human-Inspired Localisation

2019 ◽  
Vol 128 (5) ◽  
pp. 1286-1310 ◽  
Author(s):  
Oscar Mendez ◽  
Simon Hadfield ◽  
Nicolas Pugeault ◽  
Richard Bowden
Keyword(s):  
Deep Learning ◽  
Semantic Information ◽  
State Of The Art ◽  
Depth Information ◽  
Semantic Maps ◽  
Novel Method ◽  
Rgb Images ◽  
High Level ◽  
Robotic Tasks ◽  
And Robotics

Abstract The use of human-level semantic information to aid robotic tasks has recently become an important area for both Computer Vision and Robotics. This has been enabled by advances in Deep Learning that allow consistent and robust semantic understanding. Leveraging this semantic vision of the world has allowed human-level understanding to naturally emerge from many different approaches. Particularly, the use of semantic information to aid in localisation and reconstruction has been at the forefront of both fields. Like robots, humans also require the ability to localise within a structure. To aid this, humans have designed high-level semantic maps of our structures called floorplans. We are extremely good at localising in them, even with limited access to the depth information used by robots. This is because we focus on the distribution of semantic elements, rather than geometric ones. Evidence of this is that humans are normally able to localise in a floorplan that has not been scaled properly. In order to grant this ability to robots, it is necessary to use localisation approaches that leverage the same semantic information humans use. In this paper, we present a novel method for semantically enabled global localisation. Our approach relies on the semantic labels present in the floorplan. Deep Learning is leveraged to extract semantic labels from RGB images, which are compared to the floorplan for localisation. While our approach is able to use range measurements if available, we demonstrate that they are unnecessary as we can achieve results comparable to state-of-the-art without them.

scholarly journals A Meaningful Compact Key Frames Extraction in Complex Video Shots

2017 ◽  
Vol 7 (3) ◽  
pp. 818
Author(s):  
Manar Abduljabbar Mizher ◽  
Mei Choo Ang ◽  
Ahmad Abdel Jabbar Mazhar
Keyword(s):  
State Of The Art ◽  
The State ◽  
Worst Case ◽  
Key Frame Extraction ◽  
Key Frame ◽  
Easy Problem ◽  
Extraction Algorithm ◽  
Key Frames ◽  
High Level ◽  
The Relationship

Key frame extraction is an essential technique in the computer vision field. The extracted key frames should brief the salient events with an excellent feasibility, great efficiency, and with a high-level of robustness. Thus, it is not an easy problem to solve because it is attributed to many visual features. This paper intends to solve this problem by investigating the relationship between these features detection and the accuracy of key frames extraction techniques using TRIZ. An improved algorithm for key frame extraction was then proposed based on an accumulative optical flow with a self-adaptive threshold (AOF_ST) as recommended in TRIZ inventive principles. Several video shots including original and forgery videos with complex conditions are used to verify the experimental results. The comparison of our results with the-state-of-the-art algorithms results showed that the proposed extraction algorithm can accurately brief the videos and generated a meaningful compact count number of key frames. On top of that, our proposed algorithm achieves 124.4 and 31.4 for best and worst case in KTH dataset extracted key frames in terms of compression rate, while the the-state-of-the-art algorithms achieved 8.90 in the best case.

scholarly journals Pedestrian Attributes Recognition in Surveillance Scenarios Using Multi-Task Lightweight Convolutional Neural Network

2019 ◽  
Vol 9 (19) ◽  
pp. 4182 ◽  
Author(s):  
Pu Yan ◽  
Li Zhuo ◽  
Jiafeng Li ◽  
Hui Zhang ◽  
Jing Zhang
Keyword(s):  
Neural Network ◽  
Convolutional Neural Network ◽  
State Of The Art ◽  
Cross Entropy ◽  
Semantic Features ◽  
Recognition Method ◽  
Relationship Model ◽  
High Level ◽  
Fully Connected ◽  
Video Structuring

Pedestrian attributes (such as gender, age, hairstyle, and clothing) can effectively represent the appearance of pedestrians. These are high-level semantic features that are robust to illumination, deformation, etc. Therefore, they can be widely used in person re-identification, video structuring analysis and other applications. In this paper, a pedestrian attributes recognition method for surveillance scenarios using a multi-task lightweight convolutional neural network is proposed. Firstly, the labels of the attributes for each pedestrian image are integrated into a label vector. Then, a multi-task lightweight Convolutional Neural Network (CNN) is designed, which consists of five convolutional layers, three pooling layers and two fully connected layers to extract the deep features of pedestrian images. Considering that the data distribution of the datasets is unbalanced, the loss function is improved based on the sigmoid cross-entropy, and the scale factor is added to balance the amount of various attributes data. Through training the network, the mapping relationship model between the deep features of pedestrian images and the integration label vector of their attributes is established, which can be used to predict each attribute of the pedestrian. The experiments were conducted on two public pedestrian attributes datasets in surveillance scenarios, namely PETA and RAP. The results show that, compared with the state-of-the-art pedestrian attributes recognition methods, the proposed method can achieve a superior accuracy by 91.88% on PETA and 87.44% on RAP respectively.

scholarly journals LiDAR Odometry and Mapping Based on Semantic Information for Outdoor Environment

2021 ◽  
Vol 13 (15) ◽  
pp. 2864
Author(s):  
Shitong Du ◽  
Yifan Li ◽  
Xuyou Li ◽  
Menghao Wu
Keyword(s):  
Feature Extraction ◽  
Semantic Information ◽  
State Of The Art ◽  
Outdoor Environment ◽  
Geometric Features ◽  
Dynamic Object ◽  
Object Removal ◽  
Localization And Mapping ◽  
High Level ◽  
Crucial Part

Simultaneous Localization and Mapping (SLAM) in an unknown environment is a crucial part for intelligent mobile robots to achieve high-level navigation and interaction tasks. As one of the typical LiDAR-based SLAM algorithms, the Lidar Odometry and Mapping in Real-time (LOAM) algorithm has shown impressive results. However, LOAM only uses low-level geometric features without considering semantic information. Moreover, the lack of a dynamic object removal strategy limits the algorithm to obtain higher accuracy. To this end, this paper extends the LOAM pipeline by integrating semantic information into the original framework. Specifically, we first propose a two-step dynamic objects filtering strategy. Point-wise semantic labels are then used to improve feature extraction and searching for corresponding points. We evaluate the performance of the proposed method in many challenging scenarios, including highway, country and urban from the KITTI dataset. The results demonstrate that the proposed SLAM system outperforms the state-of-the-art SLAM methods in terms of accuracy and robustness.

scholarly journals A Novel Multi-Focus Image Fusion Network with U-Shape Structure

Sensors ◽  
2020 ◽  
Vol 20 (14) ◽  
pp. 3901 ◽  
Author(s):  
Tao Pan ◽  
Jiaqin Jiang ◽  
Jian Yao ◽  
Bin Wang ◽  
Bin Tan
Keyword(s):  
Image Fusion ◽  
Semantic Information ◽  
State Of The Art ◽  
Structural Similarity ◽  
Cross Entropy ◽  
Shape Structure ◽  
Focus Image ◽  
Spatial Pyramid Pooling ◽  
Global Perception ◽  
High Level

Multi-focus image fusion has become a very practical image processing task. It uses multiple images focused on various depth planes to create an all-in-focus image. Although extensive studies have been produced, the performance of existing methods is still limited by the inaccurate detection of the focus regions for fusion. Therefore, in this paper, we proposed a novel U-shape network which can generate an accurate decision map for the multi-focus image fusion. The Siamese encoder of our U-shape network can preserve the low-level cues with rich spatial details and high-level semantic information from the source images separately. Moreover, we introduce the ResBlocks to expand the receptive field, which can enhance the ability of our network to distinguish between focus and defocus regions. Moreover, in the bridge stage between the encoder and decoder, the spatial pyramid pooling is adopted as a global perception fusion module to capture sufficient context information for the learning of the decision map. Finally, we use a hybrid loss that combines the binary cross-entropy loss and the structural similarity loss for supervision. Extensive experiments have demonstrated that the proposed method can achieve the state-of-the-art performance.

scholarly journals Abstraction and Association: Cross-Modal Retrieval Based on Consistency between Semantic Structures

2020 ◽  
Vol 2020 ◽  
pp. 1-17
Author(s):  
Qibin Zheng ◽  
Xiaoguang Ren ◽  
Yi Liu ◽  
Wei Qin
Keyword(s):  
Learning Strategy ◽  
Reference Points ◽  
Training Data ◽  
Semantic Structure ◽  
Retrieval Method ◽  
Multiple Modalities ◽  
Training Samples ◽  
High Level ◽  
Active Learning Strategy ◽  
The Relationship

Cross-modal retrieval aims to find relevant data of different modalities, such as images and text. In order to bridge the modality gap, most existing methods require a lot of coupled sample pairs as training data. To reduce the demands for training data, we propose a cross-modal retrieval framework that utilizes both coupled and uncoupled samples. The framework consists of two parts: Abstraction that aims to provide high-level single-modal representations with uncoupled samples; then, Association links different modalities through a few coupled training samples. Moreover, under this framework, we implement a cross-modal retrieval method based on the consistency between the semantic structure of multiple modalities. First, both images and text are represented with the semantic structure-based representation, which represents each sample as its similarity from the reference points that are generated from single-modal clustering. Then, the reference points of different modalities are aligned through an active learning strategy. Finally, the cross-modal similarity can be measured with the consistency between the semantic structures. The experiment results demonstrate that given proper abstraction of single-modal data, the relationship between different modalities can be simplified, and even limited coupled cross-modal training data are sufficient for satisfactory retrieval accuracy.

Masking Models and Watermarking

2011 ◽  
pp. 30-63
Author(s):  
Mirko Luca Lobina ◽  
Daniele D. Giusto ◽  
Davide Mula
Keyword(s):  
Signal Processing ◽  
Intellectual Property ◽  
State Of The Art ◽  
The State ◽  
Audio Watermarking ◽  
Digital Right Management ◽  
Psychological Models ◽  
High Level ◽  
The Relationship ◽  
Work First

Many audio watermarking techniques, presented in the last years, make use of masking and psychological models derived from signal processing. Such a basic idea is winning because it guarantees a high level of robustness and bandwidth of the watermark as well as fidelity of the watermarked signal. This work first describes the relationship between Digital Right Management, Intellectual Property, and use of watermarking techniques. Then, the crossing use of watermarking and Masking Models is detailed, providing schemes, examples, and references. Finally, the authors present two strategies that make use of a Masking Model, applied to a classic watermarking technique. The joint use of classic frameworks and Masking Models seems to be one of the trends for the future of research in watermarking. Several tests on the proposed strategies with the state of the art are also offered to give an idea of how to assess the effectiveness of a watermarking technique.

scholarly journals Feature Augmented Memory with Global Attention Network for VideoQA

2020 ◽  
Author(s):  
Jiayin Cai ◽  
Chun Yuan ◽  
Cheng Shi ◽  
Lei Li ◽  
Yangyang Cheng ◽  
...  
Keyword(s):  
Question Answering ◽  
Temporal Order ◽  
State Of The Art ◽  
Memory Capacity ◽  
Coarse Grain ◽  
Fine Grained ◽  
Feature Representations ◽  
Benchmark Datasets ◽  
High Level ◽  
Video Question Answering

Recently, Recurrent Neural Network (RNN) based methods and Self-Attention (SA) based methods have achieved promising performance in Video Question Answering (VideoQA). Despite the success of these works, RNN-based methods tend to forget the global semantic contents due to the inherent drawbacks of the recurrent units themselves, while SA-based methods cannot precisely capture the dependencies of the local neighborhood, leading to insufficient modeling for temporal order. To tackle these problems, we propose a novel VideoQA framework which progressively refines the representations of videos and questions from fine to coarse grain in a sequence-sensitive manner. Specifically, our model improves the feature representations via the following two steps: (1) introducing two fine-grained feature-augmented memories to strengthen the information augmentation of video and text which can improve memory capacity by memorizing more relevant and targeted information. (2) appending the self-attention and co-attention module to the memory output thus the module is able to capture global interaction between high-level semantic informations. Experimental results show that our approach achieves state-of-the-art performance on VideoQA benchmark datasets.

scholarly journals Depression Detection on Reddit With an Emotion-Based Attention Network: Algorithm Development and Validation (Preprint)

2021 ◽  
Author(s):  
Lu Ren ◽  
Hongfei Lin ◽  
Bo Xu ◽  
Shaowu Zhang ◽  
Liang Yang ◽  
...  
Keyword(s):  
Semantic Information ◽  
State Of The Art ◽  
Emotion Understanding ◽  
Experimental Results ◽  
Network Module ◽  
Emotional Information ◽  
Attention Network ◽  
Depression Detection ◽  
Self Harm ◽  
High Level

BACKGROUND As a common mental disease, depression seriously affects people’s physical and mental health. According to the statistics of the World Health Organization, depression is one of the main reasons for suicide and self-harm events in the world. Therefore, strengthening depression detection can effectively reduce the occurrence of suicide or self-harm events so as to save more people and families. With the development of computer technology, some researchers are trying to apply natural language processing techniques to detect people who are depressed automatically. Many existing feature engineering methods for depression detection are based on emotional characteristics, but these methods do not consider high-level emotional semantic information. The current deep learning methods for depression detection cannot accurately extract effective emotional semantic information. OBJECTIVE In this paper, we propose an emotion-based attention network, including a semantic understanding network and an emotion understanding network, which can capture the high-level emotional semantic information effectively to improve the depression detection task. METHODS The semantic understanding network module is used to capture the contextual semantic information. The emotion understanding network module is used to capture the emotional semantic information. There are two units in the emotion understanding network module, including a positive emotion understanding unit and a negative emotion understanding unit, which are used to capture the positive emotional information and the negative emotional information, respectively. We further proposed a dynamic fusion strategy in the emotion understanding network module to fuse the positive emotional information and the negative emotional information. RESULTS We evaluated our method on the Reddit data set. The experimental results showed that the proposed emotion-based attention network model achieved an accuracy, precision, recall, and F-measure of 91.30%, 91.91%, 96.15%, and 93.98%, respectively, which are comparable results compared with state-of-the-art methods. CONCLUSIONS The experimental results showed that our model is competitive with the state-of-the-art models. The semantic understanding network module, the emotion understanding network module, and the dynamic fusion strategy are effective modules for depression detection. In addition, the experimental results verified that the emotional semantic information was effective in depression detection.

Sign in / Sign up

Export Citation Format

Share Document