scholarly journals Hybrid Attention Network for Language-Based Person Search

Sensors ◽  
2020 ◽  
Vol 20 (18) ◽  
pp. 5279
Author(s):  
Yang Li ◽  
Huahu Xu ◽  
Junsheng Xiao
Keyword(s):  
Image Features ◽  
Attention Mechanism ◽  
Feature Representation ◽  
Semantic Features ◽  
Retrieval Task ◽  
Attention Network ◽  
Fine Grained ◽  
Person Search ◽  
High Level ◽  
Language Description

Language-based person search retrieves images of a target person using natural language description and is a challenging fine-grained cross-modal retrieval task. A novel hybrid attention network is proposed for the task. The network includes the following three aspects: First, a cubic attention mechanism for person image, which combines cross-layer spatial attention and channel attention. It can fully excavate both important midlevel details and key high-level semantics to obtain better discriminative fine-grained feature representation of a person image. Second, a text attention network for language description, which is based on bidirectional LSTM (BiLSTM) and self-attention mechanism. It can better learn the bidirectional semantic dependency and capture the key words of sentences, so as to extract the context information and key semantic features of the language description more effectively and accurately. Third, a cross-modal attention mechanism and a joint loss function for cross-modal learning, which can pay more attention to the relevant parts between text and image features. It can better exploit both the cross-modal and intra-modal correlation and can better solve the problem of cross-modal heterogeneity. Extensive experiments have been conducted on the CUHK-PEDES dataset. Our approach obtains higher performance than state-of-the-art approaches, demonstrating the advantage of the approach we propose.

Text-Image Retrieval With Salient Features

2021 ◽  
Vol 32 (4) ◽  
pp. 1-13
Author(s):  
Xia Feng ◽  
Zhiyi Hu ◽  
Caihua Liu ◽  
W. H. Ip ◽  
Huiying Chen
Keyword(s):  
Image Retrieval ◽  
Recall Rate ◽  
Image Features ◽  
Feature Representation ◽  
Image Feature ◽  
Text And Image ◽  
Retrieval Task ◽  
Retrieval Method ◽  
Salient Features ◽  
Object Level

In recent years, deep learning has achieved remarkable results in the text-image retrieval task. However, only global image features are considered, and the vital local information is ignored. This results in a failure to match the text well. Considering that object-level image features can help the matching between text and image, this article proposes a text-image retrieval method that fuses salient image feature representation. Fusion of salient features at the object level can improve the understanding of image semantics and thus improve the performance of text-image retrieval. The experimental results show that the method proposed in the paper is comparable to the latest methods, and the recall rate of some retrieval results is better than the current work.

Distractor-Aware Tracking with Multi-Task and Dynamic Feature Learning

2020 ◽  
pp. 2150031
Author(s):  
Weichun Liu ◽  
Xiaoan Tang ◽  
Chenglin Zhao
Keyword(s):  
Correlation Filter ◽  
Coarse Grained ◽  
Dynamic Feature ◽  
Semantic Features ◽  
Low Level ◽  
Fine Grained ◽  
Semantic Embedding ◽  
Training Stage ◽  
Online Tracking ◽  
High Level

Recently, deep trackers based on the siamese networking are enjoying increasing popularity in the tracking community. Generally, those trackers learn a high-level semantic embedding space for feature representation but lose low-level fine-grained details. Meanwhile, the learned high-level semantic features are not updated during online tracking, which results in tracking drift in presence of target appearance variation and similar distractors. In this paper, we present a novel end-to-end trainable Convolutional Neural Network (CNN) based on the siamese network for distractor-aware tracking. It enhances target appearance representation in both the offline training stage and online tracking stage. In the offline training stage, this network learns both the low-level fine-grained details and high-level coarse-grained semantics simultaneously in a multi-task learning framework. The low-level features with better resolution are complementary to semantic features and able to distinguish the foreground target from background distractors. In the online stage, the learned low-level features are fed into a correlation filter layer and updated in an interpolated manner to encode target appearance variation adaptively. The learned high-level features are fed into a cross-correlation layer without online update. Therefore, the proposed tracker benefits from both the adaptability of the fine-grained correlation filter and the generalization capability of the semantic embedding. Extensive experiments are conducted on the public OTB100 and UAV123 benchmark datasets. Our tracker achieves state-of-the-art performance while running with a real-time frame-rate.

scholarly journals Improved 3D U-Net for COVID-19 Chest CT Image Segmentation

2021 ◽  
Vol 2021 ◽  
pp. 1-9
Author(s):  
Ruiyong Zheng ◽  
Yongguo Zheng ◽  
Changlei Dong-Ye
Keyword(s):  
Automatic Segmentation ◽  
Image Features ◽  
Chest Ct ◽  
Feature Representation ◽  
Global Information ◽  
Similarity Coefficients ◽  
Information Interaction ◽  
Channel Information ◽  
High Level ◽  
Different Levels

Coronavirus disease 2019 (COVID-19) has spread rapidly worldwide. The rapid and accurate automatic segmentation of COVID-19 infected areas using chest computed tomography (CT) scans is critical for assessing disease progression. However, infected areas have irregular sizes and shapes. Furthermore, there are large differences between image features. We propose a convolutional neural network, named 3D CU-Net, to automatically identify COVID-19 infected areas from 3D chest CT images by extracting rich features and fusing multiscale global information. 3D CU-Net is based on the architecture of 3D U-Net. We propose an attention mechanism for 3D CU-Net to achieve local cross-channel information interaction in an encoder to enhance different levels of the feature representation. At the end of the encoder, we design a pyramid fusion module with expanded convolutions to fuse multiscale context information from high-level features. The Tversky loss is used to resolve the problems of the irregular size and uneven distribution of lesions. Experimental results show that 3D CU-Net achieves excellent segmentation performance, with Dice similarity coefficients of 96.3% and 77.8% in the lung and COVID-19 infected areas, respectively. 3D CU-Net has high potential to be used for diagnosing COVID-19.

scholarly journals Dig into Multi-modal Cues for Video Retrieval with Hierarchical Alignment

2021 ◽  
Author(s):  
Wenzhe Wang ◽  
Mengdan Zhang ◽  
Runnan Chen ◽  
Guanyu Cai ◽  
Penghao Zhou ◽  
...  
Keyword(s):  
Video Retrieval ◽  
State Of The Art ◽  
Text Retrieval ◽  
Local Alignment ◽  
Global Alignment ◽  
Retrieval Task ◽  
Fine Grained ◽  
Global View ◽  
High Level ◽  
Semantic Correspondence

Multi-modal cues presented in videos are usually beneficial for the challenging video-text retrieval task on internet-scale datasets. Recent video retrieval methods take advantage of multi-modal cues by aggregating them to holistic high-level semantics for matching with text representations in a global view. In contrast to this global alignment, the local alignment of detailed semantics encoded within both multi-modal cues and distinct phrases is still not well conducted. Thus, in this paper, we leverage the hierarchical video-text alignment to fully explore the detailed diverse characteristics in multi-modal cues for fine-grained alignment with local semantics from phrases, as well as to capture a high-level semantic correspondence. Specifically, multi-step attention is learned for progressively comprehensive local alignment and a holistic transformer is utilized to summarize multi-modal cues for global alignment. With hierarchical alignment, our model outperforms state-of-the-art methods on three public video retrieval datasets.

Fully Convolutional DenseNet with Attention Mechanism for Liver Lesion Segmentation in MRI Images

2021 ◽  
Vol 11 (8) ◽  
pp. 2231-2242
Author(s):  
Fei Gao ◽  
Kai Qiao ◽  
Jinjin Hai ◽  
Bin Yan ◽  
Minghui Wu ◽  
...  
Keyword(s):  
Liver Tumor ◽  
Feature Fusion ◽  
Liver Tumors ◽  
Contextual Information ◽  
Liver Lesion ◽  
Attention Mechanism ◽  
Tumor Segmentation ◽  
Semantic Features ◽  
Lesion Segmentation ◽  
High Level

The goal of this research is to achieve accurate segmentation of liver tumors in noncontrast T2-weighted magnetic resonance imaging. As liver tumors and adjacent organs are represented by pixels of very similar gray intensity, segmentation is challenging, and the presence of different sizes of liver tumor makes segmentation more difficult. Differing from previous work to capture contextual information using multiscale feature fusion with concatenation, attention mechanism is added to our segmentation model to extract precise global contextual information for pixel labeling without requiring complex dilated convolution. This study describe a liver lesion segmentation model derived from FC-DenseNet with attention mechanism. Specifically, a global attention module (GAM) is added to up-sampling path, and high-level features are processed by the GAM to generating weighting information for guiding high resolution detail features recovery. High-level features are very effective for accurate category classification, but relatively weak at pixel classification and predicting restoration of the original resolution, so the fusion of high-level semantic features and low-level detail features can improve segmentation accuracy. A weighted focal loss function is used to solve the problem of lesion area occupying a relatively low proportion of the whole image, and to deal with the disequilibrium of foreground and background in the training liver lesion images. Experimental results show our segmentation model can automatically segment liver tumors from complete MRI images, and the addition of the GAM model can effectively improve liver tumor segmentation. Our algorithms have obvious advantages over other CNN algorithms and traditional manual methods of feature extraction.

scholarly journals Knowledge Graph Representation Fusion Framework for Fine-Grained Object Recognition in Smart Cities

Complexity ◽  
2021 ◽  
Vol 2021 ◽  
pp. 1-9
Author(s):  
Yang He ◽  
Ling Tian ◽  
Lizong Zhang ◽  
Xi Zeng
Keyword(s):  
Knowledge Representation ◽  
Image Classification ◽  
Visual Information ◽  
Smart Cities ◽  
Image Features ◽  
Graph Representation ◽  
Knowledge Graph ◽  
Fine Grained ◽  
High Level ◽  
Traditional Approaches

Autonomous object detection powered by cutting-edge artificial intelligent techniques has been an essential component for sustaining complex smart city systems. Fine-grained image classification focuses on recognizing subcategories of specific levels of images. As a result of the high similarity between images in the same category and the high dissimilarity in the same subcategories, it has always been a challenging problem in computer vision. Traditional approaches usually rely on exploring only the visual information in images. Therefore, this paper proposes a novel Knowledge Graph Representation Fusion (KGRF) framework to introduce prior knowledge into fine-grained image classification task. Specifically, the Graph Attention Network (GAT) is employed to learn the knowledge representation from the constructed knowledge graph modeling the categories-subcategories and subcategories-attributes associations. By introducing the Multimodal Compact Bilinear (MCB) module, the framework can fully integrate the knowledge representation and visual features for learning the high-level image features. Extensive experiments on the Caltech-UCSD Birds-200-2011 dataset verify the superiority of our proposed framework over several existing state-of-the-art methods.

scholarly journals Compound Multiscale Weak Dense Network with Hybrid Attention for Hyperspectral Image Classification

2021 ◽  
Vol 13 (16) ◽  
pp. 3305
Author(s):  
Zixian Ge ◽  
Guo Cao ◽  
Hao Shi ◽  
Youqiang Zhang ◽  
Xuesong Li ◽  
...  
Keyword(s):  
Feature Extraction ◽  
Hyperspectral Image ◽  
Spectral Feature ◽  
Research Direction ◽  
Classification Performance ◽  
Performance Comparison ◽  
Attention Mechanism ◽  
Semantic Features ◽  
Public Datasets ◽  
High Level

Recently, hyperspectral image (HSI) classification has become a popular research direction in remote sensing. The emergence of convolutional neural networks (CNNs) has greatly promoted the development of this field and demonstrated excellent classification performance. However, due to the particularity of HSIs, redundant information and limited samples pose huge challenges for extracting strong discriminative features. In addition, addressing how to fully mine the internal correlation of the data or features based on the existing model is also crucial in improving classification performance. To overcome the above limitations, this work presents a strong feature extraction neural network with an attention mechanism. Firstly, the original HSI is weighted by means of the hybrid spectral–spatial attention mechanism. Then, the data are input into a spectral feature extraction branch and a spatial feature extraction branch, composed of multiscale feature extraction modules and weak dense feature extraction modules, to extract high-level semantic features. These two features are compressed and fused using the global average pooling and concat approaches. Finally, the classification results are obtained by using two fully connected layers and one Softmax layer. A performance comparison shows the enhanced classification performance of the proposed model compared to the current state of the art on three public datasets.

Truncated attention mechanism and cascade loss for cross-modal person re-identification

2021 ◽  
pp. 1-13
Author(s):  
Shuo Shi ◽  
Changwei Huo ◽  
Yingchun Guo ◽  
Stephen Lean ◽  
Gang Yan ◽  
...  
Keyword(s):  
Natural Language ◽  
Short Term Memory ◽  
Principal Component ◽  
Image Features ◽  
Attention Mechanism ◽  
Text And Image ◽  
Level Information ◽  
Text Features ◽  
Lstm Network ◽  
Language Description

Person re-identification with natural language description is a process of retrieving the corresponding person’s image from an image dataset according to a text description of the person. The key challenge in this cross-modal task is to extract visual and text features and construct loss functions to achieve cross-modal matching between text and image. Firstly, we designed a two-branch network framework for person re-identification with natural language description. In this framework we include the following: a Bi-directional Long Short-Term Memory (Bi-LSTM) network is used to extract text features and a truncated attention mechanism is proposed to select the principal component of the text features; a MobileNet is used to extract image features. Secondly, we proposed a Cascade Loss Function (CLF), which includes cross-modal matching loss and single modal classification loss, both with relative entropy function, to fully exploit the identity-level information. The experimental results on the CUHK-PEDES dataset demonstrate that our method achieves better results in Top-5 and Top-10 than other current 10 state-of-the-art algorithms.

scholarly journals Pose-Guided Multi-Granularity Attention Network for Text-Based Person Search

2020 ◽  
Vol 34 (07) ◽  
pp. 11189-11196 ◽  
Author(s):  
Ya Jing ◽  
Chenyang Si ◽  
Junbo Wang ◽  
Wei Wang ◽  
Liang Wang ◽  
...  
Keyword(s):  
State Of The Art ◽  
Image Database ◽  
Body Part ◽  
Matching Problem ◽  
Attention Network ◽  
Fine Grained ◽  
Person Description ◽  
Person Search ◽  
Semantic Alignment ◽  
Semantic Relevance

Text-based person search aims to retrieve the corresponding person images in an image database by virtue of a describing sentence about the person, which poses great potential for various applications such as video surveillance. Extracting visual contents corresponding to the human description is the key to this cross-modal matching problem. Moreover, correlated images and descriptions involve different granularities of semantic relevance, which is usually ignored in previous methods. To exploit the multilevel corresponding visual contents, we propose a pose-guided multi-granularity attention network (PMA). Firstly, we propose a coarse alignment network (CA) to select the related image regions to the global description by a similarity-based attention. To further capture the phrase-related visual body part, a fine-grained alignment network (FA) is proposed, which employs pose information to learn latent semantic alignment between visual body part and textual noun phrase. To verify the effectiveness of our model, we perform extensive experiments on the CUHK Person Description Dataset (CUHK-PEDES) which is currently the only available dataset for text-based person search. Experimental results show that our approach outperforms the state-of-the-art methods by 15 % in terms of the top-1 metric.

scholarly journals Scene semantics involuntarily guide attention during visual search

2019 ◽  
Author(s):  
Taylor R. Hayes ◽  
John M. Henderson
Keyword(s):  
Visual Search ◽  
Search Task ◽  
Visual Search Task ◽  
Image Features ◽  
Semantic Features ◽  
Low Level ◽  
Overt Attention ◽  
Scene Viewing ◽  
High Level ◽  
Task Goals

During scene viewing, is attention primarily guided by low-level image salience or by high-level semantics? Recent evidence suggests that overt attention in scenes is primarily guided by semantic features. Here we examined whether the attentional priority given to meaningful scene regions is involuntary. Participants completed a scene-independent visual search task in which they searched for superimposed letter targets whose locations were orthogonal to both the underlying scene semantics and image salience. Critically, the analyzed scenes contained no targets, and participants were unaware of this manipulation. We then directly compared how well the distribution of semantic features and image salience accounted for the overall distribution of overt attention. The results showed that even when the task was completely independent from the scene semantics and image salience, semantics explained significantly more variance in attention than image salience and more than expected by chance. This suggests that salient image features were effectively suppressed in favor of task goals, but semantic features were not suppressed. The semantic bias was present from the very first fixation and increased non-monotonically over the course of viewing. These findings suggest that overt attention in scenes is involuntarily guided by scene semantics.

Sign in / Sign up

Export Citation Format

Share Document