Temporal convolutional network with soft thresholding and attention mechanism for machinery prognostics

Analyzing polysomnography (PSG) is an effective method for evaluating sleep health; however, the sleep stage scoring required for PSG analysis is a time-consuming effort for an experienced medical expert. When scoring sleep epochs, experts pay attention to find specific signal characteristics (e.g., K-complexes and spindles), and sometimes need to integrate information from preceding and subsequent epochs in order to make a decision. To imitate this process and to build a more interpretable deep learning model, we propose a neural network based on a convolutional network (CNN) and attention mechanism to perform automatic sleep staging. The CNN learns local signal characteristics, and the attention mechanism excels in learning inter- and intra-epoch features. In experiments on the public sleep-edf and sleep-edfx databases with different training and testing set partitioning methods, our model achieved overall accuracies of 93.7% and 82.8%, and macro-average F1-scores of 84.5 and 77.8, respectively, outperforming recently reported machine learning-based methods.

Download Full-text

Anchor free object detection with mask attention

10.21203/rs.2.24760/v2 ◽

2020 ◽

Author(s):

HE Yang ◽

Beibei Fan ◽

Ling ling Guo

Keyword(s):

Object Detection ◽

Size Variation ◽

Attention Mechanism ◽

Detection Methods ◽

Convolutional Network ◽

Heat Map ◽

Fire Model ◽

Large Size ◽

Map Generation ◽

Point Detection

Abstract The anchor-free method based on key point detection has made great progress. However, the anchor-free method is too dependent on using a convolutional network to generate a rough heat map. This is difficult to detect for objects with a large size variation and dense and overlapping objects. To solve this problem, first, we propose a mask attention mechanism for object detection methods. And make full use of the advantages of the attention mechanism to improve the accuracy of network detection heat map generation. Then, we designed an optimized fire model to reduce the size of the model. The fire model is an extension of grouped convolution. The fire model allows each group of convolutional network features to learn the same feature through purposeful grouping. In this paper, the mask attention mechanism uses object segmentation images to guide the generation of corner heat maps. Our approach achieved an accuracy of 91.84% and a recall 89.83% in the Tencent-100K dataset. Compared with the popular object detection methods, the proposed method has advantages in model size and accuracy.

Download Full-text

A Hybrid GCN and LSTM Structure Based on Attention Mechanism for Crime Prediction

CONVERTER ◽

10.17762/converter.132 ◽

2021 ◽

pp. 328-338

Author(s):

Jinming Hu

Keyword(s):

Short Term Memory ◽

The United States ◽

Typical Feature ◽

Research Field ◽

Attention Mechanism ◽

Experimental Result ◽

Good Prediction ◽

Convolutional Network ◽

Crime Prediction ◽

New Research

Globalization has been the major contributor to economy boom. While at the same time, it has stimulated the development of crime method as frequent cross-border communication allowed. With the improvement in big data and prediction system of policing work, it has become a new research field to establish an efficient crime prediction model, by which police departments could clamp down on criminal activities more accurately. Besides, this model will be quite beneficial for commanding and dispatching police force thus to improve work efficiency. This paper proposes a combination model, which uses Long Short-Term Memory Network (LSTM) and Graph Convolutional Network (GCN) to predict crime rate and takes advantage of the Attention mechanism to improve the experimental result. By extracting the spatio-temporal characteristics of crimes and increasing the proportion of typical feature, it can not only predict crime quantity, but also detect the degree of crime risk in each region. A rolling forecast of crime data for about three years in the Boston of the United States shows that our model has good prediction performance.

Download Full-text

A Remote Sensing Image Segmentation Method Based on Fusion Mechanism

Journal of Physics Conference Series ◽

10.1088/1742-6596/2138/1/012016 ◽

2021 ◽

Vol 2138 (1) ◽

pp. 012016

Author(s):

Shuangling Zhu ◽

Guli Nazi·Aili Mujiang ◽

Huxidan Jumahong ◽

Pazi Laiti·Nuer Maiti

Keyword(s):

Remote Sensing ◽

Semantic Segmentation ◽

Remote Sensing Image ◽

Detection Algorithm ◽

Attention Mechanism ◽

Segmentation Method ◽

Remote Sensing Images ◽

Convolutional Network ◽

Input Layer ◽

Basic Network

Abstract A U-Net convolutional network structure is fully capable of completing the end-to-end training with extremely little data, and can achieve better results. When the convolutional network has a short link between a near input layer and a near output layer, it can implement training in a deeper, more accurate and effective way. This paper mainly proposes a high-resolution remote sensing image change detection algorithm based on dense convolutional channel attention mechanism. The detection algorithm uses U-Net network module as the basic network to extract features, combines Dense-Net dense module to enhance U-Net, and introduces dense convolution channel attention mechanism into the basic convolution unit to highlight important features, thus completing semantic segmentation of dense convolutional remote sensing images. Simulation results have verified the effectiveness and robustness of this study.

Download Full-text

Depression Speech Recognition With a Three-Dimensional Convolutional Network

Frontiers in Human Neuroscience ◽

10.3389/fnhum.2021.713823 ◽

2021 ◽

Vol 15 ◽

Author(s):

Hongbo Wang ◽

Yu Liu ◽

Xiaoxiao Zhen ◽

Xuyan Tu

Keyword(s):

Feature Extraction ◽

Three Dimensional ◽

Objective Evaluation ◽

Extraction Methods ◽

Attention Mechanism ◽

Speech Signals ◽

Convolutional Network ◽

Depression Detection ◽

Intelligent Technology ◽

Self Learning

Depression has become one of the main afflictions that threaten people's mental health. However, the current traditional diagnosis methods have certain limitations, so it is necessary to find a method of objective evaluation of depression based on intelligent technology to assist in the early diagnosis and treatment of patients. Because the abnormal speech features of patients with depression are related to their mental state to some extent, it is valuable to use speech acoustic features as objective indicators for the diagnosis of depression. In order to solve the problem of the complexity of speech in depression and the limited performance of traditional feature extraction methods for speech signals, this article suggests a Three-Dimensional Convolutional filter bank with Highway Networks and Bidirectional GRU (Gated Recurrent Unit) with an Attention mechanism (in short 3D-CBHGA), which includes two key strategies. (1) The three-dimensional feature extraction of the speech signal can timely realize the expression ability of those depression signals. (2) Based on the attention mechanism in the GRU network, the frame-level vector is weighted to get the hidden emotion vector by self-learning. Experiments show that the proposed 3D-CBHGA can well establish mapping from speech signals to depression-related features and improve the accuracy of depression detection in speech signals.

Download Full-text

Redundant Convolutional Network With Attention Mechanism For Monaural Speech Enhancement

ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) ◽

10.1109/icassp40776.2020.9053277 ◽

2020 ◽

Author(s):

Tian Lan ◽

Yilan Lyu ◽

Guoqiang Hui ◽

Refuoe Mokhosi ◽

Sen Li ◽

...

Keyword(s):

Speech Enhancement ◽

Attention Mechanism ◽

Convolutional Network

Download Full-text

Efficient Attention Mechanism for Dynamic Convolution in Lightweight Neural Network

Applied Sciences ◽

10.3390/app11073111 ◽

2021 ◽

Vol 11 (7) ◽

pp. 3111

Author(s):

Enjie Ding ◽

Yuhao Cheng ◽

Chengcheng Xiao ◽

Zhongyu Liu ◽

Wanli Yu

Keyword(s):

Neural Networks ◽

Spatial Information ◽

Attention Mechanism ◽

Feature Representation ◽

Feature Maps ◽

Convolutional Network ◽

Dynamic Neural Networks ◽

Reduction Methods ◽

Channel Reduction ◽

Channel Information

Light-weight convolutional neural networks (CNNs) suffer limited feature representation capabilities due to low computational budgets, resulting in degradation in performance. To make CNNs more efficient, dynamic neural networks (DyNet) have been proposed to increase the complexity of the model by using the Squeeze-and-Excitation (SE) module to adaptively obtain the importance of each convolution kernel through the attention mechanism. However, the attention mechanism in the SE network (SENet) selects all channel information for calculations, which brings essential challenges: (a) interference caused by the internal redundant information; and (b) increasing number of network calculations. To address the above problems, this work proposes a dynamic convolutional network (termed as EAM-DyNet) to reduce the number of channels in feature maps by extracting only the useful spatial information. EAM-DyNet first uses the random channel reduction and channel grouping reduction methods to remove the redundancy in the information. As the downsampling of information can lead to the loss of useful information, it then applies an adaptive average pooling method to maintain the information integrity. Extensive experimental results on the baseline demonstrate that EAM-DyNet outperformed the existing approaches, thus it can achieve higher accuracy of the network test and less network parameters.

Download Full-text

Contextualized Knowledge-aware Attentive Neural Network: Enhancing Answer Selection with Knowledge

ACM Transactions on Information Systems ◽

10.1145/3457533 ◽

2022 ◽

Vol 40 (1) ◽

pp. 1-33

Author(s):

Yang Deng ◽

Yuexiang Xie ◽

Yaliang Li ◽

Min Yang ◽

Wai Lam ◽

...

Keyword(s):

Neural Network ◽

Language Processing ◽

Question Answering ◽

Representation Learning ◽

Attention Mechanism ◽

Convolutional Network ◽

Structure Information ◽

Dialog Systems ◽

External Knowledge ◽

Knowledge Based

Answer selection, which is involved in many natural language processing applications, such as dialog systems and question answering (QA), is an important yet challenging task in practice, since conventional methods typically suffer from the issues of ignoring diverse real-world background knowledge. In this article, we extensively investigate approaches to enhancing the answer selection model with external knowledge from knowledge graph (KG). First, we present a context-knowledge interaction learning framework, Knowledge-aware Neural Network, which learns the QA sentence representations by considering a tight interaction with the external knowledge from KG and the textual information. Then, we develop two kinds of knowledge-aware attention mechanism to summarize both the context-based and knowledge-based interactions between questions and answers. To handle the diversity and complexity of KG information, we further propose a Contextualized Knowledge-aware Attentive Neural Network, which improves the knowledge representation learning with structure information via a customized Graph Convolutional Network and comprehensively learns context-based and knowledge-based sentence representation via the multi-view knowledge-aware attention mechanism. We evaluate our method on four widely used benchmark QA datasets, including WikiQA, TREC QA, InsuranceQA, and Yahoo QA. Results verify the benefits of incorporating external knowledge from KG and show the robust superiority and extensive applicability of our method.

Download Full-text

Attention-Based Graph Convolutional Network for Zero-Shot Learning with Pre-Training

Mathematical Problems in Engineering ◽

10.1155/2021/7480712 ◽

2021 ◽

Vol 2021 ◽

pp. 1-13

Author(s):

Xuefei Wu ◽

Mingjiang Liu ◽

Bo Xin ◽

Zhangqing Zhu ◽

Gang Wang

Keyword(s):

Loss Function ◽

Attention Mechanism ◽

Propagation Model ◽

Knowledge Graph ◽

Learning Paradigm ◽

Convolutional Network ◽

Dense Graph ◽

Convolutional Networks ◽

Central Node ◽

Graph Propagation

Zero-shot learning (ZSL) is a powerful and promising learning paradigm for classifying instances that have not been seen in training. Although graph convolutional networks (GCNs) have recently shown great potential for the ZSL tasks, these models cannot adjust the constant connection weights between the nodes in knowledge graph and the neighbor nodes contribute equally to classify the central node. In this study, we apply an attention mechanism to adjust the connection weights adaptively to learn more important information for classifying unseen target nodes. First, we propose an attention graph convolutional network for zero-shot learning (AGCNZ) by integrating the attention mechanism and GCN directly. Then, in order to prevent the dilution of knowledge from distant nodes, we apply the dense graph propagation (DGP) model for the ZSL tasks and propose an attention dense graph propagation model for zero-shot learning (ADGPZ). Finally, we propose a modified loss function with a relaxation factor to further improve the performance of the learned classifier. Experimental results under different pre-training settings verified the effectiveness of the proposed attention-based models for ZSL.

Download Full-text