scholarly journals Capacity Visual Attention Networks

10.29007/lcmk ◽  
2018 ◽  
Author(s):  
Marcus Edel ◽  
Joscha Lausch
Keyword(s):  
Neural Networks ◽  
High Resolution ◽  
Machine Translation ◽  
Input Data ◽  
Counting Process ◽  
Model Output ◽  
Attention Networks ◽  
Attention Model ◽  
Proposed Model ◽  
Extract Information

Inspired by recent work in machine translation and object detection, we introduce an attention-based model that automatically learns to extract information from an image by adaptively assigning its capacity across different portions of the input data and only processing the selected regions of different sizes at high resolution. This is achieved by combining two modules: an attention sub-network which uses a mechanism to model a human-like counting process and a capacity sub-network. This sub-network efficiently identifies input regions for which the attention model output is most sensitive and to which we should devote more capacity and dynamically adapt the size of the region. We focus our evaluation on the Cluttered MNIST, SVHN, and Cluttered GTSRB image datasets. Our findings indicate that the proposed model is able to drastically reduce the number of computations, compared with traditional convolutional neural networks, while maintaining similar or better performance.

scholarly journals Modeling Past and Future for Neural Machine Translation

2018 ◽  
Vol 6 ◽  
pp. 145-157 ◽  
Author(s):  
Zaixiang Zheng ◽  
Hao Zhou ◽  
Shujian Huang ◽  
Lili Mou ◽  
Xinyu Dai ◽  
...  
Keyword(s):  
Machine Translation ◽  
Alignment Error ◽  
Source Information ◽  
Neural Machine Translation ◽  
Attention Model ◽  
Translation Quality ◽  
The Past ◽  
Proposed Model ◽  
Coverage Model ◽  
Translation Systems

Existing neural machine translation systems do not explicitly model what has been translated and what has not during the decoding phase. To address this problem, we propose a novel mechanism that separates the source information into two parts: translated Past contents and untranslated Future contents, which are modeled by two additional recurrent layers. The Past and Future contents are fed to both the attention model and the decoder states, which provides Neural Machine Translation (NMT) systems with the knowledge of translated and untranslated contents. Experimental results show that the proposed approach significantly improves the performance in Chinese-English, German-English, and English-German translation tasks. Specifically, the proposed model outperforms the conventional coverage model in terms of both the translation quality and the alignment error rate.

scholarly journals CellVGAE: An unsupervised scRNA-seq analysis workflow with graph attention networks

2020 ◽  
Author(s):  
David Buterez ◽  
Ioana Bica ◽  
Ifrah Tariq ◽  
Helena Andrés-Terré ◽  
Pietro Liò
Keyword(s):  
Neural Networks ◽  
High Resolution ◽  
Dimensionality Reduction ◽  
Single Cell ◽  
Rna Sequencing ◽  
Feedforward Neural Networks ◽  
Attention Networks ◽  
Single Cell Rna Sequencing ◽  
Analysis Workflow ◽  
Graph Neural Networks

AbstractCurrently, single-cell RNA sequencing (scRNA-seq) allows high-resolution views of individual cells, for libraries of up to (tens of) thousands of samples. In this study, we introduce the use of graph neural networks (GNN) in the unsupervised study of scRNA-seq data, namely for dimensionality reduction and clustering. Motivated by the success of non-neural graph-based techniques in bioinformatics, as well as the now common feedforward neural networks being applied to scRNA-seq measurements, we develop an architecture based on a variational graph autoencoder with graph attention layers that works directly on the connectivity of cells. With the help of three case studies, we show that our model, named CellVGAE, can be effectively used for exploratory analysis, even on challenging datasets, by extracting meaningful features from the data and providing the means to visualise and interpret different aspects of the model. Furthermore, we evaluate the dimensionality reduction and clustering performance on 9 well-annotated datasets, where we compare with leading neural and non-neural techniques. CellVGAE outperforms competing methods in all 9 scenarios. Finally, we show that CellVGAE is more interpretable than existing architectures by analysing the graph attention coefficients. The software and code to generate all the figures are available at https://github.com/davidbuterez/CellVGAE.

SUCCESSIVE LEARNING IN HETERO-ASSOCIATIVE MEMORY USING CHAOTIC NEURAL NETWORKS

1999 ◽  
Vol 09 (04) ◽  
pp. 285-299 ◽  
Author(s):  
YUKO OSANA ◽  
MASAFUMI HAGIWARA
Keyword(s):  
Neural Networks ◽  
Input Data ◽  
Temporal Summation ◽  
Continuous Data ◽  
Learning Method ◽  
Associative Memories ◽  
Chaotic Neural Networks ◽  
Proposed Model ◽  
The Difference ◽  
Correct Data

In this paper, we propose a successive learning method in hetero-associative memories, such as Bidirectional Associative Memories and Multidirectional Associative Memories, using chaotic neural networks. It can distinguish unknown data from the stored known data and can learn the unknown data successively. The proposed model makes use of the difference in the response to the input data in order to distinguish unknown data from the stored known data. When input data is regarded as unknown data, it is memorized. Furthermore, the proposed model can estimate and learn correct data from noisy unknown data or incomplete unknown data by considering the temporal summation of the continuous data input. In addition, similarity to the physiological facts in the olfactory bulb of a rabbit found by Freeman are observed in the behavior of the proposed model. A series of computer simulations shows the effectiveness of the proposed model.

Least-squares reverse time migration using convolutional neural networks

Geophysics ◽  
2021 ◽  
pp. 1-92
Author(s):  
Wei Zhang ◽  
Jinghuai Gao ◽  
Tao Yang ◽  
Xiudi Jiang ◽  
Wenbo Sun
Keyword(s):  
Neural Networks ◽  
High Resolution ◽  
Least Squares ◽  
Convolutional Neural Networks ◽  
Input Data ◽  
Velocity Model ◽  
Reverse Time ◽  
Reverse Time Migration ◽  
Time Migration ◽  
Reflection Image

Least-squares reverse time migration (LSRTM) has the potential to reconstruct a high-resolution image of subsurface reflectivity. However, the current data-domain LSRTM approach, which iteratively updates the subsurface reflectivity by minimizing the data residuals, is a computationally expensive task. To alleviate this problem and improve imaging quality, we develop a LSRTM approach using convolutional neural networks (CNNs), which is referred to as CNN-LSRTM. Specifically, the LSRTM problem can be implemented via a gradient-like iterative scheme, in which the updating component in each iteration is learned via a CNN model. In order to make the most of observation data and migration velocity model at hand, we utilize the common-source RTM image, the stacked RTM image, and the migration velocity model rather than only the stacked RTM image as the input data of CNN. We have successfully trained the constructed CNN model on the training data sets with a total of 5000 randomly layered and fault models. Based on the well-trained CNN model, we have proved that the proposed approach can efficiently recover the high-resolution reflection image for the layered, fault, and overthrust models. Through a marine field data experiment, it can determine the benefit of our constructed CNN model in terms of computational efficiency. In addition, we analyze the influence of input data of the constructed CNN model on the reconstruction quality of the reflection image.

scholarly journals A Multi-Attention Network for Aspect-Level Sentiment Analysis

2019 ◽  
Vol 11 (7) ◽  
pp. 157 ◽  
Author(s):  
Qiuyue Zhang ◽  
Ran Lu
Keyword(s):  
Neural Networks ◽  
Sentiment Analysis ◽  
Specific Aspect ◽  
Experimental Results ◽  
Sequence Information ◽  
Attention Network ◽  
Attention Networks ◽  
Recent Advances ◽  
Proposed Model ◽  
The Impact

Aspect-level sentiment analysis (ASA) aims at determining the sentiment polarity of specific aspect term with a given sentence. Recent advances in attention mechanisms suggest that attention models are useful in ASA tasks and can help identify focus words. Or combining attention mechanisms with neural networks are also common methods. However, according to the latest research, they often fail to extract text representations efficiently and to achieve interaction between aspect terms and contexts. In order to solve the complete task of ASA, this paper proposes a Multi-Attention Network (MAN) model which adopts several attention networks. This model not only preprocesses data by Bidirectional Encoder Representations from Transformers (BERT), but a number of measures have been taken. First, the MAN model utilizes the partial Transformer after transformation to obtain hidden sequence information. Second, because words in different location have different effects on aspect terms, we introduce location encoding to analyze the impact on distance from ASA tasks, then we obtain the influence of different words with aspect terms through the bidirectional attention network. From the experimental results of three datasets, we could find that the proposed model could achieve consistently superior results.

scholarly journals PERSIANN-CNN: Precipitation Estimation from Remotely Sensed Information Using Artificial Neural Networks–Convolutional Neural Networks

2019 ◽  
Vol 20 (12) ◽  
pp. 2273-2289 ◽  
Author(s):  
Mojtaba Sadeghi ◽  
Ata Akbari Asanjan ◽  
Mohammad Faridzad ◽  
Phu Nguyen ◽  
Kuolin Hsu ◽  
...  
Keyword(s):  
Neural Networks ◽  
Artificial Neural Networks ◽  
High Resolution ◽  
Convolutional Neural Networks ◽  
Spatial Scales ◽  
Remotely Sensed ◽  
Proposed Model ◽  
Precipitation Estimation ◽  
Artificial Neural ◽  
Rainfall Estimates

Abstract Accurate and timely precipitation estimates are critical for monitoring and forecasting natural disasters such as floods. Despite having high-resolution satellite information, precipitation estimation from remotely sensed data still suffers from methodological limitations. State-of-the-art deep learning algorithms, renowned for their skill in learning accurate patterns within large and complex datasets, appear well suited to the task of precipitation estimation, given the ample amount of high-resolution satellite data. In this study, the effectiveness of applying convolutional neural networks (CNNs) together with the infrared (IR) and water vapor (WV) channels from geostationary satellites for estimating precipitation rate is explored. The proposed model performances are evaluated during summer 2012 and 2013 over central CONUS at the spatial resolution of 0.08° and at an hourly time scale. Precipitation Estimation from Remotely Sensed Information Using Artificial Neural Networks (PERSIANN)–Cloud Classification System (CCS), which is an operational satellite-based product, and PERSIANN–Stacked Denoising Autoencoder (PERSIANN-SDAE) are employed as baseline models. Results demonstrate that the proposed model (PERSIANN-CNN) provides more accurate rainfall estimates compared to the baseline models at various temporal and spatial scales. Specifically, PERSIANN-CNN outperforms PERSIANN-CCS (and PERSIANN-SDAE) by 54% (and 23%) in the critical success index (CSI), demonstrating the detection skills of the model. Furthermore, the root-mean-square error (RMSE) of the rainfall estimates with respect to the National Centers for Environmental Prediction (NCEP) Stage IV gauge–radar data, for PERSIANN-CNN was lower than that of PERSIANN-CCS (PERSIANN-SDAE) by 37% (14%), showing the estimation accuracy of the proposed model.

A Neural N-Gram Network for Text Classification

2018 ◽  
Vol 22 (3) ◽  
pp. 380-386 ◽  
Author(s):  
Zhenguo Yan ◽  
◽  
Yue Wu
Keyword(s):  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Text Classification ◽  
Word Order ◽  
Input Data ◽  
State Of The Art ◽  
Poor Performance ◽  
Proposed Model ◽  
Classification Tasks ◽  
N Gram

Convolutional Neural Networks (CNNs) effectively extract local features from input data. However, CNN based on word embedding and convolution layers displays poor performance in text classification tasks when compared with traditional baseline methods. We address this problem and propose a model named NNGN that simplifies the convolution layer in the CNN by replacing it with a pooling layer that extracts n-gram embedding in a simpler way and obtains document representations via linear computation. We implement two settings in our model to extract n-gram features. In the first setting, which we refer to as seq-NNGN, we consider word order within each n-gram. In the second setting, BoW-NNGN, we do not consider word order. We compare the performance of these settings in different classification tasks with those of other models. The experimental results show that our proposed model achieves better performance than state-of-the-art models.

BERT-BU12 Hate Speech Detection using Bidirectional Encoder-Decoder

2022 ◽  
Vol 11 (2) ◽  
pp. 0-0
Keyword(s):  
Text Classification ◽  
Question Answering ◽  
Hate Speech ◽  
State Of The Art ◽  
Learning Models ◽  
Speech Detection ◽  
Attention Networks ◽  
Attention Model ◽  
Proposed Model ◽  
Novel Method

In the recent times transfer learning models have known to exhibited good results in the area of text classification for question-answering, summarization, next word prediction but these learning models have not been extensively used for the problem of hate speech detection yet. We anticipate that these networks may give better results in another task of text classification i.e. hate speech detection. This paper introduces a novel method of hate speech detection based on the concept of attention networks using the BERT attention model. We have conducted exhaustive experiments and evaluation over publicly available datasets using various evaluation metrics (precision, recall and F1 score). We show that our model outperforms all the state-of-the-art methods by almost 4%. We have also discussed in detail the technical challenges faced during the implementation of the proposed model.

Towards Operational Downscaling of Low Resolution Wind Fields using Neural Networks

2020 ◽  
Author(s):  
Michael Kern ◽  
Kevin Höhlein ◽  
Timothy Hewson ◽  
Rüdiger Westermann
Keyword(s):  
Neural Networks ◽  
High Resolution ◽  
Wind Field ◽  
Input Data ◽  
Vector Fields ◽  
Wind Vector ◽  
Proof Of Concept ◽  
Low Resolution ◽  
Wind Fields ◽  
Resolution Simulation

<p>Numerical weather prediction models with high resolution (of order kms or less) can deliver very accurate low-level winds. The problem is that one cannot afford to run simulations at very high resolution over global or other large domains for long periods because the computational power needed is prohibitive.</p><p>Instead, we propose using neural networks to downscale low-resolution wind-field simulations (input) to high-resolution fields (targets) to try to match a high-resolution simulation. Based on short-range forecasts of wind fields (at the 100m level) from the ECMWF ERA5 reanalysis, at 31km resolution, and the HRES (deterministic) model version, at 9km resolution, we explore two complementary approaches, in an initial “proof-of-concept” study.</p><p>In a first step, we evaluate the ability of U-Net-type convolutional neural networks to learn a one-to-one mapping of low-resolution input data to high-resolution simulation results. By creating a compressed feature-space representation of the data, networks of this kind manage to encode important flow characteristics of the input fields and assimilate information from additional data sources. Next to wind vector fields, we use topographical information to inform the network, at low and high resolution, and include additional parameters that strongly influence wind-field prediction in simulations, such as vertical stability (via the simple, compact metric of boundary layer height) and the land-sea mask. We thus infer weather-situation and location-dependent wind structures that could not be retrieved otherwise.</p><p>In some situations, however, it will be inappropriate to deliver only a single estimate for the high-resolution wind field. Especially in regions where topographic complexity fosters the emergence of complex wind patterns, a variety of different high-resolution estimates may be equally compatible with the low-resolution input, and with physical reasoning. In a second step, we therefore extend the learning task from optimizing deterministic one-to-one mappings to modelling the distribution of physically reasonable high-resolution wind-vector fields, conditioned on the given low-resolution input. Using the framework of conditional variational autoencoders, we realize a generative model, based on convolutional neural networks, which is able to learn the conditional distributions from data. Sampling multiple estimates of the high-resolution wind vector fields from the model enables us to explore multimodalities in the data and to infer uncertainties in the predictand.</p><p>In a future customer-oriented extension of this proof-of-concept work, we would envisage using a target resolution higher than 9km - say in the 1-4km range - to deliver much better representivity for users. Ensembles of low resolution input data could also be used, to deliver as output an “ensemble of ensembles”, to condense into a meaningful probabilistic format for users. The many exciting applications of this work (e.g. for wind power management) will be highlighted.</p>

scholarly journals Neural Machine Translation with Key-Value Memory-Augmented Attention

2018 ◽  
Author(s):  
Fandong Meng ◽  
Zhaopeng Tu ◽  
Yong Cheng ◽  
Haiyang Wu ◽  
Junjie Zhai ◽  
...  
Keyword(s):  
Target Word ◽  
Machine Translation ◽  
English Translation ◽  
Translation Process ◽  
Neural Machine Translation ◽  
Attention Model ◽  
Proposed Model ◽  
Source Sentence ◽  
Remarkable Progress ◽  
Source Word

Although attention-based Neural Machine Translation (NMT) has achieved remarkable progress in recent years, it still suffers from issues of repeating and dropping translations. To alleviate these issues, we propose a novel key-value memory-augmented attention model for NMT, called KVMEMATT. Specifically, we maintain a timely updated keymemory to keep track of attention history and a fixed value-memory to store the representation of source sentence throughout the whole translation process. Via nontrivial transformations and iterative interactions between the two memories, the decoder focuses on more appropriate source word(s) for predicting the next target word at each decoding step, therefore can improve the adequacy of translations. Experimental results on Chinese)English and WMT17 German,English translation tasks demonstrate the superiority of the proposed model.

Sign in / Sign up

Export Citation Format

Share Document