scholarly journals Light Attention Predicts Protein Location from the Language of Life

2021 ◽  
Author(s):  
Hannes Staerk ◽  
Christian Dallago ◽  
Michael Heinzinger ◽  
Burkhard Rost
Keyword(s):  
Network Architecture ◽  
Language Models ◽  
Sequence Length ◽  
Evolutionary Information ◽  
Biological Processes ◽  
The Novel ◽  
Neural Network Architecture ◽  
Percentage Points ◽  
Protein Functions ◽  
A Cell

Although knowing where a protein functions in a cell is important to characterize biological processes, this information remains unavailable for most known proteins. Machine learning narrows the gap through predictions from expertly chosen input features leveraging evolutionary information that is resource expensive to generate. We showcase using embeddings from protein language models for competitive localization predictions not relying on evolutionary information. Our lightweight deep neural network architecture uses a softmax weighted aggregation mechanism with linear complexity in sequence length referred to as light attention (LA). The method significantly outperformed the state-of-the-art for ten localization classes by about eight percentage points (Q10). The novel models are available as a web-service and as a stand-alone application at embed.protein.properties.

scholarly journals Jazz Bass Transcription Using a U-Net Architecture

Electronics ◽  
2021 ◽  
Vol 10 (6) ◽  
pp. 670
Author(s):  
Jakob Abeßer ◽  
Meinard Müller
Keyword(s):  
Network Architecture ◽  
Data Augmentation ◽  
Neural Network Architecture ◽  
Training Set ◽  
Pitch Estimation ◽  
Jazz Ensemble ◽  
Percentage Points ◽  
Augmentation Strategy ◽  
Augmentation Techniques ◽  
Validation Set

In this paper, we adapt a recently proposed U-net deep neural network architecture from melody to bass transcription. We investigate pitch shifting and random equalization as data augmentation techniques. In a parameter importance study, we study the influence of the skip connection strategy between the encoder and decoder layers, the data augmentation strategy, as well as of the overall model capacity on the system’s performance. Using a training set that covers various music genres and a validation set that includes jazz ensemble recordings, we obtain the best transcription performance for a downscaled version of the reference algorithm combined with skip connections that transfer intermediate activations between the encoder and decoder. The U-net based method outperforms previous knowledge-driven and data-driven bass transcription algorithms by around five percentage points in overall accuracy. In addition to a pitch estimation improvement, the voicing estimation performance is clearly enhanced.

scholarly journals CpG Transformer for imputation of single-cell methylomes

2021 ◽  
Author(s):  
Gaetan De Waele ◽  
Jim Clauwaert ◽  
Gerben Menschaert ◽  
Willem Waegeman
Keyword(s):  
Single Cell ◽  
Network Architecture ◽  
State Of The Art ◽  
Time Budget ◽  
General Purpose ◽  
Biological Processes ◽  
Neural Network Architecture ◽  
Wide Range ◽  
Incomplete Coverage ◽  
Rapid Transfer

Motivation: The adoption of current single-cell DNA methylation sequencing protocols is hindered by incomplete coverage, outlining the need for effective imputation techniques. The task of imputing single-cell (methylation) data requires models to build an understanding of underlying biological processes. Current approaches compress intercellular methylation dependencies in some way and, hence, do not provide a general-purpose way of learning interactions between neighboring CpG sites both within- and between cells. Results: We adapt the transformer neural network architecture to operate on methylation matrices through the introduction of a novel 2D sliding window self-attention. The obtained CpG Transformer displays state-of-the-art performances on a wide range of scBS-seq and scRRBS-seq datasets. Furthermore, we demonstrate the interpretability of CpG Transformer and illustrate its rapid transfer learning properties, allowing practitioners to train models on new datasets with a limited computational and time budget. Availability and Implementation: CpG Transformer is freely available at https://github.com/gdewael/cpg-transformer.

scholarly journals Optimizing Small BERTs Trained for German NER

Information ◽  
2021 ◽  
Vol 12 (11) ◽  
pp. 443
Author(s):  
Jochen Zöllner ◽  
Konrad Sperfeld ◽  
Christoph Wick ◽  
Roger Labahn
Keyword(s):  
Neural Network ◽  
Network Architecture ◽  
Fine Tuning ◽  
Language Models ◽  
Memory Usage ◽  
Memory Consumption ◽  
Neural Network Architecture ◽  
Training Techniques

Currently, the most widespread neural network architecture for training language models is the so-called BERT, which led to improvements in various NLP tasks. In general, the larger the number of parameters in a BERT model, the better the results obtained in these NLP tasks. Unfortunately, the memory consumption and the training duration drastically increases with the size of these models. In this article, we investigate various training techniques of smaller BERT models: We combine different methods from other BERT variants, such as ALBERT, RoBERTa, and relative positional encoding. In addition, we propose two new fine-tuning modifications leading to better performance: CSE tagging and a modified form of LCRF. Furthermore, we introduce WWA, which reduces BERT memory usage and leads to a small increase in performance compared to classical Multi-Head-Attention. We evaluate these techniques on five public German NER tasks, of which two are introduced by this article.

SCORING MODELING BASED ON NEURAL NETWORKS FOR DETERMINING A BANK BORROWER'S RATING

2020 ◽  
Vol 2020 (10) ◽  
pp. 54-62
Author(s):  
Oleksii VASYLIEV ◽  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Network Architecture ◽  
Statistical Data ◽  
Activation Function ◽  
Decision Making Process ◽  
Neural Network Architecture ◽  
Acceptable Accuracy ◽  
The Neural Network ◽  
Sigmoid Activation Function

The problem of applying neural networks to calculate ratings used in banking in the decision-making process on granting or not granting loans to borrowers is considered. The task is to determine the rating function of the borrower based on a set of statistical data on the effectiveness of loans provided by the bank. When constructing a regression model to calculate the rating function, it is necessary to know its general form. If so, the task is to calculate the parameters that are included in the expression for the rating function. In contrast to this approach, in the case of using neural networks, there is no need to specify the general form for the rating function. Instead, certain neural network architecture is chosen and parameters are calculated for it on the basis of statistical data. Importantly, the same neural network architecture can be used to process different sets of statistical data. The disadvantages of using neural networks include the need to calculate a large number of parameters. There is also no universal algorithm that would determine the optimal neural network architecture. As an example of the use of neural networks to determine the borrower's rating, a model system is considered, in which the borrower's rating is determined by a known non-analytical rating function. A neural network with two inner layers, which contain, respectively, three and two neurons and have a sigmoid activation function, is used for modeling. It is shown that the use of the neural network allows restoring the borrower's rating function with quite acceptable accuracy.

scholarly journals Fighting Together against the Pandemic: Learning Multiple Models on Tomography Images for COVID-19 Diagnosis

AI ◽  
2021 ◽  
Vol 2 (2) ◽  
pp. 261-273
Author(s):  
Mario Manzo ◽  
Simone Pellino
Keyword(s):  
Network Architecture ◽  
State Of The Art ◽  
Ensemble Classification ◽  
Effective Vaccine ◽  
Rt Pcr ◽  
Neural Network Architecture ◽  
Experimental Phase ◽  
Different Types ◽  
Polymerase Chain ◽  
Better Than

COVID-19 has been a great challenge for humanity since the year 2020. The whole world has made a huge effort to find an effective vaccine in order to save those not yet infected. The alternative solution is early diagnosis, carried out through real-time polymerase chain reaction (RT-PCR) tests or thorax Computer Tomography (CT) scan images. Deep learning algorithms, specifically convolutional neural networks, represent a methodology for image analysis. They optimize the classification design task, which is essential for an automatic approach with different types of images, including medical. In this paper, we adopt a pretrained deep convolutional neural network architecture in order to diagnose COVID-19 disease from CT images. Our idea is inspired by what the whole of humanity is achieving, as the set of multiple contributions is better than any single one for the fight against the pandemic. First, we adapt, and subsequently retrain for our assumption, some neural architectures that have been adopted in other application domains. Secondly, we combine the knowledge extracted from images by the neural architectures in an ensemble classification context. Our experimental phase is performed on a CT image dataset, and the results obtained show the effectiveness of the proposed approach with respect to the state-of-the-art competitors.

Sign in / Sign up

Export Citation Format

Share Document