MRlogP: Transfer Learning Enables Accurate logP Prediction Using Small Experimental Training Datasets

Small molecule lipophilicity is often included in generalized rules for medicinal chemistry. These rules aim to reduce time, effort, costs, and attrition rates in drug discovery, allowing the rejection or prioritization of compounds without the need for synthesis and testing. The availability of high quality, abundant training data for machine learning methods can be a major limiting factor in building effective property predictors. We utilize transfer learning techniques to get around this problem, first learning on a large amount of low accuracy predicted logP values before finally tuning our model using a small, accurate dataset of 244 druglike compounds to create MRlogP, a neural network-based predictor of logP capable of outperforming state of the art freely available logP prediction methods for druglike small molecules. MRlogP achieves an average root mean squared error of 0.988 and 0.715 against druglike molecules from Reaxys and PHYSPROP. We have made the trained neural network predictor and all associated code for descriptor generation freely available. In addition, MRlogP may be used online via a web interface.

Download Full-text

Auto-Colorization of Historical Images Using Deep Convolutional Neural Networks

Mathematics ◽

10.3390/math8122258 ◽

2020 ◽

Vol 8 (12) ◽

pp. 2258

Author(s):

Madhab Raj Joshi ◽

Lewis Nkenyereye ◽

Gyanendra Prasad Joshi ◽

S. M. Riazul Islam ◽

Mohammad Abdullah-Al-Wadud ◽

...

Keyword(s):

Neural Network ◽

Deep Learning ◽

User Study ◽

Mean Squared Error ◽

Color Image ◽

Machine Learning Techniques ◽

Global Features ◽

Black And White ◽

Historical Images ◽

Learning Techniques

Enhancement of Cultural Heritage such as historical images is very crucial to safeguard the diversity of cultures. Automated colorization of black and white images has been subject to extensive research through computer vision and machine learning techniques. Our research addresses the problem of generating a plausible colored photograph of ancient, historically black, and white images of Nepal using deep learning techniques without direct human intervention. Motivated by the recent success of deep learning techniques in image processing, a feed-forward, deep Convolutional Neural Network (CNN) in combination with Inception- ResnetV2 is being trained by sets of sample images using back-propagation to recognize the pattern in RGB and grayscale values. The trained neural network is then used to predict two a* and b* chroma channels given grayscale, L channel of test images. CNN vividly colorizes images with the help of the fusion layer accounting for local features as well as global features. Two objective functions, namely, Mean Squared Error (MSE) and Peak Signal-to-Noise Ratio (PSNR), are employed for objective quality assessment between the estimated color image and its ground truth. The model is trained on the dataset created by ourselves with 1.2 K historical images comprised of old and ancient photographs of Nepal, each having 256 × 256 resolution. The loss i.e., MSE, PSNR, and accuracy of the model are found to be 6.08%, 34.65 dB, and 75.23%, respectively. Other than presenting the training results, the public acceptance or subjective validation of the generated images is assessed by means of a user study where the model shows 41.71% of naturalness while evaluating colorization results.

Download Full-text

Combining Self-supervised Learning and Active Learning for Disfluency Detection

ACM Transactions on Asian and Low-Resource Language Information Processing ◽

10.1145/3487290 ◽

2022 ◽

Vol 21 (3) ◽

pp. 1-25

Author(s):

Shaolei Wang ◽

Zhongyuan Wang ◽

Wanxiang Che ◽

Sendong Zhao ◽

Ting Liu

Keyword(s):

Neural Network ◽

Active Learning ◽

Supervised Learning ◽

Large Scale ◽

Training Data ◽

Fine Tuning ◽

Training Dataset ◽

Performance Gap ◽

Annotation Costs ◽

Trained Neural Network

Spoken language is fundamentally different from the written language in that it contains frequent disfluencies or parts of an utterance that are corrected by the speaker. Disfluency detection (removing these disfluencies) is desirable to clean the input for use in downstream NLP tasks. Most existing approaches to disfluency detection heavily rely on human-annotated data, which is scarce and expensive to obtain in practice. To tackle the training data bottleneck, in this work, we investigate methods for combining self-supervised learning and active learning for disfluency detection. First, we construct large-scale pseudo training data by randomly adding or deleting words from unlabeled data and propose two self-supervised pre-training tasks: (i) a tagging task to detect the added noisy words and (ii) sentence classification to distinguish original sentences from grammatically incorrect sentences. We then combine these two tasks to jointly pre-train a neural network. The pre-trained neural network is then fine-tuned using human-annotated disfluency detection training data. The self-supervised learning method can capture task-special knowledge for disfluency detection and achieve better performance when fine-tuning on a small annotated dataset compared to other supervised methods. However, limited in that the pseudo training data are generated based on simple heuristics and cannot fully cover all the disfluency patterns, there is still a performance gap compared to the supervised models trained on the full training dataset. We further explore how to bridge the performance gap by integrating active learning during the fine-tuning process. Active learning strives to reduce annotation costs by choosing the most critical examples to label and can address the weakness of self-supervised learning with a small annotated dataset. We show that by combining self-supervised learning with active learning, our model is able to match state-of-the-art performance with just about 10% of the original training data on both the commonly used English Switchboard test set and a set of in-house annotated Chinese data.

Download Full-text

Sea Fog Identification from GOCI Images Using CNN Transfer Learning Models

Electronics ◽

10.3390/electronics9020311 ◽

2020 ◽

Vol 9 (2) ◽

pp. 311

Author(s):

Ho-Kun Jeon ◽

Seungryong Kim ◽

Jonathan Edwin ◽

Chan-Su Yang

Keyword(s):

Neural Network ◽

Transfer Learning ◽

Korean Peninsula ◽

Ocean Color ◽

Training Data ◽

Learning Models ◽

Identification Performance ◽

Coastal Regions ◽

Sea Fog ◽

Band Combinations

This study proposes an approaching method of identifying sea fog by using Geostationary Ocean Color Imager (GOCI) data through applying a Convolution Neural Network Transfer Learning (CNN-TL) model. In this study, VGG19 and ResNet50, pre-trained CNN models, are used for their high identification performance. The training and testing datasets were extracted from GOCI images for the area of coastal regions of the Korean Peninsula for six days in March 2015. With varying band combinations and changing whether Transfer Learning (TL) is applied, identification experiments were executed. TL enhanced the performance of the two models. Training data of CNN-TL showed up to 96.3% accuracy in matching, both with VGG19 and ResNet50, identically. Thus, it is revealed that CNN-TL is effective for the detection of sea fog from GOCI imagery.

Download Full-text

Approaching coupled cluster accuracy with a general-purpose neural network potential through transfer learning

10.26434/chemrxiv.6744440 ◽

2019 ◽

Cited By ~ 1

Author(s):

Justin S Smith ◽

Benjamin T. Nebgen ◽

Roman Zubatyuk ◽

Nicholas Lubbers ◽

Christian Devereux ◽

...

Keyword(s):

Neural Network ◽

Transfer Learning ◽

Materials Science ◽

Chemical Space ◽

General Purpose ◽

Learning Techniques ◽

Mechanical Methods ◽

Cluster Accuracy ◽

Large Systems ◽

Chemical And Biological Systems

<p>Computational modeling of chemical and biological systems at atomic resolution is a crucial tool in the chemist's toolset. The use of computer simulations requires a balance between cost and accuracy: quantum-mechanical methods provide high accuracy but are computationally expensive and scale poorly to large systems, while classical force fields are cheap and scalable, but lack transferability to new systems. Machine learning can be used to achieve the best of both approaches. Here we train a general-purpose neural network potential (ANI-1ccx) that approaches CCSD(T)/CBS accuracy on benchmarks for reaction thermochemistry, isomerization, and drug-like molecular torsions. This is achieved by training a network to DFT data then using transfer learning techniques to retrain on a dataset of gold standard QM calculations (CCSD(T)/CBS) that optimally spans chemical space. The resulting potential is broadly applicable to materials science, biology and chemistry, and billions of times faster<i></i>than CCSD(T)/CBS calculations. </p>

Download Full-text

Is One Teacher Model Enough to Transfer Knowledge to a Student Model?

Algorithms ◽

10.3390/a14110334 ◽

2021 ◽

Vol 14 (11) ◽

pp. 334

Author(s):

Nicola Landro ◽

Ignazio Gallo ◽

Riccardo La Grassa

Keyword(s):

Neural Network ◽

Transfer Learning ◽

Learning Problem ◽

Learning Techniques ◽

Starting Point ◽

Learning Technique ◽

Interesting Approach ◽

Classification Tasks ◽

Learned Features ◽

Fine Tune

Nowadays, the transfer learning technique can be successfully applied in the deep learning field through techniques that fine-tune the CNN’s starting point so it may learn over a huge dataset such as ImageNet and continue to learn on a fixed dataset to achieve better performance. In this paper, we designed a transfer learning methodology that combines the learned features of different teachers to a student network in an end-to-end model, improving the performance of the student network in classification tasks over different datasets. In addition to this, we tried to answer the following questions which are in any case directly related to the transfer learning problem addressed here. Is it possible to improve the performance of a small neural network by using the knowledge gained from a more powerful neural network? Can a deep neural network outperform the teacher using transfer learning? Experimental results suggest that neural networks can transfer their learning to student networks using our proposed architecture, designed to bring to light a new interesting approach for transfer learning techniques. Finally, we provide details of the code and the experimental settings.

Download Full-text

Corn Disease Classification Using Transfer Learning and Convolutional Neural Network

JUITA Jurnal Informatika ◽

10.30595/juita.v9i2.11686 ◽

2021 ◽

Vol 9 (2) ◽

pp. 211

Author(s):

Faisal Dharma Adhinata ◽

Gita Fadila Fitriana ◽

Aditya Wijayanto ◽

Muhammad Pajar Kharisma Putra

Keyword(s):

Neural Network ◽

Convolutional Neural Network ◽

Transfer Learning ◽

Disease Classification ◽

Machine Learning Techniques ◽

Early Detection Of Disease ◽

Global Average ◽

Learning Techniques ◽

Corn Plants

Indonesia is an agricultural country with abundant agricultural products. One of the crops used as a staple food for Indonesians is corn. This corn plant must be protected from diseases so that the quality of corn harvest can be optimal. Early detection of disease in corn plants is needed so that farmers can provide treatment quickly and precisely. Previous research used machine learning techniques to solve this problem. The results of the previous research were not optimal because the amount of data used was slightly and less varied. Therefore, we propose a technique that can process lots and varied data, hoping that the resulting system is more accurate than the previous research. This research uses transfer learning techniques as feature extraction combined with Convolutional Neural Network as a classification. We analysed the combination of DenseNet201 with a Flatten or Global Average Pooling layer. The experimental results show that the accuracy produced by the combination of DenseNet201 with the Global Average Pooling layer is better than DenseNet201 with Flatten layer. The accuracy obtained is 93% which proves the proposed system is more accurate than previous studies.

Download Full-text

Artificial Neural Network Model for Managing and Forecasting Water Reservoir Discharge (Hemren Reservoir as A Case Study)

Diyala Journal of Engineering Sciences ◽

10.24237/djes.2014.07409 ◽

2014 ◽

Vol 7 (4) ◽

pp. 132-143

Author(s):

ABBAS M. ABD ◽

SAAD SH. SAMMEN

Keyword(s):

Neural Network ◽

Artificial Neural Network ◽

Correlation Coefficient ◽

Mean Squared Error ◽

Water Reservoir ◽

Prediction Method ◽

Training Data ◽

Ann Model ◽

Data Set ◽

Artificial Neural

The prediction of different hydrological phenomenon (or system) plays an increasing role in the management of water resources. As engineers; it is required to predict the component of natural reservoirs’ inflow for numerous purposes. Resulting prediction techniques vary with the potential purpose, characteristics, and documented data. The best prediction method is of interest of experts to overcome the uncertainty, because the most hydrological parameters are subjected to the uncertainty. Artificial Neural Network (ANN) approach has adopted in this paper to predict Hemren reservoir inflow. Available data including monthly discharge supplied from DerbendiKhan reservoir and rain fall intensity falling on the intermediate catchment area between Hemren-DerbendiKhan dams were used.A Back Propagation (LMBP) algorithm (Levenberg-Marquardt) has been utilized to construct the ANN models. For the developed ANN model, different networks with different numbers of neurons and layers were evaluated. A total of 24 years of historical data for interval from 1980 to 2004 were used to train and test the networks. The optimum ANN network with 3 inputs, 40 neurons in both two hidden layers and one output was selected. Mean Squared Error (MSE) and the Correlation Coefficient (CC) were employed to evaluate the accuracy of the proposed model. The network was trained and converged at MSE = 0.027 by using training data subjected to early stopping approach. The network could forecast the testing data set with the accuracy of MSE = 0.031. Training and testing process showed the correlation coefficient of 0.97 and 0.77 respectively and this is refer to a high precision of that prediction technique.

Download Full-text

Enhancing Image Diagnosis by the Implementation of Transfer Classifiers

International Journal of Recent Technology and Engineering - 2 ◽

10.35940/ijrte.c4060.098319 ◽

2019 ◽

Vol 8 (3) ◽

pp. 999-1002

Keyword(s):

Supervised Learning ◽

Transfer Learning ◽

Training Data ◽

Data Sets ◽

Learning Approaches ◽

Learning Techniques ◽

Image Diagnosis ◽

Sensitivity Problem ◽

Common Application ◽

Target Data

Images generated from a variety of sources and foundations today can pose difficulty for a user to interpret similarity in them or analyze them for further use because of their segmentation policies. This unconventionality can generate many errors, because of which the previously used traditional methodologies such as supervised learning techniques less resourceful, which requires huge quantity of labelled training data which mirrors the desired target data. This paper thus puts forward the mechanism of an alternative technique i.e. transfer learning to be used in image diagnosis so that efficiency and accuracy among images can be achieved. This type of mechanism deals with variation in the desired and actual data used for training and the outlier sensitivity, which ultimately enhances the predictions by giving better results in various areas, thus leaving the traditional methodologies behind. The following analysis further discusses about three types of transfer classifiers which can be applied using only small volume of training data sets and their contrast with the traditional method which requires huge quantities of training data having attributes with slight changes. The three different separators were compared amongst them and also together from the traditional methodology being used for a very common application used in our daily life. Also, commonly occurring problems such as the outlier sensitivity problem were taken into consideration and measures were taken to recognise and improvise them. On further research it was observed that the performance of transfer learning exceeds that of the conventional supervised learning approaches being used for small amount of characteristic training data provided reducing the stratification errors to a great extent

Download Full-text

An Assessment of Machine Learning Techniques for Predicting Turbine Airfoil Component Temperatures, Using FEA Simulations for Training Data

Volume 5A: Heat Transfer ◽

10.1115/gt2019-91004 ◽

2019 ◽

Author(s):

James A. Tallman ◽

Michal Osusky ◽

Nick Magina ◽

Evan Sewall

Keyword(s):

Neural Network ◽

Machine Learning ◽

Artificial Neural Network ◽

Surrogate Model ◽

Learning Algorithms ◽

Machine Learning Algorithms ◽

Training Data ◽

Machine Learning Techniques ◽

Learning Techniques ◽

Artificial Neural

Abstract This paper provides an assessment of three different machine learning techniques for accurately reproducing a distributed temperature prediction of a high-pressure turbine airfoil. A three-dimensional Finite Element Analysis thermal model of a cooled turbine airfoil was solved repeatedly (200 instances) for various operating point settings of the corresponding gas turbine engine. The response surface created by the repeated solutions was fed into three machine learning algorithms and surrogate model representations of the FEA model’s response were generated. The machine learning algorithms investigated were a Gaussian Process, a Boosted Decision Tree, and an Artificial Neural Network. Additionally, a simple Linear Regression surrogate model was created for comparative purposes. The Artificial Neural Network model proved to be the most successful at reproducing the FEA model over the range of operating points. The mean and standard deviation differences between the FEA and the Neural Network models were 15% and 14% of a desired accuracy threshold, respectively. The Digital Thread for Design (DT4D) was used to expedite all model execution and machine learning training. A description of DT4D is also provided.

Download Full-text

Quantifying Seagrass Distribution in Coastal Water with Deep Learning Models

Remote Sensing ◽

10.3390/rs12101581 ◽

2020 ◽

Vol 12 (10) ◽

pp. 1581 ◽

Cited By ~ 2

Author(s):

Daniel Perez ◽

Kazi Islam ◽

Victoria Hill ◽

Richard Zimmerman ◽

Blake Schaeffer ◽

...

Keyword(s):

Neural Network ◽

Deep Learning ◽

Transfer Learning ◽

Satellite Images ◽

Support Vector ◽

Learning Approach ◽

Learning Models ◽

Learning Techniques ◽

The World ◽

New Locations

Coastal ecosystems are critically affected by seagrass, both economically and ecologically. However, reliable seagrass distribution information is lacking in nearly all parts of the world because of the excessive costs associated with its assessment. In this paper, we develop two deep learning models for automatic seagrass distribution quantification based on 8-band satellite imagery. Specifically, we implemented a deep capsule network (DCN) and a deep convolutional neural network (CNN) to assess seagrass distribution through regression. The DCN model first determines whether seagrass is presented in the image through classification. Second, if seagrass is presented in the image, it quantifies the seagrass through regression. During training, the regression and classification modules are jointly optimized to achieve end-to-end learning. The CNN model is strictly trained for regression in seagrass and non-seagrass patches. In addition, we propose a transfer learning approach to transfer knowledge in the trained deep models at one location to perform seagrass quantification at a different location. We evaluate the proposed methods in three WorldView-2 satellite images taken from the coastal area in Florida. Experimental results show that the proposed deep DCN and CNN models performed similarly and achieved much better results than a linear regression model and a support vector machine. We also demonstrate that using transfer learning techniques for the quantification of seagrass significantly improved the results as compared to directly applying the deep models to new locations.

Download Full-text