Hybrid Model Structure for Diabetic Retinopathy Classification

Diabetic retinopathy (DR) is one of the most common complications of diabetes and the main cause of blindness. The progression of the disease can be prevented by early diagnosis of DR. Due to differences in the distribution of medical conditions and low labor efficiency, the best time for diagnosis and treatment was missed, which results in impaired vision. Using neural network models to classify and diagnose DR can improve efficiency and reduce costs. In this work, an improved loss function and three hybrid model structures Hybrid-a, Hybrid-f, and Hybrid-c were proposed to improve the performance of DR classification models. EfficientNetB4, EfficientNetB5, NASNetLarge, Xception, and InceptionResNetV2 CNNs were chosen as the basic models. These basic models were trained using enhance cross-entropy loss and cross-entropy loss, respectively. The output of the basic models was used to train the hybrid model structures. Experiments showed that enhance cross-entropy loss can effectively accelerate the training process of the basic models and improve the performance of the models under various evaluation metrics. The proposed hybrid model structures can also improve DR classification performance. Compared with the best-performing results in the basic models, the accuracy of DR classification was improved from 85.44% to 86.34%, the sensitivity was improved from 98.48% to 98.77%, the specificity was improved from 71.82% to 74.76%, the precision was improved from 90.27% to 91.37%, and the F1 score was improved from 93.62% to 93.9% by using hybrid model structures.

Download Full-text

Individual differences among deep neural network models

10.1101/2020.01.08.898288 ◽

2020 ◽

Cited By ~ 4

Author(s):

Johannes Mehrer ◽

Courtney J. Spoerer ◽

Nikolaus Kriegeskorte ◽

Tim C. Kietzmann

Keyword(s):

Individual Differences ◽

Visual Recognition ◽

Initial Conditions ◽

Network Models ◽

Classification Performance ◽

Neural Network Models ◽

Multiple Networks ◽

Modelling Framework ◽

Neural Computations ◽

Category Exemplars

AbstractDeep neural networks (DNNs) excel at visual recognition tasks and are increasingly used as a modelling framework for neural computations in the primate brain. However, each DNN instance, just like each individual brain, has a unique connectivity and representational profile. Here, we investigate individual differences among DNN instances that arise from varying only the random initialization of the network weights. Using representational similarity analysis, we demonstrate that this minimal change in initial conditions prior to training leads to substantial differences in intermediate and higher-level network representations, despite achieving indistinguishable network-level classification performance. We locate the origins of the effects in an under-constrained alignment of category exemplars, rather than a misalignment of category centroids. Furthermore, while network regularization can increase the consistency of learned representations, considerable differences remain. These results suggest that computational neuroscientists working with DNNs should base their inferences on multiple networks instances instead of single off-the-shelf networks.

Download Full-text

Individual differences among deep neural network models

Nature Communications ◽

10.1038/s41467-020-19632-w ◽

2020 ◽

Vol 11 (1) ◽

Author(s):

Johannes Mehrer ◽

Courtney J. Spoerer ◽

Nikolaus Kriegeskorte ◽

Tim C. Kietzmann

Keyword(s):

Individual Differences ◽

Visual Recognition ◽

Initial Conditions ◽

Network Models ◽

Classification Performance ◽

Neural Network Models ◽

Modeling Framework ◽

Neural Computations ◽

Neural Information ◽

Multiple Network

AbstractDeep neural networks (DNNs) excel at visual recognition tasks and are increasingly used as a modeling framework for neural computations in the primate brain. Just like individual brains, each DNN has a unique connectivity and representational profile. Here, we investigate individual differences among DNN instances that arise from varying only the random initialization of the network weights. Using tools typically employed in systems neuroscience, we show that this minimal change in initial conditions prior to training leads to substantial differences in intermediate and higher-level network representations despite similar network-level classification performance. We locate the origins of the effects in an under-constrained alignment of category exemplars, rather than misaligned category centroids. These results call into question the common practice of using single networks to derive insights into neural information processing and rather suggest that computational neuroscientists working with DNNs may need to base their inferences on groups of multiple network instances.

Download Full-text

A Novel Analytical-ANN Hybrid Model for Borehole Heat Exchanger

Energies ◽

10.3390/en13236213 ◽

2020 ◽

Vol 13 (23) ◽

pp. 6213

Author(s):

Anjan Rao Puttige ◽

Staffan Andersson ◽

Ronny Östin ◽

Thomas Olofsson

Keyword(s):

Neural Network ◽

Heat Exchanger ◽

Analytical Model ◽

Hybrid Model ◽

Time Resolution ◽

Network Models ◽

Borehole Heat Exchanger ◽

Neural Network Models ◽

Ground Source

Optimizing the operation of ground source heat pumps requires simulation of both short-term and long-term response of the borehole heat exchanger. However, the current physical and neural network based models are not suited to handle the large range of time scales, especially for large borehole fields. In this study, we present a hybrid model for long-term simulation of BHE with high resolution in time. The model uses an analytical model with low time resolution to guide an artificial neural network model with high time resolution. We trained, tuned, and tested the hybrid model using measured data from a ground source heat pump in real operation. The performance of the hybrid model is compared with an analytical model, a calibrated analytical model, and three different types of neural network models. The hybrid model has a relative RMSE of 6% for the testing period compared to 22%, 14%, and 12% respectively for the analytical model, the calibrated analytical model, and the best of the three investigated neural network models. The hybrid model also has a reasonable computational time and was also found to be robust with regard to the model parameters used by the analytical model.

Download Full-text

Shaping the learning landscape in neural networks around wide flat minima

Proceedings of the National Academy of Sciences ◽

10.1073/pnas.1908636117 ◽

2019 ◽

Vol 117 (1) ◽

pp. 161-170 ◽

Cited By ~ 2

Author(s):

Carlo Baldassi ◽

Fabrizio Pittorino ◽

Riccardo Zecchina

Keyword(s):

Neural Networks ◽

Loss Function ◽

Critical Points ◽

Learning Process ◽

Numerical Study ◽

Cross Entropy ◽

Stochastic Gradient Descent ◽

Neural Network Models ◽

Entropy Loss ◽

Error Loss

Learning in deep neural networks takes place by minimizing a nonconvex high-dimensional loss function, typically by a stochastic gradient descent (SGD) strategy. The learning process is observed to be able to find good minimizers without getting stuck in local critical points and such minimizers are often satisfactory at avoiding overfitting. How these 2 features can be kept under control in nonlinear devices composed of millions of tunable connections is a profound and far-reaching open question. In this paper we study basic nonconvex 1- and 2-layer neural network models that learn random patterns and derive a number of basic geometrical and algorithmic features which suggest some answers. We first show that the error loss function presents few extremely wide flat minima (WFM) which coexist with narrower minima and critical points. We then show that the minimizers of the cross-entropy loss function overlap with the WFM of the error loss. We also show examples of learning devices for which WFM do not exist. From the algorithmic perspective we derive entropy-driven greedy and message-passing algorithms that focus their search on wide flat regions of minimizers. In the case of SGD and cross-entropy loss, we show that a slow reduction of the norm of the weights along the learning process also leads to WFM. We corroborate the results by a numerical study of the correlations between the volumes of the minimizers, their Hessian, and their generalization performance on real data.

Download Full-text

Early Detection of Diabetic Retinopathy

International Journal for Research in Applied Science and Engineering Technology ◽

10.22214/ijraset.2021.37918 ◽

2021 ◽

Vol 9 (IX) ◽

pp. 429-435

Author(s):

Vamsi Krishna Mekala

Keyword(s):

Diabetic Retinopathy ◽

Deep Learning ◽

Vision Loss ◽

Network Models ◽

Medical System ◽

Neural Network Models ◽

Contemporary World ◽

Common Causes ◽

Medical Photography ◽

Speed Up

Diabetic retinopathy are among the most common causes of vision loss in today's world. Visual impairment impacts about one in 3 diabetics, according to an epidemiological research. Diagnostic imaging is an important aspect of medical photography in contemporary world. Deep learning improves the eyesight for identifying illness in radiography. The goal is to use machine learning to diagnose vision loss. Deep learning in diagnostic devices might improve and speed up the diagnosis of sugar-related vision loss. This research will look at neural network models, algorithms, and simulations in order to diagnose diabetic retinopathy rapidly and help the medical system. The classifier is constructed using CNN.

Download Full-text

Adversarial Image Perturbation with a Genetic Algorithm

10.18690/978-961-286-516-0.6 ◽

2021 ◽

Author(s):

Rok Kukovec ◽

Špela Pečnik ◽

Iztok Fister Jr. ◽

Sašo Karakatič

Keyword(s):

Neural Network ◽

Genetic Algorithm ◽

Prior Knowledge ◽

Network Models ◽

Visual Assessment ◽

Training Process ◽

Neural Network Models ◽

The Neural Network ◽

Statistical Metrics

The quality of image recognition with neural network models relies heavily on filters and parameters optimized through the training process. These filters are di˙erent compared to how humans see and recognize objects around them. The di˙erence in machine and human recognition yields a noticeable gap, which is prone to exploitation. The workings of these algorithms can be compromised with adversarial perturbations of images. This is where images are seemingly modified imperceptibly, such that humans see little to no di˙erence, but the neural network classifies t he m otif i ncorrectly. This paper explores the adversarial image modifica-tion with an evolutionary algorithm, so that the AlexNet convolutional neural network cannot recognize previously clear motifs while preserving the human perceptibility of the image. The ex-periment was implemented in Python and tested on the ILSVRC dataset. Original images and their recreated counterparts were compared and contrasted using visual assessment and statistical metrics. The findings s uggest t hat t he human eye, without prior knowledge, will hardly spot the di˙erence compared to the original images.

Download Full-text

Detecting Fraudulent Bank Account Based on Convolutional Neural Network with Heterogeneous Data

Mathematical Problems in Engineering ◽

10.1155/2019/3759607 ◽

2019 ◽

Vol 2019 ◽

pp. 1-11

Author(s):

Fang Lv ◽

Wei Wang ◽

Yuliang Wei ◽

Yunxiao Sun ◽

Junheng Huang ◽

...

Keyword(s):

Neural Network ◽

Convolutional Neural Network ◽

Social Representations ◽

Network Models ◽

Heterogeneous Data ◽

Model Structure ◽

Topological Feature ◽

Neural Network Models ◽

Bank Account ◽

Bank Accounts

Detecting fraudulent accounts by using their transaction networks is helpful for proactively preventing illegal transactions in financial scenarios. In this paper, three convolutional neural network models, i.e., NTD-CNN, TTD-CNN, and HDF-CNN, are created to identify whether a bank account is fraudulent. The three models, same in model structure, are different in types of the input features. Firstly, we embed the bank accounts’ historical trading records into a general directed and weighted transaction network. And then, a DirectedWalk algorithm is proposed for learning an account’s network vector. DirectedWalk learns social representations of a network’s vertices, by modeling a stream of directed and time-related trading paths. The local topological feature, generating by accounts’ network vector, is taken as input of NTD-CNN, and TTD-CNN takes time series transaction feature as input. Finally, the two kinds of heterogeneous data, being integrated into a novel feature matrix, are fed into HDF-CNN for classifying bank accounts. The experimental results, conducted on a real bank transaction dataset, show the advantage of HDF-CNN over the existing methods.

Download Full-text

Neural classifiers with limited connectivity and recurrent readouts

10.1101/157289 ◽

2017 ◽

Author(s):

Lyudmila Kushnir ◽

Stefano Fusi

Keyword(s):

Neural Network ◽

Long Range ◽

Network Models ◽

Classification Performance ◽

Network Architectures ◽

Huge Number ◽

Neural Network Models ◽

Random Patterns ◽

Scalable Network ◽

The Brain

AbstractFor many neural network models in which neurons are trained to classify inputs like perceptrons, the number of inputs that can be classified is limited by the connectivity of each neuron, even when the total number of neurons is very large. This poses the problem of how the biological brain can take advantage of its huge number of neurons given that the connectivity is sparse. One solution is to combine multiple perceptrons together, as in committee machines. The number of classifiable random patterns would then grow linearly with the number of perceptrons, even when each perceptron has limited connectivity. However, the problem is moved to the downstream readout neurons, which would need a number of connections that is as large as the number of perceptrons. Here we propose a different approach in which the readout is implemented by connecting multiple perceptrons in a recurrent attractor neural network. We prove analytically that the number of classifiable random patterns can grow unboundedly with the number of perceptrons, even when the connectivity of each perceptron remains finite. Most importantly, both the recurrent connectivity and the connectivity of downstream readouts also remain finite. Our study shows that feed-forward neural classifiers with numerous long range afferent connections can be replaced by recurrent networks with sparse long range connectivity without sacrificing the classification performance. Our strategy could be used to design more general scalable network architectures with limited connectivity, which resemble more closely the brain neural circuits which are dominated by recurrent connectivity.

Download Full-text

Hybrid Dense Network with Dual Attention for Hyperspectral Image Classification

Remote Sensing ◽

10.3390/rs13234921 ◽

2021 ◽

Vol 13 (23) ◽

pp. 4921

Author(s):

Jinling Zhao ◽

Lei Hu ◽

Yingying Dong ◽

Linsheng Huang

Keyword(s):

Classification Accuracy ◽

Hyperspectral Image ◽

Network Models ◽

Classification Performance ◽

Dense Network ◽

Feature Maps ◽

Neural Network Models ◽

Practical Applications ◽

Training Samples ◽

3D Cnn

Hyperspectral images (HSIs) have been widely used in many fields of application, but it is still extremely challenging to obtain higher classification accuracy, especially when facing a smaller number of training samples in practical applications. It is very time-consuming and laborious to acquire enough labeled samples. Consequently, an efficient hybrid dense network was proposed based on a dual-attention mechanism, due to limited training samples and unsatisfactory classification accuracy. The stacked autoencoder was first used to reduce the dimensions of HSIs. A hybrid dense network framework with two feature-extraction branches was then established in order to extract abundant spectral–spatial features from HSIs, based on the 3D and 2D convolutional neural network models. In addition, spatial attention and channel attention were jointly introduced in order to achieve selective learning of features derived from HSIs. The feature maps were further refined, and more important features could be retained. To improve computational efficiency and prevent the overfitting, the batch normalization layer and the dropout layer were adopted. The Indian Pines, Pavia University, and Salinas datasets were selected to evaluate the classification performance; 5%, 1%, and 1% of classes were randomly selected as training samples, respectively. In comparison with the REF-SVM, 3D-CNN, HybridSN, SSRN, and R-HybridSN, the overall accuracy of our proposed method could still reach 96.80%, 98.28%, and 98.85%, respectively. Our results show that this method can achieve a satisfactory classification performance even in the case of fewer training samples.

Download Full-text

Pemodelan Generator Uap Berbasis Jaringan Saraf Tiruan dengan Algoritme Pelatihan BPGD-ALAM

IJEIS (Indonesian Journal of Electronics and Instrumentation Systems) ◽

10.22146/ijeis.10766 ◽

2016 ◽

Vol 6 (1) ◽

pp. 1

Author(s):

Fadhlia Annisa ◽

Agfianto Eko Putra

Keyword(s):

Neural Network ◽

Steam Generator ◽

Multiple Input Multiple Output ◽

Network Models ◽

Training Data ◽

Training Process ◽

Training Simulator ◽

Neural Network Models ◽

Validation Data ◽

Testing Data

Steam generator is unit plant which has nonlinear and complex system with multiple-input-multiple-output (MIMO) configuration which is hard to be modeled. Whereas, steam generator model is very useful to create simulation such as operator training simulator (OTS). The purpose of this research is to obtain model of steam generator which has 8 output parameters and 9 input parameters based neural network (NN) with BPGD-ALAM training algorithm. Data had been taken from steam generator of PT. Chevron Pacific Indonesia, Duri and it is divided into three types, i.e training data, validation data and testing data. Training data was used to obtain model for each ouput through training process. Verification model is also done for each epoch using validation data to monitor training process whether overfitting occurs or not. Eight NN model of each output which is obtained from training and verification, is tested using testing data for getting its performance. From the reseach results, architecture of neural network models are obtained with various configuration for each output with RMSE value under 9.71 %. It shows that model which has been obtained, close with steam generator real system.

Download Full-text