scholarly journals A homotopy training algorithm for fully connected neural networks

2019 ◽  
Vol 475 (2231) ◽  
pp. 20190662
Author(s):  
Qipin Chen ◽  
Wenrui Hao
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Optimization Problem ◽  
Global Minimum ◽  
Optimization Problems ◽  
Continuous Path ◽  
Training Algorithm ◽  
Test Dataset ◽  
The Neural Network ◽  
Fully Connected

In this paper, we present a homotopy training algorithm (HTA) to solve optimization problems arising from fully connected neural networks with complicated structures. The HTA dynamically builds the neural network starting from a simplified version and ending with the fully connected network via adding layers and nodes adaptively. Therefore, the corresponding optimization problem is easy to solve at the beginning and connects to the original model via a continuous path guided by the HTA, which provides a high probability of obtaining a global minimum. By gradually increasing the complexity of the model along the continuous path, the HTA provides a rather good solution to the original loss function. This is confirmed by various numerical results including VGG models on CIFAR-10. For example, on the VGG13 model with batch normalization, HTA reduces the error rate by 11.86% on the test dataset compared with the traditional method. Moreover, the HTA also allows us to find the optimal structure for a fully connected neural network by building the neutral network adaptively.

scholarly journals Discretization and machine learning approximation of BSDEs with a constraint on the Gains-process

2021 ◽  
Vol 0 (0) ◽  
Author(s):  
Idris Kharroubi ◽  
Thomas Lim ◽  
Xavier Warin
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Neural Networks ◽  
Differential Equations ◽  
Numerical Experiments ◽  
Optimization Problem ◽  
Learning Approach ◽  
The Neural Network ◽  
Machine Learning Approach ◽  
Mesh Grid

AbstractWe study the approximation of backward stochastic differential equations (BSDEs for short) with a constraint on the gains process. We first discretize the constraint by applying a so-called facelift operator at times of a grid. We show that this discretely constrained BSDE converges to the continuously constrained one as the mesh grid converges to zero. We then focus on the approximation of the discretely constrained BSDE. For that we adopt a machine learning approach. We show that the facelift can be approximated by an optimization problem over a class of neural networks under constraints on the neural network and its derivative. We then derive an algorithm converging to the discretely constrained BSDE as the number of neurons goes to infinity. We end by numerical experiments.

Estimation of Site Amplification from Geotechnical Array Data Using Neural Networks

2021 ◽  
Author(s):  
Daniel Roten ◽  
Kim B. Olsen
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Site Response ◽  
Site Amplification ◽  
Data Driven ◽  
Vertical Array ◽  
The Neural Network ◽  
Simplifying Assumptions ◽  
The Mean ◽  
Fully Connected

ABSTRACT We use deep learning to predict surface-to-borehole Fourier amplification functions (AFs) from discretized shear-wave velocity profiles. Specifically, we train a fully connected neural network and a convolutional neural network using mean AFs observed at ∼600 KiK-net vertical array sites. Compared with predictions based on theoretical SH 1D amplifications, the neural network (NN) results in up to 50% reduction of the mean squared log error between predictions and observations at sites not used for training. In the future, NNs may lead to a purely data-driven prediction of site response that is independent of proxies or simplifying assumptions.

An Interactive Neural Network for Constrained Multi-Objective Optimization with Application to the Design of Digital Filters

2012 ◽  
Vol 433-440 ◽  
pp. 2808-2816
Author(s):  
Jian Jin Zheng ◽  
You Shen Xia
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Computational Complexity ◽  
Digital Filters ◽  
Optimization Problem ◽  
Optimization Problems ◽  
Optimal Solution ◽  
Multi Objective Optimization ◽  
Multi Objective ◽  
Single Objective

This paper presents a new interactive neural network for solving constrained multi-objective optimization problems. The constrained multi-objective optimization problem is reformulated into two constrained single objective optimization problems and two neural networks are designed to obtain the optimal weight and the optimal solution of the two optimization problems respectively. The proposed algorithm has a low computational complexity and is easy to be implemented. Moreover, the proposed algorithm is well applied to the design of digital filters. Computed results illustrate the good performance of the proposed algorithm.

A BIOLOGICALLY INSPIRED NEURAL NETWORK FOR DYNAMIC PROGRAMMING

2001 ◽  
Vol 11 (06) ◽  
pp. 561-572 ◽  
Author(s):  
ROSELI A. FRANCELIN ROMERO ◽  
JANUSZ KACPRYZK ◽  
FERNANDO GOMIDE
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Dynamic Programming ◽  
Optimization Problems ◽  
Direct Method ◽  
Fuzzy Decision Making ◽  
Fuzzy Decision ◽  
Biologically Inspired ◽  
The Neural Network ◽  
The Neural Networks

An artificial neural network with a two-layer feedback topology and generalized recurrent neurons, for solving nonlinear discrete dynamic optimization problems, is developed. A direct method to assign the weights of neural networks is presented. The method is based on Bellmann's Optimality Principle and on the interchange of information which occurs during the synaptic chemical processing among neurons. The neural network based algorithm is an advantageous approach for dynamic programming due to the inherent parallelism of the neural networks; further it reduces the severity of computational problems that can occur in methods like conventional methods. Some illustrative application examples are presented to show how this approach works out including the shortest path and fuzzy decision making problems.

TRAINING SIMULTANEOUS RECURRENT NEURAL NETWORK WITH RESILIENT PROPAGATION FOR STATIC OPTIMIZATION

2002 ◽  
Vol 12 (03n04) ◽  
pp. 203-218 ◽  
Author(s):  
GURSEL SERPEN ◽  
JOEL CORRA
Keyword(s):  
Neural Network ◽  
Traveling Salesman Problem ◽  
Recurrent Neural Network ◽  
Optimization Problem ◽  
Optimization Problems ◽  
Traveling Salesman ◽  
Training Algorithm ◽  
Comparable Amount ◽  
Static Optimization ◽  
The Traveling Salesman Problem

This paper proposes a non-recurrent training algorithm, resilient propagation, for the Simultaneous Recurrent Neural network operating in relaxation-mode for computing high quality solutions of static optimization problems. Implementation details related to adaptation of the recurrent neural network weights through the non-recurrent training algorithm, resilient backpropagation, are formulated throughan algebraic approach. Performance of the proposed neuro-optimizer on a well-known static combinatorial optimization problem, the Traveling Salesman Problem, is evaluated on the basis of computational complexity measures and, subsequently, compared to performance of the Simultaneous Recurrent Neural network trained with the standard backpropagation, and recurrent backpropagation for the same static optimization problem. Simulation results indicate that the Simultaneous Recurrent Neural network trained with the resilient backpropagation algorithm is able to locate superior quality solutions through comparable amount of computational effort for the Traveling Salesman Problem.

scholarly journals Building a mathematical model and an algorithm for training a neural network with sparse dipole synaptic connections for image recognition

2021 ◽  
Vol 6 (4 (114)) ◽  
pp. 21-27
Author(s):  
Vasyl Lytvyn ◽  
Roman Peleshchak ◽  
Ivan Peleshchak ◽  
Oksana Cherniak ◽  
Lyubomyr Demkiv
Keyword(s):  
Neural Network ◽  
Computer Experiment ◽  
Dipole Interaction ◽  
Dipole Approximation ◽  
Synaptic Connections ◽  
Training Algorithm ◽  
Learning Time ◽  
The Neural Network ◽  
Fully Connected ◽  
Weight Coefficients

Large enough structured neural networks are used for solving the tasks to recognize distorted images involving computer systems. One such neural network that can completely restore a distorted image is a fully connected pseudospin (dipole) neural network that possesses associative memory. When submitting some image to its input, it automatically selects and outputs the image that is closest to the input one. This image is stored in the neural network memory within the Hopfield paradigm. Within this paradigm, it is possible to memorize and reproduce arrays of information that have their own internal structure. In order to reduce learning time, the size of the neural network is minimized by simplifying its structure based on one of the approaches: underlying the first is «regularization» while the second is based on the removal of synaptic connections from the neural network. In this work, the simplification of the structure of a fully connected dipole neural network is based on the dipole-dipole interaction between the nearest adjacent neurons of the network. It is proposed to minimize the size of a neural network through dipole-dipole synaptic connections between the nearest neurons, which reduces the time of the computational resource in the recognition of distorted images. The ratio for weight coefficients of synaptic connections between neurons in dipole approximation has been derived. A training algorithm has been built for a dipole neural network with sparse synaptic connections, which is based on the dipole-dipole interaction between the nearest neurons. A computer experiment was conducted that showed that the neural network with sparse dipole connections recognizes distorted images 3 times faster (numbers from 0 to 9, which are shown at 25 pixels), compared to a fully connected neural network

On Computer Memory Saving Methods in Performing Data Classification Using Fully Connected Neural Networks

Vestnik MEI ◽  
2021 ◽  
Vol 3 (3) ◽  
pp. 103-109
Author(s):  
Andrey I. Mamontov ◽  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Computing System ◽  
Classification Problem ◽  
Double Precision ◽  
Data Set ◽  
The Neural Network ◽  
Study Results ◽  
System Memory ◽  
Fully Connected

In solving the classification problem, a fully connected trainable neural network (with adjusting the parameters represented by double-precision real numbers) is used as a mathematical model. After the training is completed, the neural network parameters are rounded and represented as fixed-point numbers (integers). The aim of the study is to reduce the required amount of the computing system memory for storing the obtained integer parameters. To reduce the amount of memory, the following methods for storing integer parameters are developed, which are based on representing the linear polynomials included in a fully connected neural network using compositions of simpler functions: - a method based on representing the considered polynomial as a sum of simpler polynomials; - a method based on separately storing the information about additions and multiplications. In the experiment with the MNIST data set, it took 1.41 MB to store real parameters of a fully connected neural network, 0.7 MB to store integer parameters without using the proposed methods, 0.47 MB in the RAM and 0.3 MB in compressed form on the disk when using the first method, and 0.25 MB on the disk when using the second method. In the experiment with the USPS data set, it took 0.25 MB to store real parameters of a fully connected neural network, 0.1 MB to store integer parameters without using the proposed methods, 0.05 MB in the RAM and approximately the same amount in compressed form on the disk when using the first method, and 0.03 MB on the disk when using the second method. The study results can be applied in using fully connected neural networks to solve various recognition problems under the conditions of limited hardware capacities.

scholarly journals Optimization of Artificial Neural Networks with Genetic Algorithms for Biometric Pattern Recognition

2019 ◽  
Vol 24 (3) ◽  
pp. 256-264
Author(s):  
Gábor Werner ◽  
László Hanka
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Pattern Recognition ◽  
Genetic Algorithms ◽  
Artificial Neural Networks ◽  
Processing Time ◽  
Optimization Problems ◽  
Physical Environments ◽  
The Neural Network ◽  
Artificial Neural

Abstract The process of pattern recognition in the biometrics is particularly important. The patterns can differ from each other a lot, and even the samples can be significantly different from the templates. The Artificial Neural Networks can be applied as a universal approximator to recognize the patterns with more flexibility. However the topology of the networks determines the processing time and complexity of the hardware of the physical environments. The Genetic Algorithms can be used with success in optimization problems like in this situation, the topology of the neural network is more optimal if we apply the Genetic Algorithms. This study introduce an algorithm in which a tailor made algorithm correcting the topology to enhance the effectiveness of the process.

scholarly journals Construction of neural network classification of Fisher irises based on JavaScript

2019 ◽  
Author(s):  
A.М. Заяц ◽  
С.П. Хабаров
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Back Propagation ◽  
Data Set ◽  
Test Dataset ◽  
Neural Network Classification ◽  
The Neural Network ◽  
Multilayer Neural Networks ◽  
Hidden Layer

Рассматривается процедура выбора структуры и параметров нейронной сети для классификации набора данных, известного как Ирисы Фишера, который включает в себя данные о 150 экземплярах растений трех различных видов. Предложен подход к решению данной задачи без использования дополнительных программных средств и мощных нейросетевых пакетов с использованием только средств стандартного браузера ОС. Это потребовало реализации ряда процедур на JavaScript c их подгрузкой в разработанную интерфейсную HTML-страницу. Исследование большого числа различных структур многослойных нейронных сетей, обучаемых на основе алгоритма обратного распространения ошибки, позволило выбрать для тестового набора данных структуру нейронной сети всего с одним скрытым слоем из трех нейронов. Это существенно упрощает реализацию классификатора Ирисов Фишера, позволяя его оформить в виде загружаемой с сервера HTML-страницы. The procedure for selecting the structure and parameters of the neural network for the classification of a data set known as Iris Fisher, which includes data on 150 plant specimens of three different species, is considered. An approach to solving this problem without using additional software and powerful neural network packages using only the tools of the standard OS browser is proposed. This required the implementation of a number of JavaScript procedures with their loading into the developed HTML interface page. The study of a large number of different structures of multilayer neural networks, trained on the basis of the back-propagation error algorithm, made it possible to choose the structure of a neural network with only one hidden layer of three neurons for a test dataset. This greatly simplifies the implementation of the Fisher Iris classifier, allowing it to be formatted as an HTML page downloaded from the server.

SCORING MODELING BASED ON NEURAL NETWORKS FOR DETERMINING A BANK BORROWER'S RATING

2020 ◽  
Vol 2020 (10) ◽  
pp. 54-62
Author(s):  
Oleksii VASYLIEV ◽  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Network Architecture ◽  
Statistical Data ◽  
Activation Function ◽  
Decision Making Process ◽  
Neural Network Architecture ◽  
Acceptable Accuracy ◽  
The Neural Network ◽  
Sigmoid Activation Function

The problem of applying neural networks to calculate ratings used in banking in the decision-making process on granting or not granting loans to borrowers is considered. The task is to determine the rating function of the borrower based on a set of statistical data on the effectiveness of loans provided by the bank. When constructing a regression model to calculate the rating function, it is necessary to know its general form. If so, the task is to calculate the parameters that are included in the expression for the rating function. In contrast to this approach, in the case of using neural networks, there is no need to specify the general form for the rating function. Instead, certain neural network architecture is chosen and parameters are calculated for it on the basis of statistical data. Importantly, the same neural network architecture can be used to process different sets of statistical data. The disadvantages of using neural networks include the need to calculate a large number of parameters. There is also no universal algorithm that would determine the optimal neural network architecture. As an example of the use of neural networks to determine the borrower's rating, a model system is considered, in which the borrower's rating is determined by a known non-analytical rating function. A neural network with two inner layers, which contain, respectively, three and two neurons and have a sigmoid activation function, is used for modeling. It is shown that the use of the neural network allows restoring the borrower's rating function with quite acceptable accuracy.

Sign in / Sign up

Export Citation Format

Share Document