Training Feedforward Neural Networks with Gain Constraints

2000 ◽  
Vol 12 (4) ◽  
pp. 811-829 ◽  
Author(s):  
Eric Hartman
Keyword(s):  
Neural Networks ◽  
Objective Function ◽  
Network Models ◽  
Feedforward Neural Networks ◽  
Bound Constraints ◽  
Neural Network Models ◽  
Optimization Control ◽  
Input Variables ◽  
Derivatives Of ◽  
And Training

Inaccurate input-output gains (partial derivatives of outputs with respect to inputs) are common in neural network models when input variables are correlated or when data are incomplete or inaccurate. Accurate gains are essential for optimization, control, and other purposes. We develop and explore a method for training feedforward neural networks subject to inequality or equality-bound constraints on the gains of the learned mapping. Gain constraints are implemented as penalty terms added to the objective function, and training is done using gradient descent. Adaptive and robust procedures are devised for balancing the relative strengths of the various terms in the objective function, which is essential when the constraints are inconsistent with the data. The approach has the virtue that the model domain of validity can be extended via extrapolation training, which can dramatically improve generalization. The algorithm is demonstrated here on artificial and real-world problems with very good results and has been advantageously applied to dozens of models currently in commercial use.

scholarly journals Multistage ensemble of feedforward neural networks for prediction of heating energy consumption

2016 ◽  
Vol 20 (4) ◽  
pp. 1321-1331 ◽  
Author(s):  
Radisa Jovanovic ◽  
Aleksandra Sretenovic ◽  
Branislav Zivkovic
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Energy Consumption ◽  
Network Models ◽  
Feedforward Neural Networks ◽  
Neural Network Ensemble ◽  
University Campus ◽  
Neural Network Models ◽  
Feed Forward Neural Networks ◽  
Heating Energy Consumption

Feedforward neural network models are created for prediction of heating energy consumption of a university campus. Actual measured data are used for training and testing the models. Multistage neural network ensemble is proposed for the possible improvement of prediction accuracy. Previously trained feed-forward neural networks are first separated into clusters, using k-means algorithm, and then the best network of each cluster is chosen as a member of the ensemble. Three different averaging methods (simple, weighted and median) for obtaining ensemble output are applied. Besides this conventional approach, single radial basis neural network in the second level is used to aggregate the selected ensemble members. It is shown that heating energy consumption can be predicted with better accuracy by using ensemble of neural networks than using the best trained single neural network, while the best results are achieved with multistage ensemble.

Data processing using deep learning of the generative-adversarial neural network (GAN)

2021 ◽  
Author(s):  
V.Y. Ilichev ◽  
I.V. Chukhraev
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Neural Networks ◽  
Deep Learning ◽  
Effective Means ◽  
Network Models ◽  
Neural Network Models ◽  
High Quality ◽  
Python Language ◽  
And Training

The article is devoted to the consideration of one of the areas of application of modern and promising computer technology – machine learning. This direction is based on the creation of models consisting of neural networks and their deep learning. At present, there is a need to generate new, not yet existing, images of objects of different types. Most often, text files or images act as such objects. To achieve a high quality of results, a generation method based on the adversarial work of two neural networks (generator and discriminator) was once worked out. This class of neural network models is distinguished by the complexity of topography, since it is necessary to correctly organize the structure of neural layers in order to achieve maximum accuracy and minimal error. The described program is created using the Python language and special libraries that extend the set of commands for performing additional functions: working with neural networks Keras (main library), integrating with the operating system Os, outputting graphs Matplotlib, working with data arrays Numpy and others. A description is given of the type and features of each neural layer, as well as the use of library connection functions, input of initial data, compilation and training of the obtained model. Next, the implementation of the procedure for outputting the results of evaluating the errors of the generator and discriminator and the accuracy achieved by the model depending on the number of cycles (eras) of its training is considered. Based on the results of the work, conclusions were drawn and recommendations were made for the use and development of the considered methodology for creating and training generative and adversarial neural networks. Studies have demonstrated the procedure for operating with comparatively simple and accessible, but effective means of a universal Python language with the Keras library to create and teach a complex neural network model. In fact, it has been proved that the use of this method allows to achieve high-quality results of machine learning, previously achievable only when using special software systems for working with neural networks.

Issues in Bayesian Analysis of Neural Network Models

1998 ◽  
Vol 10 (3) ◽  
pp. 749-770 ◽  
Author(s):  
Peter Müller ◽  
David Rios Insua
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Monte Carlo ◽  
Markov Chain ◽  
Markov Chain Monte Carlo ◽  
Bayesian Analysis ◽  
Network Models ◽  
Feedforward Neural Networks ◽  
Data Driven ◽  
Neural Network Models

Stemming from work by Buntine and Weigend (1991) and MacKay (1992), there is a growing interest in Bayesian analysis of neural network models. Although conceptually simple, this problem is computationally involved. We suggest a very efficient Markov chain Monte Carlo scheme for inference and prediction with fixed-architecture feedforward neural networks. The scheme is then extended to the variable architecture case, providing a data-driven procedure to identify sensible architectures.

Predicting Collapse Potential of Soils with Neural Networks

1997 ◽  
Vol 1582 (1) ◽  
pp. 22-28 ◽  
Author(s):  
C. H. Juang ◽  
David J. Elton
Keyword(s):  
Neural Networks ◽  
Network Models ◽  
Field Testing ◽  
Network Architectures ◽  
Training Algorithms ◽  
Index Properties ◽  
Neural Network Models ◽  
Collapse Potential ◽  
Collapsible Soils ◽  
And Training

Collapsible soils are known to experience a dramatic decrease in volume upon wetting. This can be very detrimental to structures founded on collapsible soils. Whereas field testing might be the most reliable way to determine collapse potential, the engineer often sees it as the last resort. Neural network models for predicting the collapse potential of soils on the basis of basic index properties are presented. Field data, consisting of index properties and collapse potential, are used to train and test neural networks. Various network architectures and training algorithms are examined and compared. The trained networks are shown to be able to identify the collapsible soils and predict the collapse potential.

FEEDFORWARD NEURAL NETWORK MODELS FOR HANDLING CLASS OVERLAP AND CLASS IMBALANCE

2005 ◽  
Vol 15 (05) ◽  
pp. 323-338 ◽  
Author(s):  
RALF KRETZSCHMAR ◽  
NICOLAOS B. KARAYIANNIS ◽  
FRITZ EGGIMANN
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Error Function ◽  
Class Imbalance ◽  
Network Models ◽  
Feedforward Neural Networks ◽  
Feedforward Neural Network ◽  
Neural Network Models ◽  
Real World Data ◽  
Special Case

This paper proposes a framework for training feedforward neural network models capable of handling class overlap and imbalance by minimizing an error function that compensates for such imperfections of the training set. A special case of the proposed error function can be used for training variance-controlled neural networks (VCNNs), which are developed to handle class overlap by minimizing an error function involving the class-specific variance (CSV) computed at their outputs. Another special case of the proposed error function can be used for training class-balancing neural networks (CBNNs), which are developed to handle class imbalance by relying on class-specific correction (CSC). VCNNs and CBNNs are compared with conventional feedforward neural networks (FFNNs), quantum neural networks (QNNs), and resampling techniques. The properties of VCNNs and CBNNs are illustrated by experiments on artificial data. Various experiments involving real-world data reveal the advantages offered by VCNNs and CBNNs in the presence of class overlap and class imbalance.

scholarly journals Monthly Precipitation Forecasts Using Wavelet Neural Networks Models in a Semiarid Environment

Water ◽  
2020 ◽  
Vol 12 (7) ◽  
pp. 1909 ◽  
Author(s):  
Javier Estévez ◽  
Juan Antonio Bellido-Jiménez ◽  
Xiaodong Liu ◽  
Amanda Penélope García-Marín
Keyword(s):  
Neural Networks ◽  
Correlation Coefficients ◽  
Network Models ◽  
Semiarid Region ◽  
Monthly Precipitation ◽  
Neural Network Models ◽  
Non Linear ◽  
Effective Performance ◽  
Accurate Forecast ◽  
Input Variables

Accurate forecast of hydrological data such as precipitation is critical in order to provide useful information for water resources management, playing a key role in different sectors. Traditional forecasting methods present many limitations due to the high-stochastic property of precipitation and its strong variability in time and space: not identifying non-linear dynamics or not solving the instability of local weather situations. In this work, several alternative models based on the combination of wavelet analysis (multiscalar decomposition) with artificial neural networks have been developed and evaluated at sixteen locations in Southern Spain (semiarid region of Andalusia), representative of different climatic and geographical conditions. Based on the capability of wavelets to describe non-linear signals, ten wavelet neural network models (WNN) have been applied to predict monthly precipitation by using short-term thermo-pluviometric time series. Overall, the forecasting results show differences between the ten models, although an effective performance (i.e., correlation coefficients ranged from 0.76 to 0.90 and Root Mean Square Error values ranged from 6.79 to 29.82 mm) was obtained at each of the locations assessed. The most appropriate input variables to obtain the best forecasts are analyzed, according to the geo-climatic characteristics of the sixteen sites studied.

scholarly journals Opening the black box of neural networks: methods for interpreting neural network models in clinical applications

2018 ◽  
Vol 6 (11) ◽  
pp. 216-216 ◽  
Author(s):  
Zhongheng Zhang ◽  
◽  
Marcus W. Beck ◽  
David A. Winkler ◽  
Bin Huang ◽  
...  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Network Models ◽  
Black Box ◽  
Clinical Applications ◽  
Neural Network Models

Predicting the exchange rate in Sudan using neural networks models during the period (1960 - 2017): التنبؤ بسعر الصرف في السودان باستخدام نماذج الشبكات العصبية خلال الفترة (1960م – 2017م)

2020 ◽  
Vol 4 (14) ◽  
Author(s):  
Fathi Ahmed Ali Adam, Mahmoud Mohamed Abdel Aziz Gamal El-Di
Keyword(s):  
Neural Networks ◽  
Artificial Neural Networks ◽  
Exchange Rate ◽  
Network Models ◽  
Neural Network Models ◽  
Classical Models ◽  
Artificial Neural ◽  
The Exchange Rate ◽  
Long Term Forecasting ◽  
Artificial Neural Network Models

The study examined the use of artificial neural network models to predict the exchange rate in Sudan through annual exchange rate data between the US dollar and the Sudanese pound. This study aimed to formulate the models of artificial neural networks in which the exchange rate can be predicted in the coming period. The importance of the study is that it is necessary to use modern models to predict instead of other classical models. The study hypothesized that the models of artificial neural networks have a high ability to predict the exchange rate. Use models of artificial neural networks. The most important results ability of artificial neural networks models to predict the exchange rate accurately, Form MLP (1-1-1) is the best model chosen for that purpose. The study recommended the development of the proposed model for long-term forecasting.

scholarly journals Optical Recognition of Handwritten Logic Formulas Using Neural Networks

Electronics ◽  
2021 ◽  
Vol 10 (22) ◽  
pp. 2761
Author(s):  
Vaios Ampelakiotis ◽  
Isidoros Perikos ◽  
Ioannis Hatzilygeroudis ◽  
George Tsihrintzis
Keyword(s):  
Neural Networks ◽  
Character Recognition ◽  
Gradient Descent ◽  
Feedforward Neural Networks ◽  
Stochastic Gradient ◽  
Stochastic Gradient Descent ◽  
Training Algorithms ◽  
Gradient Descent Algorithm ◽  
Two Stages ◽  
And Training

In this paper, we present a handwritten character recognition (HCR) system that aims to recognize first-order logic handwritten formulas and create editable text files of the recognized formulas. Dense feedforward neural networks (NNs) are utilized, and their performance is examined under various training conditions and methods. More specifically, after three training algorithms (backpropagation, resilient propagation and stochastic gradient descent) had been tested, we created and trained an NN with the stochastic gradient descent algorithm, optimized by the Adam update rule, which was proved to be the best, using a trainset of 16,750 handwritten image samples of 28 × 28 each and a testset of 7947 samples. The final accuracy achieved is 90.13%. The general methodology followed consists of two stages: the image processing and the NN design and training. Finally, an application has been created that implements the methodology and automatically recognizes handwritten logic formulas. An interesting feature of the application is that it allows for creating new, user-oriented training sets and parameter settings, and thus new NN models.

scholarly journals Robotic grasp detection using a novel two-stage approach

2021 ◽  
Vol 1 (1) ◽  
pp. 19-29
Author(s):  
Zhe Chu ◽  
Mengkai Hu ◽  
Xiangyu Chen
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Network Models ◽  
Particle Swarm Optimizer ◽  
Neural Network Models ◽  
Two Stage ◽  
The Neural Network ◽  
End To End ◽  
Small Change ◽  
Robotic Grasp

Recently, deep learning has been successfully applied to robotic grasp detection. Based on convolutional neural networks (CNNs), there have been lots of end-to-end detection approaches. But end-to-end approaches have strict requirements for the dataset used for training the neural network models and it’s hard to achieve in practical use. Therefore, we proposed a two-stage approach using particle swarm optimizer (PSO) candidate estimator and CNN to detect the most likely grasp. Our approach achieved an accuracy of 92.8% on the Cornell Grasp Dataset, which leaped into the front ranks of the existing approaches and is able to run at real-time speeds. After a small change of the approach, we can predict multiple grasps per object in the meantime so that an object can be grasped in a variety of ways.

Sign in / Sign up

Export Citation Format

Share Document