scholarly journals Efficient Nonlinear Activation Function in Optical Neural Network

2020 ◽  
Author(s):  
H. Zhang ◽  
L. X. Wan ◽  
M. Gu ◽  
X. D. Jiang ◽  
J. Thompson ◽  
...  
Keyword(s):  
Neural Network ◽  
Activation Function ◽  
Optical Neural Network ◽  
Nonlinear Activation Function

Relating the Slope of the Activation Function and the Learning Rate Within a Recurrent Neural Network

1999 ◽  
Vol 11 (5) ◽  
pp. 1069-1077 ◽  
Author(s):  
Danilo P. Mandic ◽  
Jonathon A. Chambers
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Recurrent Neural Network ◽  
Recurrent Neural Networks ◽  
Degrees Of Freedom ◽  
Learning Algorithm ◽  
Activation Function ◽  
Learning Rate ◽  
Optimization Task ◽  
Nonlinear Activation Function

A relationship between the learning rate η in the learning algorithm, and the slope β in the nonlinear activation function, for a class of recurrent neural networks (RNNs) trained by the real-time recurrent learning algorithm is provided. It is shown that an arbitrary RNN can be obtained via the referent RNN, with some deterministic rules imposed on its weights and the learning rate. Such relationships reduce the number of degrees of freedom when solving the nonlinear optimization task of finding the optimal RNN parameters.

scholarly journals Nonlinear All-Optical Diffractive Deep Neural Network with 10.6 μm Wavelength for Image Classification

2021 ◽  
Vol 2021 ◽  
pp. 1-16
Author(s):  
Yichen Sun ◽  
Mingli Dong ◽  
Mingxin Yu ◽  
Jiabin Xia ◽  
Xu Zhang ◽  
...  
Keyword(s):  
Neural Network ◽  
Artificial Intelligence ◽  
Image Classification ◽  
Deep Neural Network ◽  
Activation Function ◽  
Optical Nonlinearity ◽  
Optical Diffraction ◽  
Model Based ◽  
All Optical ◽  
Optical Neural Network

A photonic artificial intelligence chip is based on an optical neural network (ONN), low power consumption, low delay, and strong antiinterference ability. The all-optical diffractive deep neural network has recently demonstrated its inference capabilities on the image classification task. However, the size of the physical model does not have miniaturization and integration, and the optical nonlinearity is not incorporated into the diffraction neural network. By introducing the nonlinear characteristics of the network, complex tasks can be completed with high accuracy. In this study, a nonlinear all-optical diffraction deep neural network (N-D2NN) model based on 10.6 μm wavelength is constructed by combining the ONN and complex-valued neural networks with the nonlinear activation function introduced into the structure. To be specific, the improved activation function of the rectified linear unit (ReLU), i.e., Leaky-ReLU, parametric ReLU (PReLU), and randomized ReLU (RReLU), is selected as the activation function of the N-D2NN model. Through numerical simulation, it is proved that the N-D2NN model based on 10.6 μm wavelength has excellent representation ability, which enables them to perform classification learning tasks of the MNIST handwritten digital dataset and Fashion-MNIST dataset well, respectively. The results show that the N-D2NN model with the RReLU activation function has the highest classification accuracy of 97.86% and 89.28%, respectively. These results provide a theoretical basis for the preparation of miniaturized and integrated N-D2NN model photonic artificial intelligence chips.

scholarly journals Two novel finite time convergent recurrent neural networks for tackling complex-valued systems of linear equation

Filomat ◽  
2020 ◽  
Vol 34 (15) ◽  
pp. 5009-5018
Author(s):  
Lei Ding ◽  
Lin Xiao ◽  
Kaiqing Zhou ◽  
Yonghong Lan ◽  
Yongsheng Zhang
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Linear Equation ◽  
Recurrent Neural Networks ◽  
Activation Function ◽  
Convergence Speed ◽  
The Other ◽  
Activation Functions ◽  
Complex Valued ◽  
Nonlinear Activation Function

Compared to the linear activation function, a suitable nonlinear activation function can accelerate the convergence speed. Based on this finding, we propose two modified Zhang neural network (ZNN) models using different nonlinear activation functions to tackle the complex-valued systems of linear equation (CVSLE) problems in this paper. To fulfill this goal, we first propose a novel neural network called NRNN-SBP model by introducing the sign-bi-power activation function. Then, we propose another novel neural network called NRNN-IRN model by introducing the tunable activation function. Finally, simulative results demonstrate that the convergence speed of NRNN-SBP and the NRNN-IRN is faster than that of the FTRNN model. On the other hand, these results also reveal that different nonlinear activation function will have a different effect on the convergence rate for different CVSLE problems.

SCORING MODELING BASED ON NEURAL NETWORKS FOR DETERMINING A BANK BORROWER'S RATING

2020 ◽  
Vol 2020 (10) ◽  
pp. 54-62
Author(s):  
Oleksii VASYLIEV ◽  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Network Architecture ◽  
Statistical Data ◽  
Activation Function ◽  
Decision Making Process ◽  
Neural Network Architecture ◽  
Acceptable Accuracy ◽  
The Neural Network ◽  
Sigmoid Activation Function

The problem of applying neural networks to calculate ratings used in banking in the decision-making process on granting or not granting loans to borrowers is considered. The task is to determine the rating function of the borrower based on a set of statistical data on the effectiveness of loans provided by the bank. When constructing a regression model to calculate the rating function, it is necessary to know its general form. If so, the task is to calculate the parameters that are included in the expression for the rating function. In contrast to this approach, in the case of using neural networks, there is no need to specify the general form for the rating function. Instead, certain neural network architecture is chosen and parameters are calculated for it on the basis of statistical data. Importantly, the same neural network architecture can be used to process different sets of statistical data. The disadvantages of using neural networks include the need to calculate a large number of parameters. There is also no universal algorithm that would determine the optimal neural network architecture. As an example of the use of neural networks to determine the borrower's rating, a model system is considered, in which the borrower's rating is determined by a known non-analytical rating function. A neural network with two inner layers, which contain, respectively, three and two neurons and have a sigmoid activation function, is used for modeling. It is shown that the use of the neural network allows restoring the borrower's rating function with quite acceptable accuracy.

scholarly journals Prediction of Stress in Power Transformer Winding Conductors Using Artificial Neural Networks: Hyperparameter Analysis

Energies ◽  
2021 ◽  
Vol 14 (14) ◽  
pp. 4242
Author(s):  
Fausto Valencia ◽  
Hugo Arcos ◽  
Franklin Quilumba
Keyword(s):  
Neural Network ◽  
Artificial Neural Network ◽  
Finite Element ◽  
Power Transformer ◽  
Network Models ◽  
Activation Function ◽  
Neural Network Models ◽  
Transformer Winding ◽  
Artificial Neural ◽  
Femm Software

The purpose of this research is the evaluation of artificial neural network models in the prediction of stresses in a 400 MVA power transformer winding conductor caused by the circulation of fault currents. The models were compared considering the training, validation, and test data errors’ behavior. Different combinations of hyperparameters were analyzed based on the variation of architectures, optimizers, and activation functions. The data for the process was created from finite element simulations performed in the FEMM software. The design of the Artificial Neural Network was performed using the Keras framework. As a result, a model with one hidden layer was the best suited architecture for the problem at hand, with the optimizer Adam and the activation function ReLU. The final Artificial Neural Network model predictions were compared with the Finite Element Method results, showing good agreement but with a much shorter solution time.

scholarly journals Freely scalable and reconfigurable optical hardware for deep learning

2021 ◽  
Vol 11 (1) ◽  
Author(s):  
Liane Bernstein ◽  
Alexander Sludds ◽  
Ryan Hamerly ◽  
Vivienne Sze ◽  
Joel Emer ◽  
...  
Keyword(s):  
Neural Network ◽  
Energy Consumption ◽  
Data Transfer ◽  
Power Delivery ◽  
Optical Data ◽  
Optical Multicast ◽  
Optical Neural Network ◽  
Fully Connected ◽  
Management Power

AbstractAs deep neural network (DNN) models grow ever-larger, they can achieve higher accuracy and solve more complex problems. This trend has been enabled by an increase in available compute power; however, efforts to continue to scale electronic processors are impeded by the costs of communication, thermal management, power delivery and clocking. To improve scalability, we propose a digital optical neural network (DONN) with intralayer optical interconnects and reconfigurable input values. The path-length-independence of optical energy consumption enables information locality between a transmitter and a large number of arbitrarily arranged receivers, which allows greater flexibility in architecture design to circumvent scaling limitations. In a proof-of-concept experiment, we demonstrate optical multicast in the classification of 500 MNIST images with a 3-layer, fully-connected network. We also analyze the energy consumption of the DONN and find that digital optical data transfer is beneficial over electronics when the spacing of computational units is on the order of $$>10\,\upmu $$ > 10 μ m.

Sign in / Sign up

Export Citation Format

Share Document