A weight initialization based on the linear product structure for neural networks

Network science can offer fundamental insights into the structural and functional properties of complex systems. For example, it is widely known that neuronal circuits tend to organize into basic functional topological modules, called network motifs. In this article, we show that network science tools can be successfully applied also to the study of artificial neural networks operating according to self-organizing (learning) principles. In particular, we study the emergence of network motifs in multi-layer perceptrons, whose initial connectivity is defined as a stack of fully-connected, bipartite graphs. Simulations show that the final network topology is shaped by learning dynamics, but can be strongly biased by choosing appropriate weight initialization schemes. Overall, our results suggest that non-trivial initialization strategies can make learning more effective by promoting the development of useful network motifs, which are often surprisingly consistent with those observed in general transduction networks.

Download Full-text

An inhibitory weight initialization improves the speed and quality of recurrent neural networks learning

Neurocomputing ◽

10.1016/s0925-2312(97)00032-5 ◽

1997 ◽

Vol 16 (3) ◽

pp. 207-224 ◽

Cited By ~ 3

Author(s):

J.P. Draye ◽

D. Pavisic ◽

G. Cheron ◽

G. Libert

Keyword(s):

Neural Networks ◽

Recurrent Neural Networks ◽

Weight Initialization

Download Full-text

Thesis Overview Invariance and Same-Equivariance Measures for Convolutional Neural Networks

Journal of Computer Science and Technology ◽

10.24215/16666038.20.e06 ◽

2020 ◽

Vol 20 (1) ◽

pp. e06

Author(s):

Facundo Quiroga ◽

Laura Lanzarini

Keyword(s):

Neural Network ◽

Neural Networks ◽

Comparative Analysis ◽

Sign Language ◽

Data Augmentation ◽

Size Analysis ◽

Kernel Size ◽

Weight Initialization ◽

Latent Representations ◽

The Impact

The main contributions of this thesis include: A comparative analysis of Neural Network based models for sign language handshape classification. An analysis of strategies to achieve equivariance to rotations in neural networks for: Comparing the performance of strategies based on data augmentation and specially designed networks and layers. Determining strategies to retrain networks so that they acquire equivariance to rotations. A set of measures to empirically analyze the equivariance of Neural Networks, as well as any other model based on latent representations, and the corresponding: Validation of the measures to establish if they are indeed measuring the purported quantity. Analysis of the different variants of the proposed measures. Analysis of the properties of the measures, in terms of their variability to transformations, models and weight initialization. Analysis of the impact of several hyperparameters of the models on the structure of their equivariance, including Max Pooling layers, Batch Normalization, and kernel size. Analysis of the structure of the equivariance in several well known CNN models such as ResNet, All Convolutional and VGG. Analysis of the impact on the equivariance of using specialized models to obtain equivariance such as Transformational Invariance Pooling. Analysis of the class dependency of equivariance. Analysis of the effect of varying the complexity and diversity of the transformations on the measures.

Download Full-text

Implementasi Metode Backpropogation dengan Inisialisasi Bobot Nguyen Widrow untuk Peramalan Harga Saham

Jurnal Teknologi Informasi dan Ilmu Komputer ◽

10.25126/jtiik.201961904 ◽

2019 ◽

Vol 6 (1) ◽

pp. 49

Author(s):

Eliv Kurniawan ◽

Hari Wibawanto ◽

Djoko Adi Widodo

Keyword(s):

Neural Network ◽

Neural Networks ◽

Artificial Neural Network ◽

Artificial Neural Networks ◽

Stock Prices ◽

Stock Price ◽

Daily Data ◽

Backpropagation Method ◽

Weight Initialization ◽

Artificial Neural

Jaringan saraf tiruan merupakan suatu ilmu yang terus berkembang pesat hingga saat ini. Jaringan saraf tiruan merupakan suatu ilmu komputasi yang didasarkan dan terinspirasi dari cara kerja sistem saraf manusia. Sama halnya dengan sistem saraf manusia, jaringan saraf tiruan bekerja melalui proses pembelajaran terhadap data-data yang sudah ada untuk memformulakan keluaran dari data-data baru. Jaringan saraf tiruan dengan metode backpropagation mampu melakukan peramalan untuk data nonlinear seperti bentuk data harian harga saham. Salah satu algoritma inisialisasi bobot yang dapat meningkatkan waktu eksekusi adalah nguyen-widrow. Pada penelitian ini akan dilakukan implementasi metode backpropagation dengan inisialisasi bobot nguyen widrow untuk meramalkan harga saham. Proses implementasi melalui 3 tahapan, yaitu preprosesing data, pelatihan jaringan, dan pengujian jaringan. Hasil dari penelitian ini menunjukkan bahwa pelatihan jaringan saraf tiruan dengan jumlah dataset yang banyak membutuhkan perhitungan yang kompleks, sehingga jaringan saraf tiruan dengan arsitektur jaringan yang sederhana kurang efektif dan dapat terjebak pada titik lokal minimum. Hasil peramalan untuk harga close saham BBCA.JK memiliki nilai MAPE 0,85% dan untuk harga close saham AALI.JK memiliki nilai MAPE sebesar 1,84%.AbstractArtificial neural network is a hot topic and invite a lot of admiration in the last decade. Artificial Neural Network is one of the artificial representations of the humans brain who always try to simulate the learning process of the humans brain. Artificial neural network with backpropagation method is able to forecast nonlinear data such as daily data form stock price. One of the weight initialization algorithms that can be increase the execution time is nguyen-widrow. In this research will be implemented backpropagation method with nguyen widrow weight initialization to forecast stock prices. The process of implementation through 3 stages, that is preprosesing data, training, and testing or simulate. The results of this research indicate that the training of artificial neural networks with many datasets required a complex calculations, so the artificial neural network with simple architectures is less effective and can get stuck at minimum local points. The results forecasting for the close price of BBCA.JK have a MAPE value 0.85% and for the close price of AALI.JK have 1.84% of MAPE value

Download Full-text