A Volatility Measure for Annealing in Feedback Neural Networks

1992 ◽  
Vol 4 (2) ◽  
pp. 191-195 ◽  
Author(s):  
Joshua Alspector ◽  
Torsten Zeppenfeld ◽  
Stephan Luna
Keyword(s):  
Neural Networks ◽  
Specific Heat ◽  
Reliable Method ◽  
Mean Field ◽  
Annealing Process ◽  
Pattern Learning ◽  
Annealing Schedule ◽  
Speed Up ◽  
Static Pattern ◽  
Feedback Neural Networks

In feedback neural networks, especially for static pattern learning, a reliable method of settling is required. Simulated annealing has been used but it is often difficult to determine how to set the annealing schedule. Often the specific heat is used as a measure of when to slow down the annealing process, but this is difficult to measure. We propose another measure, volatility, which is easy to measure and related to the Edwards-Anderson model in spin-glass physics. This paper presents the concept of volatility, an argument for its similarity to specific heat, simulations of dynamics in Boltzmann and mean-field networks, and a method of using it to speed up learning.

scholarly journals Stochastic mean-field formulation of the dynamics of diluted neural networks

2015 ◽  
Vol 16 (S1) ◽  
Author(s):  
David Angulo-Garcia ◽  
Alessandro Torcini
Keyword(s):  
Neural Networks ◽  
Mean Field

Analyzing and Accelerating the Bottlenecks of Training Deep SNNs With Backpropagation

2020 ◽  
Vol 32 (12) ◽  
pp. 2557-2600
Author(s):  
Ruizhi Chen ◽  
Ling Li
Keyword(s):  
Neural Networks ◽  
Training Algorithms ◽  
Minimization Method ◽  
Data Set ◽  
Ultra Low Power ◽  
Training Performance ◽  
Speed Up ◽  
Pros And Cons ◽  
Accuracy Increase ◽  
Initialization Algorithm

Spiking neural networks (SNNs) with the event-driven manner of transmitting spikes consume ultra-low power on neuromorphic chips. However, training deep SNNs is still challenging compared to convolutional neural networks (CNNs). The SNN training algorithms have not achieved the same performance as CNNs. In this letter, we aim to understand the intrinsic limitations of SNN training to design better algorithms. First, the pros and cons of typical SNN training algorithms are analyzed. Then it is found that the spatiotemporal backpropagation algorithm (STBP) has potential in training deep SNNs due to its simplicity and fast convergence. Later, the main bottlenecks of the STBP algorithm are analyzed, and three conditions for training deep SNNs with the STBP algorithm are derived. By analyzing the connection between CNNs and SNNs, we propose a weight initialization algorithm to satisfy the three conditions. Moreover, we propose an error minimization method and a modified loss function to further improve the training performance. Experimental results show that the proposed method achieves 91.53% accuracy on the CIFAR10 data set with 1% accuracy increase over the STBP algorithm and decreases the training epochs on the MNIST data set to 15 epochs (over 13 times speed-up compared to the STBP algorithm). The proposed method also decreases classification latency by over 25 times compared to the CNN-SNN conversion algorithms. In addition, the proposed method works robustly for very deep SNNs, while the STBP algorithm fails in a 19-layer SNN.

scholarly journals Deep Learning Strategies for ProtoDUNE Raw Data Denoising

2022 ◽  
Vol 6 (1) ◽  
Author(s):  
Marco Rossi ◽  
Sofia Vallecorsa
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Learning Strategies ◽  
Simulation Data ◽  
Raw Data ◽  
Digital Detector ◽  
Speed Up ◽  
Neural Network Hardware ◽  
Graph Neural Networks ◽  
High Level

AbstractIn this work, we investigate different machine learning-based strategies for denoising raw simulation data from the ProtoDUNE experiment. The ProtoDUNE detector is hosted by CERN and it aims to test and calibrate the technologies for DUNE, a forthcoming experiment in neutrino physics. The reconstruction workchain consists of converting digital detector signals into physical high-level quantities. We address the first step in reconstruction, namely raw data denoising, leveraging deep learning algorithms. We design two architectures based on graph neural networks, aiming to enhance the receptive field of basic convolutional neural networks. We benchmark this approach against traditional algorithms implemented by the DUNE collaboration. We test the capabilities of graph neural network hardware accelerator setups to speed up training and inference processes.

scholarly journals Parallelization of Neural Network Training for NLP with Hogwild!

2017 ◽  
Vol 109 (1) ◽  
pp. 29-38 ◽  
Author(s):  
Valentin Deyringer ◽  
Alexander Fraser ◽  
Helmut Schmid ◽  
Tsuyoshi Okita
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Suitable Method ◽  
Neural Network Training ◽  
Training Time ◽  
Pos Tagging ◽  
Network Training ◽  
Speed Up

Abstract Neural Networks are prevalent in todays NLP research. Despite their success for different tasks, training time is relatively long. We use Hogwild! to counteract this phenomenon and show that it is a suitable method to speed up training Neural Networks of different architectures and complexity. For POS tagging and translation we report considerable speedups of training, especially for the latter. We show that Hogwild! can be an important tool for training complex NLP architectures.

scholarly journals Neutral Stability, Rate Propagation, and Critical Branching in Feedforward Networks

2013 ◽  
Vol 25 (7) ◽  
pp. 1768-1806 ◽  
Author(s):  
N. Alex Cayco-Gajic ◽  
Eric Shea-Brown
Keyword(s):  
Neural Networks ◽  
Markov Chain ◽  
Mean Field ◽  
Neutral Stability ◽  
Feedforward Networks ◽  
Firing Rates ◽  
Wide Range ◽  
Complex Models ◽  
Dynamical Properties ◽  
Parameter Ranges

Recent experimental and computational evidence suggests that several dynamical properties may characterize the operating point of functioning neural networks: critical branching, neutral stability, and production of a wide range of firing patterns. We seek the simplest setting in which these properties emerge, clarifying their origin and relationship in random, feedforward networks of McCullochs-Pitts neurons. Two key parameters are the thresholds at which neurons fire spikes and the overall level of feedforward connectivity. When neurons have low thresholds, we show that there is always a connectivity for which the properties in question all occur, that is, these networks preserve overall firing rates from layer to layer and produce broad distributions of activity in each layer. This fails to occur, however, when neurons have high thresholds. A key tool in explaining this difference is the eigenstructure of the resulting mean-field Markov chain, as this reveals which activity modes will be preserved from layer to layer. We extend our analysis from purely excitatory networks to more complex models that include inhibition and local noise, and find that both of these features extend the parameter ranges over which networks produce the properties of interest.

Sign in / Sign up

Export Citation Format

Share Document