scholarly journals Nonlinear reconfiguration of network edges, topology and information content during an artificial learning task

2021 ◽  
Vol 8 (1) ◽  
Author(s):  
James M. Shine ◽  
Mike Li ◽  
Oluwasanmi Koyejo ◽  
Ben Fulcher ◽  
Joseph T. Lizier
Keyword(s):  
Neural Network ◽  
Information Content ◽  
Network Structure ◽  
Activity Patterns ◽  
Learning Task ◽  
Edge Weight ◽  
Information Theoretic ◽  
Systems Neuroscience ◽  
Multi Stage ◽  
Artificial Neural

AbstractHere, we combine network neuroscience and machine learning to reveal connections between the brain’s network structure and the emerging network structure of an artificial neural network. Specifically, we train a shallow, feedforward neural network to classify hand-written digits and then used a combination of systems neuroscience and information-theoretic tools to perform ‘virtual brain analytics’ on the resultant edge weights and activity patterns of each node. We identify three distinct phases of network reconfiguration across learning, each of which are characterized by unique topological and information-theoretic signatures. Each phase involves aligning the connections of the neural network with patterns of information contained in the input dataset or preceding layers (as relevant). We also observe a process of low-dimensional category separation in the network as a function of learning. Our results offer a systems-level perspective of how artificial neural networks function—in terms of multi-stage reorganization of edge weights and activity patterns to effectively exploit the information content of input data during edge-weight training—while simultaneously enriching our understanding of the methods used by systems neuroscience.

scholarly journals Topological Augmentation of Latent Information Streams in Feed-Forward Neural Networks

2020 ◽  
Author(s):  
James M. Shine ◽  
Mike Li ◽  
Oluwasanmi Koyejo ◽  
Ben Fulcher ◽  
Joseph T. Lizier
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Neural Networks ◽  
Activity Patterns ◽  
Edge Weight ◽  
Feed Forward ◽  
Information Theoretic ◽  
Systems Neuroscience ◽  
Feed Forward Network ◽  
Low Dimensional

AbstractThe algorithmic rules that define deep neural networks are clearly defined, however the principles that define their performance remain poorly understood. Here, we use systems neuroscience and information theoretic approaches to analyse a feedforward neural network as it is trained to classify handwritten digits. By tracking the topology of the network as it learns, we identify three distinct phases of topological reconfiguration. Each phase brings the connections of the neural network into alignment with patterns of information contained in the input dataset, as well as the preceding layers. Performing dimensionality reduction on the data reveals a process of low-dimensional category separation as a function of learning. Our results enable a systems-level understanding of how deep neural networks function, and provide evidence of how neural networks reorganize edge weights and activity patterns so as to most effectively exploit the information theoretic content of input data during edge-weight training.SummaryTrained neural networks are capable of remarkable performance on complex categorization tasks, however the precise rules according to which the network reconfigures during training remain poorly understood. We used a combination of systems neuroscience and information theoretic analyses to interrogate the network topology of a simple, feed-forward network as it was trained on a digitclassification task. Over the course of training, the hidden layers of the network reconfigured in characteristic ways that were reminiscent of key results in network neuroscience studies of human brain imaging. In addition, we observed a strong correspondence between the topological changes at different learning phases and information theoretic signatures of the data that were entered into the network. In this way, we show how neural networks learn.

PRUNING ARTIFICIAL NEURAL NETWORKS USING NEURAL COMPLEXITY MEASURES

2008 ◽  
Vol 18 (05) ◽  
pp. 389-403 ◽  
Author(s):  
THOMAS D. JORGENSEN ◽  
BARRY P. HAYNES ◽  
CHARLOTTE C. F. NORLUND
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Artificial Neural Networks ◽  
Robot Control ◽  
Complexity Measures ◽  
Information Theoretic ◽  
Pruning Method ◽  
The Neural Network ◽  
Artificial Neural ◽  
Pruning Technique

This paper describes a new method for pruning artificial neural networks, using a measure of the neural complexity of the neural network. This measure is used to determine the connections that should be pruned. The measure computes the information-theoretic complexity of a neural network, which is similar to, yet different from previous research on pruning. The method proposed here shows how overly large and complex networks can be reduced in size, whilst retaining learnt behaviour and fitness. The technique proposed here helps to discover a network topology that matches the complexity of the problem it is meant to solve. This novel pruning technique is tested in a robot control domain, simulating a racecar. It is shown, that the proposed pruning method is a significant improvement over the most commonly used pruning method Magnitude Based Pruning. Furthermore, some of the pruned networks prove to be faster learners than the benchmark network that they originate from. This means that this pruning method can also help to unleash hidden potential in a network, because the learning time decreases substantially for a pruned a network, due to the reduction of dimensionality of the network.

scholarly journals A multi-stage methodology for selecting input variables in ANN forecasting of river flows

2017 ◽  
Vol 19 (1) ◽  
pp. 49-57 ◽  
Keyword(s):  
Neural Network ◽  
Artificial Neural Network ◽  
Atmospheric Circulation ◽  
River Flow ◽  
Neural Network Models ◽  
Prediction Ability ◽  
River Flows ◽  
Multi Stage ◽  
Artificial Neural ◽  
Input Variables

<p>The scientific community has recognized the necessity for more efficiently selected inputs in artificial neural network models (ANNs) in river flows and has worked on this despite some shortcomings. Moreover, there is none or limited inclusion of ANN inputs coupled with atmospheric circulation under various patterns arising from the need of data downscaling for climate change predictions in hydrology domain. This paper presents the results of a novel multi-stage methodology for selecting input variables used in artificial neural network (ANN) models for river flow forecasting. The proposed methodology makes use of data correlations together with a set of crucial statistical indices for optimizing model performance, both in terms of ANN structure (e.g. neurons, momentum rate, learning rate, activation functions, etc), but also in terms of inputs selection. The latter include various previous time steps of daily areal precipitation and temperature data coupled with atmospheric circulation in the form of circulation patterns, observed river flow data and time expressed via functions of sine and cosine. Additionally, the no-linear behavior between river flow and the respective inputs is investigated by the ANN configuration itself and not only by correlation indices (or other equivalent contingency tools). The proposed methodology revealed the river flow of past four days, the precipitation of past three days and the seasonality as robust input variables. However, the temperature of three past days should be considered as an alternative against the seasonality. The produced models forecasting ability was validated by comparing its one-step ahead flow prediction ability to two other approaches (an auto regressive model and a genetic algorithm (GA)-optimized single input ANN).&nbsp;</p>

Application of Artificial Neural Networks for Diagnosing Acute Appendicitis

2013 ◽  
Vol 479-480 ◽  
pp. 445-450
Author(s):  
Sung Yun Park ◽  
Sangjoon Lee ◽  
Jae Hoon Jeong ◽  
Sung Min Kim
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Artificial Neural Networks ◽  
Acute Appendicitis ◽  
Network Structure ◽  
Scoring System ◽  
Probabilistic Neural Network ◽  
Clinical Scoring ◽  
Neural Network Structure ◽  
Artificial Neural

The purpose of this study is to develop an appendicitis diagnosis system, by using artificial neural networks (ANNs). Acute appendicitis is one of the most common surgical emergencies of the abdomen. Various methods have been developed to diagnose appendicitis, but these methods have not shown good performance in the Middle East and Asia, or even in the West. We used the structures of ANNs with 801 patients. These various structures are a multilayer neural network structure (MLNN), a radial basis function neural network structure (RBF), and a probabilistic neural network structure (PNN). The Alvarado clinical scoring system was used for comparison with the ANNs. The accuracy of MLNN, RBF, PNN, and Alvarado was 97.84%, 99.80%, 99.41% and 72.19%, respectively. The AUC of MLNN, RBF, PNN, and Alvarado was 0.985, 0.998, 0.993, and 0.633, respectively. The performance of ANNs was significantly better than the Alvarado clinical scoring system (P<0.001). The models developed to diagnose appendicitis using ANNs showed good performance. We consider that the developed models can help junior clinical surgeons diagnose appendicitis.

Sign in / Sign up

Export Citation Format

Share Document