Architectural Bias in Recurrent Neural Networks: Fractal Analysis

2003 ◽  
Vol 15 (8) ◽  
pp. 1931-1957 ◽  
Author(s):  
Peter Tiňo ◽  
Barbara Hammer
Keyword(s):  
Neural Networks ◽  
Fractal Analysis ◽  
Recurrent Neural Networks ◽  
Markov Models ◽  
Fractal Dimensions ◽  
Transition Diagram ◽  
Activation Patterns ◽  
Box Counting ◽  
Fractal Clusters ◽  
Finite State

We have recently shown that when initialized with “small” weights, recurrent neural networks (RNNs) with standard sigmoid-type activation functions are inherently biased toward Markov models; even prior to any training, RNN dynamics can be readily used to extract finite memory machines (Hammer & Tiňo, 2002; Tiňo, Čerňanský, &Beňušková, 2002a, 2002b). Following Christiansen and Chater (1999), we refer to this phenomenon as the architectural bias of RNNs. In this article, we extend our work on the architectural bias in RNNs by performing a rigorous fractal analysis of recurrent activation patterns. We assume the network is driven by sequences obtained by traversing an underlying finite-state transition diagram&a scenario that has been frequently considered in the past, for example, when studying RNN-based learning and implementation of regular grammars and finite-state transducers. We obtain lower and upper bounds on various types of fractal dimensions, such as box counting and Hausdorff dimensions. It turns out that not only can the recurrent activations inside RNNs with small initial weights be explored to build Markovian predictive models, but also the activations form fractal clusters, the dimension of which can be bounded by the scaled entropy of the underlying driving source. The scaling factors are fixed and are given by the RNN parameters.

EXPERIMENTAL COMPARISON OF THE EFFECT OF ORDER IN RECURRENT NEURAL NETWORKS

1993 ◽  
Vol 07 (04) ◽  
pp. 849-872 ◽  
Author(s):  
CLIFFORD B. MILLER ◽  
C. LEE GILES
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Internal State ◽  
Second Order ◽  
Convergence Time ◽  
Experimental Comparison ◽  
Grammatical Inference ◽  
Neural Net ◽  
First Order ◽  
Finite State

There has been much interest in increasing the computational power of neural networks. In addition there has been much interest in “designing” neural networks better suited to particular problems. Increasing the “order” of the connectivity of a neural network permits both. Though order has played a significant role in feedforward neural networks, its role in dynamically driven recurrent networks is still being understood. This work explores the effect of order in learning grammars. We present an experimental comparison of first order and second order recurrent neural networks, as applied to the task of grammatical inference. We show that for the small grammars studied these two neural net architectures have comparable learning and generalization power, and that both are reasonably capable of extracting the correct finite state automata for the language in question. However, for a larger randomly-generated ten-state grammar, second order networks significantly outperformed the first order networks, both in convergence time and generalization capability. We show that these networks learn faster the more neurons they have (our experiments used up to 10 hidden neurons), but that the solutions found by smaller networks are usually of better quality (in terms of generalization performance after training). Second order nets have the advantage that they converge more quickly to a solution and can find it more reliably than first order nets, but that the second order solutions tend to be of poorer quality than those of the first order if both architectures are trained to the same error tolerance. Despite this, second order nets can more successfully extract finite state machines using heuristic clustering techniques applied to the internal state representations. We speculate that this may be due to restrictions on the ability of first order architecture to fully make use of its internal state representation power and that this may have implications for the performance of the two architectures when scaled up to larger problems.

scholarly journals FRACTAL ANALYSIS OF TRABECULAR BONE: A STANDARDISED METHODOLOGY

2011 ◽  
Vol 19 (1) ◽  
pp. 45 ◽  
Author(s):  
Ian Parkinson ◽  
Nick Fazzalari
Keyword(s):  
Fractal Dimension ◽  
Trabecular Bone ◽  
Fractal Analysis ◽  
Fractal Dimensions ◽  
Box Counting ◽  
Cell Tissue ◽  
Image Analyser ◽  
Model Independent ◽  
Box Counting Method ◽  
Image Orientation

A standardised methodology for the fractal analysis of histological sections of trabecular bone has been established. A modified box counting method has been developed for use on a PC based image analyser (Quantimet 500MC, Leica Cambridge). The effect of image analyser settings, magnification, image orientation and threshold levels, was determined. Also, the range of scale over which trabecular bone is effectively fractal was determined and a method formulated to objectively calculate more than one fractal dimension from the modified Richardson plot. The results show that magnification, image orientation and threshold settings have little effect on the estimate of fractal dimension. Trabecular bone has a lower limit below which it is not fractal (λ<25 μm) and the upper limit is 4250 μm. There are three distinct fractal dimensions for trabecular bone (sectional fractals), with magnitudes greater than 1.0 and less than 2.0. It has been shown that trabecular bone is effectively fractal over a defined range of scale. Also, within this range, there is more than 1 fractal dimension, describing spatial structural entities. Fractal analysis is a model independent method for describing a complex multifaceted structure, which can be adapted for the study of other biological systems. This may be at the cell, tissue or organ level and compliments conventional histomorphometric and stereological techniques.

FRACTAL ANALYSIS OF DENDRITIC ARBORISATION PATTERNS OF STALKED AND ISLET NEURONS IN SUBSTANTIA GELATINOSA OF DIFFERENT SPECIES

Fractals ◽  
2007 ◽  
Vol 15 (01) ◽  
pp. 1-7 ◽  
Author(s):  
NEBOJŠA T. MILOŠEVIĆ ◽  
DUŠAN RISTANOVIĆ ◽  
JOVAN B. STANKOVIĆ ◽  
RADMILA GUDOVIĆ
Keyword(s):  
Fractal Dimension ◽  
Fractal Analysis ◽  
Islet Cells ◽  
Fractal Dimensions ◽  
Substantia Gelatinosa ◽  
Neuronal Populations ◽  
Box Counting ◽  
Morphological Descriptor ◽  
The Mean ◽  
Box Counting Method

Through analysis of the morphology of dendritic arborisation of neurons from the substantia gelatinosa of dorsal horns from four different species, we have established that two types of cells (stalked and islet) are always present. The aim of the study was to perform the intra- and/or inter-species comparison of these two neuronal populations by fractal analysis, as well as to clarify the importance of the fractal dimension as an objective and usable morphological parameter. Fractal analysis was carried out adopting the box-counting method. We have shown that the mean fractal dimensions for the stalked cells are significantly different between species. The same is true for the mean fractal dimensions of the islet cells. Still, no significant differences were found for the fractal dimensions of the stalked and islet cells within a particular species. The human species has shown as the only exception where fractal dimensions of these two types of cells differ significantly. This study shows once more that the fractal dimension is a useful and sensitive morphological descriptor of neuronal structures and differences between them.

Rule Extraction from Recurrent Neural Networks: ATaxonomy and Review

2005 ◽  
Vol 17 (6) ◽  
pp. 1223-1263 ◽  
Author(s):  
Henrik Jacobsson
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Ad Hoc ◽  
Rule Extraction ◽  
Performance Evaluations ◽  
Research Issues ◽  
Open Research ◽  
Finite State ◽  
Push Forward ◽  
Quantitative Performance

Rule extraction (RE) from recurrent neural networks (RNNs) refers to finding models of the underlying RNN, typically in the form of finite state machines, that mimic the network to a satisfactory degree while having the advantage of being more transparent. RE from RNNs can be argued to allow a deeper and more profound form of analysis of RNNs than other, more or less ad hoc methods. RE may give us understanding of RNNs in the intermediate levels between quite abstract theoretical knowledge of RNNs as a class of computing devices and quantitative performance evaluations of RNN instantiations. The development of techniques for extraction of rules from RNNs has been an active field since the early 1990s. This article reviews the progress of this development and analyzes it in detail. In order to structure the survey and evaluate the techniques, a taxonomy specifically designed for this purpose has been developed. Moreover, important open research issues are identified that, if addressed properly, possibly can give the field a significant push forward.

Sign in / Sign up

Export Citation Format

Share Document