Error Resilient Neural Networks on Low-Dimensional Manifolds

2018 ◽  
pp. 49-59
Author(s):  
Alexander Petukhov ◽  
Inna Kozlov
Keyword(s):  
Neural Networks ◽  
Error Resilient ◽  
Low Dimensional

CHAOS AND HYPERCHAOS IN A CLASS OF SIMPLE CELLULAR NEURAL NETWORKS MODELED BY O.D.E.

2006 ◽  
Vol 16 (09) ◽  
pp. 2729-2736 ◽  
Author(s):  
XIAO-SONG YANG ◽  
YAN HUANG
Keyword(s):  
Neural Networks ◽  
Differential Equations ◽  
Ordinary Differential Equations ◽  
Lyapunov Exponents ◽  
Cellular Neural Networks ◽  
New Class ◽  
Connection Matrices ◽  
Low Dimensional

This paper presents a new class of chaotic and hyperchaotic low dimensional cellular neural networks modeled by ordinary differential equations with some simple connection matrices. The chaoticity of these neural networks is indicated by positive Lyapunov exponents calculated by a computer.

scholarly journals Generalization-Based Acquisition of Training Data for Motor Primitive Learning by Neural Networks

2021 ◽  
Vol 11 (3) ◽  
pp. 1013
Author(s):  
Zvezdan Lončarević ◽  
Rok Pahič ◽  
Aleš Ude ◽  
Andrej Gams
Keyword(s):  
Neural Networks ◽  
Dimensionality Reduction ◽  
Gaussian Process Regression ◽  
Search Space ◽  
Robot Learning ◽  
Training Data ◽  
Practical Applications ◽  
Latent Space ◽  
Real Robot ◽  
Low Dimensional

Autonomous robot learning in unstructured environments often faces the problem that the dimensionality of the search space is too large for practical applications. Dimensionality reduction techniques have been developed to address this problem and describe motor skills in low-dimensional latent spaces. Most of these techniques require the availability of a sufficiently large database of example task executions to compute the latent space. However, the generation of many example task executions on a real robot is tedious, and prone to errors and equipment failures. The main result of this paper is a new approach for efficient database gathering by performing a small number of task executions with a real robot and applying statistical generalization, e.g., Gaussian process regression, to generate more data. We have shown in our experiments that the data generated this way can be used for dimensionality reduction with autoencoder neural networks. The resulting latent spaces can be exploited to implement robot learning more efficiently. The proposed approach has been evaluated on the problem of robotic throwing at a target. Simulation and real-world results with a humanoid robot TALOS are provided. They confirm the effectiveness of generalization-based database acquisition and the efficiency of learning in a low-dimensional latent space.

scholarly journals Exploiting node metadata to predict interactions in large networks using graph embedding and neural networks

2021 ◽  
Author(s):  
Rogini Runghen ◽  
Daniel B Stouffer ◽  
Giulio Valentino Dalla Riva
Keyword(s):  
Machine Learning ◽  
Neural Networks ◽  
Link Prediction ◽  
Graph Embedding ◽  
Feature Space ◽  
Machine Learning Techniques ◽  
Large Networks ◽  
Data Set ◽  
Learning Techniques ◽  
Low Dimensional

Collecting network interaction data is difficult. Non-exhaustive sampling and complex hidden processes often result in an incomplete data set. Thus, identifying potentially present but unobserved interactions is crucial both in understanding the structure of large scale data, and in predicting how previously unseen elements will interact. Recent studies in network analysis have shown that accounting for metadata (such as node attributes) can improve both our understanding of how nodes interact with one another, and the accuracy of link prediction. However, the dimension of the object we need to learn to predict interactions in a network grows quickly with the number of nodes. Therefore, it becomes computationally and conceptually challenging for large networks. Here, we present a new predictive procedure combining a graph embedding method with machine learning techniques to predict interactions on the base of nodes' metadata. Graph embedding methods project the nodes of a network onto a---low dimensional---latent feature space. The position of the nodes in the latent feature space can then be used to predict interactions between nodes. Learning a mapping of the nodes' metadata to their position in a latent feature space corresponds to a classic---and low dimensional---machine learning problem. In our current study we used the Random Dot Product Graph model to estimate the embedding of an observed network, and we tested different neural networks architectures to predict the position of nodes in the latent feature space. Flexible machine learning techniques to map the nodes onto their latent positions allow to account for multivariate and possibly complex nodes' metadata. To illustrate the utility of the proposed procedure, we apply it to a large dataset of tourist visits to destinations across New Zealand. We found that our procedure accurately predicts interactions for both existing nodes and nodes newly added to the network, while being computationally feasible even for very large networks. Overall, our study highlights that by exploiting the properties of a well understood statistical model for complex networks and combining it with standard machine learning techniques, we can simplify the link prediction problem when incorporating multivariate node metadata. Our procedure can be immediately applied to different types of networks, and to a wide variety of data from different systems. As such, both from a network science and data science perspective, our work offers a flexible and generalisable procedure for link prediction.

scholarly journals The Goldilocks Zone: Towards Better Understanding of Neural Network Loss Landscapes

2019 ◽  
Vol 33 ◽  
pp. 3574-3581
Author(s):  
Stanislav Fort ◽  
Adam Scherlis
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Configuration Space ◽  
Loss Function ◽  
Positive Curvature ◽  
Local Convexity ◽  
Convolutional Networks ◽  
Hollow Spherical Shell ◽  
Low Dimensional ◽  
Fully Connected

We explore the loss landscape of fully-connected and convolutional neural networks using random, low-dimensional hyperplanes and hyperspheres. Evaluating the Hessian, H, of the loss function on these hypersurfaces, we observe 1) an unusual excess of the number of positive eigenvalues of H, and 2) a large value of Tr(H)/||H|| at a well defined range of configuration space radii, corresponding to a thick, hollow, spherical shell we refer to as the Goldilocks zone. We observe this effect for fully-connected neural networks over a range of network widths and depths on MNIST and CIFAR-10 datasets with the ReLU and tanh non-linearities, and a similar effect for convolutional networks. Using our observations, we demonstrate a close connection between the Goldilocks zone, measures of local convexity/prevalence of positive curvature, and the suitability of a network initialization. We show that the high and stable accuracy reached when optimizing on random, low-dimensional hypersurfaces is directly related to the overlap between the hypersurface and the Goldilocks zone, and as a corollary demonstrate that the notion of intrinsic dimension is initialization-dependent. We note that common initialization techniques initialize neural networks in this particular region of unusually high convexity/prevalence of positive curvature, and offer a geometric intuition for their success. Furthermore, we demonstrate that initializing a neural network at a number of points and selecting for high measures of local convexity such as Tr(H)/||H||, number of positive eigenvalues of H, or low initial loss, leads to statistically significantly faster training on MNIST. Based on our observations, we hypothesize that the Goldilocks zone contains an unusually high density of suitable initialization configurations.

Feature Extraction of Video Using Artificial Neural Network

2017 ◽  
Vol 11 (2) ◽  
pp. 25-40 ◽  
Author(s):  
Yoshihiro Hayakawa ◽  
Takanori Oonuma ◽  
Hideyuki Kobayashi ◽  
Akiko Takahashi ◽  
Shinji Chiba ◽  
...  
Keyword(s):  
Neural Networks ◽  
Feature Extraction ◽  
Image Recognition ◽  
High Performance ◽  
Feature Space ◽  
Middle Layer ◽  
Image Feature ◽  
Consecutive Series ◽  
Form Analysis ◽  
Low Dimensional

In deep neural networks, which have been gaining attention in recent years, the features of input images are expressed in a middle layer. Using the information on this feature layer, high performance can be demonstrated in the image recognition field. In the present study, we achieve image recognition, without using convolutional neural networks or sparse coding, through an image feature extraction function obtained when identity mapping learning is applied to sandglass-style feed-forward neural networks. In sports form analysis, for example, a state trajectory is mapped in a low-dimensional feature space based on a consecutive series of actions. Here, we discuss ideas related to image analysis by applying the above method.

scholarly journals Intrinsic motivation and episodic memories for robot exploration of high-dimensional sensory spaces

2020 ◽  
pp. 105971232092291
Author(s):  
Guido Schillaci ◽  
Antonio Pico Villalpando ◽  
Verena V Hafner ◽  
Peter Hanappe ◽  
David Colliaux ◽  
...  
Keyword(s):  
Neural Networks ◽  
Episodic Memory ◽  
Intrinsic Motivation ◽  
Computational Models ◽  
Deep Neural Networks ◽  
Image Sensor ◽  
Forward Kinematics ◽  
High Dimensional ◽  
Episodic Memories ◽  
Low Dimensional

This work presents an architecture that generates curiosity-driven goal-directed exploration behaviours for an image sensor of a microfarming robot. A combination of deep neural networks for offline unsupervised learning of low-dimensional features from images and of online learning of shallow neural networks representing the inverse and forward kinematics of the system have been used. The artificial curiosity system assigns interest values to a set of pre-defined goals and drives the exploration towards those that are expected to maximise the learning progress. We propose the integration of an episodic memory in intrinsic motivation systems to face catastrophic forgetting issues, typically experienced when performing online updates of artificial neural networks. Our results show that adopting an episodic memory system not only prevents the computational models from quickly forgetting knowledge that has been previously acquired but also provides new avenues for modulating the balance between plasticity and stability of the models.

A graph auto-encoder model for miRNA-disease associations prediction

2020 ◽  
Author(s):  
Zhengwei Li ◽  
Jiashu Li ◽  
Ru Nie ◽  
Zhu-Hong You ◽  
Wenzheng Bao
Keyword(s):  
Neural Networks ◽  
Clinical Medicine ◽  
Area Under The Curve ◽  
Heterogeneous Information ◽  
Source Codes ◽  
Differentially Expressed Mirnas ◽  
Disease Associations ◽  
Graph Neural Networks ◽  
New Biomarkers ◽  
Low Dimensional

Abstract Emerging evidence indicates that the abnormal expression of miRNAs involves in the evolution and progression of various human complex diseases. Identifying disease-related miRNAs as new biomarkers can promote the development of disease pathology and clinical medicine. However, designing biological experiments to validate disease-related miRNAs is usually time-consuming and expensive. Therefore, it is urgent to design effective computational methods for predicting potential miRNA-disease associations. Inspired by the great progress of graph neural networks in link prediction, we propose a novel graph auto-encoder model, named GAEMDA, to identify the potential miRNA-disease associations in an end-to-end manner. More specifically, the GAEMDA model applies a graph neural networks-based encoder, which contains aggregator function and multi-layer perceptron for aggregating nodes’ neighborhood information, to generate the low-dimensional embeddings of miRNA and disease nodes and realize the effective fusion of heterogeneous information. Then, the embeddings of miRNA and disease nodes are fed into a bilinear decoder to identify the potential links between miRNA and disease nodes. The experimental results indicate that GAEMDA achieves the average area under the curve of $93.56\pm 0.44\%$ under 5-fold cross-validation. Besides, we further carried out case studies on colon neoplasms, esophageal neoplasms and kidney neoplasms. As a result, 48 of the top 50 predicted miRNAs associated with these diseases are confirmed by the database of differentially expressed miRNAs in human cancers and microRNA deregulation in human disease database, respectively. The satisfactory prediction performance suggests that GAEMDA model could serve as a reliable tool to guide the following researches on the regulatory role of miRNAs. Besides, the source codes are available at https://github.com/chimianbuhetang/GAEMDA.

scholarly journals Convergence Behavior of DNNs with Mutual-Information-Based Regularization

Entropy ◽  
2020 ◽  
Vol 22 (7) ◽  
pp. 727 ◽  
Author(s):  
Hlynur Jónsson ◽  
Giovanni Cherubini ◽  
Evangelos Eleftheriou
Keyword(s):  
Neural Networks ◽  
Mutual Information ◽  
Low Complexity ◽  
High Dimensional ◽  
Test Accuracy ◽  
Compression Phase ◽  
Hidden Layer ◽  
Low Dimensional ◽  
Fully Connected ◽  
Fully Connected Networks

Information theory concepts are leveraged with the goal of better understanding and improving Deep Neural Networks (DNNs). The information plane of neural networks describes the behavior during training of the mutual information at various depths between input/output and hidden-layer variables. Previous analysis revealed that most of the training epochs are spent on compressing the input, in some networks where finiteness of the mutual information can be established. However, the estimation of mutual information is nontrivial for high-dimensional continuous random variables. Therefore, the computation of the mutual information for DNNs and its visualization on the information plane mostly focused on low-complexity fully connected networks. In fact, even the existence of the compression phase in complex DNNs has been questioned and viewed as an open problem. In this paper, we present the convergence of mutual information on the information plane for a high-dimensional VGG-16 Convolutional Neural Network (CNN) by resorting to Mutual Information Neural Estimation (MINE), thus confirming and extending the results obtained with low-dimensional fully connected networks. Furthermore, we demonstrate the benefits of regularizing a network, especially for a large number of training epochs, by adopting mutual information estimates as additional terms in the loss function characteristic of the network. Experimental results show that the regularization stabilizes the test accuracy and significantly reduces its variance.

scholarly journals DLDTI: a learning-based framework for drug-target interaction identification using neural networks and network representation

2020 ◽  
Vol 18 (1) ◽  
Author(s):  
Yihan Zhao ◽  
Kai Zheng ◽  
Baoyi Guan ◽  
Mengmeng Guo ◽  
Lei Song ◽  
...  
Keyword(s):  
Neural Networks ◽  
Molecular Mechanism ◽  
Drug Target ◽  
Drug Repositioning ◽  
Biological Data ◽  
Mapping Space ◽  
Target Interaction ◽  
Network Representation ◽  
Combination Methods ◽  
Low Dimensional

Abstract Background Drug repositioning, the strategy of unveiling novel targets of existing drugs could reduce costs and accelerate the pace of drug development. To elucidate the novel molecular mechanism of known drugs, considering the long time and high cost of experimental determination, the efficient and feasible computational methods to predict the potential associations between drugs and targets are of great aid. Methods A novel calculation model for drug-target interaction (DTI) prediction based on network representation learning and convolutional neural networks, called DLDTI, was generated. The proposed approach simultaneously fused the topology of complex networks and diverse information from heterogeneous data sources, and coped with the noisy, incomplete, and high-dimensional nature of large-scale biological data by learning the low-dimensional and rich depth features of drugs and proteins. The low-dimensional feature vectors were used to train DLDTI to obtain the optimal mapping space and to infer new DTIs by ranking candidates according to their proximity to the optimal mapping space. More specifically, based on the results from the DLDTI, we experimentally validated the predicted targets of tetramethylpyrazine (TMPZ) on atherosclerosis progression in vivo. Results The experimental results showed that the DLDTI model achieved promising performance under fivefold cross-validations with AUC values of 0.9172, which was higher than the methods using different classifiers or different feature combination methods mentioned in this paper. For the validation study of TMPZ on atherosclerosis, a total of 288 targets were identified and 190 of them were involved in platelet activation. The pathway analysis indicated signaling pathways, namely PI3K/Akt, cAMP and calcium pathways might be the potential targets. Effects and molecular mechanism of TMPZ on atherosclerosis were experimentally confirmed in animal models. Conclusions DLDTI model can serve as a useful tool to provide promising DTI candidates for experimental validation. Based on the predicted results of DLDTI model, we found TMPZ could attenuate atherosclerosis by inhibiting signal transductions in platelets. The source code and datasets explored in this work are available at https://github.com/CUMTzackGit/DLDTI.

SYNCHRONOUS CHAOS IN HIGH-DIMENSIONAL MODULAR NEURAL NETWORKS

1996 ◽  
Vol 06 (11) ◽  
pp. 2055-2067 ◽  
Author(s):  
THOMAS WENNEKERS ◽  
FRANK PASEMANN
Keyword(s):  
Neural Networks ◽  
Single Unit ◽  
System Size ◽  
Single Units ◽  
High Dimensional ◽  
Coexisting Attractors ◽  
Modular Neural Networks ◽  
Coupled Maps ◽  
Low Dimensional ◽  
The Relationship

The relationship between certain types of high-dimensional neural networks and low-dimensional prototypical equations (neuromodules) is investigated. The high-dimensional systems consist of finitely many pools containing identical, dissipative and nonlinear single-units operating in discrete time. Under the assumption of random connections inside and between pools, the system can be reduced to a set of only a few equations, which — asymptotically in time and system size — describe the behavior of every single unit arbitrarily well. This result can be viewed as synchronization of the single units in each pool. It is stated as a theorem on systems of nonlinear coupled maps, which gives explicit conditions on the single unit dynamics and the nature of the random connections. As an application we compare a 2-pool network with the corresponding two-dimensional dynamics. The bifurcation diagrams of both systems become very similar even for moderate system size (N=50) and large disorder in the connection strengths (50% of mean), despite the fact, that the systems exhibit fairly complex behavior (quasiperiodicity, chaos, coexisting attractors).

Sign in / Sign up

Export Citation Format

Share Document