scholarly journals Resonator Networks, 2: Factorization Performance and Capacity Compared to Optimization-Based Methods

2020 ◽  
Vol 32 (12) ◽  
pp. 2332-2388 ◽  
Author(s):  
Spencer J. Kent ◽  
E. Paxon Frady ◽  
Friedrich T. Sommer ◽  
Bruno A. Olshausen
Keyword(s):  
Hadamard Product ◽  
Solution Space ◽  
Local Information ◽  
Alternative Methods ◽  
High Dimensional ◽  
Factorization Problem ◽  
Gradient Based ◽  
New Type ◽  
Alternative Approaches ◽  
Theoretical Foundations

We develop theoretical foundations of resonator networks, a new type of recurrent neural network introduced in Frady, Kent, Olshausen, and Sommer ( 2020 ), a companion article in this issue, to solve a high-dimensional vector factorization problem arising in Vector Symbolic Architectures. Given a composite vector formed by the Hadamard product between a discrete set of high-dimensional vectors, a resonator network can efficiently decompose the composite into these factors. We compare the performance of resonator networks against optimization-based methods, including Alternating Least Squares and several gradient-based algorithms, showing that resonator networks are superior in several important ways. This advantage is achieved by leveraging a combination of nonlinear dynamics and searching in superposition, by which estimates of the correct solution are formed from a weighted superposition of all possible solutions. While the alternative methods also search in superposition, the dynamics of resonator networks allow them to strike a more effective balance between exploring the solution space and exploiting local information to drive the network toward probable solutions. Resonator networks are not guaranteed to converge, but within a particular regime they almost always do. In exchange for relaxing the guarantee of global convergence, resonator networks are dramatically more effective at finding factorizations than all alternative approaches considered.

Resonator Networks, 1: An Efficient Solution for Factoring High-Dimensional, Distributed Representations of Data Structures

2020 ◽  
Vol 32 (12) ◽  
pp. 2311-2331
Author(s):  
E. Paxon Frady ◽  
Spencer J. Kent ◽  
Bruno A. Olshausen ◽  
Friedrich T. Sommer
Keyword(s):  
Data Structure ◽  
Data Structures ◽  
Efficient Solution ◽  
High Dimensional ◽  
Search Problem ◽  
Combinatorial Search ◽  
Distributed Representations ◽  
Factorization Problem ◽  
Analysis And Evaluation ◽  
New Type

The ability to encode and manipulate data structures with distributed neural representations could qualitatively enhance the capabilities of traditional neural networks by supporting rule-based symbolic reasoning, a central property of cognition. Here we show how this may be accomplished within the framework of Vector Symbolic Architectures (VSAs) (Plate, 1991 ; Gayler, 1998 ; Kanerva, 1996 ), whereby data structures are encoded by combining high-dimensional vectors with operations that together form an algebra on the space of distributed representations. In particular, we propose an efficient solution to a hard combinatorial search problem that arises when decoding elements of a VSA data structure: the factorization of products of multiple codevectors. Our proposed algorithm, called a resonator network, is a new type of recurrent neural network that interleaves VSA multiplication operations and pattern completion. We show in two examples—parsing of a tree-like data structure and parsing of a visual scene—how the factorization problem arises and how the resonator network can solve it. More broadly, resonator networks open the possibility of applying VSAs to myriad artificial intelligence problems in real-world domains. The companion article in this issue (Kent, Frady, Sommer, & Olshausen, 2020 ) presents a rigorous analysis and evaluation of the performance of resonator networks, showing it outperforms alternative approaches.

Additive and Alternative Approaches to Treating Obsessive Compulsive Disorder

2012 ◽  
Author(s):  
Jordana Muroff ◽  
Abigail Ross ◽  
Joseph Rothfarb
Keyword(s):  
Obsessive Compulsive Disorder ◽  
Behavioral Therapy ◽  
Alternative Methods ◽  
Herbal Remedies ◽  
Future Research ◽  
Obsessive Compulsive ◽  
Compulsive Disorder ◽  
Specific Alternative ◽  
Alternative Approaches ◽  
Complementary And Alternative

While cognitive-behavioral therapy (CBT) and pharmacotherapy are “gold standard” treatments for obsessive-compulsive disorder (OCD), complementary and alternative treatments are frequently sought for anxiety disorders. The purpose of this chapter is to review and discuss the available research on the application, efficacy and effectiveness of complementary and alternative methods for treating OCD. The first section identifies and reviews studies focusing on specific alternative and complementary treatments that are independent from, or work in conjunction with CBT, such as yoga, herbal remedies, motivational strategies, and bibliotherapy. The second section discusses alternative and complementary methods of more mainstream CBT and related techniques, with a particular focus on technology-supported approaches. The chapter concludes with a discussion of the methodological issues in the existing research on complementary and alternative methods in the treatment of OCD, questions for future research, and implications for providers.

The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions

1998 ◽  
Vol 06 (02) ◽  
pp. 107-116 ◽  
Author(s):  
Sepp Hochreiter
Keyword(s):  
Time Series ◽  
Time Series Prediction ◽  
Time Lag ◽  
Neural Nets ◽  
Alternative Methods ◽  
Reasonable Time ◽  
Practical Applications ◽  
Learning Time ◽  
Long Time ◽  
Gradient Based

Recurrent nets are in principle capable to store past inputs to produce the currently desired output. Because of this property recurrent nets are used in time series prediction and process control. Practical applications involve temporal dependencies spanning many time steps, e.g. between relevant inputs and desired outputs. In this case, however, gradient based learning methods take too much time. The extremely increased learning time arises because the error vanishes as it gets propagated back. In this article the de-caying error flow is theoretically analyzed. Then methods trying to overcome vanishing gradients are briefly discussed. Finally, experiments comparing conventional algorithms and alternative methods are presented. With advanced methods long time lag problems can be solved in reasonable time.

A Comparison of Numerical Optimizers in Developing High Dimensional Surrogate Models

2019 ◽  
Author(s):  
Yanwen Xu ◽  
Pingfeng Wang
Keyword(s):  
Gaussian Process ◽  
Process Model ◽  
Likelihood Function ◽  
Surrogate Models ◽  
Kernel Functions ◽  
Function Optimization ◽  
High Dimensional ◽  
Model Parameters ◽  
Gradient Based ◽  
Gp Model

Abstract The Gaussian Process (GP) model has become one of the most popular methods to develop computationally efficient surrogate models in many engineering design applications, including simulation-based design optimization and uncertainty analysis. When more observations are used for high dimensional problems, estimating the best model parameters of Gaussian Process model is still an essential yet challenging task due to considerable computation cost. One of the most commonly used methods to estimate model parameters is Maximum Likelihood Estimation (MLE). A common bottleneck arising in MLE is computing a log determinant and inverse over a large positive definite matrix. In this paper, a comparison of five commonly used gradient based and non-gradient based optimizers including Sequential Quadratic Programming (SQP), Quasi-Newton method, Interior Point method, Trust Region method and Pattern Line Search for likelihood function optimization of high dimension GP surrogate modeling problem is conducted. The comparison has been focused on the accuracy of estimation, the efficiency of computation and robustness of the method for different types of Kernel functions.

scholarly journals Reconsidering the Evidence That Systematic Phonics Is More Effective Than Alternative Methods of Reading Instruction

2020 ◽  
Vol 32 (3) ◽  
pp. 681-705 ◽  
Author(s):  
Jeffrey S. Bowers
Keyword(s):  
Systematic Review ◽  
Reading Instruction ◽  
Whole Language ◽  
Language Instruction ◽  
Research Community ◽  
Alternative Methods ◽  
Alternative Approaches ◽  
Meta Analyses ◽  
Initial Reading ◽  
State Schools

AbstractThere is a widespread consensus in the research community that reading instruction in English should first focus on teaching letter (grapheme) to sound (phoneme) correspondences rather than adopt meaning-based reading approaches such as whole language instruction. That is, initial reading instruction should emphasize systematic phonics. In this systematic review, I show that this conclusion is not justified based on (a) an exhaustive review of 12 meta-analyses that have assessed the efficacy of systematic phonics and (b) summarizing the outcomes of teaching systematic phonics in all state schools in England since 2007. The failure to obtain evidence in support of systematic phonics should not be taken as an argument in support of whole language and related methods, but rather, it highlights the need to explore alternative approaches to reading instruction.

Constraining solution space to improve generalization

1997 ◽  
Vol 20 (1) ◽  
pp. 67-68
Author(s):  
John A. Bullinaria
Keyword(s):  
Learning Algorithms ◽  
Solution Space ◽  
Local Information ◽  
Learning Processes ◽  
Point Of View ◽  
Heavy Duty ◽  
Hidden Regularities ◽  
Evolutionary Point

I suggest that the difficulties inherent in discovering the hidden regularities in realistic (type-2) problems can often be resolved by learning algorithms employing simple constraints (such as symmetry and the importance of local information) that are natural from an evolutionary point of view. Neither “heavy-duty nativism” nor “representational recoding” appear to offer totally appropriate descriptions of such natural learning processes.

scholarly journals Semantic domains in Picasso’s poetry

2018 ◽  
Vol 34 (Supplement_1) ◽  
pp. i123-i128
Author(s):  
Luis Meneses ◽  
Enrique Mallen
Keyword(s):  
Semantic Networks ◽  
Alternative Methods ◽  
The Political ◽  
Pablo Picasso ◽  
Verbal Expression ◽  
Semantic Categories ◽  
Visual Composition ◽  
Alternative Approaches ◽  
Concrete Concepts ◽  
Political Turmoil

Abstract The question of why Pablo Picasso dedicated a considerable amount of his time to writing around 1935 is open to speculation. Many have cited, among possible causes: the Spanish artist’s emotional crisis, the political turmoil in Europe in the period between the two wars, and the menace of a confrontation in Spain. All of these views are predicated on an assumed irreducible conflict between visual composition and verbal expression. However, we cannot forget that Picasso’s interest in alternative methods of expression might have started with his fascination for linguistic structure as a whole during his cubist period. In this article, we explore the possibility that the transition into poetry that we observe in Picasso is simply one more manifestation of his pursuit of alternative approaches to language as a means of representation. In this sense, one thing that remained to be determined was how concrete concepts in both languages cluster into representative semantic categories and how these categories interact with each other in semantic networks.

Models of Movement: A Review of Alternative Methods

1980 ◽  
Vol 12 (12) ◽  
pp. 1383-1404 ◽  
Author(s):  
A Pickles
Keyword(s):  
Stochastic Process ◽  
Transition Rate ◽  
Process Models ◽  
Alternative Methods ◽  
Occupancy Models ◽  
Alternative Approaches ◽  
Fixed State

This paper reviews methods available to analyse movement and in particular migration. Stochastic process models seem able to provide a framework for microanalysis which can incorporate much of the complexity of such processes. However, a consideration of the effect of macro-constraints, in the form of limited opportunities for movement and of interhousehold competition, leads to a distinction between fixed transition rate and fixed state occupancy models. Alternative approaches to fixed state occupancy models are considered, and some of their potential strengths and weaknesses are discussed.

Sign in / Sign up

Export Citation Format

Share Document