scholarly journals A mathematical theory of semantic development in deep neural networks

2019 ◽  
Vol 116 (23) ◽  
pp. 11537-11546 ◽  
Author(s):  
Andrew M. Saxe ◽  
James L. McClelland ◽  
Surya Ganguli
Keyword(s):  
Neural Networks ◽  
Semantic Processing ◽  
Neural Model ◽  
Semantic Knowledge ◽  
Neural Representation ◽  
Learning Dynamics ◽  
Abstract Knowledge ◽  
Semantic Development ◽  
Extensive Body ◽  
Conceptual Question

An extensive body of empirical research has revealed remarkable regularities in the acquisition, organization, deployment, and neural representation of human semantic knowledge, thereby raising a fundamental conceptual question: What are the theoretical principles governing the ability of neural networks to acquire, organize, and deploy abstract knowledge by integrating across many individual experiences? We address this question by mathematically analyzing the nonlinear dynamics of learning in deep linear networks. We find exact solutions to this learning dynamics that yield a conceptual explanation for the prevalence of many disparate phenomena in semantic cognition, including the hierarchical differentiation of concepts through rapid developmental transitions, the ubiquity of semantic illusions between such transitions, the emergence of item typicality and category coherence as factors controlling the speed of semantic processing, changing patterns of inductive projection over development, and the conservation of semantic similarity in neural representations across species. Thus, surprisingly, our simple neural model qualitatively recapitulates many diverse regularities underlying semantic development, while providing analytic insight into how the statistical structure of an environment can interact with nonlinear deep-learning dynamics to give rise to these regularities.

The TP Chromatic Aberration Estimation by Using Neural Networks

2011 ◽  
Vol 121-126 ◽  
pp. 4239-4243 ◽  
Author(s):  
Du Jou Huang ◽  
Yu Ju Chen ◽  
Huang Chu Huang ◽  
Yu An Lin ◽  
Rey Chue Hwang
Keyword(s):  
Neural Networks ◽  
Learning Algorithm ◽  
Back Propagation ◽  
Physical Property ◽  
Chromatic Aberration ◽  
Neural Model ◽  
Artificial Intelligent ◽  
Error Back Propagation ◽  
The Neural Networks ◽  
Simulation Results

The chromatic aberration estimations of touch panel (TP) film by using neural networks are presented in this paper. The neural networks with error back-propagation (BP) learning algorithm were used to catch the complex relationship between the chromatic aberration, i.e., L.A.B. values, and the relative parameters of TP decoration film. An artificial intelligent (AI) estimator based on neural model for the estimation of physical property of TP film is expected to be developed. From the simulation results shown, the estimations of chromatic aberration of TP film are very accurate. In other words, such an AI estimator is quite promising and potential in commercial using.

scholarly journals Semiotic Aggregation in Deep Learning

Entropy ◽  
2020 ◽  
Vol 22 (12) ◽  
pp. 1365
Author(s):  
Bogdan Muşat ◽  
Răzvan Andonie
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Deep Learning ◽  
Decision Model ◽  
Deep Neural Networks ◽  
Neural Model ◽  
Network Layers ◽  
Saliency Maps ◽  
Spatial Entropy ◽  
Insight Into

Convolutional neural networks utilize a hierarchy of neural network layers. The statistical aspects of information concentration in successive layers can bring an insight into the feature abstraction process. We analyze the saliency maps of these layers from the perspective of semiotics, also known as the study of signs and sign-using behavior. In computational semiotics, this aggregation operation (known as superization) is accompanied by a decrease of spatial entropy: signs are aggregated into supersign. Using spatial entropy, we compute the information content of the saliency maps and study the superization processes which take place between successive layers of the network. In our experiments, we visualize the superization process and show how the obtained knowledge can be used to explain the neural decision model. In addition, we attempt to optimize the architecture of the neural model employing a semiotic greedy technique. To the extent of our knowledge, this is the first application of computational semiotics in the analysis and interpretation of deep neural networks.

scholarly journals Validated measures of semantic knowledge and semantic control: Normative data from young and older adults for more than 300 semantic judgements

2021 ◽  
Author(s):  
Wei Wu ◽  
Paul Hoffman
Keyword(s):  
Normative Data ◽  
Semantic Processing ◽  
Feature Matching ◽  
Cognitive Architecture ◽  
Later Life ◽  
Semantic Knowledge ◽  
Matching Task ◽  
Large Set ◽  
Semantic Cognition ◽  
Control Processes

Recent studies suggest that knowledge representations and control processes are the two key components underpinning semantic cognition, and are also crucial indicators of the shifting cognitive architecture of semantics in later life. Although there are many standardized assessments that provide measures of the quantity of semantic knowledge participants possess, normative data for tasks that probe semantic control processes are not yet available. Here, we present normative data from more than 200 young and older participants on a large set of stimuli in two semantic tasks, which probe controlled semantic processing (feature-matching task) and semantic knowledge (synonym judgement task). We verify the validity of our norms by replicating established age- and psycholinguistic-property-related effects on semantic cognition. Specifically, we find that older people have more detailed semantic knowledge than young people but have less effective semantic control processes. We also obtain expected effects of word frequency and inter-item competition on performance. Parametrically varied difficulty levels are defined for half of the stimuli based on participants’ behavioral performance, allowing future studies to produce customized sets of experimental stimuli based on our norms. We provide all stimuli, data and code used for analysis, in the hope that they are useful to other researchers.

scholarly journals Prediction of Slot Shape and Slot Size for Improving the Performance of Microstrip Antennas Using Knowledge-Based Neural Networks

2014 ◽  
Vol 2014 ◽  
pp. 1-9 ◽  
Author(s):  
Taimoor Khan ◽  
Asok De
Keyword(s):  
Neural Networks ◽  
Artificial Neural Networks ◽  
Prior Knowledge ◽  
Neural Model ◽  
Microstrip Antennas ◽  
Performance Parameters ◽  
Dual Frequency ◽  
Knowledge Based ◽  
Slot Size ◽  
Good Agreement

In the last decade, artificial neural networks have become very popular techniques for computing different performance parameters of microstrip antennas. The proposed work illustrates a knowledge-based neural networks model for predicting the appropriate shape and accurate size of the slot introduced on the radiating patch for achieving desired level of resonance, gain, directivity, antenna efficiency, and radiation efficiency for dual-frequency operation. By incorporating prior knowledge in neural model, the number of required training patterns is drastically reduced. Further, the neural model incorporated with prior knowledge can be used for predicting response in extrapolation region beyond the training patterns region. For validation, a prototype is also fabricated and its performance parameters are measured. A very good agreement is attained between measured, simulated, and predicted results.

Semantic Cognition: Semantic Memory and Semantic Control

2021 ◽  
Author(s):  
Elizabeth Jefferies ◽  
Xiuyi Wang
Keyword(s):  
Semantic Memory ◽  
Semantic Processing ◽  
Inferior Frontal Gyrus ◽  
Semantic Knowledge ◽  
Human Cognition ◽  
Factual Knowledge ◽  
Conceptual Representations ◽  
Semantic Cognition ◽  
Control Processes ◽  
Functional Features

Semantic processing is a defining feature of human cognition, central not only to language, but also to object recognition, the generation of appropriate actions, and the capacity to use knowledge in reasoning, planning, and problem-solving. Semantic memory refers to our repository of conceptual or factual knowledge about the world. This semantic knowledge base is typically viewed as including “general knowledge” as well as schematic representations of objects and events distilled from multiple experiences and retrieved independently from their original spatial or temporal context. Semantic cognition refers to our ability to flexibly use this knowledge to produce appropriate thoughts and behaviors. Semantic cognition includes at least two interactive components: a long-term store of semantic knowledge and semantic control processes, each supported by a different network. Conceptual representations are organized according to the semantic relationships between items, with different theories proposing different key organizational principles, including sensory versus functional features, domain-specific theory, embodied distributed concepts, and hub-and-spoke theory, in which distributed features are integrated within a heteromodal hub in the anterior temporal lobes. The activity within the network for semantic representation must often be controlled to ensure that the system generates representations and inferences that are suited to the immediate task or context. Semantic control is thought to include both controlled retrieval processes, in which knowledge relevant to the goal or context is accessed in a top-down manner when automatic retrieval is insufficient for the task, and post-retrieval selection to resolve competition between simultaneously active representations. Control of semantic retrieval is supported by a strongly left-lateralized brain network, which partially overlaps with the bilateral network that supports domain-general control, but extends beyond these sites to include regions not typically associated with executive control, including anterior inferior frontal gyrus and posterior middle temporal gyrus. The interaction of semantic control processes with conceptual representations allows meaningful thoughts and behavior to emerge, even when the context requires non-dominant features of the concept to be brought to the fore.

scholarly journals Neural Patterns in Linguistic Cortices Discriminate the Content of Verbal Working Memory

2019 ◽  
Vol 30 (5) ◽  
pp. 2997-3014 ◽  
Author(s):  
Benjamin Kowialiewski ◽  
Laurens Van Calster ◽  
Lucie Attout ◽  
Christophe Phillips ◽  
Steve Majerus
Keyword(s):  
Working Memory ◽  
Semantic Processing ◽  
Semantic Knowledge ◽  
Linguistic Knowledge ◽  
Long Term Memory ◽  
Short Term ◽  
Term Memory ◽  
Direct Activation ◽  
Term Maintenance

Abstract An influential theoretical account of working memory (WM) considers that WM is based on direct activation of long-term memory knowledge. While there is empirical support for this position in the visual WM domain, direct evidence is scarce in the verbal WM domain. This question is critical for models of verbal WM, as the question of whether short-term maintenance of verbal information relies on direct activation within the long-term linguistic knowledge base or not is still debated. In this study, we examined the extent to which short-term maintenance of lexico-semantic knowledge relies on neural activation patterns in linguistic cortices, and this by using a fast encoding running span task for word and nonword stimuli minimizing strategic encoding mechanisms. Multivariate analyses showed specific neural patterns for the encoding and maintenance of word versus nonword stimuli. These patterns were not detectable anymore when participants were instructed to stop maintaining the memoranda. The patterns involved specific regions within the dorsal and ventral pathways, which are considered to support phonological and semantic processing to various degrees. This study provides novel evidence for a role of linguistic cortices in the representation of long-term memory linguistic knowledge during WM processing.

MREM, Discrete Recurrent Network for Optimization

2011 ◽  
pp. 1112-1120
Author(s):  
Enrique Mérida-Casermeiro ◽  
Domingo López-Rodríguez ◽  
Juan M. Ortiz-de-Lazcano-Lobato
Keyword(s):  
Neural Networks ◽  
Combinatorial Optimization ◽  
Image Recognition ◽  
Single Neuron ◽  
Optimization Problems ◽  
Neural Model ◽  
Recurrent Network ◽  
Neural Models ◽  
Seminal Work ◽  
Combinatorial Optimization Problems

Since McCulloch and Pitts’ seminal work (McCulloch & Pitts, 1943), several models of discrete neural networks have been proposed, many of them presenting the ability of assigning a discrete value (other than unipolar or bipolar) to the output of a single neuron. These models have focused on a wide variety of applications. One of the most important models was developed by J. Hopfield in (Hopfield, 1982), which has been successfully applied in fields such as pattern and image recognition and reconstruction (Sun et al., 1995), design of analogdigital circuits (Tank & Hopfield, 1986), and, above all, in combinatorial optimization (Hopfield & Tank, 1985) (Takefuji, 1992) (Takefuji & Wang, 1996), among others. The purpose of this work is to review some applications of multivalued neural models to combinatorial optimization problems, focusing specifically on the neural model MREM, since it includes many of the multivalued models in the specialized literature.

Perturbations, Accuracy and Robustness in Neural Networks

2005 ◽  
pp. 2282-2287
Author(s):  
Cesare Alippi ◽  
Giovanni Vanini
Keyword(s):  
Neural Networks ◽  
Robustness Analysis ◽  
Theoretical Aspect ◽  
Neural Model ◽  
Knowledge Space

A robustness analysis for neural networks, namely the evaluation of the effects induced by perturbations affecting the network weights, is a relevant theoretical aspect since weights characterise the “knowledge space” of the neural model and, hence, its inner nature.

Sign in / Sign up

Export Citation Format

Share Document