scholarly journals Learning Hierarchical Representations with Spike-and-Slab Inception Network

Sensors ◽  
2021 ◽  
Vol 21 (19) ◽  
pp. 6382
Author(s):  
Weizheng Qiao ◽  
Xiaojun Bi
Keyword(s):  
Latent Variables ◽  
Network Architecture ◽  
Recognition Task ◽  
Model Parameters ◽  
Neural Network Architecture ◽  
Deep Convolutional Neural Networks ◽  
Parameter Redundancy ◽  
Feature Mining ◽  
Feature Expression ◽  
Hierarchical Representations

Recently, deep convolutional neural networks (CNN) with inception modules have attracted much attention due to their excellent performances on diverse domains. Nevertheless, the basic CNN can only capture a univariate feature, which is essentially linear. It leads to a weak ability in feature expression, further resulting in insufficient feature mining. In view of this issue, researchers incessantly deepened the network, bringing parameter redundancy and model over-fitting. Hence, whether we can employ this efficient deep neural network architecture to improve CNN and enhance the capacity of image recognition task still remains unknown. In this paper, we introduce spike-and-slab units to the modified inception module, enabling our model to capture dual latent variables and the average and covariance information. This operation further enhances the robustness of our model to variations of image intensity without increasing the model parameters. The results of several tasks demonstrated that dual variable operations can be well-integrated into inception modules, and excellent results have been achieved.

scholarly journals Gated recurrence enables simple and accurate sequence prediction in stochastic, changing, and structured environments

2021 ◽  
Author(s):  
Cedric Foucault ◽  
Florent Meyniel
Keyword(s):  
Latent Variables ◽  
Network Architecture ◽  
Real Life ◽  
Latent Structure ◽  
Neural Network Architecture ◽  
Internal Representations ◽  
Sequence Prediction ◽  
Effective Learning ◽  
Multiple Levels ◽  
The Brain

From decision making to perception to language, predicting what is coming next is crucial. It is also challenging in stochastic, changing, and structured environments; yet the brain makes accurate predictions in many situations. What computational architecture could enable this feat? Bayesian inference makes optimal predictions but is prohibitively difficult to compute. Here, we show that a specific recurrent neural network architecture enables simple and accurate solutions in several environments. To this end, a set of three mechanisms suffices: gating, lateral connections, and recurrent weight tuning. Like the human brain, such networks develop internal representations of their changing environment (including estimates of the environment's latent variables and the precision of these estimates), leverage multiple levels of latent structure, and adapt their effective learning rate to changes without changing their connection weights. Being ubiquitous in the brain, gated recurrence could therefore serve as a generic building block to predict in real-life environments.

scholarly journals Flexible Working Memory Through Selective Gating and Attentional Tagging

2021 ◽  
Vol 33 (1) ◽  
pp. 1-40 ◽  
Author(s):  
Wouter Kruijne ◽  
Sander M. Bohte ◽  
Pieter R. Roelfsema ◽  
Christian N. L. Olivers
Keyword(s):  
Neural Network ◽  
Working Memory ◽  
Network Architecture ◽  
Neural Circuit ◽  
Recognition Task ◽  
Learning Rule ◽  
Task Demands ◽  
Neural Network Architecture ◽  
Complex Tasks ◽  
Memory Content

Working memory is essential: it serves to guide intelligent behavior of humans and nonhuman primates when task-relevant stimuli are no longer present to the senses. Moreover, complex tasks often require that multiple working memory representations can be flexibly and independently maintained, prioritized, and updated according to changing task demands. Thus far, neural network models of working memory have been unable to offer an integrative account of how such control mechanisms can be acquired in a biologically plausible manner. Here, we present WorkMATe, a neural network architecture that models cognitive control over working memory content and learns the appropriate control operations needed to solve complex working memory tasks. Key components of the model include a gated memory circuit that is controlled by internal actions, encoding sensory information through untrained connections, and a neural circuit that matches sensory inputs to memory content. The network is trained by means of a biologically plausible reinforcement learning rule that relies on attentional feedback and reward prediction errors to guide synaptic updates. We demonstrate that the model successfully acquires policies to solve classical working memory tasks, such as delayed recognition and delayed pro-saccade/anti-saccade tasks. In addition, the model solves much more complex tasks, including the hierarchical 12-AX task or the ABAB ordered recognition task, both of which demand an agent to independently store and updated multiple items separately in memory. Furthermore, the control strategies that the model acquires for these tasks subsequently generalize to new task contexts with novel stimuli, thus bringing symbolic production rule qualities to a neural network architecture. As such, WorkMATe provides a new solution for the neural implementation of flexible memory control.

Application of convolutional neural networks for monitoring of marine objects

2020 ◽  
pp. 53-61
Author(s):  
Н.А. Полковникова ◽  
Е.В. Тузинкевич ◽  
А.Н. Попов
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Computer Vision ◽  
Convolutional Neural Network ◽  
Convolutional Neural Networks ◽  
Network Architecture ◽  
Research Process ◽  
Probability Of Detection ◽  
Neural Network Architecture ◽  
Deep Convolutional Neural Networks

В статье рассмотрены технологии компьютерного зрения на основе глубоких свёрточных нейронных сетей. Применение нейронных сетей особенно эффективно для решения трудно формализуемых задач. Разработана архитектура свёрточной нейронной сети применительно к задаче распознавания и классификации морских объектов на изображениях. В ходе исследования выполнен ретроспективный анализ технологий компьютерного зрения и выявлен ряд проблем, связанных с применением нейронных сетей: «исчезающий» градиент, переобучение и вычислительная сложность. При разработке архитектуры нейросети предложено использовать функцию активации RELU, обучение некоторых случайно выбранных нейронов и нормализацию с целью упрощения архитектуры нейросети. Сравнение используемых в нейросети функций активации ReLU, LeakyReLU, Exponential ReLU и SOFTMAX выполнено в среде Matlab R2020a. На основе свёрточной нейронной сети разработана программа на языке программирования Visual C# в среде MS Visual Studio для распознавания морских объектов. Программапредназначена для автоматизированной идентификации морских объектов, производит детектирование (нахождение объектов на изображении) и распознавание объектов с высокой вероятностью обнаружения. The article considers computer vision technologies based on deep convolutional neural networks. Application of neural networks is particularly effective for solving difficult formalized problems. As a result convolutional neural network architecture to the problem of recognition and classification of marine objects on images is implemented. In the research process a retrospective analysis of computer vision technologies was performed and a number of problems associated with the use of neural networks were identified: vanishing gradient, overfitting and computational complexity. To solve these problems in neural network architecture development, it was proposed to use RELU activation function, training some randomly selected neurons and normalization for simplification of neural network architecture. Comparison of ReLU, LeakyReLU, Exponential ReLU, and SOFTMAX activation functions used in the neural network implemented in Matlab R2020a.The computer program based on convolutional neural network for marine objects recognition implemented in Visual C# programming language in MS Visual Studio integrated development environment. The program is designed for automated identification of marine objects, produces detection (i.e., presence of objects on image), and objects recognition with high probability of detection.

scholarly journals Fast Pre-Diagnosis of Neoplastic Changes in Cytology Images Using Machine Learning

2021 ◽  
Vol 11 (16) ◽  
pp. 7181
Author(s):  
Jakub Caputa ◽  
Daria Łukasik ◽  
Maciej Wielgosz ◽  
Michał Karwatowski ◽  
Rafał Frączek ◽  
...  
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Network Architecture ◽  
State Of The Art ◽  
Prototype Model ◽  
Model Parameters ◽  
Round Cell ◽  
Neural Network Architecture ◽  
Original Dataset ◽  
Automated Tools

We present the experiment results to use the YOLOv3 neural network architecture to automatically detect tumor cells in cytological samples taken from the skin in canines. A rich dataset of 1219 smeared sample images with 28,149 objects was gathered and annotated by the vet doctor to perform the experiments. It covers three types of common round cell neoplasms: mastocytoma, histiocytoma, and lymphoma. The dataset has been thoroughly described in the paper and is publicly available. The YOLOv3 neural network architecture was trained using various schemes involving original dataset modification and the different model parameters. The experiments showed that the prototype model achieved 0.7416 mAP, which outperforms the state-of-the-art machine learning and human estimated results. We also provided a series of analyses that may facilitate ML-based solutions by casting more light on some aspects of its performance. We also presented the main discrepancies between ML-based and human-based diagnoses. This outline may help depict the scenarios and how the automated tools may support the diagnosis process.

scholarly journals Gated recurrence enables simple and accurate sequence prediction in stochastic, changing, and structured environments

eLife ◽  
2021 ◽  
Vol 10 ◽  
Author(s):  
Cédric Foucault ◽  
Florent Meyniel
Keyword(s):  
Latent Variables ◽  
Network Architecture ◽  
Real Life ◽  
Optimal Solution ◽  
Neural Network Architecture ◽  
Internal Representations ◽  
Sequence Prediction ◽  
Effective Learning ◽  
Multiple Levels ◽  
The Brain

From decision making to perception to language, predicting what is coming next is crucial. It is also challenging in stochastic, changing, and structured environments; yet the brain makes accurate predictions in many situations. What computational architecture could enable this feat? Bayesian inference makes optimal predictions but is prohibitively difficult to compute. Here, we show that a specific recurrent neural network architecture enables simple and accurate solutions in several environments. This architecture relies on three mechanisms: gating, lateral connections, and recurrent weight training. Like the optimal solution and the human brain, such networks develop internal representations of their changing environment (including estimates of the environment's latent variables and the precision of these estimates), leverage multiple levels of latent structure, and adapt their effective learning rate to changes without changing their connection weights. Being ubiquitous in the brain, gated recurrence could therefore serve as a generic building block to predict in real-life environments.

SCORING MODELING BASED ON NEURAL NETWORKS FOR DETERMINING A BANK BORROWER'S RATING

2020 ◽  
Vol 2020 (10) ◽  
pp. 54-62
Author(s):  
Oleksii VASYLIEV ◽  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Network Architecture ◽  
Statistical Data ◽  
Activation Function ◽  
Decision Making Process ◽  
Neural Network Architecture ◽  
Acceptable Accuracy ◽  
The Neural Network ◽  
Sigmoid Activation Function

The problem of applying neural networks to calculate ratings used in banking in the decision-making process on granting or not granting loans to borrowers is considered. The task is to determine the rating function of the borrower based on a set of statistical data on the effectiveness of loans provided by the bank. When constructing a regression model to calculate the rating function, it is necessary to know its general form. If so, the task is to calculate the parameters that are included in the expression for the rating function. In contrast to this approach, in the case of using neural networks, there is no need to specify the general form for the rating function. Instead, certain neural network architecture is chosen and parameters are calculated for it on the basis of statistical data. Importantly, the same neural network architecture can be used to process different sets of statistical data. The disadvantages of using neural networks include the need to calculate a large number of parameters. There is also no universal algorithm that would determine the optimal neural network architecture. As an example of the use of neural networks to determine the borrower's rating, a model system is considered, in which the borrower's rating is determined by a known non-analytical rating function. A neural network with two inner layers, which contain, respectively, three and two neurons and have a sigmoid activation function, is used for modeling. It is shown that the use of the neural network allows restoring the borrower's rating function with quite acceptable accuracy.

Sign in / Sign up

Export Citation Format

Share Document