scholarly journals Deep Learning: Implications for Human Learning and Memory

2020 ◽  
Author(s):  
James Lloyd McClelland ◽  
Matthew M. Botvinick
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Learning And Memory ◽  
Deep Neural Networks ◽  
Human Learning ◽  
Machine Intelligence ◽  
Learning Approaches ◽  
Human Capabilities ◽  
Wide Range ◽  
Learning Research

Recent years have seen an explosion of interest in deep learning and deep neural networks. Deep learning lies at the heart of unprecedented feats of machine intelligence as well as software people use every day. Systems built on deep learning have surpassed human capabilities in complex strategy games like go and chess, and we use them for speech recognition, image captioning, and a wide range of other applications. A consideration of deep learning is crucial for a Handbook of Human Memory, since human brains are deep neural networks, and an understanding of artificial deep learning systems may contribute to our understanding of how humans and animals learn and remember. Deep neural networks are complex, structured systems that process information in a parallel, distributed, and context sensitive fashion, and deep learning is the effort to use these systems to acquire capabilities we associate with intelligence through an experience dependent learning process. Within the field of Artificial Intelligence, work in deep learning is typically directed toward the goal of creating and understanding intelligence using all available tools and resources without consideration of their biological plausibility. Many of the ideas, however, at the heart of deep learning draw their inspiration from the brain and from characteristics of human intelligence we believe are best captured by these brain-inspired systems (Rumelhart, McClelland, and the PDP Research Group, 1986). Furthermore, ideas emerging from deep learning research can help inform us about memory and learning in humans and animals. Thus, deep learning research can be seen as fertile ground for engagement between researchers who work on related issues with implications for both biological and machine intelligence.We begin by introducing the basic constructs employed in deep learning and then consider several of the widely used learning paradigms and architectures used in these systems. We then turn to a consideration of how the constructs of deep learning relate to traditional constructs in the psychological literature on learning and memory. Next, we consider recent developments in the field of reinforcement learning that have broad implications for human learning and memory. We conclude with a consideration of areas where human capabilities still far exceed current deep learning approaches, and describe possible future directions toward understanding how these abilities might best be captured.

Bearing fault diagnosis using deep learning techniques coupled with handcrafted feature extraction: A comparative study

2020 ◽  
pp. 107754632092914
Author(s):  
Mohammed Alabsi ◽  
Yabin Liao ◽  
Ala-Addin Nabulsi
Keyword(s):  
Neural Networks ◽  
Feature Extraction ◽  
Deep Learning ◽  
Comparative Study ◽  
Domain Knowledge ◽  
Deep Neural Networks ◽  
Performance Limits ◽  
Data Repositories ◽  
Learning Techniques ◽  
Wide Range

Deep learning has seen tremendous growth over the past decade. It has set new performance limits for a wide range of applications, including computer vision, speech recognition, and machinery health monitoring. With the abundance of instrumentation data and the availability of high computational power, deep learning continues to prove itself as an efficient tool for the extraction of micropatterns from machinery big data repositories. This study presents a comparative study for feature extraction capabilities using stacked autoencoders considering the use of expert domain knowledge. Case Western Reserve University bearing dataset was used for the study, and a classifier was trained and tested to extract and visualize features from 12 different failure classes. Based on the raw data preprocessing, four different deep neural network structures were studied. Results indicated that integrating domain knowledge with deep learning techniques improved feature extraction capabilities and reduced the deep neural networks size and computational requirements without the need for exhaustive deep neural networks architecture tuning and modification.

Exemplar models are useful and deep neural networks overcome their limitations: A commentary on Ambridge (2020)

2020 ◽  
Vol 40 (5-6) ◽  
pp. 612-615
Author(s):  
James L. McClelland
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Language Processing ◽  
Deep Neural Networks ◽  
Machine Intelligence ◽  
Human Language ◽  
Language Usage ◽  
Primitive Elements ◽  
Exemplar Models ◽  
Learning Architectures

Humans are sensitive to the properties of individual items, and exemplar models are useful for capturing this sensitivity. I am a proponent of an extension of exemplar-based architectures that I briefly describe. However, exemplar models are very shallow architectures in which it is necessary to stipulate a set of primitive elements that make up each example, and such architectures have not been as successful as deep neural networks in capturing language usage and meaning. More work is needed bringing contemporary deep learning architectures used in machine intelligence to the effort to understand human language processing.

Network attributes describe a similarity between deep neural networks and large scale brain networks

2019 ◽  
Author(s):  
Kosuke Takagi
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Human Brain ◽  
Large Scale ◽  
Deep Neural Networks ◽  
Distribution Model ◽  
Common Mechanism ◽  
Learning Models ◽  
Connection Weight ◽  
Wide Range

Abstract Despite the recent success of deep learning models in solving various problems, their ability is still limited compared with human intelligence, which has the flexibility to adapt to a changing environment. To obtain a model which achieves adaptability to a wide range of problems and tasks is a challenging problem. To achieve this, an issue that must be addressed is identification of the similarities and differences between the human brain and deep neural networks. In this article, inspired by the human flexibility which might suggest the existence of a common mechanism allowing solution of different kinds of tasks, we consider a general learning process in neural networks, on which no specific conditions and constraints are imposed. Subsequently, we theoretically show that, according to the learning progress, the network structure converges to the state, which is characterized by a unique distribution model with respect to network quantities such as the connection weight and node strength. Noting that the empirical data indicate that this state emerges in the large scale network in the human brain, we show that the same state can be reproduced in a simple example of deep learning models. Although further research is needed, our findings provide an insight into the common inherent mechanism underlying the human brain and deep learning. Thus, our findings provide suggestions for designing efficient learning algorithms for solving a wide variety of tasks in the future.

scholarly journals Integrating Deep Neural Networks and Symbolic Inference for Organic Reactivity Prediction

2020 ◽  
Author(s):  
Wesley Wei Qian ◽  
Nathan T. Russell ◽  
Claire L. W. Simons ◽  
Yunan Luo ◽  
Martin D. Burke ◽  
...  
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Probability Distribution ◽  
Deep Neural Networks ◽  
Machine Intelligence ◽  
Data Availability ◽  
Learning Models ◽  
Expert Performance ◽  
Covalent Bonds ◽  
Chemical Reasoning

<div>Accurate <i>in silico</i> models for the prediction of novel chemical reaction outcomes can be used to guide the rapid discovery of new reactivity and enable novel synthesis strategies for newly discovered lead compounds. Recent advances in machine learning, driven by deep learning models and data availability, have shown utility throughout synthetic organic chemistry as a data-driven method for reaction prediction. Here we present a machine-intelligence approach to predict the products of an organic reaction by integrating deep neural networks with a probabilistic and symbolic inference that flexibly enforces chemical constraints and accounts for prior chemical knowledge. We first train a graph convolutional neural network to estimate the likelihood of changes in covalent bonds, hydrogen counts, and formal charges. These estimated likelihoods govern a probability distribution over potential products. Integer Linear Programming is then used to infer the most probable products from the probability distribution subject to heuristic rules such as the octet rule and chemical constraints that reflect a user's prior knowledge. Our approach outperforms previous graph-based neural networks by predicting products with more than 90% accuracy, demonstrates intuitive chemical reasoning through a learned attention mechanism, and provides generalizability across various reaction types. Furthermore, we demonstrate the potential for even higher model accuracy when complemented by expert chemists contributing to the system, boosting both machine and expert performance. The results show the advantages of empowering deep learning models with chemical intuition and knowledge to expedite the drug discovery process.</div>

scholarly journals SIAM: Chiplet-based Scalable In-Memory Acceleration with Mesh for Deep Neural Networks

2021 ◽  
Vol 20 (5s) ◽  
pp. 1-24
Author(s):  
Gokul Krishnan ◽  
Sumit K. Mandal ◽  
Manvitha Pannala ◽  
Chaitali Chakrabarti ◽  
Jae-Sun Seo ◽  
...  
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Design Space Exploration ◽  
Deep Neural Networks ◽  
Feasible Solution ◽  
Computing System ◽  
Efficient Design ◽  
Simulation Speed ◽  
Wide Range ◽  
On Chip

In-memory computing (IMC) on a monolithic chip for deep learning faces dramatic challenges on area, yield, and on-chip interconnection cost due to the ever-increasing model sizes. 2.5D integration or chiplet-based architectures interconnect multiple small chips (i.e., chiplets) to form a large computing system, presenting a feasible solution beyond a monolithic IMC architecture to accelerate large deep learning models. This paper presents a new benchmarking simulator, SIAM, to evaluate the performance of chiplet-based IMC architectures and explore the potential of such a paradigm shift in IMC architecture design. SIAM integrates device, circuit, architecture, network-on-chip (NoC), network-on-package (NoP), and DRAM access models to realize an end-to-end system. SIAM is scalable in its support of a wide range of deep neural networks (DNNs), customizable to various network structures and configurations, and capable of efficient design space exploration. We demonstrate the flexibility, scalability, and simulation speed of SIAM by benchmarking different state-of-the-art DNNs with CIFAR-10, CIFAR-100, and ImageNet datasets. We further calibrate the simulation results with a published silicon result, SIMBA. The chiplet-based IMC architecture obtained through SIAM shows 130 and 72 improvement in energy-efficiency for ResNet-50 on the ImageNet dataset compared to Nvidia V100 and T4 GPUs.

Integrating Deep Neural Networks and Symbolic Inference for Organic Reactivity Prediction

2020 ◽  
Author(s):  
Wesley Wei Qian ◽  
Nathan T. Russell ◽  
Claire L. W. Simons ◽  
Yunan Luo ◽  
Martin D. Burke ◽  
...  
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Probability Distribution ◽  
Deep Neural Networks ◽  
Machine Intelligence ◽  
Data Availability ◽  
Learning Models ◽  
Expert Performance ◽  
Covalent Bonds ◽  
Chemical Reasoning

<div>Accurate <i>in silico</i> models for the prediction of novel chemical reaction outcomes can be used to guide the rapid discovery of new reactivity and enable novel synthesis strategies for newly discovered lead compounds. Recent advances in machine learning, driven by deep learning models and data availability, have shown utility throughout synthetic organic chemistry as a data-driven method for reaction prediction. Here we present a machine-intelligence approach to predict the products of an organic reaction by integrating deep neural networks with a probabilistic and symbolic inference that flexibly enforces chemical constraints and accounts for prior chemical knowledge. We first train a graph convolutional neural network to estimate the likelihood of changes in covalent bonds, hydrogen counts, and formal charges. These estimated likelihoods govern a probability distribution over potential products. Integer Linear Programming is then used to infer the most probable products from the probability distribution subject to heuristic rules such as the octet rule and chemical constraints that reflect a user's prior knowledge. Our approach outperforms previous graph-based neural networks by predicting products with more than 90% accuracy, demonstrates intuitive chemical reasoning through a learned attention mechanism, and provides generalizability across various reaction types. Furthermore, we demonstrate the potential for even higher model accuracy when complemented by expert chemists contributing to the system, boosting both machine and expert performance. The results show the advantages of empowering deep learning models with chemical intuition and knowledge to expedite the drug discovery process.</div>

Recent Progresses of Deep Learning in Drug Discovery

2021 ◽  
Vol 27 ◽  
Author(s):  
Feng Wang ◽  
XiaoMin Diao ◽  
Shan Chang ◽  
Lei Xu
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Drug Discovery ◽  
Deep Neural Networks ◽  
De Novo ◽  
De Novo Design ◽  
Learning Approaches ◽  
Property Prediction ◽  
Design Property ◽  
Future Direction

: Deep learning, an emerging field of artificial intelligence based on neural network in machine learning, has been applied in various fields and is highly valued. Herein we mainly review several mainstream architectures in deep learning, including deep neural networks, convolutional neural networks and recurrent neural networks in the field of drug discovery. The applications of several architectures in molecular de novo design, property prediction, biomedical imaging and synthetic planning have also been explored. We also discuss the future direction of the deep learning approaches and the main challenges we need to address.

scholarly journals Leveraging Deep Learning for Visual Odometry Using Optical Flow

Sensors ◽  
2021 ◽  
Vol 21 (4) ◽  
pp. 1313
Author(s):  
Tejas Pandey ◽  
Dexmont Pena ◽  
Jonathan Byrne ◽  
David Moloney
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Optical Flow ◽  
Recurrent Neural Networks ◽  
Deep Neural Networks ◽  
Visual Odometry ◽  
Camera Motion ◽  
Learning Approaches ◽  
Outlier Rejection ◽  
Entire Trajectory

In this paper, we study deep learning approaches for monocular visual odometry (VO). Deep learning solutions have shown to be effective in VO applications, replacing the need for highly engineered steps, such as feature extraction and outlier rejection in a traditional pipeline. We propose a new architecture combining ego-motion estimation and sequence-based learning using deep neural networks. We estimate camera motion from optical flow using Convolutional Neural Networks (CNNs) and model the motion dynamics using Recurrent Neural Networks (RNNs). The network outputs the relative 6-DOF camera poses for a sequence, and implicitly learns the absolute scale without the need for camera intrinsics. The entire trajectory is then integrated without any post-calibration. We evaluate the proposed method on the KITTI dataset and compare it with traditional and other deep learning approaches in the literature.

scholarly journals Deep neural networks in hydrology: the new generation of universal and efficient models

2021 ◽  
Vol 66 (1) ◽  
Author(s):  
Georgy V. Ayzel ◽  
◽  
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Deep Neural Networks ◽  
Hydrological Modeling ◽  
Recent Change ◽  
Model Parameters ◽  
Learning Technology ◽  
Water Balance Components ◽  
Wide Range ◽  
New Generation

For around a decade, deep learning – the sub-field of machine learning that refers to artificial neural networks comprised of many computational layers – modifies the landscape of statistical model development in many research areas, such as image classification, machine translation, and speech recognition. Geoscientific disciplines in general and the field of hydrology in particular, also do not stand aside from this movement. Recently, the proliferation of modern deep learning-based techniques and methods has been actively gaining popularity for solving a wide range of hydrological problems: modeling and forecasting of river runoff, hydrological model parameters regionalization, assessment of available water resources, identification of the main drivers of the recent change in water balance components. This growing popularity of deep neural networks is primarily due to their high universality and efficiency. The presented qualities, together with the rapidly growing amount of accumulated environmental information, as well as increasing availability of computing facilities and resources, allow us to speak about deep neural networks as a new generation of mathematical models designed to, if not to replace existing solutions, but significantly enrich the field of geophysical processes modeling. This paper provides a brief overview of the current state of the field of development and application of deep neural networks in hydrology. Also in the following study, the qualitative long-term forecast regarding the development of deep learning technology for managing the corresponding hydrological modeling challenges is provided based on the use of “Gartner Hype Curve”, which in the general details describes a life cycle of modern technologies.

scholarly journals Deep convolutional neural networks for cardiovascular vulnerable plaque detection

2019 ◽  
Vol 277 ◽  
pp. 02024 ◽  
Author(s):  
Lincan Li ◽  
Tong Jia ◽  
Tianqi Meng ◽  
Yizhe Liu
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Convolutional Neural Networks ◽  
Vulnerable Plaque ◽  
Recall Rate ◽  
Superior Performance ◽  
Learning Approaches ◽  
Deep Convolutional Neural Networks ◽  
Vulnerable Plaques ◽  
Plaque Detection

In this paper, an accurate two-stage deep learning method is proposed to detect vulnerable plaques in ultrasonic images of cardiovascular. Firstly, a Fully Convonutional Neural Network (FCN) named U-Net is used to segment the original Intravascular Optical Coherence Tomography (IVOCT) cardiovascular images. We experiment on different threshold values to find the best threshold for removing noise and background in the original images. Secondly, a modified Faster RCNN is adopted to do precise detection. The modified Faster R-CNN utilize six-scale anchors (122,162,322,642,1282,2562) instead of the conventional one scale or three scale approaches. First, we present three problems in cardiovascular vulnerable plaque diagnosis, then we demonstrate how our method solve these problems. The proposed method in this paper apply deep convolutional neural networks to the whole diagnostic procedure. Test results show the Recall rate, Precision rate, IoU (Intersection-over-Union) rate and Total score are 0.94, 0.885, 0.913 and 0.913 respectively, higher than the 1st team of CCCV2017 Cardiovascular OCT Vulnerable Plaque Detection Challenge. AP of the designed Faster RCNN is 83.4%, higher than conventional approaches which use one-scale or three-scale anchors. These results demonstrate the superior performance of our proposed method and the power of deep learning approaches in diagnose cardiovascular vulnerable plaques.

Sign in / Sign up

Export Citation Format

Share Document