scholarly journals Maximum entropy methods for extracting the learned features of deep neural networks

2017 ◽  
Author(s):  
Alex Finnegan ◽  
Jun S. Song
Keyword(s):  
Neural Networks ◽  
Maximum Entropy ◽  
Statistical Physics ◽  
Deep Neural Networks ◽  
Gc Content ◽  
Input Sequence ◽  
Biological Sequence ◽  
Maximum Entropy Distribution ◽  
On Chip ◽  
Learned Features

AbstractNew architectures of multilayer artificial neural networks and new methods for training them are rapidly revolutionizing the application of machine learning in diverse fields, including business, social science, physical sciences, and biology. Interpreting deep neural networks, however, currently remains elusive, and a critical challenge lies in understanding which meaningful features a network is actually learning. We present a general method for interpreting deep neural networks and extracting network-learned features from input data. We describe our algorithm in the context of biological sequence analysis. Our approach, based on ideas from statistical physics, samples from the maximum entropy distribution over possible sequences, anchored at an input sequence and subject to constraints implied by the empirical function learned by a network. Using our framework, we demonstrate that local transcription factor binding motifs can be identified from a network trained on ChIP-seq data and that nucleosome positioning signals are indeed learned by a network trained on chemical cleavage nucleosome maps. Imposing a further constraint on the maximum entropy distribution also allows us to probe whether a network is learning global sequence features, such as the high GC content in nucleosome-rich regions. This work thus provides valuable mathematical tools for interpreting and extracting learned features from feed-forward neural networks.

scholarly journals A Scalable System-on-Chip Acceleration for Deep Neural Networks

IEEE Access ◽  
2021 ◽  
pp. 1-1
Author(s):  
Faisal Shehzad ◽  
Muhammad Rashid ◽  
Mohammed H Sinky ◽  
Saud S Alotaibi ◽  
Muhammad Yousuf Irfan Zia
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
System On Chip ◽  
Scalable System ◽  
On Chip

Analyzing networks-on-chip based deep neural networks

2019 ◽  
Author(s):  
Giuseppe Ascia ◽  
Vincenzo Catania ◽  
Salvatore Monteleone ◽  
Maurizio Palesi ◽  
Davide Patti ◽  
...  
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Networks On Chip ◽  
On Chip

scholarly journals SIAM: Chiplet-based Scalable In-Memory Acceleration with Mesh for Deep Neural Networks

2021 ◽  
Vol 20 (5s) ◽  
pp. 1-24
Author(s):  
Gokul Krishnan ◽  
Sumit K. Mandal ◽  
Manvitha Pannala ◽  
Chaitali Chakrabarti ◽  
Jae-Sun Seo ◽  
...  
Keyword(s):  
Neural Networks ◽  
Deep Learning ◽  
Design Space Exploration ◽  
Deep Neural Networks ◽  
Feasible Solution ◽  
Computing System ◽  
Efficient Design ◽  
Simulation Speed ◽  
Wide Range ◽  
On Chip

In-memory computing (IMC) on a monolithic chip for deep learning faces dramatic challenges on area, yield, and on-chip interconnection cost due to the ever-increasing model sizes. 2.5D integration or chiplet-based architectures interconnect multiple small chips (i.e., chiplets) to form a large computing system, presenting a feasible solution beyond a monolithic IMC architecture to accelerate large deep learning models. This paper presents a new benchmarking simulator, SIAM, to evaluate the performance of chiplet-based IMC architectures and explore the potential of such a paradigm shift in IMC architecture design. SIAM integrates device, circuit, architecture, network-on-chip (NoC), network-on-package (NoP), and DRAM access models to realize an end-to-end system. SIAM is scalable in its support of a wide range of deep neural networks (DNNs), customizable to various network structures and configurations, and capable of efficient design space exploration. We demonstrate the flexibility, scalability, and simulation speed of SIAM by benchmarking different state-of-the-art DNNs with CIFAR-10, CIFAR-100, and ImageNet datasets. We further calibrate the simulation results with a published silicon result, SIMBA. The chiplet-based IMC architecture obtained through SIAM shows 130 and 72 improvement in energy-efficiency for ResNet-50 on the ImageNet dataset compared to Nvidia V100 and T4 GPUs.

scholarly journals Broad Autoencoder Features Learning for Classification Problem

2021 ◽  
Vol 15 (4) ◽  
pp. 0-0
Keyword(s):  
Neural Networks ◽  
Deep Neural Networks ◽  
Classification Problem ◽  
Activation Function ◽  
Activation Functions ◽  
Classification Problems ◽  
Stacked Autoencoders ◽  
Learned Features ◽  
Sigmoid Functions ◽  
Nonlinear Mappings

Activation functions such as Tanh and Sigmoid functions are widely used in Deep Neural Networks (DNNs) and pattern classification problems. To take advantages of different activation functions, the Broad Autoencoder Features (BAF) is proposed in this work. The BAF consists of four parallel-connected Stacked Autoencoders (SAEs) and each of them uses a different activation function, including Sigmoid, Tanh, ReLU, and Softplus. The final learned features can merge such features by various nonlinear mappings from original input features with such a broad setting. This helps to excavate more information from the original input features. Experimental results show that the BAF yields better-learned features and classification performances.

scholarly journals Probabilistic Models with Deep Neural Networks

Entropy ◽  
2021 ◽  
Vol 23 (1) ◽  
pp. 117
Author(s):  
Andrés R. Masegosa ◽  
Rafael Cabañas ◽  
Helge Langseth ◽  
Thomas D. Nielsen ◽  
Antonio Salmerón
Keyword(s):  
Neural Networks ◽  
Learning Community ◽  
Statistical Physics ◽  
Deep Neural Networks ◽  
Probabilistic Models ◽  
Broad Class ◽  
Probabilistic Inference ◽  
Probabilistic Modeling ◽  
Stochastic Gradient Descent ◽  
Modeling Framework

Recent advances in statistical inference have significantly expanded the toolbox of probabilistic modeling. Historically, probabilistic modeling has been constrained to very restricted model classes, where exact or approximate probabilistic inference is feasible. However, developments in variational inference, a general form of approximate probabilistic inference that originated in statistical physics, have enabled probabilistic modeling to overcome these limitations: (i) Approximate probabilistic inference is now possible over a broad class of probabilistic models containing a large number of parameters, and (ii) scalable inference methods based on stochastic gradient descent and distributed computing engines allow probabilistic modeling to be applied to massive data sets. One important practical consequence of these advances is the possibility to include deep neural networks within probabilistic models, thereby capturing complex non-linear stochastic relationships between the random variables. These advances, in conjunction with the release of novel probabilistic modeling toolboxes, have greatly expanded the scope of applications of probabilistic models, and allowed the models to take advantage of the recent strides made by the deep learning community. In this paper, we provide an overview of the main concepts, methods, and tools needed to use deep neural networks within a probabilistic modeling framework.

Sign in / Sign up

Export Citation Format

Share Document