A Review of Algorithms and Hardware Implementations for Spiking Neural Networks

Deep Learning (DL) has contributed to the success of many applications in recent years. The applications range from simple ones such as recognizing tiny images or simple speech patterns to ones with a high level of complexity such as playing the game of Go. However, this superior performance comes at a high computational cost, which made porting DL applications to conventional hardware platforms a challenging task. Many approaches have been investigated, and Spiking Neural Network (SNN) is one of the promising candidates. SNN is the third generation of Artificial Neural Networks (ANNs), where each neuron in the network uses discrete spikes to communicate in an event-based manner. SNNs have the potential advantage of achieving better energy efficiency than their ANN counterparts. While generally there will be a loss of accuracy on SNN models, new algorithms have helped to close the accuracy gap. For hardware implementations, SNNs have attracted much attention in the neuromorphic hardware research community. In this work, we review the basic background of SNNs, the current state and challenges of the training algorithms for SNNs and the current implementations of SNNs on various hardware platforms.

Download Full-text

An efficient pruning scheme of deep neural networks for Internet of Things applications

EURASIP Journal on Advances in Signal Processing ◽

10.1186/s13634-021-00744-4 ◽

2021 ◽

Vol 2021 (1) ◽

Author(s):

Chen Qi ◽

Shibo Shen ◽

Rongpeng Li ◽

Zhifeng Zhao ◽

Qing Liu ◽

...

Keyword(s):

Neural Network ◽

Neural Networks ◽

Internet Of Things ◽

Deep Neural Networks ◽

Computational Cost ◽

Superior Performance ◽

Compact Structure ◽

Resource Limited ◽

Benchmark Datasets ◽

Iot Devices

AbstractNowadays, deep neural networks (DNNs) have been rapidly deployed to realize a number of functionalities like sensing, imaging, classification, recognition, etc. However, the computational-intensive requirement of DNNs makes it difficult to be applicable for resource-limited Internet of Things (IoT) devices. In this paper, we propose a novel pruning-based paradigm that aims to reduce the computational cost of DNNs, by uncovering a more compact structure and learning the effective weights therein, on the basis of not compromising the expressive capability of DNNs. In particular, our algorithm can achieve efficient end-to-end training that transfers a redundant neural network to a compact one with a specifically targeted compression rate directly. We comprehensively evaluate our approach on various representative benchmark datasets and compared with typical advanced convolutional neural network (CNN) architectures. The experimental results verify the superior performance and robust effectiveness of our scheme. For example, when pruning VGG on CIFAR-10, our proposed scheme is able to significantly reduce its FLOPs (floating-point operations) and number of parameters with a proportion of 76.2% and 94.1%, respectively, while still maintaining a satisfactory accuracy. To sum up, our scheme could facilitate the integration of DNNs into the common machine-learning-based IoT framework and establish distributed training of neural networks in both cloud and edge.

Download Full-text

Exploring Optimized Spiking Neural Network Architectures for Classification Tasks on Embedded Platforms

Sensors ◽

10.3390/s21093240 ◽

2021 ◽

Vol 21 (9) ◽

pp. 3240

Author(s):

Tehreem Syed ◽

Vijay Kakani ◽

Xuenan Cui ◽

Hakil Kim

Keyword(s):

Neural Networks ◽

Gradient Descent ◽

Spiking Neural Networks ◽

License Plate ◽

Training Techniques ◽

Neuromorphic Hardware ◽

Private And Public ◽

Embedded Platforms ◽

Public Datasets ◽

Event Based

In recent times, the usage of modern neuromorphic hardware for brain-inspired SNNs has grown exponentially. In the context of sparse input data, they are undertaking low power consumption for event-based neuromorphic hardware, specifically in the deeper layers. However, using deep ANNs for training spiking models is still considered as a tedious task. Until recently, various ANN to SNN conversion methods in the literature have been proposed to train deep SNN models. Nevertheless, these methods require hundreds to thousands of time-steps for training and still cannot attain good SNN performance. This work proposes a customized model (VGG, ResNet) architecture to train deep convolutional spiking neural networks. In this current study, the training is carried out using deep convolutional spiking neural networks with surrogate gradient descent backpropagation in a customized layer architecture similar to deep artificial neural networks. Moreover, this work also proposes fewer time-steps for training SNNs with surrogate gradient descent. During the training with surrogate gradient descent backpropagation, overfitting problems have been encountered. To overcome these problems, this work refines the SNN based dropout technique with surrogate gradient descent. The proposed customized SNN models achieve good classification results on both private and public datasets. In this work, several experiments have been carried out on an embedded platform (NVIDIA JETSON TX2 board), where the deployment of customized SNN models has been extensively conducted. Performance validations have been carried out in terms of processing time and inference accuracy between PC and embedded platforms, showing that the proposed customized models and training techniques are feasible for achieving a better performance on various datasets such as CIFAR-10, MNIST, SVHN, and private KITTI and Korean License plate dataset.

Download Full-text

Deep Learning

International Journal of Semantic Computing ◽

10.1142/s1793351x16500045 ◽

2016 ◽

Vol 10 (03) ◽

pp. 417-439 ◽

Cited By ~ 28

Author(s):

Xing Hao ◽

Guigang Zhang ◽

Shang Ma

Keyword(s):

Machine Learning ◽

Neural Networks ◽

Deep Learning ◽

Complex Structures ◽

Training Algorithms ◽

High Level ◽

And Training

Deep learning is a branch of machine learning that tries to model high-level abstractions of data using multiple layers of neurons consisting of complex structures or non-liner transformations. With the increase of the amount of data and the power of computation, neural networks with more complex structures have attracted widespread attention and been applied to various fields. This paper provides an overview of deep learning in neural networks including popular architecture models and training algorithms.

Download Full-text

Exploiting Heterogeneous Graph Neural Networks with Latent Worker/Task Correlation Information for Label Aggregation in Crowdsourcing

ACM Transactions on Knowledge Discovery from Data ◽

10.1145/3460865 ◽

2022 ◽

Vol 16 (2) ◽

pp. 1-18

Author(s):

Hanlu Wu ◽

Tengfei Ma ◽

Lingfei Wu ◽

Fangli Xu ◽

Shouling Ji

Keyword(s):

Neural Network ◽

Neural Networks ◽

State Of The Art ◽

Superior Performance ◽

True Label ◽

Label Aggregation ◽

Correlation Information ◽

Real World Datasets ◽

Graph Neural Networks ◽

High Level

Crowdsourcing has attracted much attention for its convenience to collect labels from non-expert workers instead of experts. However, due to the high level of noise from the non-experts, a label aggregation model that infers the true label from noisy crowdsourced labels is required. In this article, we propose a novel framework based on graph neural networks for aggregating crowd labels. We construct a heterogeneous graph between workers and tasks and derive a new graph neural network to learn the representations of nodes and the true labels. Besides, we exploit the unknown latent interaction between the same type of nodes (workers or tasks) by adding a homogeneous attention layer in the graph neural networks. Experimental results on 13 real-world datasets show superior performance over state-of-the-art models.

Download Full-text

Nanosecond Photodynamics Simulations of a Cis-Trans Isomerization Are Enabled by Machine Learning

10.26434/chemrxiv.13047863 ◽

2020 ◽

Author(s):

Jingbai Li ◽

Patrick Reiser ◽

André Eberhard ◽

Pascal Friederich ◽

Steven Lopez

Keyword(s):

Machine Learning ◽

Neural Networks ◽

Excited State ◽

Adaptive Sampling ◽

Computational Cost ◽

Ground Truth ◽

Absolute Error ◽

Photochemical Reactions ◽

Computational Techniques ◽

Full Potential

Photochemical reactions are being increasingly used to construct complex molecular architectures with mild and straightforward reaction conditions. Computational techniques are increasingly important to understand the reactivities and chemoselectivities of photochemical isomerization reactions because they offer molecular bonding information along the excited-state(s) of photodynamics. These photodynamics simulations are resource-intensive and are typically limited to 1–10 picoseconds and 1,000 trajectories due to high computational cost. Most organic photochemical reactions have excited-state lifetimes exceeding 1 picosecond, which places them outside possible computational studies. Westermeyr et al. demonstrated that a machine learning approach could significantly lengthen photodynamics simulation times for a model system, methylenimmonium cation (CH2NH2+).We have developed a Python-based code, Python Rapid Artificial Intelligence Ab Initio Molecular Dynamics (PyRAI2MD), to accomplish the unprecedented 10 ns cis-trans photodynamics of trans-hexafluoro-2-butene (CF3–CH=CH–CF3) in 3.5 days. The same simulation would take approximately 58 years with ground-truth multiconfigurational dynamics. We proposed an innovative scheme combining Wigner sampling, geometrical interpolations, and short-time quantum chemical trajectories to effectively sample the initial data, facilitating the adaptive sampling to generate an informative and data-efficient training set with 6,232 data points. Our neural networks achieved chemical accuracy (mean absolute error of 0.032 eV). Our 4,814 trajectories reproduced the S1 half-life (60.5 fs), the photochemical product ratio (trans: cis = 2.3: 1), and autonomously discovered a pathway towards a carbene. The neural networks have also shown the capability of generalizing the full potential energy surface with chemically incomplete data (trans → cis but not cis → trans pathways) that may offer future automated photochemical reaction discoveries.

Download Full-text

BAND NN: A Deep Learning Framework For Energy Prediction and Geometry Optimization of Organic Small Molecules

10.26434/chemrxiv.9763094 ◽

2019 ◽

Author(s):

Siddhartha Laghuvarapu ◽

Yashaswi Pathak ◽

U. Deva Priyakumar

Keyword(s):

Machine Learning ◽

Density Functional ◽

Computational Cost ◽

Geometry Optimization ◽

Dft Methods ◽

Energy Prediction ◽

Machine Learning Model ◽

Equilibrium Structures ◽

High Level ◽

Non Equilibrium

Recent advances in artificial intelligence along with development of large datasets of energies calculated using quantum mechanical (QM)/density functional theory (DFT) methods have enabled prediction of accurate molecular energies at reasonably low computational cost. However, machine learning models that have been reported so far requires the atomic positions obtained from geometry optimizations using high level QM/DFT methods as input in order to predict the energies, and do not allow for geometry optimization. In this paper, a transferable and molecule-size independent machine learning model (BAND NN) based on a chemically intuitive representation inspired by molecular mechanics force fields is presented. The model predicts the atomization energies of equilibrium and non-equilibrium structures as sum of energy contributions from bonds (B), angles (A), nonbonds (N) and dihedrals (D) at remarkable accuracy. The robustness of the proposed model is further validated by calculations that span over the conformational, configurational and reaction space. The transferability of this model on systems larger than the ones in the dataset is demonstrated by performing calculations on select large molecules. Importantly, employing the BAND NN model, it is possible to perform geometry optimizations starting from non-equilibrium structures along with predicting their energies.

Download Full-text

Neural networks approached for modelling river suspended sediment concentration due to tropical storms

Global NEST Journal ◽

10.30955/gnj.000628 ◽

2013 ◽

Vol 11 (4) ◽

pp. 457-466

Keyword(s):

Neural Network ◽

Neural Networks ◽

Suspended Sediment ◽

Suspended Sediment Concentration ◽

Time Series Data ◽

Water Discharge ◽

Sediment Concentration ◽

Series Data ◽

Generalized Regression Neural Network ◽

Event Based

Artificial neural networks are one of the advanced technologies employed in hydrology modelling. This paper investigates the potential of two algorithm networks, the feed forward backpropagation (BP) and generalized regression neural network (GRNN) in comparison with the classical regression for modelling the event-based suspended sediment concentration at Jiasian diversion weir in Southern Taiwan. For this study, the hourly time series data comprised of water discharge, turbidity and suspended sediment concentration during the storm events in the year of 2002 are taken into account in the models. The statistical performances comparison showed that both BP and GRNN are superior to the classical regression in the weir sediment modelling. Additionally, the turbidity was found to be a dominant input variable over the water discharge for suspended sediment concentration estimation. Statistically, both neural network models can be successfully applied for the event-based suspended sediment concentration modelling in the weir studied herein when few data are available.

Download Full-text

ФОРМИРОВАНИЕ ПРОФЕССИОНАЛЬНОЙ КУЛЬТУРЫ КУРСАНТОВ КАК НЕОБХОДИМОЕ УСЛОВИЕ СТАНОВЛЕНИЯ СОТРУДНИКА УИС

Vestnik Samarskogo iuridicheskogo instituta ◽

10.37523/sui.2019.36.5.019 ◽

2020 ◽

Author(s):

Alexander Votinov

Keyword(s):

Russian Federation ◽

Professional Culture ◽

Professional Activity ◽

Teaching Staff ◽

Successful Performance ◽

Current State ◽

Professional Self ◽

The Russian Federation ◽

Executive System ◽

High Level

Современное состояние и развитие уголовно-исполнительной системы Российской Федерации диктует необходимость овладения будущими специалистами комплексом определенных знаний, умений и навыков, позволяющих им эффективно решать служебные задачи. Одним из путей повышения профессионального уровня специалистов является формирование и развитие профессиональной культуры. Проведенный в статье анализ понятия «профессиональная культура» позволяет констатировать сложность его содержания, что связано с особенностями профессиональной деятельности сотрудников УИС, многообразием решаемых задач. Автором подробно исследуется процесс формирования профессиональной культуры в вузах ФСИН России, рассматриваются особенности работы в данном направлении профессорско-преподавательского состава, командиров строевых подразделений, сотрудников отделов по работе с личным составом, приводятся возникающие при этом проблемы и предлагаются возможные пути решения. Отмечается, что успешность формирования профессиональной культуры курсантов зависит от их профессионализма, дисциплинированности, инициативности, настойчивости и личного примера сотрудников. В заключение подчеркивается, что высокий уровень профессиональной культуры сотрудника УИС является условием успешной служебной деятельности и целью дальнейшего профессионального самосовершенствования.The current state and development of the criminal Executive system of the Russian Federation dictates the need for future specialists to master a set of certain knowledge, skills and abilities that allow them to solve official tasks effectively. One of the ways to improve the professional level of specialists is the formation and development of professional culture. The analysis of the concept of «professional culture» in the article allows us to state the complexity of its content, which is associated with the peculiarities of professional activity of employees of the UIS, the variety of tasks to be solved. The author studies in detail the process of formation of professional culture in the universities of the Federal penitentiary service of Russia, examines the features of work in this direction of the teaching staff, commanders of combat units, employees of departments for work with personnel, presents the problems arising in this case and suggests possible solutions. It is noted that the success of the formation of professional culture of cadets depends on their professionalism, discipline, initiative, perseverance and personal example. In conclusion, it is emphasized that the high level of professional culture of the employee is a condition of successful performance and the purpose of further professional self-improvement.

Download Full-text

TLBO-FLN: Teaching-Learning Based Optimization of Functional Link Neural Networks for Stock Closing Price Prediction

International Journal of Sensors Wireless Communications and Control ◽

10.2174/2210327909666191202113015 ◽

2020 ◽

Vol 10 (4) ◽

pp. 522-532 ◽

Cited By ~ 1

Author(s):

Sarat Chandra Nayak ◽

Subhranginee Das ◽

Mohammad Dilsad Ansari

Keyword(s):

Neural Networks ◽

Computational Cost ◽

Optimization Techniques ◽

Fine Tuning ◽

Functional Link ◽

Price Prediction ◽

Closing Price ◽

Teaching Learning Based Optimization ◽

Artificial Neural ◽

Teaching Learning

Background and Objective: Stock closing price prediction is enormously complicated. Artificial Neural Networks (ANN) are excellent approximation algorithms applied to this area. Several nature-inspired evolutionary optimization techniques are proposed and used in the literature to search the optimum parameters of ANN based forecasting models. However, most of them need fine-tuning of several control parameters as well as algorithm specific parameters to achieve optimal performance. Improper tuning of such parameters either leads toward additional computational cost or local optima. Methods: Teaching Learning Based Optimization (TLBO) is a newly proposed algorithm which does not necessitate any parameters specific to it. The intrinsic capability of Functional Link Artificial Neural Network (FLANN) to recognize the multifaceted nonlinear relationship present in the historical stock data made it popular and got wide applications in the stock market prediction. This article presents a hybrid model termed as Teaching Learning Based Optimization of Functional Neural Networks (TLBO-FLN) by combining the advantages of both TLBO and FLANN. Results and Conclusion: The model is evaluated by predicting the short, medium, and long-term closing prices of four emerging stock markets. The performance of the TLBO-FLN model is measured through Mean Absolute Percentage of Error (MAPE), Average Relative Variance (ARV), and coefficient of determination (R2); compared with that of few other state-of-the-art models similarly trained and found superior.

Download Full-text

Evaluation of Mixed Deep Neural Networks for Reverberant Speech Enhancement

Biomimetics ◽

10.3390/biomimetics5010001 ◽

2019 ◽

Vol 5 (1) ◽

pp. 1 ◽

Cited By ~ 1

Author(s):

Michelle Gutiérrez-Muñoz ◽

Astryd González-Salazar ◽

Marvin Coto-Jiménez

Keyword(s):

Neural Networks ◽

Short Term Memory ◽

Computational Cost ◽

Real Life ◽

Fixed Number ◽

Training Procedure ◽

Statistical Validation ◽

Significant Drop ◽

Training Time ◽

Important Solution

Speech signals are degraded in real-life environments, as a product of background noise or other factors. The processing of such signals for voice recognition and voice analysis systems presents important challenges. One of the conditions that make adverse quality difficult to handle in those systems is reverberation, produced by sound wave reflections that travel from the source to the microphone in multiple directions. To enhance signals in such adverse conditions, several deep learning-based methods have been proposed and proven to be effective. Recently, recurrent neural networks, especially those with long short-term memory (LSTM), have presented surprising results in tasks related to time-dependent processing of signals, such as speech. One of the most challenging aspects of LSTM networks is the high computational cost of the training procedure, which has limited extended experimentation in several cases. In this work, we present a proposal to evaluate the hybrid models of neural networks to learn different reverberation conditions without any previous information. The results show that some combinations of LSTM and perceptron layers produce good results in comparison to those from pure LSTM networks, given a fixed number of layers. The evaluation was made based on quality measurements of the signal’s spectrum, the training time of the networks, and statistical validation of results. In total, 120 artificial neural networks of eight different types were trained and compared. The results help to affirm the fact that hybrid networks represent an important solution for speech signal enhancement, given that reduction in training time is on the order of 30%, in processes that can normally take several days or weeks, depending on the amount of data. The results also present advantages in efficiency, but without a significant drop in quality.

Download Full-text