When Massive GPU Parallelism Ain’t Enough: A Novel Hardware Architecture of 2D-LSTM Neural Network

Multidimensional Long Short-Term Memory (MD-LSTM) neural network is an extension of one-dimensional LSTM for data with more than one dimension. MD-LSTM achieves state-of-the-art results in various applications, including handwritten text recognition, medical imaging, and many more. However, its implementation suffers from the inherently sequential execution that tremendously slows down both training and inference compared to other neural networks. The main goal of the current research is to provide acceleration for inference of MD-LSTM. We advocate that Field-Programmable Gate Array (FPGA) is an alternative platform for deep learning that can offer a solution when the massive parallelism of GPUs does not provide the necessary performance required by the application. In this article, we present the first hardware architecture for MD-LSTM. We conduct a systematic exploration to analyze a tradeoff between precision and accuracy. We use a challenging dataset for semantic segmentation, namely historical document image binarization from the DIBCO 2017 contest and a well-known MNIST dataset for handwritten digit recognition. Based on our new architecture, we implement FPGA-based accelerators that outperform Nvidia Geforce RTX 2080 Ti with respect to throughput by up to 9.9 and Nvidia Jetson AGX Xavier with respect to energy efficiency by up to 48 . Our accelerators achieve higher throughput, energy efficiency, and resource efficiency than FPGA-based implementations of convolutional neural networks (CNNs) for semantic segmentation tasks. For the handwritten digit recognition task, our FPGA implementations provide higher accuracy and can be considered as a solution when accuracy is a priority. Furthermore, they outperform earlier FPGA implementations of one-dimensional LSTMs with respect to throughput, energy efficiency, and resource efficiency.

Optical neuromorphic processing with soliton crystal Kerr microcombs: Scaling the network in size and speed

Convolutional neural networks (CNNs), inspired by biological visual cortex systems, are a powerful category of artificial neural networks that can extract the hierarchical features of raw data to greatly reduce the network parametric complexity and enhance the predicting accuracy. They are of significant interest for machine learning tasks such as computer vision, speech recognition, playing board games and medical diagnosis [1-7]. Optical neural networks offer the promise of dramatically accelerating computing speed to overcome the inherent bandwidth bottleneck of electronics. Here, we demonstrate a universal optical vector convolutional accelerator operating beyond 10 Tera-OPS (TOPS - operations per second), generating convolutions of images of 250,000 pixels with 8-bit resolution for 10 kernels simultaneously — enough for facial image recognition. We then use the same hardware to sequentially form a deep optical CNN with ten output neurons, achieving successful recognition of full 10 digits with 900 pixel handwritten digit images with 88% accuracy. Our results are based on simultaneously interleaving temporal, wavelength and spatial dimensions enabled by an integrated microcomb source. We show that this approach is scalable and trainable to much more complex networks for demanding applications such as unmanned vehicle and real-time video recognition.Keywords: Optical neural networks, neuromorphic processor, microcomb, convolutional accelerator

An optical neural network using less than 1 photon per multiplication

Deep learning has become a widespread tool in both science and industry. However, continued progress is hampered by the rapid growth in energy costs of ever-larger deep neural networks. Optical neural networks provide a potential means to solve the energy-cost problem faced by deep learning. Here, we experimentally demonstrate an optical neural network based on optical dot products that achieves 99% accuracy on handwritten-digit classification using ~3.1 detected photons per weight multiplication and ~90% accuracy using ~0.66 photons (~2.5 × 10−19 J of optical energy) per weight multiplication. The fundamental principle enabling our sub-photon-per-multiplication demonstration—noise reduction from the accumulation of scalar multiplications in dot-product sums—is applicable to many different optical-neural-network architectures. Our work shows that optical neural networks can achieve accurate results using extremely low optical energies.

Comparison of classical machine learning algorithms in the task of handwritten digits classification

The purpose of this paper is to compare classical machine learning algorithms for handwritten number classification. The following algorithms were chosen for comparison: Logistic Regression, SVM, Decision Tree, Random Forest and k-NN. MNIST handwritten digit database is used in the task of training and testing the above algorithms. The dataset consists of 70,000 images of numbers from 0 to 9. The algorithms are compared considering such criteria as the learning speed, prediction construction speed, host machine load, and classification accuracy. Each algorithm went through the training and testing phases 100 times, with the desired KPIs retained at each iteration. The results were averaged to reach reliable outcomes.

An Efficient Handwritten Digit Recognition based on Convolutional Neural Networks with Orthogonal Learning Strategies

The predictions of characters/text/digits from the handwritten images have made the research community spotlight towards recognition. There are enormous applications and ambiguity that made prediction possible with Deep Learning (DL) approaches. Primarily, there are four necessary steps to be carried out with handwriting prediction. First, consideration of a dataset that is more appropriate for DL validation an inefficient manner. Here, Special Database 1 and Special Database 2 are used, which are combined and modified by the National Institute of Standards and Technology (NIST). Next is pre-processing of input handwritten digit recognition data by data normalization, extraction of efficient features which provides better prediction accuracy. The proposed idea uses pixel values as features with the analysis of hyper-parameters to enhance near-human performance. With SVM, non-linear and linear models are built to extract the appropriate features for further processing. The features are separate and placed over the Bag of Features (BoF), which is used by the next processing stage. Finally, a novel Convolutional Neural Network (CNN) is by built modifying the network structure with Orthogonal Learning Particle Swarm Optimization (CNN-OLPSO). This modification is adopted for evolutionarily optimizing the number of hyper-parameters. This proposed optimizer predicts the optimal values from the fitness computation and shows better efficiency when compared to various other conventional approaches. The novelty which relies on CNN adoption is to endeavor a suitable path towards digitalization and preserve the handwritten structure and help automatic feature extraction using CNN by offering better computation accuracy. The optimization approach helps to avoid over-fitting and under-fitting issues. Here, metrics like accuracy, elapsed time, recall, precision, and [Formula: see text]-measure are evaluated. The results of CNN-OLPSO give better accuracy, reduced error rate and better execution time (s) compared to other existing methods. Thus, the proposed model shows better tradeoff in the recognition rate of handwritten digits.

Digit recognition using decimal coding and artificial neural network

Current artificial neural network image recognition techniques use all the pixels of an image as input. In this paper, we present an efficient method for handwritten digit recognition that involves extracting the characteristics of a digit image by coding each row of the image as a decimal value, i.e., by transforming the binary representation into a decimal value. This method is called the decimal coding of rows. The set of decimal values calculated from the initial image is arranged as a vector and normalized; these values represent the inputs to the artificial neural network. The approach proposed in this work uses a multilayer perceptron neural network for the classification, recognition, and prediction of handwritten digits from 0 to 9. In this study, a dataset of 1797 samples were obtained from a digit database imported from the Scikit-learn library. Backpropagation was used as a learning algorithm to train the multilayer perceptron neural network. The results show that the proposed approach achieves better performance than two other schemes in terms of recognition accuracy and execution time.

Research on Mnist Handwritten Numbers Recognition based on CNN

In view of the increasing demand for handwritten digit recognition, a handwritten digit recognition model based on convolutional neural network is proposed. The model includes 1 input layer and 2 convolutional layers (5*5 convolution Core), 2 pooling layers (2*2 pooling core), 1 fully connected layer, 1 output layer, and use the mnist data set for model training and prediction. After a lot of training and participation, the accuracy rate of the training set was finally reached to 100%, and the accuracy rate of 99.25% was also achieved on the test set, which can meet the requirements of recognizing handwritten digits.