scholarly journals Evaluating the Impact of Optical Interconnects on a Multi-Chip Machine-Learning Architecture

Electronics ◽  
2018 ◽  
Vol 7 (8) ◽  
pp. 130 ◽  
Author(s):  
Yuhwan Ro ◽  
Eojin Lee ◽  
Jung Ahn
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Neural Networks ◽  
Optical Interconnects ◽  
Performance Model ◽  
Training Time ◽  
Performance Improvements ◽  
Cluster Architecture ◽  
The Neural Network ◽  
The Impact

Following trends that emphasize neural networks for machine learning, many studies regarding computing systems have focused on accelerating deep neural networks. These studies often propose utilizing the accelerator specialized in a neural network and the cluster architecture composed of interconnected accelerator chips. We observed that inter-accelerator communication within a cluster has a significant impact on the training time of the neural network. In this paper, we show the advantages of optical interconnects for multi-chip machine-learning architecture by demonstrating performance improvements through replacing electrical interconnects with optical ones in an existing multi-chip system. We propose to use highly practical optical interconnect implementation and devise an arithmetic performance model to fairly assess the impact of optical interconnects on a machine-learning accelerator platform. In our evaluation of nine Convolutional Neural Networks with various input sizes, 100 and 400 Gbps optical interconnects reduce the training time by an average of 20.6% and 35.6%, respectively, compared to the baseline system with 25.6 Gbps electrical ones.

scholarly journals Discretization and machine learning approximation of BSDEs with a constraint on the Gains-process

2021 ◽  
Vol 0 (0) ◽  
Author(s):  
Idris Kharroubi ◽  
Thomas Lim ◽  
Xavier Warin
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Neural Networks ◽  
Differential Equations ◽  
Numerical Experiments ◽  
Optimization Problem ◽  
Learning Approach ◽  
The Neural Network ◽  
Machine Learning Approach ◽  
Mesh Grid

AbstractWe study the approximation of backward stochastic differential equations (BSDEs for short) with a constraint on the gains process. We first discretize the constraint by applying a so-called facelift operator at times of a grid. We show that this discretely constrained BSDE converges to the continuously constrained one as the mesh grid converges to zero. We then focus on the approximation of the discretely constrained BSDE. For that we adopt a machine learning approach. We show that the facelift can be approximated by an optimization problem over a class of neural networks under constraints on the neural network and its derivative. We then derive an algorithm converging to the discretely constrained BSDE as the number of neurons goes to infinity. We end by numerical experiments.

Signal processing algorithm for neural networks with integrodifferential splines as an activation function and its particular case of image classification

2021 ◽  
Author(s):  
T.K. Biryukova
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Image Classification ◽  
Activation Function ◽  
Experimental Comparison ◽  
Training Time ◽  
Operation Speed ◽  
The Neural Network ◽  
Linear Algebraic Equations ◽  
Network Operation

Classic neural networks suppose trainable parameters to include just weights of neurons. This paper proposes parabolic integrodifferential splines (ID-splines), developed by author, as a new kind of activation function (AF) for neural networks, where ID-splines coefficients are also trainable parameters. Parameters of ID-spline AF together with weights of neurons are vary during the training in order to minimize the loss function thus reducing the training time and increasing the operation speed of the neural network. The newly developed algorithm enables software implementation of the ID-spline AF as a tool for neural networks construction, training and operation. It is proposed to use the same ID-spline AF for neurons in the same layer, but different for different layers. In this case, the parameters of the ID-spline AF for a particular layer change during the training process independently of the activation functions (AFs) of other network layers. In order to comply with the continuity condition for the derivative of the parabolic ID-spline on the interval (x x0, n) , its parameters fi (i= 0,...,n) should be calculated using the tridiagonal system of linear algebraic equations: To solve the system it is necessary to use two more equations arising from the boundary conditions for specific problems. For exam- ple the values of the grid function (if they are known) in the points (x x0, n) may be used for solving the system above: f f x0 = ( 0) , f f xn = ( n) . The parameters Iii+1 (i= 0,...,n−1 ) are used as trainable parameters of neural networks. The grid boundaries and spacing of the nodes of ID-spline AF are best chosen experimentally. The optimal selection of grid nodes allows improving the quality of results produced by the neural network. The formula for a parabolic ID-spline is such that the complexity of the calculations does not depend on whether the grid of nodes is uniform or non-uniform. An experimental comparison of the results of image classification from the popular FashionMNIST dataset by convolutional neural 0, x< 0 networks with the ID-spline AFs and the well-known ReLUx( ) =AF was carried out. The results reveal that the usage x x, ≥ 0 of the ID-spline AFs provides better accuracy of neural network operation than the ReLU AF. The training time for two convolutional layers network with two ID-spline AFs is just about 2 times longer than with two instances of ReLU AF. Doubling of the training time due to complexity of the ID-spline formula is the acceptable price for significantly better accuracy of the network. Wherein the difference of an operation speed of the networks with ID-spline and ReLU AFs will be negligible. The use of trainable ID-spline AFs makes it possible to simplify the architecture of neural networks without losing their efficiency. The modification of the well-known neural networks (ResNet etc.) by replacing traditional AFs with ID-spline AFs is a promising approach to increase the neural network operation accuracy. In a majority of cases, such a substitution does not require to train the network from scratch because it allows to use pre-trained on large datasets neuron weights supplied by standard software libraries for neural network construction thus substantially shortening training time.

XCSR Learning from Compressed Data Acquired by Deep Neural Network

2017 ◽  
Vol 21 (5) ◽  
pp. 856-867 ◽  
Author(s):  
Kazuma Matsumoto ◽  
Takato Tatsumi ◽  
Hiroyuki Sato ◽  
Tim Kovacs ◽  
Keiki Takadama ◽  
...  
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Neural Networks ◽  
Learning Classifier System ◽  
Rule Based ◽  
Classifier System ◽  
Learning Classifier ◽  
The Neural Network ◽  
Compressed Data

The correctness rate of classification of neural networks is improved by deep learning, which is machine learning of neural networks, and its accuracy is higher than the human brain in some fields. This paper proposes the hybrid system of the neural network and the Learning Classifier System (LCS). LCS is evolutionary rule-based machine learning using reinforcement learning. To increase the correctness rate of classification, we combine the neural network and the LCS. This paper conducted benchmark experiments to verify the proposed system. The experiment revealed that: 1) the correctness rate of classification of the proposed system is higher than the conventional LCS (XCSR) and normal neural network; and 2) the covering mechanism of XCSR raises the correctness rate of proposed system.

Bolt Tightening Using Impact Wrench Based on Neural Networks

2000 ◽  
Vol 12 (6) ◽  
pp. 706-711
Author(s):  
Toru Fujinaka ◽  
◽  
Hirofumi Nakano ◽  
Michifumi Yoshioka ◽  
Sigeru Omatu
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Clamping Force ◽  
The Neural Network ◽  
The Impact

A method for controlling the tightening operation of bolts using an impact wrench is proposed, where the neural network is employed for achieving proper clamping force. The characteristics of the clamping force depend on the kind of work to which bolts are tightened, thus a neural network is used for classification of the work. The clamping force, which can only be measured during the test run, is estimated online, using another neural network. Then appropriate input to the actuator of the impact wrench is determined, based on the estimated value of the clamping force.

scholarly journals Identification of Baikal phytoplankton inferred from computer vision methods and machine learning

2021 ◽  
pp. 1143-1146
Author(s):  
A.V. Lysenko ◽  
◽  
◽  
M.S. Oznobikhin ◽  
E.A. Kireev ◽  
...  
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Neural Networks ◽  
Computer Vision ◽  
Object Recognition ◽  
Light Microscope ◽  
Optimal Size ◽  
Optimal Parameters ◽  
The Neural Network

Abstract. This study discusses the problem of phytoplankton classification using computer vision methods and convolutional neural networks. We created a system for automatic object recognition consisting of two parts: analysis and primary processing of phytoplankton images and development of the neural network based on the obtained information about the images. We developed software that can detect particular objects in images from a light microscope. We trained a convolutional neural network in transfer learning and determined optimal parameters of this neural network and the optimal size of using dataset. To increase accuracy for these groups of classes, we created three neural networks with the same structure. The obtained accuracy in the classification of Baikal phytoplankton by these neural networks was up to 80%.

scholarly journals Multi-expert annotation of Crohn’s disease images of the small bowel for automatic detection using a convolutional recurrent attention neural network

2021 ◽  
Vol 09 (07) ◽  
pp. E1136-E1144
Author(s):  
Astrid de Maissin ◽  
Remi Vallée ◽  
Mathurin Flamant ◽  
Marie Fondain-Bossiere ◽  
Catherine Le Berre ◽  
...  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Crohn’S Disease ◽  
Crohn's Disease ◽  
Diagnostic Tools ◽  
The Neural Network ◽  
Before And After ◽  
Expert Annotation ◽  
The Impact

Abstract Background and study aims Computer-aided diagnostic tools using deep neural networks are efficient for detection of lesions in endoscopy but require a huge number of images. The impact of the quality of annotation has not been tested yet. Here we describe a multi-expert annotated dataset of images extracted from capsules from Crohn’s disease patients and the impact of the quality of annotations on the accuracy of a recurrent attention neural network. Methods Images of capsule were annotated by a reader first and then reviewed by three experts in inflammatory bowel disease. Concordance analysis between experts was evaluated by Fleiss’ kappa and all the discordant images were, again, read by all the endoscopists to obtain a consensus annotation. A recurrent attention neural network developed for the study was tested before and after the consensus annotation. Available neural networks (ResNet and VGGNet) were also tested under the same conditions. Results The final dataset included 3498 images with 2124 non-pathological (60.7 %), 1360 pathological (38.9 %), and 14 (0.4 %) inconclusive. Agreement of the experts was good for distinguishing pathological and non-pathological images with a kappa of 0.79 (P < 0.0001). The accuracy of our classifier and the available neural networks increased after the consensus annotation with a precision of 93.7 %, sensitivity of 93 %, and specificity of 95 %. Conclusions The accuracy of the neural network increased with improved annotations, suggesting that the number of images needed for the development of these systems could be diminished using a well-designed dataset.

scholarly journals Toward accurate platform-aware performance modeling for deep neural networks

2021 ◽  
Vol 21 (1) ◽  
pp. 50-61
Author(s):  
Chuan-Chi Wang ◽  
Ying-Chiao Liao ◽  
Ming-Chang Kao ◽  
Wen-Yew Liang ◽  
Shih-Hao Hung
Keyword(s):  
Machine Learning ◽  
Neural Networks ◽  
Network Performance ◽  
Percentage Error ◽  
Training Time ◽  
Fine Grain ◽  
The Neural Network ◽  
Machine Learning Model ◽  
The Neural Networks ◽  
System Developer

In this paper, we provide a fine-grain machine learning-based method, PerfNetV2, which improves the accuracy of our previous work for modeling the neural network performance on a variety of GPU accelerators. Given an application, the proposed method can be used to predict the inference time and training time of the convolutional neural networks used in the application, which enables the system developer to optimize the performance by choosing the neural networks and/or incorporating the hardware accelerators to deliver satisfactory results in time. Furthermore, the proposed method is capable of predicting the performance of an unseen or non-existing device, e.g. a new GPU which has a higher operating frequency with less processor cores, but more memory capacity. This allows a system developer to quickly search the hardware design space and/or fine-tune the system configuration. Compared to the previous works, PerfNetV2 delivers more accurate results by modeling detailed host-accelerator interactions in executing the full neural networks and improving the architecture of the machine learning model used in the predictor. Our case studies show that PerfNetV2 yields a mean absolute percentage error within 13.1% on LeNet, AlexNet, and VGG16 on NVIDIA GTX-1080Ti, while the error rate on a previous work published in ICBD 2018 could be as large as 200%.

scholarly journals Study of Convolutional Neural Networks applied to Image Stereo Matching

2020 ◽  
Author(s):  
João Pedro Poloni Ponce ◽  
Ricardo Suyama
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Convolutional Neural Networks ◽  
Error Rate ◽  
Stereo Matching ◽  
Structural Parameters ◽  
Neural Model ◽  
Temporal Difference ◽  
Training Time ◽  
The Neural Network

Stereo images are images formed from two or more sources that capture the same scene so that it is possible to infer the depth of the scene under analysis. The use of convolutional neural networks to compute these images has been shown to be a viable alternative due to its speed in finding the correspondence between the images. This raises questions related to the influence of structural parameters, such as size of kernel, stride and pooling policy on the performance of the neural network. To this end, this work sought to reproduce an article that deals with the topic and to explore the influence of the parameters mentioned above in function of the results of error rate and losses of the neural model. The results obtained reveal improvements. The influence of the parameters on the training time of the models was also notable, even using the GPU, the temporal difference in the training period between the maximum and minimum limits reached a ratio of six times.

scholarly journals A Study on the Effect of DropConnect to Control Overfitting in Designing Neural Networks

2020 ◽  
Author(s):  
Hyun-il Lim
Keyword(s):  
Neural Network ◽  
Machine Learning ◽  
Neural Networks ◽  
Neural Network Model ◽  
Network Models ◽  
Training Data ◽  
Neural Network Models ◽  
Structural Problems ◽  
The Neural Network ◽  
Training Neural Network

The neural network is an approach of machine learning by training the connected nodes of a model to predict the results of specific problems. The prediction model is trained by using previously collected training data. In training neural network models, overfitting problems can occur from the excessively dependent training of data and the structural problems of the models. In this paper, we analyze the effect of DropConnect for controlling overfitting in neural networks. It is analyzed according to the DropConnect rates and the number of nodes in designing neural networks. The analysis results of this study help to understand the effect of DropConnect in neural networks. To design an effective neural network model, the DropConnect can be applied with appropriate parameters from the understanding of the effect of the DropConnect in neural network models.

SYNTHESIS OF NEURAL NETWORK BASED ON PETRY NET USING AS EXAMPLE THE PROBLEM OF MOVEMENT AND STABILIZATION OF STRUCTURE GROUPS OF UNMANNED AERIAL VEHICLES

2018 ◽  
pp. 26-33
Author(s):  
Dmitry Olegovich Romannikov ◽  
Alexander Aleksandrovich Voevoda
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Petri Nets ◽  
Unmanned Aerial Vehicles ◽  
Training Time ◽  
Initial Values ◽  
Network Training ◽  
Aerial Vehicles ◽  
The Neural Network ◽  
Parameter Values

The article focuses on the approach to forming the structure of a neural network with application of a pre-built algorithm using Petri nets which represent a well-characterized body of mathematics and help to describe algorithms, in particular, distributed asynchronous systems. According to the proposed approach, the model built in Petri nets serves as the basis for further developing the neural network. There was proposed the idea of informal transformation, which makes sense because the structure of Petri net provides substantiation for the structure of the neural network. This fact leads to decreasing the number of training parameters in the neural network (in the example described in the article the decrease was more than twice: from 650 to 254), increasing the time of the network training and getting initial values for the training parameters. It has been stated that with the initial values obtained the training time grows even more and, thus, training process acts as fine-adjusting values of parameters. Transformation can be explained by the fact that both Petri nets and neural networks act as languages for describing functions, and differ only in the case of neural networks, where the presented function must be trained first (or to find parameter values). The above-mentioned approach is illustrated by the example of the problem of automatic formation of a group of unmanned aerial vehicles (UAV) and their movement. In this problem, identical instances of the neural network are located on each UAV and interact in asynchronous mode.

Sign in / Sign up

Export Citation Format

Share Document