scholarly journals Binary Quantization Analysis of Neural Networks Weights on MNIST Dataset

Author(s):  
Zoran H. Peric ◽  
Bojan D. Denic ◽  
Milan S. Savic ◽  
Nikola J. Vucic ◽  
Nikola B. Simic

This paper considers the design of a binary scalar quantizer of Laplacian source and its application in compressed neural networks. The quantizer performance is investigated in a wide dynamic range of data variances, and for that purpose, we derive novel closed-form expressions. Moreover, we propose two selection criteria for the variance range of interest. Binary quantizers are further implemented for compressing neural network weights and its performance is analysed for a simple classification task. Good matching between theory and experiment is observed and a great possibility for implementation is indicated.

2021 ◽  
Vol 4 (1) ◽  
pp. 71-79
Author(s):  
Borys Igorovych Tymchenko

Nowadays, means of preventive management in various spheres of human life are actively developing. The task of automated screening is to detect hidden problems at an early stage without human intervention, while the cost of responding to them is low. Visual inspection is often used to perform a screening task. Deep artificial neural networks are especially popular in image processing. One of the main problems when working with them is the need for a large amount of well-labeled data for training. In automated screening systems, available neural network approaches have limitations on the reliability of predictions due to the lack of accurately marked training data, as obtaining quality markup from professionals is very expensive, and sometimes not possible in principle. Therefore, there is a contradiction between increasing the requirements for the precision of predictions of neural network models without increasing the time spent on the one hand, and the need to reduce the cost of obtaining the markup of educational data. In this paper, we propose the parametric model of the segmentation dataset, which can be used to generate training data for model selection and benchmarking; and the multi-task learning method for training and inference of deep neural networks for semantic segmentation. Based on the proposed method, we develop a semi-supervised approach for segmentation of salient regions for classification task. The main advantage of the proposed method is that it uses semantically-similar general tasks, that have better labeling than original one, what allows users to reduce the cost of the labeling process. We propose to use classification task as a more general to the problem of semantic segmentation. As semantic segmentation aims to classify each pixel in the input image, classification aims to assign a class to all of the pixels in the input image. We evaluate our methods using the proposed dataset model, observing the Dice score improvement by seventeen percent. Additionally, we evaluate the robustness of the proposed method to different amount of the noise in labels and observe consistent improvement over baseline version.


Author(s):  
Laurin Luttmann ◽  
Paolo Mercorelli

This work describes and compares the backpropagation algorithm with the Extended Kalman filter, a second-order training method which can be applied to the problem of learning neural network parameters and is known to converge in only a few iterations. The algorithms are compared with respect to their effectiveness and speed of convergence using simulated data for both, a regression and a classification task.


1994 ◽  
Vol 04 (02n03) ◽  
pp. 131-136 ◽  
Author(s):  
S. IWASAKI ◽  
S.H. FUKUDA ◽  
K. YOSHIZAKI ◽  
M. KITAMURA ◽  
K. ISHII

A new technique of spectrum analysis for PIXE based on the linear associative neural network has been developed. The network has high capability in the association of almost all elements across the periodic table in samples, in the quantification with wide dynamic range of the concentrations, and in the analysis speed.


Author(s):  
P. Chintrakulchai ◽  
N. Nintunze ◽  
A. Wu ◽  
J. Meador

2011 ◽  
Vol 63-64 ◽  
pp. 205-208
Author(s):  
Yi Ding ◽  
Tian Jiang Wang ◽  
Xian Fu

The penalty functions are introduced in the negative correlation learning for finding a neural network in an ensemble. It is based on the average output of the ensemble. The idea of penalty function based on the average output is to make each individual network has the different output value to that of the ensemble on the same input. Experiments on a classification task show how the negative correlation learning generates a neural network with penalty functions.


Entropy ◽  
2021 ◽  
Vol 23 (12) ◽  
pp. 1699
Author(s):  
Jelena Nikolić ◽  
Zoran Perić ◽  
Danijela Aleksić ◽  
Stefan Tomić ◽  
Aleksandra Jovanović

Driven by the need for the compression of weights in neural networks (NNs), which is especially beneficial for edge devices with a constrained resource, and by the need to utilize the simplest possible quantization model, in this paper, we study the performance of three-bit post-training uniform quantization. The goal is to put various choices of the key parameter of the quantizer in question (support region threshold) in one place and provide a detailed overview of this choice’s impact on the performance of post-training quantization for the MNIST dataset. Specifically, we analyze whether it is possible to preserve the accuracy of the two NN models (MLP and CNN) to a great extent with the very simple three-bit uniform quantizer, regardless of the choice of the key parameter. Moreover, our goal is to answer the question of whether it is of the utmost importance in post-training three-bit uniform quantization, as it is in quantization, to determine the optimal support region threshold value of the quantizer to achieve some predefined accuracy of the quantized neural network (QNN). The results show that the choice of the support region threshold value of the three-bit uniform quantizer does not have such a strong impact on the accuracy of the QNNs, which is not the case with two-bit uniform post-training quantization, when applied in MLP for the same classification task. Accordingly, one can anticipate that due to this special property, the post-training quantization model in question can be greatly exploited.


2021 ◽  
Vol 50 (3) ◽  
pp. 558-569
Author(s):  
Zoran Peric ◽  
Bojan Denic ◽  
Milan Savic ◽  
Milan Dincic ◽  
Darko Mihajlov

Quantization and compression of neural network parameters using the uniform scalar quantization is carried out in this paper. The attractiveness of the uniform scalar quantizer is reflected in a low complexity and relatively good performance, making it the most popular quantization model. We present a design approach for the memoryless Laplacian source with zero-mean and unit variance, which is based on iterative rule and uses the minimal mean-squared error distortion as a performance criterion. In addition, we derive closed-form expressions for SQNR (Signal to Quantization Noise Ratio) in a wide dynamic range of variance of input data. To show effectiveness on real data, the proposed quantizer is used to compress the weights of neural networks using bit rates from 9 to 16 bps (bits/sample) instead of standardly used 32 bps full precision bit rate. The impact of weights compression on the NN (neural network) performance is analyzed, indicating good matching with the theoretical results and showing negligible decreasing of the prediction accuracy of the NN even in the case of high variance-mismatch between the variance of NN weights and the variance used for the design of quantizer, if the value of the bit-rate is properly chosen according to the rule proposed in the paper.


Author(s):  
F. Ouyang ◽  
D. A. Ray ◽  
O. L. Krivanek

Electron backscattering Kikuchi diffraction patterns (BKDP) reveal useful information about the structure and orientation of crystals under study. With the well focused electron beam in a scanning electron microscope (SEM), one can use BKDP as a microanalysis tool. BKDPs have been recorded in SEMs using a phosphor screen coupled to an intensified TV camera through a lens system, and by photographic negatives. With the development of fiber-optically coupled slow scan CCD (SSC) cameras for electron beam imaging, one can take advantage of their high sensitivity and wide dynamic range for observing BKDP in SEM.We have used the Gatan 690 SSC camera to observe backscattering patterns in a JEOL JSM-840A SEM. The CCD sensor has an active area of 13.25 mm × 8.83 mm and 576 × 384 pixels. The camera head, which consists of a single crystal YAG scintillator fiber optically coupled to the CCD chip, is located inside the SEM specimen chamber. The whole camera head is cooled to about -30°C by a Peltier cooler, which permits long integration times (up to 100 seconds).


2020 ◽  
Vol 2020 (10) ◽  
pp. 54-62
Author(s):  
Oleksii VASYLIEV ◽  

The problem of applying neural networks to calculate ratings used in banking in the decision-making process on granting or not granting loans to borrowers is considered. The task is to determine the rating function of the borrower based on a set of statistical data on the effectiveness of loans provided by the bank. When constructing a regression model to calculate the rating function, it is necessary to know its general form. If so, the task is to calculate the parameters that are included in the expression for the rating function. In contrast to this approach, in the case of using neural networks, there is no need to specify the general form for the rating function. Instead, certain neural network architecture is chosen and parameters are calculated for it on the basis of statistical data. Importantly, the same neural network architecture can be used to process different sets of statistical data. The disadvantages of using neural networks include the need to calculate a large number of parameters. There is also no universal algorithm that would determine the optimal neural network architecture. As an example of the use of neural networks to determine the borrower's rating, a model system is considered, in which the borrower's rating is determined by a known non-analytical rating function. A neural network with two inner layers, which contain, respectively, three and two neurons and have a sigmoid activation function, is used for modeling. It is shown that the use of the neural network allows restoring the borrower's rating function with quite acceptable accuracy.


2019 ◽  
Vol 2019 (1) ◽  
pp. 153-158
Author(s):  
Lindsay MacDonald

We investigated how well a multilayer neural network could implement the mapping between two trichromatic color spaces, specifically from camera R,G,B to tristimulus X,Y,Z. For training the network, a set of 800,000 synthetic reflectance spectra was generated. For testing the network, a set of 8,714 real reflectance spectra was collated from instrumental measurements on textiles, paints and natural materials. Various network architectures were tested, with both linear and sigmoidal activations. Results show that over 85% of all test samples had color errors of less than 1.0 ΔE2000 units, much more accurate than could be achieved by regression.


Sign in / Sign up

Export Citation Format

Share Document