scholarly journals Whether the Support Region of Three-Bit Uniform Quantizer Has a Strong Impact on Post-Training Quantization for MNIST Dataset?

Entropy ◽  
2021 ◽  
Vol 23 (12) ◽  
pp. 1699
Author(s):  
Jelena Nikolić ◽  
Zoran Perić ◽  
Danijela Aleksić ◽  
Stefan Tomić ◽  
Aleksandra Jovanović
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Special Property ◽  
Threshold Value ◽  
Classification Task ◽  
Strong Impact ◽  
Uniform Quantization ◽  
Key Parameter

Driven by the need for the compression of weights in neural networks (NNs), which is especially beneficial for edge devices with a constrained resource, and by the need to utilize the simplest possible quantization model, in this paper, we study the performance of three-bit post-training uniform quantization. The goal is to put various choices of the key parameter of the quantizer in question (support region threshold) in one place and provide a detailed overview of this choice’s impact on the performance of post-training quantization for the MNIST dataset. Specifically, we analyze whether it is possible to preserve the accuracy of the two NN models (MLP and CNN) to a great extent with the very simple three-bit uniform quantizer, regardless of the choice of the key parameter. Moreover, our goal is to answer the question of whether it is of the utmost importance in post-training three-bit uniform quantization, as it is in quantization, to determine the optimal support region threshold value of the quantizer to achieve some predefined accuracy of the quantized neural network (QNN). The results show that the choice of the support region threshold value of the three-bit uniform quantizer does not have such a strong impact on the accuracy of the QNNs, which is not the case with two-bit uniform post-training quantization, when applied in MLP for the same classification task. Accordingly, one can anticipate that due to this special property, the post-training quantization model in question can be greatly exploited.

NEURAL NETWORK METHODS FOR PLANAR IMAGE ANALYSIS IN AUTOMATED SCREENING SYSTEMS

2021 ◽  
Vol 4 (1) ◽  
pp. 71-79
Author(s):  
Borys Igorovych Tymchenko
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Early Stage ◽  
Semantic Segmentation ◽  
Input Image ◽  
Training Data ◽  
Classification Task ◽  
Automated Screening ◽  
The Cost ◽  
Screening Systems

Nowadays, means of preventive management in various spheres of human life are actively developing. The task of automated screening is to detect hidden problems at an early stage without human intervention, while the cost of responding to them is low. Visual inspection is often used to perform a screening task. Deep artificial neural networks are especially popular in image processing. One of the main problems when working with them is the need for a large amount of well-labeled data for training. In automated screening systems, available neural network approaches have limitations on the reliability of predictions due to the lack of accurately marked training data, as obtaining quality markup from professionals is very expensive, and sometimes not possible in principle. Therefore, there is a contradiction between increasing the requirements for the precision of predictions of neural network models without increasing the time spent on the one hand, and the need to reduce the cost of obtaining the markup of educational data. In this paper, we propose the parametric model of the segmentation dataset, which can be used to generate training data for model selection and benchmarking; and the multi-task learning method for training and inference of deep neural networks for semantic segmentation. Based on the proposed method, we develop a semi-supervised approach for segmentation of salient regions for classification task. The main advantage of the proposed method is that it uses semantically-similar general tasks, that have better labeling than original one, what allows users to reduce the cost of the labeling process. We propose to use classification task as a more general to the problem of semantic segmentation. As semantic segmentation aims to classify each pixel in the input image, classification aims to assign a class to all of the pixels in the input image. We evaluate our methods using the proposed dataset model, observing the Dice score improvement by seventeen percent. Additionally, we evaluate the robustness of the proposed method to different amount of the noise in labels and observe consistent improvement over baseline version.

scholarly journals MASKS: A Multi-Artificial Neural Networks System's verification approach

2020 ◽  
Author(s):  
Amirhoshang Hoseinpour Dehkordi ◽  
Majid Alizadeh ◽  
Ebrahim Ardeshir-Larijani ◽  
Ali Movaghar
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Artificial Neural Network ◽  
Artificial Neural Networks ◽  
Special Property ◽  
External Information ◽  
Multi Agent Systems ◽  
Classification Problems ◽  
Multi Agent ◽  
Artificial Neural

<div>Artificial Neural networks are one of the most widely applied approaches for classification problems. However, developing an errorless artificial neural network is in practice impossible, due to the statistical nature of such networks. The employment of artificial neural networks in critical applications has rendered any such emerging errors, in these systems, incredibly more significant. Nevertheless, the real consequences of such errors have not been addressed, especially due to lacking verification approaches. This study aims to develop a verification method that eliminates errors through the integration of multiple artificial neural networks. In order to do this, first of all, a special property has been defined, by the authors, to extract the knowledge of these artificial neural networks. </div><div>Furthermore, a multi-agent system has been designed, itself comprised of multiple artificial neural networks, in order to check whether the aforementioned special property has been satisfied, or not. Also, in order to help examine the reasoning concerning the aggregation of the distributed knowledge, itself gained through the combined effort of separate artificial neural networks and acquired external information sources, a dynamic epistemic logic-based method has been proposed.</div><div>Finally, we believe aggregated knowledge may lead to self-awareness for the system. As a result, our model shall be capable of verifying specific inputs, if the cumulative knowledge of the entire system proves its correctness. </div><div>In conclusion, and formulated for multi-agent systems, a knowledge-sharing algorithm (Abbr. MASKS) has been developed. Which after being applied on the MNIST dataset successfully reduced the error rate to roughly one-eighth of previous runs on individual artificial neural network in the same model. </div>

scholarly journals Comparison of Backpropagation and Kalman Filter-based Training for Neural Networks

2021 ◽  
Author(s):  
Laurin Luttmann ◽  
Paolo Mercorelli
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Kalman Filter ◽  
Extended Kalman Filter ◽  
Simulated Data ◽  
Second Order ◽  
Classification Task ◽  
Training Method ◽  
Speed Of Convergence ◽  
Backpropagation Algorithm

This work describes and compares the backpropagation algorithm with the Extended Kalman filter, a second-order training method which can be applied to the problem of learning neural network parameters and is known to converge in only a few iterations. The algorithms are compared with respect to their effectiveness and speed of convergence using simulated data for both, a regression and a classification task.

scholarly journals Binary Quantization Analysis of Neural Networks Weights on MNIST Dataset

2021 ◽  
Author(s):  
Zoran H. Peric ◽  
Bojan D. Denic ◽  
Milan S. Savic ◽  
Nikola J. Vucic ◽  
Nikola B. Simic
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Closed Form ◽  
Selection Criteria ◽  
Dynamic Range ◽  
Classification Task ◽  
Wide Dynamic Range ◽  
Simple Classification ◽  
Great Possibility ◽  
Theory And Experiment

This paper considers the design of a binary scalar quantizer of Laplacian source and its application in compressed neural networks. The quantizer performance is investigated in a wide dynamic range of data variances, and for that purpose, we derive novel closed-form expressions. Moreover, we propose two selection criteria for the variance range of interest. Binary quantizers are further implemented for compressing neural network weights and its performance is analysed for a simple classification task. Good matching between theory and experiment is observed and a great possibility for implementation is indicated.

The Research of Penalty Functions Based on Neural Networks

2011 ◽  
Vol 63-64 ◽  
pp. 205-208
Author(s):  
Yi Ding ◽  
Tian Jiang Wang ◽  
Xian Fu
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Negative Correlation ◽  
Penalty Function ◽  
Penalty Functions ◽  
Classification Task ◽  
Average Output ◽  
Negative Correlation Learning ◽  
Correlation Learning

The penalty functions are introduced in the negative correlation learning for finding a neural network in an ensemble. It is based on the average output of the ensemble. The idea of penalty function based on the average output is to make each individual network has the different output value to that of the ensemble on the same input. Experiments on a classification task show how the negative correlation learning generates a neural network with penalty functions.

scholarly journals MASKS: A Multi-Artificial Neural Networks System's verification approach

2020 ◽  
Author(s):  
Amirhoshang Hoseinpour Dehkordi ◽  
Majid Alizadeh ◽  
Ebrahim Ardeshir-Larijani ◽  
Ali Movaghar
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Artificial Neural Network ◽  
Artificial Neural Networks ◽  
Special Property ◽  
External Information ◽  
Multi Agent Systems ◽  
Classification Problems ◽  
Multi Agent ◽  
Artificial Neural

<div>Artificial Neural networks are one of the most widely applied approaches for classification problems. However, developing an errorless artificial neural network is in practice impossible, due to the statistical nature of such networks. The employment of artificial neural networks in critical applications has rendered any such emerging errors, in these systems, incredibly more significant. Nevertheless, the real consequences of such errors have not been addressed, especially due to lacking verification approaches. This study aims to develop a verification method that eliminates errors through the integration of multiple artificial neural networks. In order to do this, first of all, a special property has been defined, by the authors, to extract the knowledge of these artificial neural networks. </div><div>Furthermore, a multi-agent system has been designed, itself comprised of multiple artificial neural networks, in order to check whether the aforementioned special property has been satisfied, or not. Also, in order to help examine the reasoning concerning the aggregation of the distributed knowledge, itself gained through the combined effort of separate artificial neural networks and acquired external information sources, a dynamic epistemic logic-based method has been proposed.</div><div>Finally, we believe aggregated knowledge may lead to self-awareness for the system. As a result, our model shall be capable of verifying specific inputs, if the cumulative knowledge of the entire system proves its correctness. </div><div>In conclusion, and formulated for multi-agent systems, a knowledge-sharing algorithm (Abbr. MASKS) has been developed. Which after being applied on the MNIST dataset successfully reduced the error rate to roughly one-eighth of previous runs on individual artificial neural network in the same model. </div>

scholarly journals Quantization noise in low bit quantization and iterative adaptation to quantization noise in quantizable neural networks

2021 ◽  
Vol 2134 (1) ◽  
pp. 012004
Author(s):  
D Chudakov ◽  
A Goncharenko ◽  
S Alyamkin ◽  
A Densidov
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Quantization Noise ◽  
Uniform Quantization ◽  
Resource Requirements ◽  
The Moment ◽  
Iterative Adaptation

Abstract Quantization is one of the most popular and widely used methods of speeding up a neural network. At the moment, the standard is 8-bit uniform quantization. Nevertheless, the use of uniform low-bit quantization (4- and 6-bit quantization) has significant advantages in speed and resource requirements for inference. We present our quantization algorithm that offers advantages when using uniform low-bit quantization. It is faster than quantization-aware training from scratch and more accurate than methods aimed only at selecting thresholds and reducing noise from quantization. We also investigated quantization noise in neural networks for low-bit quantization and concluded that quantization noise is not always a good metric for quantization quality.

SCORING MODELING BASED ON NEURAL NETWORKS FOR DETERMINING A BANK BORROWER'S RATING

2020 ◽  
Vol 2020 (10) ◽  
pp. 54-62
Author(s):  
Oleksii VASYLIEV ◽  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Network Architecture ◽  
Statistical Data ◽  
Activation Function ◽  
Decision Making Process ◽  
Neural Network Architecture ◽  
Acceptable Accuracy ◽  
The Neural Network ◽  
Sigmoid Activation Function

The problem of applying neural networks to calculate ratings used in banking in the decision-making process on granting or not granting loans to borrowers is considered. The task is to determine the rating function of the borrower based on a set of statistical data on the effectiveness of loans provided by the bank. When constructing a regression model to calculate the rating function, it is necessary to know its general form. If so, the task is to calculate the parameters that are included in the expression for the rating function. In contrast to this approach, in the case of using neural networks, there is no need to specify the general form for the rating function. Instead, certain neural network architecture is chosen and parameters are calculated for it on the basis of statistical data. Importantly, the same neural network architecture can be used to process different sets of statistical data. The disadvantages of using neural networks include the need to calculate a large number of parameters. There is also no universal algorithm that would determine the optimal neural network architecture. As an example of the use of neural networks to determine the borrower's rating, a model system is considered, in which the borrower's rating is determined by a known non-analytical rating function. A neural network with two inner layers, which contain, respectively, three and two neurons and have a sigmoid activation function, is used for modeling. It is shown that the use of the neural network allows restoring the borrower's rating function with quite acceptable accuracy.

Color Space Transformation using Neural Networks

2019 ◽  
Vol 2019 (1) ◽  
pp. 153-158
Author(s):  
Lindsay MacDonald
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Color Space ◽  
Reflectance Spectra ◽  
Network Architectures ◽  
Color Spaces ◽  
Natural Materials ◽  
Space Transformation ◽  
Color Space Transformation

We investigated how well a multilayer neural network could implement the mapping between two trichromatic color spaces, specifically from camera R,G,B to tristimulus X,Y,Z. For training the network, a set of 800,000 synthetic reflectance spectra was generated. For testing the network, a set of 8,714 real reflectance spectra was collated from instrumental measurements on textiles, paints and natural materials. Various network architectures were tested, with both linear and sigmoidal activations. Results show that over 85% of all test samples had color errors of less than 1.0 ΔE2000 units, much more accurate than could be achieved by regression.

Estimating Pigment Concentrations from Spectral Images Using an Encoder‐Decoder Neural Network

2020 ◽  
Vol 64 (3) ◽  
pp. 30502-1-30502-15
Author(s):  
Kensuke Fukumoto ◽  
Norimichi Tsumura ◽  
Roy Berns
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Absorption Coefficient ◽  
Spectral Data ◽  
High Accuracy ◽  
Pigment Concentration ◽  
Scattering Coefficient ◽  
A Value ◽  
Input And Output ◽  
Pigment Concentrations

Abstract A method is proposed to estimate the concentration of pigments mixed in a painting, using the encoder‐decoder model of neural networks. The model is trained to output a value that is the same as its input, and its middle output extracts a certain feature as compressed information about the input. In this instance, the input and output are spectral data of a painting. The model is trained with pigment concentration as the middle output. A dataset containing the scattering coefficient and absorption coefficient of each of 19 pigments was used. The Kubelka‐Munk theory was applied to the coefficients to obtain many patterns of synthetic spectral data, which were used for training. The proposed method was tested using spectral images of 33 paintings, which showed that the method estimates, with high accuracy, the concentrations that have a similar spectrum of the target pigments.

Sign in / Sign up

Export Citation Format

Share Document