scholarly journals Quantization of Weights of Neural Networks with Negligible Decreasing of Prediction Accuracy

2021 ◽  
Vol 50 (3) ◽  
pp. 558-569
Author(s):  
Zoran Peric ◽  
Bojan Denic ◽  
Milan Savic ◽  
Milan Dincic ◽  
Darko Mihajlov
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Prediction Accuracy ◽  
Network Performance ◽  
Dynamic Range ◽  
Mean Squared Error ◽  
Real Data ◽  
Quantization Noise ◽  
Bit Rate ◽  
The Impact

Quantization and compression of neural network parameters using the uniform scalar quantization is carried out in this paper. The attractiveness of the uniform scalar quantizer is reflected in a low complexity and relatively good performance, making it the most popular quantization model. We present a design approach for the memoryless Laplacian source with zero-mean and unit variance, which is based on iterative rule and uses the minimal mean-squared error distortion as a performance criterion. In addition, we derive closed-form expressions for SQNR (Signal to Quantization Noise Ratio) in a wide dynamic range of variance of input data. To show effectiveness on real data, the proposed quantizer is used to compress the weights of neural networks using bit rates from 9 to 16 bps (bits/sample) instead of standardly used 32 bps full precision bit rate. The impact of weights compression on the NN (neural network) performance is analyzed, indicating good matching with the theoretical results and showing negligible decreasing of the prediction accuracy of the NN even in the case of high variance-mismatch between the variance of NN weights and the variance used for the design of quantizer, if the value of the bit-rate is properly chosen according to the rule proposed in the paper.

scholarly journals Design and Analysis of Binary Scalar Quantizer of Laplacian Source with Applications

Information ◽  
2020 ◽  
Vol 11 (11) ◽  
pp. 501
Author(s):  
Zoran Peric ◽  
Bojan Denic ◽  
Milan Savic ◽  
Vladimir Despotovic
Keyword(s):  
Neural Network ◽  
Dynamic Range ◽  
Mean Squared Error ◽  
Theoretical Models ◽  
Detailed Comparison ◽  
Quantization Noise ◽  
Scalar Quantization ◽  
Model Compression ◽  
Processing Power ◽  
Image Quantization

A compression method based on non-uniform binary scalar quantization, designed for the memoryless Laplacian source with zero-mean and unit variance, is analyzed in this paper. Two quantizer design approaches are presented that investigate the effect of clipping with the aim of reducing the quantization noise, where the minimal mean-squared error distortion is used to determine the optimal clipping factor. A detailed comparison of both models is provided, and the performance evaluation in a wide dynamic range of input data variances is also performed. The observed binary scalar quantization models are applied in standard signal processing tasks, such as speech and image quantization, but also to quantization of neural network parameters. The motivation behind the binary quantization of neural network weights is the model compression by a factor of 32, which is crucial for implementation in mobile or embedded devices with limited memory and processing power. The experimental results follow well the theoretical models, confirming their applicability in real-world applications.

scholarly journals Trade impacts of the New Silk Road in Africa: Insight from Neural Networks Analysis

2021 ◽  
Vol 17 (2) ◽  
pp. 13-26
Author(s):  
Dumor Koffi ◽  
Komlan Gbongli
Keyword(s):  
Neural Networks ◽  
Gravity Model ◽  
Fixed Effects ◽  
Prediction Accuracy ◽  
Bilateral Trade ◽  
Mean Squared Error ◽  
Ordinary Least Squares ◽  
Machine Learning Algorithms ◽  
Accurate Estimation ◽  
The Impact

The Belt and Road Initiative (BRI) is aimed to strengthen the preferential reciprocal trade between China and the Belt-Road nations. Quantitative evaluations of BRI to determine whether it can explicitly provide more insight into China’s bilateral trade among its partners are needed. Hence, improving prediction accuracy while using more superior algorithms for sustainable decision-making remains essential since decision-makers have been interested in predicting the future. Machine learning algorithms, such as supervised artificial neural networks (ANN), outperform several econometric procedures in predictions; therefore, they are potentially powerful techniques to evaluate BRI. This study uses detailed China’s bilateral export data from 1990 to 2017 to analyze and evaluate the impact of BRI on bilateral trade using gravity model estimations and ANN analysis techniques. The finding suggests that China’s bilateral export flow among the BRI countries results in a slight increase in inter-regional trade. The study provides a comparison view on the different estimation procedures of the gravity model – ordinary least squares (OLS) and Poisson pseudo-maximum likelihood (PPML) with the ANN. The ANN associated with fixed country effects reveals a more accurate estimation compared to a baseline model and with country-year fixed effects. Contrarily, the OLS estimator and PPML showed mixed results. Grounded on the study dataset, the ANN estimation of the gravity equation was superior over the other procedures to explain the variability of the dependent variable (export) regarding the prediction accuracy using root mean squared error (RMSE) and R-square.

scholarly journals Quantitative Dynamics Effects of Belt and Road Economies Trade Using Structural Gravity and Neural Networks

SAGE Open ◽  
2021 ◽  
Vol 11 (3) ◽  
pp. 215824402110326
Author(s):  
Koffi Dumor ◽  
Li Yao ◽  
Jean-Paul Ainam ◽  
Edem Koffi Amouzou ◽  
Williams Ayivi
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Network Analysis ◽  
Bilateral Trade ◽  
Belt And Road Initiative ◽  
Neural Network Analysis ◽  
Analysis Techniques ◽  
Belt And Road ◽  
The Impact ◽  
Trade Partners

Recent research suggests that China’s Belt and Road Initiative (BRI) would improve the bilateral trade between China and its partners. This article uses detailed bilateral export data from 1990 to 2017 to investigate the impact of China’s BRI on its trade partners using neural network analysis techniques and structural gravity model estimations. Our main findings suggest that the BRI countries would raise exports by a modest 5.053%. This indicates that export and network upgrades should be considered from economic and policy perspectives. The results also show that neural networks is more robust compared with structural gravity framework.

Prognosis of water levels in a moor groundwater system influenced by hydrology and water extraction using an artificial neural network 

2021 ◽  
Author(s):  
Sascha Flaig ◽  
Timothy Praditia ◽  
Alexander Kissinger ◽  
Ulrich Lang ◽  
Sergey Oladyshkin ◽  
...  
Keyword(s):  
Neural Network ◽  
Artificial Neural Network ◽  
Mean Squared Error ◽  
Meteorological Data ◽  
Water Levels ◽  
Typical Structure ◽  
Water Abstraction ◽  
Physical Knowledge ◽  
Artificial Neural ◽  
The Impact

<p>In order to prevent possible negative impacts of water abstraction in an ecologically sensitive moor south of Munich (Germany), a “predictive control” scheme is in place. We design an artificial neural network (ANN) to provide predictions of moor water levels and to separate hydrological from anthropogenic effects. As the moor is a dynamic system, we adopt the „Long short-term memory“ architecture.</p><p>To find the best LSTM setup, we train, test and compare LSTMs with two different structures: (1) the non-recurrent one-to-one structure, where the series of inputs are accumulated and fed into the LSTM; and (2) the recurrent many-to-many structure, where inputs gradually enter the LSTM (including LSTM forecasts from previous forecast time steps). The outputs of our LSTMs then feed into a readout layer that converts the hidden states into water level predictions. We hypothesize that the recurrent structure is the better structure because it better resembles the typical structure of differential equations for dynamic systems, as they would usually be used for hydro(geo)logical systems. We evaluate the comparison with the mean squared error as test metric, and conclude that the recurrent many-to-many LSTM performs better for the analyzed complex situations. It also produces plausible predictions with reasonable accuracy for seven days prediction horizon.</p><p>Furthermore, we analyze the impact of preprocessing meteorological data to evapotranspiration data using typical ETA models. Inserting knowledge into the LSTM in the form of ETA models (rather than implicitly having the LSTM learn the ETA relations) leads to superior prediction results. This finding aligns well with current ideas on physically-inspired machine learning.</p><p>As an additional validation step, we investigate whether our ANN is able to correctly identify both anthropogenic and natural influences and their interaction. To this end, we investigate two comparable pumping events under different meteorological conditions. Results indicate that all individual and combined influences of input parameters on water levels can be represented well. The neural networks recognize correctly that the predominant precipitation and lower evapotranspiration during one pumping event leads to a lower decrease of the hydrograph.</p><p>To further demonstrate the capability of the trained neural network, scenarios of pumping events are created and simulated.</p><p>In conclusion, we show that more robust and accurate predictions of moor water levels can be obtained if available physical knowledge of the modeled system is used to design and train the neural network. The artificial neural network can be a useful instrument to assess the impact of water abstraction by quantifying the anthropogenic influence.</p>

scholarly journals Impact of Low Resolution on Image Recognition with Deep Neural Networks: An Experimental Study

2018 ◽  
Vol 28 (4) ◽  
pp. 735-744 ◽  
Author(s):  
Michał Koziarski ◽  
Bogusław Cyganek
Keyword(s):  
Neural Networks ◽  
Image Recognition ◽  
Classification Accuracy ◽  
Deep Neural Networks ◽  
Dynamic Range ◽  
Super Resolution ◽  
Image Resolution ◽  
Quality Data ◽  
Low Resolution ◽  
The Impact

Abstract Due to the advances made in recent years, methods based on deep neural networks have been able to achieve a state-of-the-art performance in various computer vision problems. In some tasks, such as image recognition, neural-based approaches have even been able to surpass human performance. However, the benchmarks on which neural networks achieve these impressive results usually consist of fairly high quality data. On the other hand, in practical applications we are often faced with images of low quality, affected by factors such as low resolution, presence of noise or a small dynamic range. It is unclear how resilient deep neural networks are to the presence of such factors. In this paper we experimentally evaluate the impact of low resolution on the classification accuracy of several notable neural architectures of recent years. Furthermore, we examine the possibility of improving neural networks’ performance in the task of low resolution image recognition by applying super-resolution prior to classification. The results of our experiments indicate that contemporary neural architectures remain significantly affected by low image resolution. By applying super-resolution prior to classification we were able to alleviate this issue to a large extent as long as the resolution of the images did not decrease too severely. However, in the case of very low resolution images the classification accuracy remained considerably affected.

scholarly journals Analyzing and interpreting neural networks for NLP: A report on the first BlackboxNLP workshop

2019 ◽  
Vol 25 (4) ◽  
pp. 543-557 ◽  
Author(s):  
Afra Alishahi ◽  
Grzegorz Chrupała ◽  
Tal Linzen
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Natural Language Processing ◽  
Language Processing ◽  
Performance Testing ◽  
Network Architectures ◽  
Empirical Methods ◽  
Neural Models ◽  
The Impact ◽  
Systematic Manipulation

AbstractThe Empirical Methods in Natural Language Processing (EMNLP) 2018 workshop BlackboxNLP was dedicated to resources and techniques specifically developed for analyzing and understanding the inner-workings and representations acquired by neural models of language. Approaches included: systematic manipulation of input to neural networks and investigating the impact on their performance, testing whether interpretable knowledge can be decoded from intermediate representations acquired by neural networks, proposing modifications to neural network architectures to make their knowledge state or generated output more explainable, and examining the performance of networks on simplified or formal languages. Here we review a number of representative studies in each category.

A COMPARISON OF SCORING METRICS FOR PREDICTING THE NEXT NAVIGATION STEP WITH MARKOV MODEL-BASED SYSTEMS

2010 ◽  
Vol 09 (04) ◽  
pp. 547-573 ◽  
Author(s):  
JOSÉ BORGES ◽  
MARK LEVENE
Keyword(s):  
Markov Model ◽  
Prediction Accuracy ◽  
Prediction Models ◽  
Markov Models ◽  
Real Data ◽  
Absolute Error ◽  
Brier Score ◽  
Data Sets ◽  
Extensive Evaluation ◽  
The Impact

The problem of predicting the next request during a user's navigation session has been extensively studied. In this context, higher-order Markov models have been widely used to model navigation sessions and to predict the next navigation step, while prediction accuracy has been mainly evaluated with the hit and miss score. We claim that this score, although useful, is not sufficient for evaluating next link prediction models with the aim of finding a sufficient order of the model, the size of a recommendation set, and assessing the impact of unexpected events on the prediction accuracy. Herein, we make use of a variable length Markov model to compare the usefulness of three alternatives to the hit and miss score: the Mean Absolute Error, the Ignorance Score, and the Brier score. We present an extensive evaluation of the methods on real data sets and a comprehensive comparison of the scoring methods.

Attempt at Resilient Modulus Modeling Using Artificial Neural Networks

1996 ◽  
Vol 1540 (1) ◽  
pp. 1-6 ◽  
Author(s):  
Erol Tutumluer ◽  
Roger W. Meier
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Artificial Neural Networks ◽  
Network Performance ◽  
Resilient Modulus ◽  
Quality Of Data ◽  
Unsuccessful Attempt ◽  
Independent Test ◽  
Artificial Neural

The pitfalls inherent in the indiscriminate application of artificial neural networks to numerical modeling problems are illustrated. An example is used of an apparently successful (but ultimately unsuccessful) attempt at training a neural network constitutive model for computing the resilient modulus of gravels as a function of stress state and various material properties. Issues such as the quantity and quality of data needed to successfully train a neural network are explored, and the importance of an independent test set to verify network performance is examined.

scholarly journals 137 Gb/s PAM-4 Transmissions at 850 nm over 40 cm Optical Backplane with 25 G Devices with Improved Neural Network-Based Equalization

2019 ◽  
Vol 9 (23) ◽  
pp. 5095
Author(s):  
Qianwu Zhang ◽  
Yuntong Jiang ◽  
Hai Zhou ◽  
Chuanlu Deng ◽  
Shuaihang Duan ◽  
...  
Keyword(s):  
Neural Network ◽  
Forward Error Correction ◽  
Optical Power ◽  
Pulse Amplitude ◽  
Performance Comparison ◽  
Decision Feedback ◽  
Bit Rate ◽  
Optical Backplane ◽  
The Impact ◽  
Equalization Method

An improved neural network-based equalization method is proposed and experimentally demonstrated. The up-to-137 Gb/s transmission of four level pulse amplitude modulation (PAM-4) signals with 25 G class 850 nm optical devices is achieved over an in-house fabricated 40 cm optical backplane. An in-depth investigation is conducted regarding the impact of delayed taps and spans on equalization performance. A performance comparison of the proposed method with the traditional maximum likelihood sequence estimation (MLSE) and decision feedback equalization (DFE) is also undertaken. For the bit rate from 80 to 100 Gb/s, the proposed method achieves an adopted hard-decision forward error correction (HD-FEC) requirement at a received optical power (RoP) of −9 and −8 dBm, while DFE and MLSE cannot meet the HD-FEC requirement. When the bit rate increases from 120 to 137 Gb/s, the proposed equalization method still successfully maintains the acceptable system performance at an RoP of −4 and −2.5 dBm. Furthermore, the specific bit error rate (BER) performances for varied maximum achievable bit rate under different RoPs by applying MLSE and the proposed method are also analyzed. This provides an important potential solution to realize the future data centers.

scholarly journals A Computational Approach for the Identification of Small GTPases Based on Preprocessed Amino Acid Sequences

2009 ◽  
Vol 8 (5) ◽  
pp. 333-341 ◽  
Author(s):  
Dominik Heider ◽  
Jessica Appelmann ◽  
Tuygun Bayro ◽  
Winfried Dreckmann ◽  
Andreas Held ◽  
...  
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Secondary Structure ◽  
Prediction Accuracy ◽  
Biological Activities ◽  
Small Gtpases ◽  
Computational Approach ◽  
Small Gtpase ◽  
Amino Acid Sequences ◽  
Network Cluster

The prediction of essential biological features based on a given protein sequence is a challenging task in computational biology. To limit the amount of in vitro verification, the prediction of essential biological activities gives the opportunity to detect so far unknown sequences with similar properties. Besides the application within the identification of proteins being involved in tumorigenesis, other functional classes of proteins can be predicted. The prediction accuracy depends on the selected machine learning approach and even more on the composition of the descriptor set used. A computational approach based on feedforward neural networks was applied for the prediction of small GTPases. Consequently, this was realized by taking secondary structure and hydrophobicity information as a preprocessing architecture and thus, as descriptors for the neural networks. We developed a neural network cluster, which consists of a filter network and four subfamily networks. The filter network was trained to identify small GTPases and the subfamily networks were trained to assign a small GTPase to one of the subfamilies. The accuracy of the prediction, whether a given sequence represents a small GTPase is very high (98.25%). The classifications of the subfamily networks yield comparable accuracy. The high prediction accuracy of the neural network cluster developed, gives the opportunity to suggest the use of hydrophobicity and secondary structure prediction in combination with a neural network cluster, as a promising method for the prediction of essential biological activities.

Sign in / Sign up

Export Citation Format

Share Document