PRIMA: general and precise neural network certification via scalable convex hull approximations

Formal verification of neural networks is critical for their safe adoption in real-world applications. However, designing a precise and scalable verifier which can handle different activation functions, realistic network architectures and relevant specifications remains an open and difficult challenge. In this paper, we take a major step forward in addressing this challenge and present a new verification framework, called PRIMA. PRIMA is both (i) general: it handles any non-linear activation function, and (ii) precise: it computes precise convex abstractions involving multiple neurons via novel convex hull approximation algorithms that leverage concepts from computational geometry. The algorithms have polynomial complexity, yield fewer constraints, and minimize precision loss. We evaluate the effectiveness of PRIMA on a variety of challenging tasks from prior work. Our results show that PRIMA is significantly more precise than the state-of-the-art, verifying robustness to input perturbations for up to 20%, 30%, and 34% more images than existing work on ReLU-, Sigmoid-, and Tanh-based networks, respectively. Further, PRIMA enables, for the first time, the precise verification of a realistic neural network for autonomous driving within a few minutes.

Download Full-text

New Autonomous Intelligent Sensor Design Approach for Multiple Parameter Inference

Engineering Proceedings ◽

10.3390/engproc2020002096 ◽

2021 ◽

Vol 2 (1) ◽

pp. 96

Author(s):

Umberto Michelucci ◽

Francesca Venturini

Keyword(s):

Neural Network ◽

Machine Learning ◽

Training Dataset ◽

Network Architectures ◽

Single Measurement ◽

Intelligent Sensor ◽

Multiple Parameters ◽

Two Parameters ◽

First Time

The determination of multiple parameters via luminescence sensing is of great interest for many applications in different fields, like biosensing and biological imaging, medicine, and diagnostics. The typical approach consists in measuring multiple quantities and in applying complex and frequently just approximated mathematical models to characterize the sensor response. The use of machine learning to extract information from measurements in sensors have been tried in several forms before. But one of the problems with the approaches so far, is the difficulty in getting a training dataset that is representative of the measurements done by the sensor. Additionally, extracting multiple parameters from a single measurement has been so far an impossible problem to solve efficiently in luminescence. In this work a new approach is described for building an autonomous intelligent sensor, which is able to produce the training dataset self-sufficiently, use it for training a neural network, and then use the trained model to do inference on measurements done on the same hardware. For the first time the use of machine learning additionally allows to extract two parameters from one single measurement using multitask learning neural network architectures. This is demonstrated here by a dual oxygen concentration and temperature sensor.

Download Full-text

Efficient Quantization for Neural Networks with Binary Weights and Low Bitwidth Activations

Proceedings of the AAAI Conference on Artificial Intelligence ◽

10.1609/aaai.v33i01.33013854 ◽

2019 ◽

Vol 33 ◽

pp. 3854-3861

Author(s):

Kun Huang ◽

Bingbing Ni ◽

Xiaokang Yang

Keyword(s):

Neural Network ◽

Neural Networks ◽

Deep Neural Network ◽

Activation Function ◽

Limited Resources ◽

Portable Devices ◽

Training Procedure ◽

Activation Functions ◽

First Time ◽

And Training

Quantization has shown stunning efficiency on deep neural network, especially for portable devices with limited resources. Most existing works uncritically extend weight quantization methods to activations. However, we take the view that best performance can be obtained by applying different quantization methods to weights and activations respectively. In this paper, we design a new activation function dubbed CReLU from the quantization perspective and further complement this design with appropriate initialization method and training procedure. Moreover, we develop a specific quantization strategy in which we formulate the forward and backward approximation of weights with binary values and quantize the activations to low bitwdth using linear or logarithmic quantizer. We show, for the first time, our final quantized model with binary weights and ultra low bitwidth activations outperforms the previous best models by large margins on ImageNet as well as achieving nearly a 10.85× theoretical speedup with ResNet-18. Furthermore, ablation experiments and theoretical analysis demonstrate the effectiveness and robustness of CReLU in comparison with other activation functions.

Download Full-text

Fourier neural networks: A comparative study

Intelligent Data Analysis ◽

10.3233/ida-195050 ◽

2020 ◽

Vol 24 (5) ◽

pp. 1107-1120

Author(s):

Malika Uteuliyeva ◽

Abylay Zhumekenov ◽

Rustem Takhanov ◽

Zhenisbek Assylbekov ◽

Alejandro J. Castro ◽

...

Keyword(s):

Neural Network ◽

Neural Networks ◽

Fourier Series ◽

Real World ◽

Approximation Error ◽

Activation Function ◽

Network Architectures ◽

Multiple Variables ◽

Truncated Fourier Series ◽

Sigmoid Activation Function

We review neural network architectures which were motivated by Fourier series and integrals and which are referred to as Fourier neural networks. These networks are empirically evaluated in synthetic and real-world tasks. Neither of them outperforms the standard neural network with sigmoid activation function in the real-world tasks. All neural networks, both Fourier and the standard one, empirically demonstrate lower approximation error than the truncated Fourier series when it comes to approximation of a known function of multiple variables.

Download Full-text

The Compact Support Neural Network

Sensors ◽

10.3390/s21248494 ◽

2021 ◽

Vol 21 (24) ◽

pp. 8494

Author(s):

Adrian Barbu ◽

Hongyu Mou

Keyword(s):

Neural Network ◽

Neural Networks ◽

Compact Support ◽

Shape Parameter ◽

Autonomous Driving ◽

Activation Function ◽

Training Data ◽

Benchmark Datasets ◽

The Neural Networks ◽

Experimental Findings

Neural networks are popular and useful in many fields, but they have the problem of giving high confidence responses for examples that are away from the training data. This makes the neural networks very confident in their prediction while making gross mistakes, thus limiting their reliability for safety-critical applications such as autonomous driving and space exploration, etc. This paper introduces a novel neuron generalization that has the standard dot-product-based neuron and the radial basis function (RBF) neuron as two extreme cases of a shape parameter. Using a rectified linear unit (ReLU) as the activation function results in a novel neuron that has compact support, which means its output is zero outside a bounded domain. To address the difficulties in training the proposed neural network, it introduces a novel training method that takes a pretrained standard neural network that is fine-tuned while gradually increasing the shape parameter to the desired value. The theoretical findings of the paper are bound on the gradient of the proposed neuron and proof that a neural network with such neurons has the universal approximation property. This means that the network can approximate any continuous and integrable function with an arbitrary degree of accuracy. The experimental findings on standard benchmark datasets show that the proposed approach has smaller test errors than the state-of-the-art competing methods and outperforms the competing methods in detecting out-of-distribution samples on two out of three datasets.

Download Full-text

SCORING MODELING BASED ON NEURAL NETWORKS FOR DETERMINING A BANK BORROWER'S RATING

Economy of Ukraine ◽

10.15407/economyukr.2020.10.054 ◽

2020 ◽

Vol 2020 (10) ◽

pp. 54-62

Author(s):

Oleksii VASYLIEV ◽

Keyword(s):

Neural Network ◽

Neural Networks ◽

Network Architecture ◽

Statistical Data ◽

Activation Function ◽

Decision Making Process ◽

Neural Network Architecture ◽

Acceptable Accuracy ◽

The Neural Network ◽

Sigmoid Activation Function

The problem of applying neural networks to calculate ratings used in banking in the decision-making process on granting or not granting loans to borrowers is considered. The task is to determine the rating function of the borrower based on a set of statistical data on the effectiveness of loans provided by the bank. When constructing a regression model to calculate the rating function, it is necessary to know its general form. If so, the task is to calculate the parameters that are included in the expression for the rating function. In contrast to this approach, in the case of using neural networks, there is no need to specify the general form for the rating function. Instead, certain neural network architecture is chosen and parameters are calculated for it on the basis of statistical data. Importantly, the same neural network architecture can be used to process different sets of statistical data. The disadvantages of using neural networks include the need to calculate a large number of parameters. There is also no universal algorithm that would determine the optimal neural network architecture. As an example of the use of neural networks to determine the borrower's rating, a model system is considered, in which the borrower's rating is determined by a known non-analytical rating function. A neural network with two inner layers, which contain, respectively, three and two neurons and have a sigmoid activation function, is used for modeling. It is shown that the use of the neural network allows restoring the borrower's rating function with quite acceptable accuracy.

Download Full-text

Color Space Transformation using Neural Networks

Color and Imaging Conference ◽

10.2352/issn.2169-2629.2019.27.29 ◽

2019 ◽

Vol 2019 (1) ◽

pp. 153-158

Author(s):

Lindsay MacDonald

Keyword(s):

Neural Network ◽

Neural Networks ◽

Color Space ◽

Reflectance Spectra ◽

Network Architectures ◽

Color Spaces ◽

Natural Materials ◽

Space Transformation ◽

Color Space Transformation

We investigated how well a multilayer neural network could implement the mapping between two trichromatic color spaces, specifically from camera R,G,B to tristimulus X,Y,Z. For training the network, a set of 800,000 synthetic reflectance spectra was generated. For testing the network, a set of 8,714 real reflectance spectra was collated from instrumental measurements on textiles, paints and natural materials. Various network architectures were tested, with both linear and sigmoidal activations. Results show that over 85% of all test samples had color errors of less than 1.0 ΔE2000 units, much more accurate than could be achieved by regression.

Download Full-text

Prediction of Stress in Power Transformer Winding Conductors Using Artificial Neural Networks: Hyperparameter Analysis

Energies ◽

10.3390/en14144242 ◽

2021 ◽

Vol 14 (14) ◽

pp. 4242

Author(s):

Fausto Valencia ◽

Hugo Arcos ◽

Franklin Quilumba

Keyword(s):

Neural Network ◽

Artificial Neural Network ◽

Finite Element ◽

Power Transformer ◽

Network Models ◽

Activation Function ◽

Neural Network Models ◽

Transformer Winding ◽

Artificial Neural ◽

Femm Software

The purpose of this research is the evaluation of artificial neural network models in the prediction of stresses in a 400 MVA power transformer winding conductor caused by the circulation of fault currents. The models were compared considering the training, validation, and test data errors’ behavior. Different combinations of hyperparameters were analyzed based on the variation of architectures, optimizers, and activation functions. The data for the process was created from finite element simulations performed in the FEMM software. The design of the Artificial Neural Network was performed using the Keras framework. As a result, a model with one hidden layer was the best suited architecture for the problem at hand, with the optimizer Adam and the activation function ReLU. The final Artificial Neural Network model predictions were compared with the Finite Element Method results, showing good agreement but with a much shorter solution time.

Download Full-text