Fourier neural networks: A comparative study

Malika Uteuliyeva; Abylay Zhumekenov; Rustem Takhanov; Zhenisbek Assylbekov; Alejandro J. Castro; Olzhas Kabdolov

doi:10.3233/ida-195050

Fourier neural networks: A comparative study

Intelligent Data Analysis ◽

10.3233/ida-195050 ◽

2020 ◽

Vol 24 (5) ◽

pp. 1107-1120

Author(s):

Malika Uteuliyeva ◽

Abylay Zhumekenov ◽

Rustem Takhanov ◽

Zhenisbek Assylbekov ◽

Alejandro J. Castro ◽

...

Keyword(s):

Neural Network ◽

Neural Networks ◽

Fourier Series ◽

Real World ◽

Approximation Error ◽

Activation Function ◽

Network Architectures ◽

Multiple Variables ◽

Truncated Fourier Series ◽

Sigmoid Activation Function

We review neural network architectures which were motivated by Fourier series and integrals and which are referred to as Fourier neural networks. These networks are empirically evaluated in synthetic and real-world tasks. Neither of them outperforms the standard neural network with sigmoid activation function in the real-world tasks. All neural networks, both Fourier and the standard one, empirically demonstrate lower approximation error than the truncated Fourier series when it comes to approximation of a known function of multiple variables.

SCORING MODELING BASED ON NEURAL NETWORKS FOR DETERMINING A BANK BORROWER'S RATING

Economy of Ukraine ◽

10.15407/economyukr.2020.10.054 ◽

2020 ◽

Vol 2020 (10) ◽

pp. 54-62

Author(s):

Oleksii VASYLIEV ◽

Keyword(s):

Neural Network ◽

Neural Networks ◽

Network Architecture ◽

Statistical Data ◽

Activation Function ◽

Decision Making Process ◽

Neural Network Architecture ◽

Acceptable Accuracy ◽

The Neural Network ◽

Sigmoid Activation Function

The problem of applying neural networks to calculate ratings used in banking in the decision-making process on granting or not granting loans to borrowers is considered. The task is to determine the rating function of the borrower based on a set of statistical data on the effectiveness of loans provided by the bank. When constructing a regression model to calculate the rating function, it is necessary to know its general form. If so, the task is to calculate the parameters that are included in the expression for the rating function. In contrast to this approach, in the case of using neural networks, there is no need to specify the general form for the rating function. Instead, certain neural network architecture is chosen and parameters are calculated for it on the basis of statistical data. Importantly, the same neural network architecture can be used to process different sets of statistical data. The disadvantages of using neural networks include the need to calculate a large number of parameters. There is also no universal algorithm that would determine the optimal neural network architecture. As an example of the use of neural networks to determine the borrower's rating, a model system is considered, in which the borrower's rating is determined by a known non-analytical rating function. A neural network with two inner layers, which contain, respectively, three and two neurons and have a sigmoid activation function, is used for modeling. It is shown that the use of the neural network allows restoring the borrower's rating function with quite acceptable accuracy.

On Transformative Adaptive Activation Functions in Neural Networks for Gene Expression Inference

10.1101/587287 ◽

2019 ◽

Cited By ~ 1

Author(s):

Vladimír Kunc ◽

Jiří Kléma

Keyword(s):

Neural Network ◽

Gene Expression ◽

Neural Networks ◽

Expression Profiling ◽

Mean Absolute Error ◽

Absolute Error ◽

Activation Function ◽

Activation Functions ◽

Hyperbolic Tangent ◽

Sigmoid Activation Function

AbstractMotivationGene expression profiling was made cheaper by the NIH LINCS program that profiles only ~1, 000 selected landmark genes and uses them to reconstruct the whole profile. The D–GEX method employs neural networks to infer the whole profile. However, the original D–GEX can be further significantly improved.ResultsWe have analyzed the D–GEX method and determined that the inference can be improved using a logistic sigmoid activation function instead of the hyperbolic tangent. Moreover, we propose a novel transformative adaptive activation function that improves the gene expression inference even further and which generalizes several existing adaptive activation functions. Our improved neural network achieves average mean absolute error of 0.1340 which is a significant improvement over our reimplementation of the original D–GEX which achieves average mean absolute error 0.1637

Color Space Transformation using Neural Networks

Color and Imaging Conference ◽

10.2352/issn.2169-2629.2019.27.29 ◽

2019 ◽

Vol 2019 (1) ◽

pp. 153-158

Author(s):

Lindsay MacDonald

Keyword(s):

Neural Network ◽

Neural Networks ◽

Color Space ◽

Reflectance Spectra ◽

Network Architectures ◽

Color Spaces ◽

Natural Materials ◽

Space Transformation ◽

Color Space Transformation

We investigated how well a multilayer neural network could implement the mapping between two trichromatic color spaces, specifically from camera R,G,B to tristimulus X,Y,Z. For training the network, a set of 800,000 synthetic reflectance spectra was generated. For testing the network, a set of 8,714 real reflectance spectra was collated from instrumental measurements on textiles, paints and natural materials. Various network architectures were tested, with both linear and sigmoidal activations. Results show that over 85% of all test samples had color errors of less than 1.0 ΔE2000 units, much more accurate than could be achieved by regression.

Deep neural network based on generalized neo-fuzzy neurons and its learning based on backpropagation

Artificial Intelligence ◽

10.15407/jai2021.01.032 ◽

2021 ◽

Vol 26 (jai2021.26(1)) ◽

pp. 32-41

Author(s):

Bodyanskiy Y ◽

◽

Antonenko T ◽

Keyword(s):

Neural Network ◽

Neural Networks ◽

Learning Process ◽

Activation Function ◽

Point Of View ◽

Basic Unit ◽

Theoretical Point ◽

Activation Functions ◽

Approximation Properties ◽

Network Training

Modern approaches in deep neural networks have a number of issues related to the learning process and computational costs. This article considers the architecture grounded on an alternative approach to the basic unit of the neural network. This approach achieves optimization in the calculations and gives rise to an alternative way to solve the problems of the vanishing and exploding gradient. The main issue of the article is the usage of the deep stacked neo-fuzzy system, which uses a generalized neo-fuzzy neuron to optimize the learning process. This approach is non-standard from a theoretical point of view, so the paper presents the necessary mathematical calculations and describes all the intricacies of using this architecture from a practical point of view. From a theoretical point, the network learning process is fully disclosed. Derived all necessary calculations for the use of the backpropagation algorithm for network training. A feature of the network is the rapid calculation of the derivative for the activation functions of neurons. This is achieved through the use of fuzzy membership functions. The paper shows that the derivative of such function is a constant, and this is a reason for the statement of increasing in the optimization rate in comparison with neural networks which use neurons with more common activation functions (ReLU, sigmoid). The paper highlights the main points that can be improved in further theoretical developments on this topic. In general, these issues are related to the calculation of the activation function. The proposed methods cope with these points and allow approximation using the network, but the authors already have theoretical justifications for improving the speed and approximation properties of the network. The results of the comparison of the proposed network with standard neural network architectures are shown

Prediction of slip in cable-drum systems using structured neural networks

Proceedings of the Institution of Mechanical Engineers Part C Journal of Mechanical Engineering Science ◽

10.1177/0954406213487471 ◽

2013 ◽

Vol 228 (3) ◽

pp. 441-456 ◽

Cited By ~ 2

Author(s):

Ergin Kilic ◽

Melik Dolen

Keyword(s):

Neural Network ◽

Neural Networks ◽

Network Size ◽

Network Architectures ◽

Network Development ◽

Low Velocity ◽

Error Band ◽

Artificial Neural ◽

Recurrent Type ◽

Selection Of

This study focuses on the slip prediction in a cable-drum system using artificial neural networks for the prospect of developing linear motion sensing scheme for such mechanisms. Both feed-forward and recurrent-type artificial neural network architectures are considered to capture the slip dynamics of cable-drum mechanisms. In the article, the network development is presented in a progressive (step-by-step) fashion for the purpose of not only making the design process transparent to the readers but also highlighting the corresponding challenges associated with the design phase (i.e. selection of architecture, network size, training process parameters, etc.). Prediction performances of the devised networks are evaluated rigorously via an experimental study. Finally, a structured neural network, which embodies the network with the best prediction performance, is further developed to overcome the drift observed at low velocity. The study illustrates that the resulting structured neural network could predict the slip in the mechanism within an error band of 100 µm when an absolute reference is utilized.

Signal processing algorithm for neural networks with integrodifferential splines as an activation function and its particular case of image classification

Highly available systems ◽

10.18127/j20729472-202102-02 ◽

2021 ◽

Author(s):

T.K. Biryukova

Keyword(s):

Neural Network ◽

Neural Networks ◽

Image Classification ◽

Activation Function ◽

Experimental Comparison ◽

Training Time ◽

Operation Speed ◽

The Neural Network ◽

Linear Algebraic Equations ◽

Network Operation

Classic neural networks suppose trainable parameters to include just weights of neurons. This paper proposes parabolic integrodifferential splines (ID-splines), developed by author, as a new kind of activation function (AF) for neural networks, where ID-splines coefficients are also trainable parameters. Parameters of ID-spline AF together with weights of neurons are vary during the training in order to minimize the loss function thus reducing the training time and increasing the operation speed of the neural network. The newly developed algorithm enables software implementation of the ID-spline AF as a tool for neural networks construction, training and operation. It is proposed to use the same ID-spline AF for neurons in the same layer, but different for different layers. In this case, the parameters of the ID-spline AF for a particular layer change during the training process independently of the activation functions (AFs) of other network layers. In order to comply with the continuity condition for the derivative of the parabolic ID-spline on the interval (x x0, n) , its parameters fi (i= 0,...,n) should be calculated using the tridiagonal system of linear algebraic equations: To solve the system it is necessary to use two more equations arising from the boundary conditions for specific problems. For exam- ple the values of the grid function (if they are known) in the points (x x0, n) may be used for solving the system above: f f x0 = ( 0) , f f xn = ( n) . The parameters Iii+1 (i= 0,...,n−1 ) are used as trainable parameters of neural networks. The grid boundaries and spacing of the nodes of ID-spline AF are best chosen experimentally. The optimal selection of grid nodes allows improving the quality of results produced by the neural network. The formula for a parabolic ID-spline is such that the complexity of the calculations does not depend on whether the grid of nodes is uniform or non-uniform. An experimental comparison of the results of image classification from the popular FashionMNIST dataset by convolutional neural 0, x< 0 networks with the ID-spline AFs and the well-known ReLUx( ) =AF was carried out. The results reveal that the usage x x, ≥ 0 of the ID-spline AFs provides better accuracy of neural network operation than the ReLU AF. The training time for two convolutional layers network with two ID-spline AFs is just about 2 times longer than with two instances of ReLU AF. Doubling of the training time due to complexity of the ID-spline formula is the acceptable price for significantly better accuracy of the network. Wherein the difference of an operation speed of the networks with ID-spline and ReLU AFs will be negligible. The use of trainable ID-spline AFs makes it possible to simplify the architecture of neural networks without losing their efficiency. The modification of the well-known neural networks (ResNet etc.) by replacing traditional AFs with ID-spline AFs is a promising approach to increase the neural network operation accuracy. In a majority of cases, such a substitution does not require to train the network from scratch because it allows to use pre-trained on large datasets neuron weights supplied by standard software libraries for neural network construction thus substantially shortening training time.

Forecasting Stock Returns with Artificial Neural Networks

Neural Networks in Business Forecasting ◽

10.4018/978-1-59140-176-6.ch003 ◽

2011 ◽

pp. 47-79 ◽

Cited By ~ 5

Author(s):

Suraphan Thawornwong ◽

David Enke

Keyword(s):

Neural Network ◽

Neural Networks ◽

Artificial Neural Network ◽

Artificial Neural Networks ◽

Stock Returns ◽

Future Research ◽

Network Architectures ◽

Network Applications ◽

Artificial Neural ◽

Neural Network Applications

During the last few years there has been growing literature on applications of artificial neural networks to business and financial domains. In fact, a great deal of attention has been placed in the area of stock return forecasting. This is due to the fact that once artificial neural network applications are successful, monetary rewards will be substantial. Many studies have reported promising results in successfully applying various types of artificial neural network architectures for predicting stock returns. This chapter reviews and discusses various neural network research methodologies used in 45 journal articles that attempted to forecast stock returns. Modeling techniques and suggestions from the literature are also compiled and addressed. The results show that artificial neural networks are an emerging and promising computational technology that will continue to be a challenging tool for future research.

Evaluation of Parameter Settings for Training Neural Networks Using Backpropagation Algorithms

10.4018/978-1-6684-2408-7.ch009 ◽

2022 ◽

pp. 202-226

Author(s):

Leema N. ◽

Khanna H. Nehemiah ◽

Elgin Christo V. R. ◽

Kannan A.

Keyword(s):

Neural Network ◽

Neural Networks ◽

Activation Function ◽

Neural Network Training ◽

Network Parameter ◽

Network Parameters ◽

Network Training ◽

Rate Minimum ◽

Hidden Layer ◽

Function Number

Artificial neural networks (ANN) are widely used for classification, and the training algorithm commonly used is the backpropagation (BP) algorithm. The major bottleneck faced in the backpropagation neural network training is in fixing the appropriate values for network parameters. The network parameters are initial weights, biases, activation function, number of hidden layers and the number of neurons per hidden layer, number of training epochs, learning rate, minimum error, and momentum term for the classification task. The objective of this work is to investigate the performance of 12 different BP algorithms with the impact of variations in network parameter values for the neural network training. The algorithms were evaluated with different training and testing samples taken from the three benchmark clinical datasets, namely, Pima Indian Diabetes (PID), Hepatitis, and Wisconsin Breast Cancer (WBC) dataset obtained from the University of California Irvine (UCI) machine learning repository.

Research on improved convolutional wavelet neural network

Scientific Reports ◽

10.1038/s41598-021-97195-6 ◽

2021 ◽

Vol 11 (1) ◽

Author(s):

Jingwei Liu ◽

Peixuan Li ◽

Xuehan Tang ◽

Jiaxin Li ◽

Jiaming Chen

Keyword(s):

Neural Network ◽

Neural Networks ◽

Convolutional Neural Network ◽

Radial Basis Function ◽

Error Rate ◽

Basis Function ◽

Activation Function ◽

Wavelet Neural Network ◽

Mean Square ◽

The Mean

AbstractArtificial neural networks (ANN) which include deep learning neural networks (DNN) have problems such as the local minimal problem of Back propagation neural network (BPNN), the unstable problem of Radial basis function neural network (RBFNN) and the limited maximum precision problem of Convolutional neural network (CNN). Performance (training speed, precision, etc.) of BPNN, RBFNN and CNN are expected to be improved. Main works are as follows: Firstly, based on existing BPNN and RBFNN, Wavelet neural network (WNN) is implemented in order to get better performance for further improving CNN. WNN adopts the network structure of BPNN in order to get faster training speed. WNN adopts the wavelet function as an activation function, whose form is similar to the radial basis function of RBFNN, in order to solve the local minimum problem. Secondly, WNN-based Convolutional wavelet neural network (CWNN) method is proposed, in which the fully connected layers (FCL) of CNN is replaced by WNN. Thirdly, comparative simulations based on MNIST and CIFAR-10 datasets among the discussed methods of BPNN, RBFNN, CNN and CWNN are implemented and analyzed. Fourthly, the wavelet-based Convolutional Neural Network (WCNN) is proposed, where the wavelet transformation is adopted as the activation function in Convolutional Pool Neural Network (CPNN) of CNN. Fifthly, simulations based on CWNN are implemented and analyzed on the MNIST dataset. Effects are as follows: Firstly, WNN can solve the problems of BPNN and RBFNN and have better performance. Secondly, the proposed CWNN can reduce the mean square error and the error rate of CNN, which means CWNN has better maximum precision than CNN. Thirdly, the proposed WCNN can reduce the mean square error and the error rate of CWNN, which means WCNN has better maximum precision than CWNN.

Analyzing and interpreting neural networks for NLP: A report on the first BlackboxNLP workshop

Natural Language Engineering ◽

10.1017/s135132491900024x ◽

2019 ◽

Vol 25 (4) ◽

pp. 543-557 ◽

Cited By ~ 3

Author(s):

Afra Alishahi ◽

Grzegorz Chrupała ◽

Tal Linzen

Keyword(s):

Neural Network ◽

Neural Networks ◽

Natural Language Processing ◽

Language Processing ◽

Performance Testing ◽

Network Architectures ◽

Empirical Methods ◽

Neural Models ◽

The Impact ◽

Systematic Manipulation

AbstractThe Empirical Methods in Natural Language Processing (EMNLP) 2018 workshop BlackboxNLP was dedicated to resources and techniques specifically developed for analyzing and understanding the inner-workings and representations acquired by neural models of language. Approaches included: systematic manipulation of input to neural networks and investigating the impact on their performance, testing whether interpretable knowledge can be decoded from intermediate representations acquired by neural networks, proposing modifications to neural network architectures to make their knowledge state or generated output more explainable, and examining the performance of networks on simplified or formal languages. Here we review a number of representative studies in each category.