Automatic Fault Detection for Deep Learning Programs Using Graph Transformations

Nowadays, we are witnessing an increasing demand in both corporates and academia for exploiting Deep Learning ( DL ) to solve complex real-world problems. A DL program encodes the network structure of a desirable DL model and the process by which the model learns from the training dataset. Like any software, a DL program can be faulty, which implies substantial challenges of software quality assurance, especially in safety-critical domains. It is therefore crucial to equip DL development teams with efficient fault detection techniques and tools. In this article, we propose NeuraLint , a model-based fault detection approach for DL programs, using meta-modeling and graph transformations. First, we design a meta-model for DL programs that includes their base skeleton and fundamental properties. Then, we construct a graph-based verification process that covers 23 rules defined on top of the meta-model and implemented as graph transformations to detect faults and design inefficiencies in the generated models (i.e., instances of the meta-model). First, the proposed approach is evaluated by finding faults and design inefficiencies in 28 synthesized examples built from common problems reported in the literature. Then NeuraLint successfully finds 64 faults and design inefficiencies in 34 real-world DL programs extracted from Stack Overflow posts and GitHub repositories. The results show that NeuraLint effectively detects faults and design issues in both synthesized and real-world examples with a recall of 70.5% and a precision of 100%. Although the proposed meta-model is designed for feedforward neural networks, it can be extended to support other neural network architectures such as recurrent neural networks. Researchers can also expand our set of verification rules to cover more types of issues in DL programs.

Short range SW monsoon rainfall forecasting over India using neural networks

Feedforward Neural Networks are used for daily precipitation forecast using several test stations all over India. The six year European Centre of Medium Range Weather Forecasting (ECMWF) data is used with the training set consisting of the four year data from 1985-1988 and validation set consisting of the data from 1989-1990. Neural networks are used to develop a concurrent relationship between precipitation and other atmospheric variables. No attempt is made to select optimal variables for this study and the inputs are chosen to be same as the ones obtained earlier at National Center for Medium Range Weather Forecasting (NCMRWF) in developing a linear regression model. Neural networks are found to yield results which are atleast as good as linear regression and in several cases yield 10 - 20 % improvement. This is encouraging since the variable selection has so far been optimized for linear regression.

Multiple IoT based Network Attacks Discrimination by Multilayer Feedforward Neural Networks

In this paper a new neural model for detection of multiple network IoT-based attacks, such as DDoS TCP, UDP, and HHTP flood, is presented. It consists of feedforward multilayer network with back propagation. A general algorithm for its optimization during training is proposed, leading to proper number of neurons in the hidden layers. The Scaled Gradient Descent algorithm and the Adam optimization are studied with better classification results, obtained by the developed classifiers, using the latter. Tangent hyperbolic function appears to be proper selection for the hidden neurons. Two sets of features, gathered from aggregated records of the network traffic, are tested, containing 8 and 10 components. While more accurate results are obtained for the 10-feature set, the 8-feature set offers twice lower training time and seems applicable for real-world applications. The detection rate for 7 of 10 different network attacks, primarily various types of floods, is higher than 90% and for 3 of them – mainly reconnaissance and keylogging activities with low intensity of the generated traffic, deviates between 57% and 68%. The classifier is considered applicable for industrial implementation.

Detection of IoT based DDoS Attacks by Network Traffic Analysis using Feedforward Neural Networks

In this paper an optimized feedforward neural network model is proposed for detection of IoT based DDoS attacks by network traffic analysis aimed towards a specific target which could be constantly monitored by a tap. The proposed model is applicable for DoS and DDoS attacks which consist of TCP, UDP and HTTP flood and also against keylogging, data exfiltration, OS fingerprint and service scan activities. It simply differentiates such kind of network traffic from normal network flows. The neural network uses Adam optimization as a solver and the hyperbolic tangent activation function in all neurons from a single hidden layer. The number of hidden neurons could be varied, depending on targeted accuracy and processing speed. Testing over the Bot IoT dataset reveals that developed models are applicable using 8 or 10 features and achieved discrimination error of 4.91.10-3%.

Optimizing connection weights in neural networks using hybrid metaheuristics algorithm

The learning process of artificial neural networks is an important and complex task in the supervised learning field. The main difficulty of training a neural network is the process of fine-tuning the best set of control parameters in terms of weight and bias. This paper presents a new training method based on hybrid particle swarm optimization with Multi-Verse Optimization (PMVO) to train the feedforward neural networks. The hybrid algorithm is utilized to search better in solution space which proves its efficiency in reducing the problems of trapping in local minima. The performance of the proposed approach was compared with five evolutionary techniques and the standard momentum backpropagation and adaptive learning rate. The comparison was benchmarked and evaluated using six bio-medical datasets. The results of the comparative study show that PMVO outperformed other training methods in most datasets and can be an alternative to other training methods.

Collocation polynomial neural forms and domain fragmentation for solving initial value problems

Several neural network approaches for solving differential equations employ trial solutions with a feedforward neural network. There are different means to incorporate the trial solution in the construction, for instance, one may include them directly in the cost function. Used within the corresponding neural network, the trial solutions define the so-called neural form. Such neural forms represent general, flexible tools by which one may solve various differential equations. In this article, we consider time-dependent initial value problems, which require to set up the neural form framework adequately. The neural forms presented up to now in the literature for such a setting can be considered as first-order polynomials. In this work, we propose to extend the polynomial order of the neural forms. The novel collocation-type construction includes several feedforward neural networks, one for each order. Additionally, we propose the fragmentation of the computational domain into subdomains. The neural forms are solved on each subdomain, whereas the interfacing grid points overlap in order to provide initial values over the whole fragmentation. We illustrate in experiments that the combination of collocation neural forms of higher order and the domain fragmentation allows to solve initial value problems over large domains with high accuracy and reliability.

Chapter 5. Answering Natural-Language Questions with Neuro-Symbolic Knowledge Bases

Symbolic reasoning systems based on first-order logics are computationally powerful, and feedforward neural networks are computationally efficient, so unless P=NP, neural networks cannot, in general, emulate symbolic logics. Hence bridging the gap between neural and symbolic methods requires achieving a delicate balance: one needs to incorporate just enough of symbolic reasoning to be useful for a task, but not so much as to cause computational intractability. In this chapter we first present results that make this claim precise, and then use these formal results to inform the choice of a neuro-symbolic knowledge-based reasoning system, based on a set-based dataflow query language. We then present experimental results with a number of variants of this neuro-symbolic reasoner, and also show that this neuro-symbolic reasoner can be closely integrated into modern neural language models.

Study of Nonlinear Models of Oscillatory Systems by Applying an Intelligent Computational Technique

In this paper, we have analyzed the mathematical model of various nonlinear oscillators arising in different fields of engineering. Further, approximate solutions for different variations in oscillators are studied by using feedforward neural networks (NNs) based on the backpropagated Levenberg–Marquardt algorithm (BLMA). A data set for different problem scenarios for the supervised learning of BLMA has been generated by the Runge–Kutta method of order 4 (RK-4) with the “NDSolve” package in Mathematica. The worth of the approximate solution by NN-BLMA is attained by employing the processing of testing, training, and validation of the reference data set. For each model, convergence analysis, error histograms, regression analysis, and curve fitting are considered to study the robustness and accuracy of the design scheme.