scholarly journals An Empirical Evaluation of Rule Extraction from Recurrent Neural Networks

2018 ◽  
Vol 30 (9) ◽  
pp. 2568-2591 ◽  
Author(s):  
Qinglong Wang ◽  
Kaixuan Zhang ◽  
Alexander G. Ororbia II ◽  
Xinyu Xing ◽  
Xue Liu ◽  
...  
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Credit Scoring ◽  
Empirical Evaluation ◽  
Rule Extraction ◽  
Production Rules ◽  
Recursive Models ◽  
Box Models ◽  
Highly Nonlinear ◽  
Black Box Models

Rule extraction from black box models is critical in domains that require model validation before implementation, as can be the case in credit scoring and medical diagnosis. Though already a challenging problem in statistical learning in general, the difficulty is even greater when highly nonlinear, recursive models, such as recurrent neural networks (RNNs), are fit to data. Here, we study the extraction of rules from second-order RNNs trained to recognize the Tomita grammars. We show that production rules can be stably extracted from trained RNNs and that in certain cases, the rules outperform the trained RNNs.

scholarly journals One-Dimensional Convolutional Neural Networks with Feature Selection for Highly Concise Rule Extraction from Credit Scoring Datasets with Heterogeneous Attributes

Electronics ◽  
2020 ◽  
Vol 9 (8) ◽  
pp. 1318
Author(s):  
Yoichi Hayashi ◽  
Naoki Takano
Keyword(s):  
Neural Networks ◽  
Credit Scoring ◽  
Extraction Methods ◽  
Rule Extraction ◽  
Financial Industry ◽  
New Approach ◽  
New Era ◽  
One Dimensional ◽  
Recursive Rule ◽  
Fully Connected

Convolution neural networks (CNNs) have proven effectiveness, but they are not applicable to all datasets, such as those with heterogeneous attributes, which are often used in the finance and banking industries. Such datasets are difficult to classify, and to date, existing high-accuracy classifiers and rule-extraction methods have not been able to achieve sufficiently high classification accuracies or concise classification rules. This study aims to provide a new approach for achieving transparency and conciseness in credit scoring datasets with heterogeneous attributes by using a one-dimensional (1D) fully-connected layer first CNN combined with the Recursive-Rule Extraction (Re-RX) algorithm with a J48graft decision tree (hereafter 1D FCLF-CNN). Based on a comparison between the proposed 1D FCLF-CNN and existing rule extraction methods, our architecture enabled the extraction of the most concise rules (6.2) and achieved the best accuracy (73.10%), i.e., the highest interpretability–priority rule extraction. These results suggest that the 1D FCLF-CNN with Re-RX with J48graft is very effective for extracting highly concise rules for heterogeneous credit scoring datasets. Although it does not completely overcome the accuracy–interpretability dilemma for deep learning, it does appear to resolve this issue for credit scoring datasets with heterogeneous attributes, and thus, could lead to a new era in the financial industry.

scholarly journals Modelling Acetification with Artificial Neural Networks and Comparison with Alternative Procedures

Processes ◽  
2020 ◽  
Vol 8 (7) ◽  
pp. 749 ◽  
Author(s):  
Jorge E. Jiménez-Hornero ◽  
Inés María Santos-Dueñas ◽  
Isidoro García-García
Keyword(s):  
Neural Networks ◽  
Acetic Acid ◽  
Artificial Neural Networks ◽  
First Principles ◽  
Goodness Of Fit ◽  
Black Box ◽  
Operating Conditions ◽  
Box Models ◽  
Artificial Neural ◽  
Black Box Models

Modelling techniques allow certain processes to be characterized and optimized without the need for experimentation. One of the crucial steps in vinegar production is the biotransformation of ethanol into acetic acid by acetic bacteria. This step has been extensively studied by using two predictive models: first-principles models and black-box models. The fact that first-principles models are less accurate than black-box models under extreme bacterial growth conditions suggests that the kinetic equations used by the former, and hence their goodness of fit, can be further improved. By contrast, black-box models predict acetic acid production accurately enough under virtually any operating conditions. In this work, we trained black-box models based on Artificial Neural Networks (ANNs) of the multilayer perceptron (MLP) type and containing a single hidden layer to model acetification. The small number of data typically available for a bioprocess makes it rather difficult to identify the most suitable type of ANN architecture in terms of indices such as the mean square error (MSE). This places ANN methodology at a disadvantage against alternative techniques and, especially, polynomial modelling.

Rule Extraction from Recurrent Neural Networks: ATaxonomy and Review

2005 ◽  
Vol 17 (6) ◽  
pp. 1223-1263 ◽  
Author(s):  
Henrik Jacobsson
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Ad Hoc ◽  
Rule Extraction ◽  
Performance Evaluations ◽  
Research Issues ◽  
Open Research ◽  
Finite State ◽  
Push Forward ◽  
Quantitative Performance

Rule extraction (RE) from recurrent neural networks (RNNs) refers to finding models of the underlying RNN, typically in the form of finite state machines, that mimic the network to a satisfactory degree while having the advantage of being more transparent. RE from RNNs can be argued to allow a deeper and more profound form of analysis of RNNs than other, more or less ad hoc methods. RE may give us understanding of RNNs in the intermediate levels between quite abstract theoretical knowledge of RNNs as a class of computing devices and quantitative performance evaluations of RNN instantiations. The development of techniques for extraction of rules from RNNs has been an active field since the early 1990s. This article reviews the progress of this development and analyzes it in detail. In order to structure the survey and evaluate the techniques, a taxonomy specifically designed for this purpose has been developed. Moreover, important open research issues are identified that, if addressed properly, possibly can give the field a significant push forward.

Real-time daily flow forecasting using black-box models, diffusion processes, and neural networks

2000 ◽  
Vol 27 (4) ◽  
pp. 671-682 ◽  
Author(s):  
N Lauzon ◽  
J Rousselle ◽  
S Birikundavyi ◽  
H T Trung
Keyword(s):  
Neural Network ◽  
Neural Networks ◽  
Diffusion Process ◽  
Diffusion Processes ◽  
Black Box ◽  
Linear Type ◽  
Modeling Approaches ◽  
Box Models ◽  
Almost All ◽  
Black Box Models

The purpose of this study is to compare three modeling approaches used for the prediction of daily natural flows 1-7 days ahead. Linear black-box models, which have been commonly used for modeling flows, constitute the first approach. The second approach, a linear type in the context of our application, is less known in the water resources field and is identified by the term diffusion process. The third approach uses models called neural networks, which have gained interest in many fields. All these approaches were tested on 15 watersheds from the Saguenay - Lac-Saint-Jean hydrographic system, located in the province of Quebec, Canada. Because the watersheds possess different physical characteristics, the models were tested under several runoff conditions. In this article, the focus is on results; all approaches along with their conditions of use have been detailed elsewhere in the literature. The results obtained showed that neural networks constitute, for almost all the watersheds studied, the best approach to forecast daily natural flows. The more flexible structure of neural networks allows a best reproduction of complex runoff conditions. However, neural networks are more sensitive to outliers present in observed natural flow series, which are used as inputs in the three models tested. In practice, to model flows at specific periods of the year, it seems preferable to establish seasonal models. If a neural network has an inadequate structure for the period under consideration, then it may produce less convincing results than the other two modeling approaches tested in this study.Key words: forecasts, flows, black-box model, diffusion process, neural network.

scholarly journals Continual learning for recurrent neural networks: An empirical evaluation

2021 ◽  
Author(s):  
Andrea Cossu ◽  
Antonio Carta ◽  
Vincenzo Lomonaco ◽  
Davide Bacciu
Keyword(s):  
Neural Networks ◽  
Recurrent Neural Networks ◽  
Empirical Evaluation ◽  
Continual Learning
Sign in / Sign up

Export Citation Format

Share Document