Research on Multi-Dimensional Bayesian Network Classifiers Based on ICA Dimension Reduction

Multi-dimensional Bayesian network classifiers (MBCs) are probabilistic graphical models proposed to solve classification problems. However, in data analysis and preprocessing tasks, one is often confronted with the problem of selecting features from very high dimensional data. To resolve this problem, the covariance analysis and the FastICA algorithm are applied to decrease the dimension and remove redundant information. And then, we only need to construct class subgraph and bridge subgraph of the MBC model with algorithm and mutual information from the processed data, since the new feature variables satisfy independence assumption. The experiment was tested on three benchmark data sets. The theoretically and experimental results show that our method outperforms other state-of-the-art algorithms for multi-dimensional classification in accuracy.

Download Full-text

A novel approach to fully representing the diversity in conditional dependencies for learning Bayesian network classifier

Intelligent Data Analysis ◽

10.3233/ida-194959 ◽

2021 ◽

Vol 25 (1) ◽

pp. 35-55

Author(s):

Limin Wang ◽

Peng Chen ◽

Shenglei Chen ◽

Minghui Sun

Keyword(s):

Bayesian Network ◽

Conditional Independence ◽

Structure Learning ◽

Classification Performance ◽

Data Sets ◽

Independence Assumption ◽

Bayesian Network Classifiers ◽

Novel Approach ◽

Conditional Independence Assumption ◽

Dependence Criterion

Bayesian network classifiers (BNCs) have proved their effectiveness and efficiency in the supervised learning framework. Numerous variations of conditional independence assumption have been proposed to address the issue of NP-hard structure learning of BNC. However, researchers focus on identifying conditional dependence rather than conditional independence, and information-theoretic criteria cannot identify the diversity in conditional (in)dependencies for different instances. In this paper, the maximum correlation criterion and minimum dependence criterion are introduced to sort attributes and identify conditional independencies, respectively. The heuristic search strategy is applied to find possible global solution for achieving the trade-off between significant dependency relationships and independence assumption. Our extensive experimental evaluation on widely used benchmark data sets reveals that the proposed algorithm achieves competitive classification performance compared to state-of-the-art single model learners (e.g., TAN, KDB, KNN and SVM) and ensemble learners (e.g., ATAN and AODE).

Download Full-text

MODULAR FEATURE SELECTION USING RELATIVE IMPORTANCE FACTORS

International Journal of Computational Intelligence and Applications ◽

10.1142/s1469026804001021 ◽

2004 ◽

Vol 04 (01) ◽

pp. 57-75 ◽

Cited By ~ 1

Author(s):

SHENG-UEI GUAN ◽

FANGMING ZHU ◽

PENG LI

Keyword(s):

Feature Selection ◽

Classification Accuracy ◽

Feature Space ◽

Data Sets ◽

Relative Importance ◽

Classification Problems ◽

Feature Selection Technique ◽

Importance Factor ◽

Rule Sets ◽

New Feature

Feature selection plays an important role in finding relevant or irrelevant features in classification. Genetic algorithms (GAs) have been used as conventional methods for classifiers to adaptively evolve solutions for classification problems. In this paper, we explore the use of feature selection in modular GA-based classification. We propose a new feature selection technique, Relative Importance Factor (RIF), to find irrelevant features in the feature space of each module. By removing these features, we aim to improve classification accuracy and reduce the dimensionality of classification problems. Benchmark classification data sets are used to evaluate the proposed approaches. The experiment results show that RIF can be used to determine irrelevant features and help achieve higher classification accuracy with the feature space dimension reduced. The complexity of the resulting rule sets is also reduced which means the modular classifiers with irrelevant features removed will be able to classify data with a higher throughput.

Download Full-text

Building Bayesian network classifiers through a Bayesian complexity monitoring system

Proceedings of the Institution of Mechanical Engineers Part C Journal of Mechanical Engineering Science ◽

10.1243/09544062jmes1243 ◽

2008 ◽

Vol 223 (3) ◽

pp. 743-755 ◽

Cited By ~ 10

Author(s):

G A Ruz ◽

D T Pham

Keyword(s):

Bayesian Network ◽

Training Data ◽

Machine Learning Techniques ◽

Data Sets ◽

Learning Method ◽

Benchmark Data ◽

Bayesian Network Classifiers ◽

Tree Construction ◽

Learning Techniques ◽

Bayesian Network Classifier

Nowadays, the need for practical yet efficient machine learning techniques for engineering applications are highly in demand. A new learning method for building Bayesian network classifiers is presented in this article. The proposed method augments the naive Bayesian (NB) classifier by using the Chow and Liu tree construction method, but introducing a Bayesian approach to control the accuracy and complexity of the resulting network, which yields simple structures that are not necessarily a spanning tree. Experiments by using benchmark data sets show that the number of augmenting edges by using the proposed learning method depends on the number of training data used. The classification accuracy was better, or at least equal, to the NB and the tree augmented NB models when tested on 10 benchmark data sets. The evaluation on a real industrial application showed that the simple Bayesian network classifier outperformed the C4.5 and the random forest algorithms and achieved competitive results against C5.0 and a neural network.

Download Full-text

A lazy feature selection method for multi-label classification

Intelligent Data Analysis ◽

10.3233/ida-194878 ◽

2021 ◽

Vol 25 (1) ◽

pp. 21-34

Author(s):

Rafael B. Pereira ◽

Alexandre Plastino ◽

Bianca Zadrozny ◽

Luiz H.C. Merschmann

Keyword(s):

Feature Selection ◽

Text Categorization ◽

Feature Selection Method ◽

Selection Method ◽

Video Classification ◽

Classification Problems ◽

Class Label ◽

New Feature ◽

Feature Selection Techniques ◽

Biomolecular Analysis

In many important application domains, such as text categorization, biomolecular analysis, scene or video classification and medical diagnosis, instances are naturally associated with more than one class label, giving rise to multi-label classification problems. This has led, in recent years, to a substantial amount of research in multi-label classification. More specifically, feature selection methods have been developed to allow the identification of relevant and informative features for multi-label classification. This work presents a new feature selection method based on the lazy feature selection paradigm and specific for the multi-label context. Experimental results show that the proposed technique is competitive when compared to multi-label feature selection techniques currently used in the literature, and is clearly more scalable, in a scenario where there is an increasing amount of data.

Download Full-text

A branch & bound algorithm to determine optimal bivariate splits for oblique decision tree induction

Applied Intelligence ◽

10.1007/s10489-021-02281-x ◽

2021 ◽

Author(s):

Ferdinand Bollwein ◽

Stephan Westphal

Keyword(s):

Decision Tree ◽

Feature Space ◽

Classification Problems ◽

Decision Tree Induction ◽

Single Attribute ◽

Global Optimal ◽

The Individual ◽

Tree Building ◽

Very High ◽

Multiclass Classification Problems

AbstractUnivariate decision tree induction methods for multiclass classification problems such as CART, C4.5 and ID3 continue to be very popular in the context of machine learning due to their major benefit of being easy to interpret. However, as these trees only consider a single attribute per node, they often get quite large which lowers their explanatory value. Oblique decision tree building algorithms, which divide the feature space by multidimensional hyperplanes, often produce much smaller trees but the individual splits are hard to interpret. Moreover, the effort of finding optimal oblique splits is very high such that heuristics have to be applied to determine local optimal solutions. In this work, we introduce an effective branch and bound procedure to determine global optimal bivariate oblique splits for concave impurity measures. Decision trees based on these bivariate oblique splits remain fairly interpretable due to the restriction to two attributes per split. The resulting trees are significantly smaller and more accurate than their univariate counterparts due to their ability of adapting better to the underlying data and capturing interactions of attribute pairs. Moreover, our evaluation shows that our algorithm even outperforms algorithms based on heuristically obtained multivariate oblique splits despite the fact that we are focusing on two attributes only.

Download Full-text

Structure Extension of Tree-Augmented Naive Bayes

Entropy ◽

10.3390/e21080721 ◽

2019 ◽

Vol 21 (8) ◽

pp. 721 ◽

Cited By ~ 1

Author(s):

YuGuang Long ◽

LiMin Wang ◽

MingHui Sun

Keyword(s):

Naive Bayes ◽

Classification Performance ◽

Naïve Bayes ◽

Training Data ◽

Conditional Probability Distribution ◽

Independence Assumption ◽

Bayesian Network Classifiers ◽

Leibler Divergence ◽

The Difference ◽

Structure Extension

Due to the simplicity and competitive classification performance of the naive Bayes (NB), researchers have proposed many approaches to improve NB by weakening its attribute independence assumption. Through the theoretical analysis of Kullback–Leibler divergence, the difference between NB and its variations lies in different orders of conditional mutual information represented by these augmenting edges in the tree-shaped network structure. In this paper, we propose to relax the independence assumption by further generalizing tree-augmented naive Bayes (TAN) from 1-dependence Bayesian network classifiers (BNC) to arbitrary k-dependence. Sub-models of TAN that are built to respectively represent specific conditional dependence relationships may “best match” the conditional probability distribution over the training data. Extensive experimental results reveal that the proposed algorithm achieves bias-variance trade-off and substantially better generalization performance than state-of-the-art classifiers such as logistic regression.

Download Full-text