A novel approach to fully representing the diversity in conditional dependencies for learning Bayesian network classifier

2021 ◽  
Vol 25 (1) ◽  
pp. 35-55
Author(s):  
Limin Wang ◽  
Peng Chen ◽  
Shenglei Chen ◽  
Minghui Sun
Keyword(s):  
Bayesian Network ◽  
Conditional Independence ◽  
Structure Learning ◽  
Classification Performance ◽  
Data Sets ◽  
Independence Assumption ◽  
Bayesian Network Classifiers ◽  
Novel Approach ◽  
Conditional Independence Assumption ◽  
Dependence Criterion

Bayesian network classifiers (BNCs) have proved their effectiveness and efficiency in the supervised learning framework. Numerous variations of conditional independence assumption have been proposed to address the issue of NP-hard structure learning of BNC. However, researchers focus on identifying conditional dependence rather than conditional independence, and information-theoretic criteria cannot identify the diversity in conditional (in)dependencies for different instances. In this paper, the maximum correlation criterion and minimum dependence criterion are introduced to sort attributes and identify conditional independencies, respectively. The heuristic search strategy is applied to find possible global solution for achieving the trade-off between significant dependency relationships and independence assumption. Our extensive experimental evaluation on widely used benchmark data sets reveals that the proposed algorithm achieves competitive classification performance compared to state-of-the-art single model learners (e.g., TAN, KDB, KNN and SVM) and ensemble learners (e.g., ATAN and AODE).

Research on Multi-Dimensional Bayesian Network Classifiers Based on ICA Dimension Reduction

2013 ◽  
Vol 380-384 ◽  
pp. 2593-2596
Author(s):  
Xiu Fang Zhang ◽  
You Long Yang ◽  
Xing Jia Tang
Keyword(s):  
Bayesian Network ◽  
Graphical Models ◽  
Data Sets ◽  
Classification Problems ◽  
Independence Assumption ◽  
Bayesian Network Classifiers ◽  
Dimensional Classification ◽  
Fastica Algorithm ◽  
New Feature ◽  
Very High

Multi-dimensional Bayesian network classifiers (MBCs) are probabilistic graphical models proposed to solve classification problems. However, in data analysis and preprocessing tasks, one is often confronted with the problem of selecting features from very high dimensional data. To resolve this problem, the covariance analysis and the FastICA algorithm are applied to decrease the dimension and remove redundant information. And then, we only need to construct class subgraph and bridge subgraph of the MBC model with algorithm and mutual information from the processed data, since the new feature variables satisfy independence assumption. The experiment was tested on three benchmark data sets. The theoretically and experimental results show that our method outperforms other state-of-the-art algorithms for multi-dimensional classification in accuracy.

scholarly journals Efficient Heuristics for Structure Learning of k-Dependence Bayesian Classifier

Entropy ◽  
2018 ◽  
Vol 20 (12) ◽  
pp. 897 ◽  
Author(s):  
Yang Liu ◽  
Limin Wang ◽  
Minghui Sun
Keyword(s):  
Bayesian Network ◽  
Structure Learning ◽  
State Of The Art ◽  
Classification Performance ◽  
Bayesian Classifier ◽  
Bayesian Network Classifiers ◽  
Discriminative Model ◽  
Minimal Redundancy ◽  
Structure Complexity ◽  
Maximal Relevance

The rapid growth in data makes the quest for highly scalable learners a popular one. To achieve the trade-off between structure complexity and classification accuracy, the k-dependence Bayesian classifier (KDB) allows to represent different number of interdependencies for different data sizes. In this paper, we proposed two methods to improve the classification performance of KDB. Firstly, we use the minimal-redundancy-maximal-relevance analysis, which sorts the predictive features to identify redundant ones. Then, we propose an improved discriminative model selection to select an optimal sub-model by removing redundant features and arcs in the Bayesian network. Experimental results on 40 UCI datasets demonstrate that these two techniques are complementary and the proposed algorithm achieves competitive classification performance, and less classification time than other state-of-the-art Bayesian network classifiers like tree-augmented naive Bayes and averaged one-dependence estimators.

scholarly journals Structure Learning of Bayesian Network Based on Adaptive Thresholding

Entropy ◽  
2019 ◽  
Vol 21 (7) ◽  
pp. 665
Author(s):  
Yang Zhang ◽  
Limin Wang ◽  
Zhiyi Duan ◽  
Minghui Sun
Keyword(s):  
Bayesian Network ◽  
Structure Learning ◽  
Mean Squared Error ◽  
Structural Complexity ◽  
Classification Performance ◽  
Adaptive Thresholding ◽  
Bayesian Network Classifiers ◽  
Structure Reliability ◽  
Adaptive Thresholds ◽  
Error Bias

Direct dependencies and conditional dependencies in restricted Bayesian network classifiers (BNCs) are two basic kinds of dependencies. Traditional approaches, such as filter and wrapper, have proved to be beneficial to identify non-significant dependencies one by one, whereas the high computational overheads make them inefficient especially for those BNCs with high structural complexity. Study of the distributions of information-theoretic measures provides a feasible approach to identifying non-significant dependencies in batch that may help increase the structure reliability and avoid overfitting. In this paper, we investigate two extensions to the k-dependence Bayesian classifier, MI-based feature selection, and CMI-based dependence selection. These two techniques apply a novel adaptive thresholding method to filter out redundancy and can work jointly. Experimental results on 30 datasets from the UCI machine learning repository demonstrate that adaptive thresholds can help distinguish between dependencies and independencies and the proposed algorithm achieves competitive classification performance compared to several state-of-the-art BNCs in terms of 0–1 loss, root mean squared error, bias, and variance.

scholarly journals Structure Extension of Tree-Augmented Naive Bayes

Entropy ◽  
2019 ◽  
Vol 21 (8) ◽  
pp. 721 ◽  
Author(s):  
YuGuang Long ◽  
LiMin Wang ◽  
MingHui Sun
Keyword(s):  
Naive Bayes ◽  
Classification Performance ◽  
Naïve Bayes ◽  
Training Data ◽  
Conditional Probability Distribution ◽  
Independence Assumption ◽  
Bayesian Network Classifiers ◽  
Leibler Divergence ◽  
The Difference ◽  
Structure Extension

Due to the simplicity and competitive classification performance of the naive Bayes (NB), researchers have proposed many approaches to improve NB by weakening its attribute independence assumption. Through the theoretical analysis of Kullback–Leibler divergence, the difference between NB and its variations lies in different orders of conditional mutual information represented by these augmenting edges in the tree-shaped network structure. In this paper, we propose to relax the independence assumption by further generalizing tree-augmented naive Bayes (TAN) from 1-dependence Bayesian network classifiers (BNC) to arbitrary k-dependence. Sub-models of TAN that are built to respectively represent specific conditional dependence relationships may “best match” the conditional probability distribution over the training data. Extensive experimental results reveal that the proposed algorithm achieves bias-variance trade-off and substantially better generalization performance than state-of-the-art classifiers such as logistic regression.

Evaluation of Bayesian Network Structure Learning Using Elephant Swarm Water Search Algorithm

2020 ◽  
pp. 139-159
Author(s):  
Shahab Wahhab Kareem ◽  
Mehmet Cudi Okur
Keyword(s):  
Simulated Annealing ◽  
Bayesian Network ◽  
Network Structure ◽  
Structure Learning ◽  
Search Algorithm ◽  
Confusion Matrix ◽  
Data Sets ◽  
Greedy Search ◽  
Bayesian Network Structure ◽  
Bayesian Network Structure Learning

Bayesian networks are useful analytical models for designing the structure of knowledge in machine learning which can represent probabilistic dependency relationships among the variables. The authors present the Elephant Swarm Water Search Algorithm (ESWSA) for Bayesian network structure learning. In the algorithm; Deleting, Reversing, Inserting, and Moving are used to make the ESWSA for reaching the optimal structure solution. Mainly, water search strategy of elephants during drought periods is used in the ESWSA algorithm. The proposed method is compared with Pigeon Inspired Optimization, Simulated Annealing, Greedy Search, Hybrid Bee with Simulated Annealing, and Hybrid Bee with Greedy Search using BDeu score function as a metric for all algorithms. They investigated the confusion matrix performances of these techniques utilizing various benchmark data sets. As presented by the results of evaluations, the proposed algorithm achieves better performance than the other algorithms and produces better scores as well as the better values.

scholarly journals Adapting Hidden Naive Bayes for Text Classification

Mathematics ◽  
2021 ◽  
Vol 9 (19) ◽  
pp. 2378
Author(s):  
Shengfeng Gan ◽  
Shiqi Shao ◽  
Long Chen ◽  
Liangjun Yu ◽  
Liangxiao Jiang
Keyword(s):  
Text Classification ◽  
Conditional Independence ◽  
Structure Learning ◽  
Naive Bayes ◽  
Learning Algorithm ◽  
Classification Performance ◽  
Naïve Bayes ◽  
Efficiency And Effectiveness ◽  
The One ◽  
Structure Extension

Due to its simplicity, efficiency, and effectiveness, multinomial naive Bayes (MNB) has been widely used for text classification. As in naive Bayes (NB), its assumption of the conditional independence of features is often violated and, therefore, reduces its classification performance. Of the numerous approaches to alleviating its assumption of the conditional independence of features, structure extension has attracted less attention from researchers. To the best of our knowledge, only structure-extended MNB (SEMNB) has been proposed so far. SEMNB averages all weighted super-parent one-dependence multinomial estimators; therefore, it is an ensemble learning model. In this paper, we propose a single model called hidden MNB (HMNB) by adapting the well-known hidden NB (HNB). HMNB creates a hidden parent for each feature, which synthesizes all the other qualified features’ influences. For HMNB to learn, we propose a simple but effective learning algorithm without incurring a high-computational-complexity structure-learning process. Our improved idea can also be used to improve complement NB (CNB) and the one-versus-all-but-one model (OVA), and the resulting models are simply denoted as HCNB and HOVA, respectively. The extensive experiments on eleven benchmark text classification datasets validate the effectiveness of HMNB, HCNB, and HOVA.

scholarly journals Efficient Markov Network Structure Discovery Using Independence Tests

2009 ◽  
Vol 35 ◽  
pp. 449-484 ◽  
Author(s):  
F. Bromberg ◽  
D. Margaritis ◽  
V. Honavar
Keyword(s):  
Conditional Independence ◽  
Structure Learning ◽  
Statistical Tests ◽  
Likelihood Estimation ◽  
Data Sets ◽  
Statistical Independence ◽  
Real World Data ◽  
Independence Tests ◽  
Markov Network ◽  
Experimental Comparisons

We present two algorithms for learning the structure of a Markov network from data: GSMN* and GSIMN. Both algorithms use statistical independence tests to infer the structure by successively constraining the set of structures consistent with the results of these tests. Until very recently, algorithms for structure learning were based on maximum likelihood estimation, which has been proved to be NP-hard for Markov networks due to the difficulty of estimating the parameters of the network, needed for the computation of the data likelihood. The independence-based approach does not require the computation of the likelihood, and thus both GSMN* and GSIMN can compute the structure efficiently (as shown in our experiments). GSMN* is an adaptation of the Grow-Shrink algorithm of Margaritis and Thrun for learning the structure of Bayesian networks. GSIMN extends GSMN* by additionally exploiting Pearl's well-known properties of the conditional independence relation to infer novel independences from known ones, thus avoiding the performance of statistical tests to estimate them. To accomplish this efficiently GSIMN uses the Triangle theorem, also introduced in this work, which is a simplified version of the set of Markov axioms. Experimental comparisons on artificial and real-world data sets show GSIMN can yield significant savings with respect to GSMN*, while generating a Markov network with comparable or in some cases improved quality. We also compare GSIMN to a forward-chaining implementation, called GSIMN-FCH, that produces all possible conditional independences resulting from repeatedly applying Pearl's theorems on the known conditional independence tests. The results of this comparison show that GSIMN, by the sole use of the Triangle theorem, is nearly optimal in terms of the set of independences tests that it infers.

Sign in / Sign up

Export Citation Format

Share Document