A bias/variance decomposition for models using collective inference

The bias/variance decomposition of mean-squared error is well understood and relatively straightforward. In this note, a similar simple decomposition is derived, valid for any kind of error measure that, when using the appropriate probability model, can be derived from a Kullback-Leibler divergence or log-likelihood.

Download Full-text

Data-Driven Bayesian Network Learning: A Bi-Objective Approach to Address the Bias-Variance Decomposition

Mathematical and Computational Applications ◽

10.3390/mca25020037 ◽

2020 ◽

Vol 25 (2) ◽

pp. 37 ◽

Cited By ~ 1

Author(s):

Vicente-Josué Aguilera-Rueda ◽

Nicandro Cruz-Ramírez ◽

Efrén Mezura-Montes

Keyword(s):

Bayesian Networks ◽

Bayesian Network ◽

Variance Decomposition ◽

Data Driven ◽

End User ◽

Learning Problem ◽

Nsga Ii ◽

Bias Variance ◽

Core Idea ◽

Nondominated Sorting

We present a novel bi-objective approach to address the data-driven learning problem of Bayesian networks. Both the log-likelihood and the complexity of each candidate Bayesian network are considered as objectives to be optimized by our proposed algorithm named Nondominated Sorting Genetic Algorithm for learning Bayesian networks (NS2BN) which is based on the well-known NSGA-II algorithm. The core idea is to reduce the implicit selection bias-variance decomposition while identifying a set of competitive models using both objectives. Numerical results suggest that, in stark contrast to the single-objective approach, our bi-objective approach is useful to find competitive Bayesian networks especially in the complexity. Furthermore, our approach presents the end user with a set of solutions by showing different Bayesian network and their respective MDL and classification accuracy results.

Download Full-text

Bias Variance Decomposition

10.1007/springerreference_178786 ◽

2012 ◽

Keyword(s):

Variance Decomposition ◽

Bias Variance

Download Full-text

Start Simple and then Refine: Bias-Variance Decomposition as a Diagnosis Tool for Leakage Profiling

IEEE Transactions on Computers ◽

10.1109/tc.2017.2731342 ◽

2018 ◽

Vol 67 (2) ◽

pp. 268-283 ◽

Cited By ~ 5

Author(s):

Liran Lerman ◽

Nikita Veshchikov ◽

Olivier Markowitch ◽

Francois-Xavier Standaert

Keyword(s):

Variance Decomposition ◽

Diagnosis Tool ◽

Bias Variance

Download Full-text

Combining Predictors

DAIMI Report Series ◽

10.7146/dpb.v29i550.7203 ◽

2000 ◽

Vol 29 (550) ◽

Cited By ~ 1

Author(s):

Jakob Vogdrup Hansen

Keyword(s):

Machine Learning ◽

Cross Validation ◽

Error Function ◽

Variance Decomposition ◽

Ensemble Method ◽

Error Functions ◽

The Family ◽

Bias Variance ◽

The Cross ◽

Opinion Pool

The most important theoretical tool in connection with machine learning is the bias/variance decomposition of error functions. Together with Tom Heskes, I have found the family of error functions with a natural bias/variance decomposition that has target independent variance. It is shown that no other group of error functions can be decomposed in the same way. An open problem in the machine learning community is thereby solved. The error functions are derived from the deviance measure on distributions in the one-parameter exponential family. It is therefore called the deviance error family. A bias/variance decomposition can also be viewed as an ambiguity decomposition for an ensemble method. The family of error functions with a natural bias/variance decomposition that has target independent variance can therefore be of use in connection with ensemble methods. The logarithmic opinion pool ensemble method has been developed together with Anders Krogh. It is based on the logarithmic opinion pool ambiguity decomposition using the Kullback-Leibler error function. It has been extended to the cross-validation logarithmic opinion pool ensemble method. The advantage of the cross-validation logarithmic opinion pool ensemble method is that it can use unlabeled data to estimate the generalization error, while it still uses the entire labeled example set for training. The cross-validation logarithmic opinion pool ensemble method is easily reformulated for another error function, as long as the error function has an ambiguity decomposition with target independent ambiguity. It is therefore possible to use the cross-validation ensemble method on all error functions in the deviance error family.

Download Full-text

Bias-variance decomposition in Genetic Programming

Open Mathematics ◽

10.1515/math-2016-0005 ◽

2016 ◽

Vol 14 (1) ◽

pp. 62-80 ◽

Cited By ~ 2

Author(s):

Taras Kowaliw ◽

René Doursat

Keyword(s):

Genetic Programming ◽

Variance Decomposition ◽

Initial Population ◽

Linear Genetic Programming ◽

Improve Performance ◽

Training Samples ◽

Bias Variance ◽

Selection Of

AbstractWe study properties of Linear Genetic Programming (LGP) through several regression and classification benchmarks. In each problem, we decompose the results into bias and variance components, and explore the effect of varying certain key parameters on the overall error and its decomposed contributions. These parameters are the maximum program size, the initial population, and the function set used. We confirm and quantify several insights into the practical usage of GP, most notably that (a) the variance between runs is primarily due to initialization rather than the selection of training samples, (b) parameters can be reasonably optimized to obtain gains in efficacy, and (c) functions detrimental to evolvability are easily eliminated, while functions well-suited to the problem can greatly improve performance—therefore, larger and more diverse function sets are always preferable.

Download Full-text