Network scores

Description

Overview of the network scores implemented in bnlearn, with the respective reference publications.

Details

Available scores (and the respective labels) for discrete Bayesian networks (categorical variables) are:

• the multinomial log-likelihood (loglik) score, which is equivalent to the entropy measure used in Weka.

• the Akaike Information Criterion (AIC) score (aic). Note that AIC is rescaled by -2 to make it comparable with marginal log-likelihoods.

• the Bayesian Information Criterion (BIC) score (bic), which is equivalent to the Minimum Description Length (MDL) and is also known as Schwarz Information Criterion. Note that BIC is rescaled by -2 to make it comparable with marginal log-likelihoods.

  Chickering DM (1995). "A Transformational Characterization of Equivalent Bayesian Network Structures." Proceedings of the Eleventh Annual Conference on Uncertainty in Artificial Intelligence, 87–98.

• the extended Bayesian Information Criterion (ebic), which adds a second penalty to BIC to penalize dense networks. Like BIC, eBIC is rescaled by -2.

  Foygel R, Drton M (2010). "Extended Bayesian Information Criteria for Gaussian Graphical Models." NIPS 23, 604–612.

• the predictive log-likelihood (pred-loglik) computed on a separate test set.

  Chickering DM, Heckerman D (2000). "A Comparison of Scientific and Engineering Criteria for Bayesian Model Selection." Statistics and Computing, 10:55–62.

  Scutari M, Vitolo C, Tucker A (2019). "Learning Bayesian Networks from Big Data with Greedy Search: Computational Complexity and Efficient Implementation." Statistics and Computing, 25(9):1095–1108.

• the logarithm of the Bayesian Dirichlet equivalent (uniform) score (bde) (also denoted BDeu), a score equivalent Dirichlet posterior density.

  Heckerman D, Geiger D, Chickering DM (1995). "Learning Bayesian Networks: The Combination of Knowledge and Statistical Data." Machine Learning, 20(3):197–243.

  Castelo R, Siebes A (2000). "Priors on Network Structures. Biasing the Search for Bayesian Networks." International Journal of Approximate Reasoning, 24(1):39–57.

• the logarithm of the Bayesian Dirichlet sparse score (bds) (BDs), a sparsity-inducing Dirichlet posterior density (not score equivalent).

  Scutari M (2016). "An Empirical-Bayes Score for Discrete Bayesian Networks." Journal of Machine Learning Research, 52:438–448.

• the logarithm of the Bayesian Dirichlet score with Jeffrey's prior (not score equivalent).

  Suzuki J (2016). "A Theoretical Analysis of the BDeu Scores in Bayesian Network Structure Learning." Behaviormetrika, 44(1):97–116.

• the logarithm of the modified Bayesian Dirichlet equivalent score (mbde) for mixtures of experimental and observational data (not score equivalent).

  Cooper GF, Yoo C (1999). "Causal Discovery from a Mixture of Experimental and Observational Data." Proceedings of the Fifteenth Annual Conference on Uncertainty in Artificial Intelligence, 116–125.

• the logarithm of the K2 score (k2), a Dirichlet posterior density (not score equivalent).

  Korb K, Nicholson AE (2010). Bayesian Artificial Intelligence. Chapman & Hall/CRC, 2nd edition.

• the logarithm of the factorized normalized maximum likelihood score (fnml, not score equivalent).

  Silander T, Roos T, Kontkanen P, Myllymaki P (2008). "Factorized Normalized Maximum Likelihood Criterion for Learning Bayesian Network Structures." Proceedings of the 4th European Workshop on Probabilistic Graphical Models, 257–272.

• the logarithm of the quotient normalized maximum likelihood (qnml).

  Silander T, Leppa-Abo J, Jaasaari, Roos T (2018). "Quotient Normalized Maximum Likelihood Criterion for Learning Bayesian Network Structures." Proceedings of Machine Learning Research, 84:948–957.

• the node-average (log-)likelihood (nal) and the penalized node-average (log-)likelihood (pnal).

  Bodewes T, Scutari M (2021). "Learning Bayesian Networks from Incomplete Data with the Node-Averaged Likelihood." International Journal of Approximate Reasoning, 138:145–160.

Available scores (and the respective labels) for Gaussian Bayesian networks (normal variables) are:

• the multivariate Gaussian log-likelihood (loglik-g) score.

• the corresponding Akaike Information Criterion (AIC) score (aic-g). Note that AIC is rescaled by -2 to make it comparable with marginal log-likelihoods.

• the corresponding Bayesian Information Criterion (BIC) score (bic-g). Note that BIC is rescaled by -2 to make it comparable with marginal log-likelihoods.

  Geiger D, Heckerman D (1994). "Learning Gaussian Networks." Proceedings of the Tenth Annual Conference on Uncertainty in Artificial Intelligence, 235–243.

• the extended Bayesian Information Criterion (ebic-g), which adds a second penalty to BIC to penalize dense networks. Like BIC, eBIC is rescaled by -2.

  Foygel R, Drton M (2010). "Extended Bayesian Information Criteria for Gaussian Graphical Models." NIPS 23, 604–612.

• the predictive log-likelihood (pred-loglik-g) computed on a separate test set. The reference paper is the same as that for pred-loglik. It is currently implemented to be score-equivalent like pred-loglik, but that may be subject to change.

• a score equivalent Gaussian posterior density (bge).

  Kuipers J, Moffa G, Heckerman D (2014). "Addendum on the Scoring of Gaussian Directed Acyclic Graphical Models." The Annals of Statistics, 42(4):1689–1691.

• the node-average (log-)likelihood (nal-g) and the penalized node-average (log-)likelihood (pnal-g).

  Bodewes T, Scutari M (2021). "Learning Bayesian Networks from Incomplete Data with the Node-Averaged Likelihood." International Journal of Approximate Reasoning, 138:145–160.

Available scores (and the respective labels) for conditional Gaussian Bayesian networks (mixed categorical and normal variables) are:

• the conditional linear Gaussian log-likelihood (loglik-cg) score.

• the corresponding Akaike Information Criterion (AIC) score (aic-cg). Note that AIC is rescaled by -2 to make it comparable with marginal log-likelihoods.

• the corresponding Bayesian Information Criterion (BIC) score (bic-cg). Note that BIC is rescaled by -2 to make it comparable with marginal log-likelihoods.

• the extended Bayesian Information Criterion (ebic-cg), which adds a second penalty to BIC to penalize dense networks. Like BIC, eBIC is rescaled by -2.

  Foygel R, Drton M (2010). "Extended Bayesian Information Criteria for Gaussian Graphical Models." NIPS 23, 604–612.

• the predictive log-likelihood (pred-loglik-cg) computed on a separate test set. The reference paper is the same as that for pred-loglik.

• the node-average (log-)likelihood (nal-cg) and the penalized node-average (log-)likelihood (pnal-cg).

  Bodewes T, Scutari M (2021). "Learning Bayesian Networks from Incomplete Data with the Node-Averaged Likelihood." International Journal of Approximate Reasoning, 138:145–160.

Other scores (and the respective labels):

• a custom decomposable (custom) score interface that takes an R function as an argument. It can be used to trial experimental score functions without having to code them in C and hook them up to the internals of bnlearn.