Pith. sign in

REVIEW 3 major objections 4 minor 23 references

Efficient Parameter Estimation for Bayesian Network Classifiers using Hierarchical Linear Smoothing

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a log-linear regression over tree-structured coefficients can replace hierarchical Dirichlet process smoothing for Bayesian network classifiers, matching or beating its accuracy while costing orders of magnitude less.

desk verdict A practical log-linear smoother for BNC parameters that stands on its own empirical feet, even though the paper's HDP-approximation framing is more asserted than shown. read the letter →

arxiv 2505.23320 v1 pith:BGYZCJXJ submitted 2025-05-29 cs.LG stat.ML

Connor Cooper , Geoffrey I. Webb , Daniel F. Schmidt This is my paper
classification cs.LGstat.ML
keywords BayesiannetworkclassifiershierarchicalDirichletprocesseslogisticregressionparametersmoothingconditionalprobabilitytablesridgeshrinkagecategoricaldata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Bayesian network classifiers have lagged behind random forests, but recent hierarchical Dirichlet process (HDP) smoothing made them competitive on categorical data at the price of a complicated, specialized Gibbs sampler. The paper's central claim is that this smoothing can be approximated by a log-linear regression: place one coefficient on each node of the same parent-tree HDP smoothing uses, and learn all conditional probability table parameters jointly under multinomial logistic loss with shrinkage. In experiments on dozens of public categorical datasets, the resulting hierarchical linear smoothing matches or exceeds HDP-smoothed accuracy for both tree-augmented and k-dependence structures, while running orders of magnitude faster. Because the estimator is a linear model, it inherits the flexibility and tooling of standard logistic regression rather than requiring bespoke sampling.

What carries the argument

The central object is the tree-structured design matrix U, together with the identity H = U B. For a CPT with parents ordered by mutual information, each leaf's linear predictor is the sum of the coefficients of its ancestors in the parent-tree; the leaves' probabilities are softmaxes of these predictors, and B is learned by penalized multinomial logistic regression. The design matrix is sparse, containing only L×p nonzeros, which is what keeps the method fast, and the paper uses both fixed-ridge and Pólya-Gamma-based Bayesian ridge estimators to fit it.

What would settle it

Construct categorical datasets whose CPTs have strong interactions among parents—for instance XOR-like dependence where a leaf probability is high only for one specific parent combination—and compare held-out log loss of HLS versus HDP smoothing on sparse leaves. If HLS loses consistently on such structures, the linear ancestor-sum approximation is inadequate.

Watch

Extended reading notes

Core claim

The discovery, on the paper's own terms, is that the hierarchical sharing of Dirichlet priors over a conditional-probability-table tree reduces to a linear predictor. For each child node, build a design matrix U whose entries indicate whether one tree node is an ancestor of another; then the linear predictors are H = U B, with B one coefficient per tree node, and each leaf's probability vector is the softmax of its row. Fitting B by penalized logistic regression—either fixed-regularization ridge or Bayesian ridge with an inverse-gamma prior—reproduces or improves on HDP-smoothed estimates for TAN and kDB structures on the 42 datasets where HDP could be run, and extends to 50 datasets overall. This reframes joint CPT estimation as a standard linear-model problem, with no Gibbs sampling.

Load-bearing premise

The load-bearing premise is that summing one coefficient per ancestor level captures the information sharing that the hierarchical Dirichlet process achieves, so no important smoothing behavior is lost in the linear approximation.

Editorial extensions

If this is right

  • BNC parameter learning no longer requires the specialized HDP sampling process; any implementation of penalized or Bayesian logistic regression can produce competitive estimates.
  • Because HLS is a linear model, the extensive literature on linear-model learning—different penalties, sparse solvers, online updates, and interpretability tools—applies directly to BNC parameter estimation.
  • Spending a little extra computation on Bayesian ridge with an inverse-gamma prior yields the best accuracy among the variants tested, while fixed ridge at τ = 1 remains a strong, widely available baseline.
  • On log loss, BNCs with HLS rank above HDP-smoothed BNCs and above random forests with statistical significance; on zero-one loss the difference from random forests is not statistically significant, making HLS-equipped BNCs competitive overall.
  • The speed advantage over HDP grows with network size and parent cardinality, since the design matrix stays sparse while the Gibbs sampler must traverse the full hierarchy.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the linear approximation is as faithful as the aggregate results suggest, the practical value of HDP smoothing likely lies mostly in hierarchical sharing of information rather than in the nonparametric process itself; a testable consequence is that non-hierarchical shrinkage on flat CPT parameters should not match HLS on sparse leaves.
  • The method as presented only handles categorical features and bounded parent cardinalities; a natural, untested extension is to augment the design matrix with interaction or continuous-feature columns, which the linear framework would admit without changing the estimator.
  • The paper's win-draw-loss summaries show HLS winning overall but with some substantial HDP victories on individual datasets; a fair reading is that HLS is broadly competitive rather than universally dominant, and datasets with high cross-validation variance are exactly where the two disagree most.
  • Because the design matrix encodes only ancestor relations, HLS implicitly assumes smoothness along a fixed parent ordering; reordering parents or weighting ancestor contributions could become a modeling choice rather than a fixed prior, something the paper does not explore.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes hierarchical linear smoothing (HLS), a method for estimating the conditional probability tables (CPTs) of Bayesian network classifiers (BNCs) by casting parameter estimation as a regularized multinomial logistic regression. For each CPT, the method builds a design matrix U from the tree over parent configurations; the linear predictor for a leaf is the sum of coefficients of its ancestor nodes, and the CPT parameters are obtained via softmax. The coefficients are fit either by ridge regression with a fixed penalty (HLS-NB) or by a Bayesian ridge under a global-local shrinkage hierarchy with Pólya-Gamma augmentation (HLS-IG). Experiments on 50 UCI datasets compare HLS against additive smoothing, HDP smoothing, and random forests under zero-one loss and log loss, reporting win-draw-loss tables, critical difference diagrams, and timing comparisons. The paper claims that HLS approximates HDP smoothing while being orders of magnitude faster, and that BNCs with HLS remain competitive with random forests on categorical data.

Significance. If the central claim were established, the paper would make a useful contribution: it would replace the specialized Gibbs sampling used for HDP smoothing in BNCs with a standard, fast, and flexible linear-model estimation procedure, thereby broadening the applicability of hierarchical smoothing. The paper has clear strengths: it provides publicly available code, evaluates on a large collection of datasets with several BNC structures, compares against strong baselines, and is refreshingly explicit about limitations (e.g., non-rigorous timing, excluded HDP datasets, high CV variance). However, the conceptual claim that HLS 'approximates the behaviour of HDPs' is not supported by a derivation or bound, and the empirical evidence for it is mixed. Thus the significance is conditional on a missing theoretical or parameter-level validation.

major comments (3)
  1. [Section 3.1] The central claim that HLS approximates HDP smoothing is asserted but not established. The design matrix U defines leaf predictors as sums of ancestor coefficients, eta_leaf = sum_j beta_j, followed by softmax; HDP smoothing pools counts via a tree of Dirichlet priors, producing posterior means that are convex combinations in probability space. These are not equivalent transformations: HDP shrinkage acts on probabilities, while HLS shrinkage acts additively on logits, so an ancestor coefficient multiplies leaf odds by exp(beta_j). No theorem, bound, or parameter-level comparison is given to control the divergence. In a minimal two-leaf tree, a leaf with zero counts receives a pooled probability near the parent's estimate under HDP, while ridge regression shrinks its logit toward zero; the resulting probabilities can differ materially. The only evidence is the aggregate W-D-L comparison in Table 1, which is not direct evidence of approximation. The authors should either derive the relationship, bound the difference, or provide a parameter-level comparison on synthetic data where the HDP posterior mean is known.
  2. [Section 4.4.1, Table 1] The empirical support for the claim that HLS 'can actually surpass the performance of HDP smoothing' is weakened by two issues. First, HDP was run on only 42 of the 50 datasets, with the other 8 excluded 'due to issues with the implementation'; the critical difference diagrams in Figure 4 are also computed only on these 42 datasets. The authors should report which datasets were excluded and why, and assess whether the comparison is representative. Second, the W-D-L records in Table 1 are mixed: for example, under TAN 0/1 loss HLS-IG wins 24 and loses 16, and under kDB-2 0/1 loss it wins 21 and loses 20. These results do not clearly demonstrate approximation or even consistent superiority. The paper should include paired statistical tests (e.g., Wilcoxon signed-rank) or a more nuanced interpretation that distinguishes 'competitive' from 'approximates'.
  3. [Section 3.2 and Section 4.3.1] The choice of fixed ridge penalty tau = 1 is not adequately justified, despite being a load-bearing component of the main HLS-NB model. The paper reports that cross-validated ridge performs poorly (5-2-43 against fixed ridge) and that an 'improved CV' still loses 26-2-33, but no theoretical or empirical explanation is given for why tau=1 is a principled default. Since the method is presented as a way to avoid HDP's complexity, relying on an arbitrarily fixed penalty is a limitation. The authors should provide a sensitivity analysis over tau, or a data-dependent selection rule that does not rely on unstable CV, to show that the main comparisons are not sensitive to this choice.
minor comments (4)
  1. [Section 2.1.2] Typo: 'K-depence Bayes' should be 'K-dependence Bayes'.
  2. [Section 4.4.1] The text says 'Results in Figure 1 suggest that HLS can actually surpass the performance of HDP smoothing'; this should refer to Table 1, not Figure 1.
  3. [Appendix A.3] The phrase 'the same experiment as in Table 4.4.1' should be 'Table 1'.
  4. [Appendix B] The Pólya-Gamma sampler uses K=2 terms in the infinite series; given the importance of accurate logistic likelihood evaluation, the authors should briefly justify this truncation or cite a reference showing its accuracy.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: HLS is defined independently and evaluated on held-out folds against external baselines; the HDP approximation claim is an empirical contention, not a definitional identity.

full rationale

The paper's central construction H = U B (Section 3.1) is a linear predictor assembled from ancestor coefficients in a CPT tree; it is not fitted from HDP smoothing outputs, nor is HDP's posterior mean used as a target. The claim that HLS 'approximates the behaviour of HDP' is asserted as motivation and then tested by independent win-draw-loss comparisons against HDP smoothing (Table 1, Figure 7) on held-out folds of 42 UCI datasets, with additive smoothing and random forests as additional external benchmarks. No equation of the paper reduces HLS's predictive quantity to HDP's fitted values by construction. The citation of Petitjean et al. (2018), which shares an author with the present paper, supplies the tree structure and the HDP baseline; because the HDP results come from running the published implementation with default hyperparameters, this is independent empirical evidence rather than a load-bearing self-citation. A genuine scientific weakness, but not a circularity, is that the equivalence between Dirichlet pooling in probability space and additive shrinkage in logit space is asserted rather than derived or bounded; that concern belongs under correctness risk, not circularity. Therefore no circular step can be exhibited, and the score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method introduces no new probabilistic entities; it uses standard regression coefficients. The main assumptions are the linear approximation of the HDP tree, the discretization choice, and the evaluation protocol. The fixed regularization hyperparameters are the only fitted quantities, and they are chosen by hand rather than tuned on the target metric.

free parameters (3)
  • ridge regularization strength tau = 1 (fixed)
    The non-Bayesian HLS variant uses a fixed ridge penalty tau = 1, chosen as a strong baseline. Cross-validated tau performs worse, so the choice is a hyperparameter that affects results.
  • inverse-gamma shrinkage hyperparameters = IG(1/2, 1/2)
    The Bayesian HLS-IG variant fixes the inverse-gamma prior shape and scale to 1/2, 1/2. This is a modeling choice that controls the degree of shrinkage.
  • additive smoothing pseudo-count m = 1 (chosen from {0.2, 0.5, 1, 2, 5, 20})
    Add-1 was chosen because it generally performed best among the options tested in Appendix A.2; the choice is data-driven across datasets.
assumptions (3)
  • domain assumption The linear predictor H = U B over ancestor coefficients approximates the hierarchical Dirichlet process prior structure.
    Section 3.1 constructs the design matrix so that each leaf is the sum of ancestor coefficients, asserted to mirror the HDP tree. No bound or formal equivalence is given.
  • domain assumption Discretization via MDLP preserves enough information for categorical comparisons.
    Section 4.1 discretizes all numerical attributes with the MDL principle; all comparisons, including random forests, use the discretized data. The results may not transfer to raw continuous features.
  • domain assumption Win-draw-loss across 50 datasets with 10-fold CV is an adequate basis for comparing classifiers.
    The paper uses W-D-L and critical difference diagrams; it acknowledges high CV variance on some datasets and marks them in scatter plots, but does not apply a formal correction for multiple comparisons.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Efficient Parameter Estimation for Bayesian Network Classifiers using Hierarchical Linear Smoothing." pith.science (2026). https://pith.science/paper/BGYZCJXJ

@misc{pith2026250523320,
  author       = {Pith},
  title        = {Pith review of: Efficient Parameter Estimation for Bayesian Network Classifiers using Hierarchical Linear Smoothing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BGYZCJXJ}},
  note         = {Machine review of arXiv:2505.23320}
}
read the original abstract

Bayesian network classifiers (BNCs) possess a number of properties desirable for a modern classifier: They are easily interpretable, highly scalable, and offer adaptable complexity. However, traditional methods for learning BNCs have historically underperformed when compared to leading classification methods such as random forests. Recent parameter smoothing techniques using hierarchical Dirichlet processes (HDPs) have enabled BNCs to achieve performance competitive with random forests on categorical data, but these techniques are relatively inflexible, and require a complicated, specialized sampling process. In this paper, we introduce a novel method for parameter estimation that uses a log-linear regression to approximate the behaviour of HDPs. As a linear model, our method is remarkably flexible and simple to interpret, and can leverage the vast literature on learning linear models. Our experiments show that our method can outperform HDP smoothing while being orders of magnitude faster, remaining competitive with random forests on categorical data.

Figures

Figures reproduced from arXiv: 2505.23320 by the authors.

Figure 1
Figure 1. For a child node Xc with binary-valued parents X1 and X2, a tree that branches first on the values of X1 and then on the values of X2, with the corresponding: (a) parameters for HDP, (b) coefficients for HLS, and (c) design matrix for HLS. where nxc | Πxc is the observed frequency of Xc = xc when ΠXc = Πxc , and nΠxc is the total number of observations where ΠXc = Πxc . Smoothing can be applied by assigning a Dirich… view at source ↗
Figure 2
Figure 2. (a) Win-Draw-Loss records on kDB-3 for various regularization strategies vs ridge with τ = 1. (b) Scatter plot for Bayesian inverse-gamma (HLS-IG) vs ridge regression (HLS-NB) on kDB-3, under zero-one loss. Red squares indicate datasets with top-15 CV variance. 4 Experiments The aim of this section is two-fold: we wish to explore the problem of estimating our regression coefficients, and to assess how BNCs using our… view at source ↗
Figure 3
Figure 3. Scatter plots for HLS-IG on TAN vs random forests, under (a) zero-one loss and (b) log loss. Red squares indicate datasets with top-15 cross-validation variance. 10 [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Critical difference diagrams representing the mean rank of models under zero-one loss and log loss, across the 42 datasets that HDP was trained on. BNCs use TAN as a structure. 11 [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Scatter plots for HLS-NB with an intercept vs HLS-NB with no intercept, on kDB-3, under (a) zero-one loss and (b) log loss. Red squares indicate datasets with top-15 cross-validation variance. A.2 Additive smoothing In [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: Scatter plots for HLS-IG on TAN vs Add-1, under (a) zero-one loss and (b) log loss. Red squares indicate datasets with top-15 cross-validation variance. A.3 HLS vs HDP In [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]
Figure 7
Figure 7. Figure 7: Scatter plots for HLS-IG on TAN vs HDP on 42 datasets, under (a) zero-one loss and (b) log loss. Red squares indicate datasets with top-15 cross-validation variance. A.4 Horseshoe We experiment with two additional models under the GLS hierarchy: 15 [PITH_FULL_IMAGE:fi…
Figure 8
Figure 8. Figure 8: Scatter plots for HLS with HS vs HLS-IG on kDB-3, under (a) zero-one loss and (b) log loss. Red squares indicate datasets with top-15 cross-validation variance. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Scatter plots for HLS with HS-IG vs HLS-IG on kDB-3, under (a) zero-one loss and (b) log loss. Red squares indicate datasets with top-15 cross-validation variance. Appendix B. Pólya-Gamma Sampler From Polson et al. (2013), a random variable X has a Pólya-Gamma distribu…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

23 extracted references · 19 canonical work pages

  1. [1]

    Azzimonti, G

    L. Azzimonti, G. Corani, and M. Zaffalon. Hierarchical estimation of parameters in Bayesian networks. Computational Statistics & Data Analysis, 137: 0 67--91, 2019

    work page 2019

  2. [2]

    C. M. Carvalho, N. G. Polson, and J. G. Scott. The horseshoe estimator for sparse signals. Biometrika, 97 0 (2): 0 465--480, 2010

    work page 2010

  3. [3]

    J. Demšar. Statistical Comparisons of Classifiers over Multiple Data Sets . Journal of Machine Learning Research, 7, 2006

    work page 2006

  4. [4]

    Fayyad and K

    U. Fayyad and K. Irani. Multi- Interval Discretization of Continuous - Valued Attributes for Classification Learning . In International Joint Conference on Artificial Intelligence, 1993

    work page 1993

  5. [5]

    Friedman, D

    N. Friedman, D. Geiger, and M. Goldszmidt. Bayesian Network Classifiers . Machine Learning, 29 0 (2): 0 131--163, 1997

    work page 1997

  6. [6]

    Ismail Fawaz, G

    H. Ismail Fawaz, G. Forestier, J. Weber, L. Idoumghar, and P.-A. Muller. Deep learning for time series classification: a review. Data Mining and Knowledge Discovery, 33 0 (4): 0 917--963, 2019

    2019

  7. [7]

    M. Lichman. UCI Machine Learning Repository , 2013

    work page 2013

  8. [8]

    H. Lin. hlin117/mdlp-discretization, 2017. GitHub repository

    work page 2017

Show all 23 references
  1. [9]

    Makalic and D

    E. Makalic and D. F. Schmidt. High- Dimensional Bayesian Regularised Regression with the BayesReg Package , 2016. arXiv:1611.06649

    2016 arXiv

  2. [10]

    Pedregosa, G

    F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and \'E . Duchesnay. Scikit-learn: Machine Learning in Python . Journal of Machine Lear...

    2011

  3. [11]

    Petitjean, W

    F. Petitjean, W. Buntine, G. I. Webb, and N. Zaidi. Accurate parameter estimation for bayesian network classifiers using hierarchical dirichlet processes. Machine Learning, 107 0 (8): 0 1303--1331, 2018

    2018

  4. [12]

    N. G. Polson and J. G. Scott. On the Half-Cauchy Prior for a Global Scale Parameter . Bayesian Analysis, 7 0 (4): 0 887 -- 902, 2012 a

    2012

  5. [13]

    N. G. Polson and J. G. Scott. Local shrinkage rules, lévy processes and regularized regression. Journal of the Royal Statistical Society Series B: Statistical Methodology, 74 0 (2): 0 287--311, 01 2012 b . ISSN 1369-7412

    2012

  6. [14]

    N. G. Polson, J. G. Scott, and J. W. and. Bayesian inference for logistic models using pólya–gamma latent variables. Journal of the American Statistical Association, 108 0 (504): 0 1339--1349, 2013. doi:10.1080/01621459.2013.829001

    2013

  7. [15]

    F. Rijmen. Bayesian networks with a logistic regression model for the conditional probabilities. International Journal of Approximate Reasoning, pages 659--666, 2008

    2008

  8. [16]

    H. Rue. Fast sampling of Gaussian Markov random fields. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 63 0 (2): 0 325--338, 2001

    2001

  9. [17]

    M. Sahami. Learning Limited Dependence Bayesian Classifiers , 1996

    1996

  10. [18]

    D. F. Schmidt and E. Makalic. Bayesian Generalized Horseshoe Estimation of Generalized Linear Models . In Machine Learning and Knowledge Discovery in Databases , pages 598--613. Springer International Publishing, 2020

    2020

  11. [19]

    S. Y. Tew, M. Boley, and D. Schmidt. Bayes beats Cross Validation : Efficient and Accurate Ridge Regression via Expectation Maximization . Advances in Neural Information Processing Systems, 36: 0 19749--19768, 2023

    2023

  12. [20]

    W. N. van Wieringen. Lecture notes on ridge regression, 2023. arXiv:1509.09169

    2023 arXiv

  13. [21]

    Wang and N

    Y. Wang and N. G. Polson. Horseshoe Priors for Sparse Dirichlet - Multinomial Models , 2024. arXiv:2402.09583

    2024 arXiv

  14. [22]

    Zhang, F

    H. Zhang, F. Petitjean, and W. Buntine. Bayesian network classifiers using ensembles and smoothing. Knowledge and Information Systems, pages 3457--3480, 2020 a

    2020

  15. [23]

    Zhang, F

    H. Zhang, F. Petitjean, and W. Buntine. Hierarchical Gradient Smoothing for Probability Estimation Trees . In Advances in Knowledge Discovery and Data Mining , pages 222--234, 2020 b

    2020

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.