REVIEW 3 major objections 6 minor 1 cited by
Oblique Bayesian additive regression trees
T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Hyperplane-splitting BART matches or beats axis-aligned tree ensembles
desk verdict A genuinely new oblique BART construction with a real technical bug: the stated leaf prior (N(0, τ²/M)) does not match the full conditional and MH ratios in Section 3.2, which use precision τ⁻². read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the oblique decision rule $\phi^{\top} x_{\mathrm{cont}} < c$, with a hierarchical spike-and-slab prior on $\phi$: a point-mass spike at zero for unused coordinates and a standard normal slab on active coordinates, with the sparsity parameter $\theta$ given a Beta prior so the number of active features adapts to the data. At a grow move the rule is proposed from this prior, the cutpoint $c$ is drawn uniformly from the interval of valid values of $\phi^{\top} x_{\mathrm{cont}}$ at that node (found by solving two linear programs over the polytope defined by ancestor rules), and the Metropolis-Hastings acceptance probability then simplifies because prior and proposal cancel, leaving a fit-versus-complexity comparison. Categorical predictors are handled by random subset rules. This prior-proposal design is what lets the ensemble explore oblique partitions without hand-tuned rotations.
What would settle it
Re-run the 40 benchmark comparisons with 10 independent chains of 10,000 post-burn-in draws each and compute R-hat and effective sample sizes; if the posterior predictive means change enough to flip which regression datasets show statistically significant SMSE differences, then the single-chain posterior approximation is not valid and the reported performance comparisons are not settled.
Extended reading notes
Core claim
The paper's central claim is that replacing axis-aligned splits in BART with oblique rules drawn from a hierarchical spike-and-slab prior yields a valid Bayesian tree ensemble that is competitive with -- and sometimes much better than -- axis-aligned BART and other tree ensembles on regression benchmarks, while remaining competitive on classification. The gains are largest when the true function's discontinuities are not aligned with the coordinate axes, and the model adapts its sparsity automatically: on the rotated-axes synthetic problem it uses axis-aligned rules about 70 percent of the time when the truth is axis-aligned and about 53 percent when the truth is oblique. The paper does not argue for wholesale replacement of axis-aligned BART, but for a practical alternative that is often more accurate and rarely clearly worse.
Load-bearing premise
The benchmark conclusions assume that 1000 retained draws from a single MCMC chain, without convergence diagnostics, accurately represent obliqueBART's posterior predictive distribution.
Editorial extensions
If this is right
- On 18 regression benchmarks, obliqueBART posts the lowest average standardized mean squared error (0.296) among obliqueBART, BART (0.316), RF (0.330), ERT (0.332), and XGB (0.342).
- On the rotated-axes synthetic problem it beats axis-aligned BART at every rotation angle, including $\theta = 0$, and its rule ensemble shifts from 70.2% axis-aligned splits at $\theta=0$ to 52.9% at $\theta=\pi/4$.
- Rotating the input features before fitting axis-aligned methods rarely closes the gap: rotBART cannot match obliqueBART on 19 of the regression datasets even with 200 random rotations, and obliqueBART is roughly 20x faster than rotRF and 2x faster than rotBART at that rotation count.
- On 22 classification benchmarks obliqueBART is competitive, with average accuracy 0.846 vs ERT's best 0.866 and differences under 2% on 15 of the 22 datasets.
Reading between the lines
- If the posterior approximation is trustworthy, the same spike-and-slab rule prior could be dropped into BART extensions for survival, heteroskedastic regression, or treatment-effect estimation, since only the grow-proposal distribution changes.
- A direct test of robustness would be to re-run the benchmark comparison with multiple independent chains and convergence diagnostics; if the posterior means move, the reported nine-dataset advantage over BART may shrink, but the tilted-boundary gains from the synthetic experiments should persist.
- For image inputs, constraining the non-zero entries of $\phi$ to form a connected sub-region of the image would give a Bayesian analog of manifold oblique random forests, an extension the paper itself notes.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes obliqueBART, an extension of Bayesian Additive Regression Trees in which internal nodes use hyperplane splits of the form {ϕ^T x < c}, with a spike-and-slab prior on ϕ and a Beta prior on the sparsity probability θ. The authors derive a Metropolis-within-Gibbs sampler, provide an R package, and compare predictive performance against axis-aligned BART, random-rotation versions of BART/RF/ERT/XGBoost, and other tree ensembles on synthetic and benchmark regression and classification tasks. The central claim is that obliqueBART is competitive with and sometimes substantially better than axis-aligned BART and other tree ensembles, especially when the regression function has boundaries not aligned with coordinate axes.
Significance. If the derivation is corrected, obliqueBART is a conceptually clean and potentially valuable extension of BART. It introduces a coherent generative model for oblique trees, with a sparse direction prior that adapts the expected number of nonzero coefficients, and the empirical study covers a broad set of benchmarks. The paper ships an R package (https://github.com/paulhnguyen/obliqueBART), which is a strength. However, the current manuscript contains a load-bearing error in the MCMC formulas that requires the authors to either correct the paper or confirm that the implementation already implements the corrected forms; and the empirical sections need stronger MCMC diagnostics and more careful statistical inference before the claims can be accepted.
major comments (3)
- [Section 2.2 and Eqs. (1)–(3), (B1)–(B2)] The stated leaf-output prior is N(0, τ²/M), but the full conditional in Eq. (1) uses precision τ^{-2} and normalizing constant τ^{-1}, which is the full conditional for a N(0, τ²) leaf prior. Consequently, Eq. (2) should read P_ℓ = n_ℓ σ^{-2} + M τ^{-2} with an overall factor √M/τ, and the grow/prune acceptance probabilities in Eqs. (B1) and (B2) should contain √M/τ and τ/√M, respectively, instead of τ^{-1} and τ. As written, the sampler targets the posterior of a model in which the marginal prior for f(x) is N(0, M τ²) rather than N(0, τ²); with M=200 this is a 200-fold difference in prior variance, and the missing √M≈14.1 factor materially changes the growth acceptance probability. The authors must correct these formulas and either verify that the released implementation contains the corrected factors or re-run the experiments with the corrected sampler.
- [Section 4, first paragraph] The posterior means are based on a single chain of 2000 iterations with the first 1000 discarded, but the paper reports no convergence diagnostics, no effective sample sizes, and no acceptance rates for the tree proposals. Given that oblique rules dramatically expand the move space and that grow proposals are drawn entirely from the prior, the chain may mix very slowly; 1000 retained samples could yield predictions that are not representative of the posterior. Please report trace plots, ESS, and MH acceptance rates at least for a few representative datasets, and consider increasing the number of iterations or using multiple chains.
- [Section 4.2 and Tables A2–A7] The significance claims rely on one-sided paired t-tests at the 5% level applied to 40 datasets (and to many rotation counts), without any multiple-testing correction; the 'number of rotations needed to match' statements in Tables A6–A7 are based on the same unadjusted tests and therefore inherit this problem. Also, average SMSEs and accuracies are reported without standard errors or confidence intervals, making it difficult to assess the strength of the headline comparisons. Please either adjust for multiplicity, or downgrade the language to point estimates, and report measures of uncertainty for the averages.
minor comments (6)
- [Section 1.2 and Figure 2] The text refers to 'the sinusoid function (Figure 2a)', but the sinusoid appears in Figure 2d; please fix the cross-reference.
- [Section 2.2, first paragraph] 'fits of other tres' should be 'fits of other trees'.
- [Section 4.1] The synthetic predictors are drawn from Uniform([0,1]^2), while the model is defined for predictors in [-1,1]^p; please state explicitly how the data are scaled before fitting.
- [Section 4.2] The average SMSE for obliqueBART is reported as 0.296 in the comparison with axis-aligned methods and 0.297 in the random-rotation comparison; clarify whether these averages are computed over the same datasets, given that some rotated methods could not be run on all datasets.
- [Section 3.1] The phrase 'data-adaptive decision rule prior' may overstate the case because the rule prior is not itself data-dependent; the sparsity probability θ is updated from the data. Consider rephrasing to avoid confusion.
- [Throughout] Typos: 'In principal' should be 'In principle'; 'implementating' should be 'implementing'; 'diamons' should be 'diamonds'; 'the the smallest' should be 'the smallest'; and the dataset name 'hepatitis' in Table A3 should be 'hepatitis'.
Circularity Check
No significant circularity: the central oblique-BART model is fully generative and is evaluated against external benchmarks; the only self-citation is a minor design justification.
full rationale
The paper's derivation chain is self-contained: Section 3.1 specifies a full generative prior (branching process for T, spike-and-slab prior for phi, N(0, tau^2/M) leaf outputs, inverse-gamma for sigma^2), and Section 3.2 derives the conditional posteriors and Metropolis-Hastings acceptance probabilities from that prior plus the likelihood. The empirical claims in Section 4 compare posterior means from this model to BART, RF, ERT, and XGB on synthetic and external benchmark datasets, so the reported 'competitive or better' performance is measured against independently fit methods, not recovered from the model's own fitted parameters. The only direct self-citation, to Deshpande (2024), is invoked for the categorical decision-rule prior and for intuition about why over-informed proposals slow MCMC; neither is the load-bearing premise of the paper's main claim, and the categorical-rule component is not what drives the oblique-geometry results. A genuine correctness concern exists independently of circularity: the leaf-output prior is stated as N(0, tau^2/M) in Sections 2.2 and 3.1, while Eq. (1), Eq. (B1), and Eq. (B2) treat the leaf precision as tau^{-2} rather than M tau^{-2}, which changes the target posterior (grow/prune acceptance terms are off by a factor involving sqrt(M)). That is a bug or typo, not a circular reduction, so it does not affect the circularity score. Overall: no fitted parameter is renamed as a prediction, no uniqueness claim is imported from the authors' prior work, and the comparisons are external. Score 1 reflects only the minor self-citation that is not load-bearing.
Assumptions & free parameters
free parameters (2)
- theta hyperparameters (a_theta, b_theta) =
a_theta = M = 200; b_theta set so prior mean of theta is 1/pcont
- BART defaults (tau, lambda, M) =
tau so prior f(x) covers 95% of Y range; lambda so 90% prior probability sigma < sd(Y); M = 200
assumptions (4)
- domain assumption The branching process tree prior with depth penalty alpha(1+d)^(-beta) and normal leaf priors from Chipman et al. (2010) is appropriate for oblique rules.
- standard math The region of the predictor space reaching a node is a convex polytope, so the valid cutpoint interval for phi^T x can be computed by two linear programs.
- standard math The Metropolis-Hastings acceptance probability in Equation (B1) is correctly derived under grow/prune proposals that draw new rules from the prior.
- domain assumption The single MCMC chain converges within 2000 iterations.
Cite this review
Pith. "Pith review of Oblique Bayesian additive regression trees." pith.science (2026). https://pith.science/paper/YCQKTKTH
@misc{pith2026241108849,
author = {Pith},
title = {Pith review of: Oblique Bayesian additive regression trees},
year = {2026},
howpublished = {\url{https://pith.science/paper/YCQKTKTH}},
note = {Machine review of arXiv:2411.08849}
}
read the original abstract
Current implementations of Bayesian Additive Regression Trees (BART) are based on axis-aligned decision rules that recursively partition the feature space using a single feature at a time. Several authors have demonstrated that oblique trees, whose decision rules are based on linear combinations of features, can sometimes yield better predictions than axis-aligned trees and exhibit excellent theoretical properties. We develop an oblique version of BART that leverages a data-adaptive decision rule prior that recursively partitions the feature space along random hyperplanes. Using several synthetic and real-world benchmark datasets, we systematically compared our oblique BART implementation to axis-aligned BART and other tree ensemble methods, finding that oblique BART was competitive with -- and sometimes much better than -- those methods.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 1 Pith paper
-
Breaking the Curse with BAND: Nonparametric Distribution Estimation in High Dimensions
Sparse Bayesian-network factorization plus sparsity-aware regression yields polynomial TV rates for high-dimensional mixed-type distribution estimation, beating classical histogram rates under sparsity.
Reference graph
Works this paper leans on
-
[1]
Albert, J. H. and Chib, S. (1993). Bayesian analysis of binary and polychotomous response data. Journal of the American Statistical Association , 88(422):669--679
1993
-
[2]
Bertsimas, D. and Dunn, J. (2017). Optimal classification trees. Machine Learning , 106(7):1039--1082
work page 2017
-
[3]
Bertsimas, D., Dunn, J., and Wang, Y. (2021). Near-optimal nonlinear regression trees. Operations Research Letters , 49(2):201--206
work page 2021
-
[4]
Blaser, R. and Fryzlewicz, P. (2016). Random rotation ensembles. Journal of Machine Learning Research , 17(1):126--151
work page 2016
-
[5]
Bochkanov, S. (2023). ALGLIB . www.alglib.net
work page 2023
-
[6]
Breiman, L. (2001). Random forests. Machine Learning , 45:5--32
2001
-
[7]
Breiman, L., Friedman, J. H., Olshen, R. A., and Stone, C. J. (1984). Classification and Regression Trees . Routledge
work page 1984
-
[8]
D., Chandak, R., and Klusowski, J
Cattaneo, M. D., Chandak, R., and Klusowski, J. M. (2024). Convergence rates of oblique regression trees for flexible function libraries. Annals of Statistics , 52(2):466--490
work page 2024
Show all 45 references
-
[9]
Center for High Throughput Computing
Center for High Throughput Computing (2006). Center for High Throughput Computing
2006
-
[10]
and Loh, W.-Y
Chan, K.-Y. and Loh, W.-Y. (2004). Lotus: An algorithm for building accurate and comprehensible logistic regression trees. Journal of Computational and Graphical Statistics , 13(4):826--852
2004
-
[11]
Chen, T., He, T., Benesty, M., Khotilovich, V., Tang, Y., Cho, H., Chen, K., Mitchell, R., Cano, I., Zhou, T., Li, M., Xie, J., Lin, M., Geng, Y., Li, Y., and Yuan, J. (2024). xgboost: Extreme Gradient Boosting . R package version 1.7.8.1
2024
-
[12]
A., George, E
Chipman, H. A., George, E. I., and McCulloch, R. E. (1998). Bayesian CART model search. Journal of the American Statistical Association , 93(443):935--948
1998
-
[13]
A., George, E
Chipman, H. A., George, E. I., and McCulloch, R. E. (2010). BART: B ayesian additive regression trees . Annals of Applied Statistics , 4(1):266--298
2010
-
[14]
Deshpande, S. K. (2024). flexBART : Flexible B ayesian regression trees with categorical predictors. arXiv:2211.04459 https://arxiv.org/abs/2211.04459
2024 arXiv
-
[15]
Dorie, V., Hill, J., Shalit, U., Scott, M., and Cervone, D. (2019). Automated versus do-it-yourself methods for causal inference: Lessons learned from a data analysis competition. Statistical Science , 34(1):43--68
2019
-
[16]
Friedman, J. H. (2001). Greedy function approximation: A gradient boosting machine. Annals of Statistics , 29(5):1189--1232
2001
-
[17]
Geurts, P., Ernst, D., and Wehenkel, L. (2006). Extremely randomized trees. Machine Learning , 63:3--42
2006
-
[18]
Gramacy, R. B. and Lee, H. K. (2008). Bayesian treed Gaussian process models with an application to computer modeling. Journal of the American Statistical Association , 103(483):1119--1130
2008
-
[19]
R., Murray, J
Hahn, P. R., Murray, J. S., and Carvalho, C. M. (2020). Bayesian regression models for causal inference: Regularization, confounding, and heterogeneous effects. Bayesian Analysis , 15(3):965--1056
2020
-
[20]
and Friedman, R
Hastie, T. and Friedman, R. T. J. (2005). The Elements of Statistical Learning: Data Mining, Inference, and Prediction . Springer
2005
-
[21]
G., Kasif, S., and Salzberg, S
Heath, D. G., Kasif, S., and Salzberg, S. L. (1993). Induction of oblique decision trees. In International Joint Conference on Artificial Intelligence
1993
-
[22]
Hill, J. L. (2011). Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics , 20(1):217--240
2011
-
[23]
R., Saarinen, T
K\" u nzel, S. R., Saarinen, T. F., Liu, E. W., and Sekhon, J. S. (2022). Linear aggregation in tree-based estimators. Journal of Computational and Graphical Statistics , 31(3):917--934
2022
-
[24]
Lamprinakou, S., Barahona, M., Flaxman, S., Filippi, S., Gandy, A., and McCoy, E. J. (2023). BART -based inference for poisson processes. Computational Statistics & Data Analysis , 180:107658
2023
-
[25]
Landwehr, N., Hall, M., and Frank, E. (2005). Logistic model trees. Machine Learning , 59:161--205
2005
-
[26]
M., Mehta, R., Arroyo, J., Patsolic, J., Falk, B., Sarma, S., and Vogelstein, J
Li, A., Perry, R., Huynh, C., Tomita, T. M., Mehta, R., Arroyo, J., Patsolic, J., Falk, B., Sarma, S., and Vogelstein, J. (2023a). Manifold oblique random forests: Towards closing the gap on convolutional deep networks. SIAM Journal on Mathematics of Data Science , 5(1):77--96
2023
-
[27]
R., and Murray, J
Li, Y., Linero, A. R., and Murray, J. (2023b). Adaptive conditional distribution estimation with Bayesian decision tree ensembles. Journal of the American Statistical Association , 118(543):2129--2142
2023
-
[28]
and Wiener, M
Liaw, A. and Wiener, M. (2002). Classification and regression by randomForest . R News , 2(3):18--22
2002
-
[29]
R., Basak, P., Li, Y., and Sinha, D
Linero, A. R., Basak, P., Li, Y., and Sinha, D. (2022). Bayesian survival tree ensembles with submodal shrinkage. Bayesian Analysis , 17(3):997--1020
2022
-
[30]
R., Sinha, D., and Lipsitz, S
Linero, A. R., Sinha, D., and Lipsitz, S. R. (2020). Semiparametric mixed-scale models using shared B ayesian forests. Biometrics , 76(1):131--144
2020
-
[31]
Maia, M., Murphy, K., and Parnell, A. C. (2024). GP-BART : A novel B ayesian additive regression trees approach using gaussian processes. Computational Statistics & Data Analysis , 190:107858
2024
-
[32]
H., Kelm, B
Menze, B. H., Kelm, B. M., Splitthoff, D. N., Koethe, U., and Hamprecht, F. A. (2011). On oblique random forests. In Machine Learning and Knowledge Discovery in Databases
2011
-
[33]
R., and Volfovsky, A
Orlandi, V., Murray, J., Linero, A. R., and Volfovsky, A. (2021). Density regression with Bayesian additive regression trees. arXiv:2112.12259 https://arxiv.org/abs/2112.12259
2021 arXiv
-
[34]
T., Chipman, H
Pratola, M. T., Chipman, H. A., George, E. I., and McCulloch, R. E. (2020). Heteroscedastic BART using multiplicative regression trees. Journal of Computational and Graphical Statistics , 29(2):405--417
2020
-
[35]
Quinlan, J. R. (1992). Learning with continuous classes. In 5^ th Australian Joint Conference on Artificial Intelligence
1992
-
[36]
and Wood, F
Rainforth, T. and Wood, F. (2015). Canonical correlation forests. arXiv:1507.05444 https://arxiv.org/abs/1507.05444
2015 arXiv
-
[37]
Rodriguez, J., Kuncheva, L., and Alonso, C. (2006). Rotation forest: A new classifier ensemble method. IEEE Transactions on Pattern Analysis and Machine Intelligence , 28(10):1619--1630
2006
-
[38]
R., McCulloch, R
Sparapani, R., Logan, B. R., McCulloch, R. E., and Laud, P. W. (2016). Nonparametric survival analysis using Bayesian A dditive R egression T rees. Statistics in Medicine , 35(16):2741--2753
2016
-
[39]
Sparapani, R., Spanbauer, C., and McCulloch, R. (2021). Nonparametric machine learning and efficient computation with B ayesian additive regression trees: The BART R package. Journal of Statistical Software , 97(1):1--66
2021
-
[40]
E., Murray, J
Starling, J. E., Murray, J. S., Carvalho, C. M., Bukowski, R. K., and Scott, J. G. (2020). BART with targeted smoothing: An analysis of patient-specific stillbirth risk. Annals of Applied Statistics , 14(1):28--50
2020
-
[41]
M., Browne, J., Shen, C., Chung, J., Patsolic, J
Tomita, T. M., Browne, J., Shen, C., Chung, J., Patsolic, J. L., Falk, B., Priebe, C. E., Yim, J., Burns, R., Maggioni, M., and Vogelstein, J. T. (2020). Sparse projection oblique randomer forests. Journal of Machine Learning Research , 21(104):1--39
2020
-
[42]
Wright, M. N. and Ziegler, A. (2017). ranger : A fast implementation of random forests for high dimensional data in C++ and R . Journal of Statistical Software , 77(1):1--17
2017
-
[43]
C., Santner, T
Yannotty, J. C., Santner, T. J., Furnstahl, R. J., and Pratola, M. T. (2024a). Model mixing using Bayesian additive regression trees. Technometrics , 66(2):196--207
2024
-
[44]
C., Santner, T
Yannotty, J. C., Santner, T. J., Li, B., and Pratola, M. T. (2024b). Combining climate models using Bayesian regression trees and random paths. arXiv 2407.13169 https://arxiv.org/abs/2407.13169
2024
-
[45]
and Suganthan, P
Zhang, L. and Suganthan, P. N. (2014). Random forests with ensemble of feature spaces. Pattern Recognition , 47(10):3429--3437
2014
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.