REVIEW 3 major objections 4 minor 50 references
SIRUS: Stable and Interpretable RUle Set for Classification
T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read SIRUS turns a random forest into a short rule list whose contents converge: with enough data, the same rules are selected on every fresh sample, with accuracy close to the forest.
desk verdict A genuinely useful rule-set method with strong empirical stability, but the headline theorem is proved for a subsampling variant that the released code does not use. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the path space $\Pi$, made finite by $q$-quantile discretization: a path is a sequence of triplets $(j_k,r_k,s_k)$ encoding the variable, quantile index, and side of each split. Finiteness is what gives each path a nonzero, well-defined occurrence probability in a randomized tree; without discretization, continuous cut points make almost every path unique to a single sample, which is exactly why ordinary rule extraction from trees is unstable. The argument is carried by two convergence results: the empirical CART splitting criterion computed at empirical quantiles converges to the theoretical criterion at population quantiles, and the bagged occurrence-frequency estimates have vanishing variance by a U-statistic bound. Together these show $\hat p_{M,n}(P)\to p^\star(P)$ in probability, which by the threshold selection and deterministic post-treatment yields convergence of the whole rule set.
What would settle it
Simulate data satisfying the continuity assumptions, run SIRUS on two independent samples of increasing size, and measure the Dice–Sorensen index of the two rule lists; if the index does not approach 1 for a threshold away from the tie values, Corollary 1 is false. A sharper boundary check is to engineer a distribution where two splits tie in the population CART criterion and set $p_0$ exactly at their common probability, a case the theorem excludes; the rule lists should flip between samples at that threshold.
Extended reading notes
Core claim
To make rule extraction stable, SIRUS restricts every split of every tree to a fixed set of empirical $q$-quantiles of each input variable (typically $q=10$), so the universe of possible tree paths is finite and independent of the sample: a path is a sequence of triplets specifying the variable, the quantile index, and the side of each split. The forest then estimates, for each path $P$, the probability $\hat p_{M,n}(P)$ that a randomized tree contains $P$, and the method keeps all paths whose estimated probability exceeds a threshold $p_0$. The paper proves that under assumptions (A1)–(A3) these estimated probabilities converge in probability to theoretical probabilities $p^\star(P)$ defined from population quantiles and the population CART criterion; consequently the selected path set converges to the deterministic set $P^\star_{p_0}$, and the Dice–Sorensen stability index converges to 1 in probability. The post-treatment that removes redundant rules and the final averaging of rules are deterministic functions of the selected paths, so they inherit the convergence. The net claim is that on continuous data the rule list itself is a reproducible estimate of a population-level object, not a sample-dependent artifact.
Load-bearing premise
The stability proof assumes every input variable is continuous, with no ties and no categories; for discrete or categorical data the theorem does not directly apply and the paper relies on practical workarounds.
Editorial extensions
If this is right
- On data satisfying the assumptions, repeated runs of SIRUS on independent samples converge to the same rule list, so the model's output can be read as a stable statement about the data-generating process.
- Because deeper paths have exponentially more competitors in the finite path space, the selected rules are almost always one- or two-split rules, which keeps the final model readable.
- On the UCI and manufacturing datasets tested, SIRUS keeps 1-AUC close to random forests while cutting the model to a handful of rules, so interpretability is not bought with a large accuracy loss.
- The stability comparison indicates that competing tree-based rule methods such as RuleFit and Node harvest are unstable by design when splits are continuous; the quantile discretization is the step that makes convergence possible.
Reading between the lines
- Beyond the paper: the same quantile-discretization device is portable to regression and to other tree ensembles such as boosting or extremely randomized trees, so a stable rule-surrogate method could be built for any tree black box; regression is flagged as future work in the conclusion.
- Beyond the paper: because the stability guarantee is proven only for continuous inputs, a practitioner-facing diagnostic suggests itself—plot the Dice–Sorensen index against sample size on real data; if it fails to approach 1, the data are likely outside the continuous regime and the theoretical protection does not apply.
- Beyond the paper: the theorem excludes thresholds $p_0$ that equal a theoretical path probability, and those boundary values are exactly where finite-sample rule lists should flip; users could avoid unstable outputs by inspecting the histogram of path frequencies and not choosing $p_0$ inside a dense cluster of near-tied probabilities.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces SIRUS, a rule-based classification algorithm that extracts a short list of rules from a random forest by selecting paths that occur with frequency above a threshold p0 in a forest grown with split values restricted to empirical quantiles. The selected rules are averaged to produce a class probability estimate. The authors argue that SIRUS is interpretable under a simplicity-stability-predictivity triptych and support this with three types of evidence: (i) a theoretical analysis in Section 4 showing asymptotic stability under assumptions (A1)-(A3), i.e., that the estimated rule set converges in probability to a deterministic set P*_{p0} and that the Dice-Sorensen stability index converges to 1; (ii) extensive experiments on 19 UCI datasets plus the SECOM case study comparing SIRUS with CART, C5.0, RIPPER, PART, FOIL, CBA, BRL, RuleFit, Node harvest, and random forests; and (iii) an open-source R/C++ implementation available on CRAN. The theoretical proofs in Appendix C are detailed for one- and two-split paths, with the extension to arbitrary depth asserted by recursion.
Significance. If the central claims hold, SIRUS would be a valuable contribution to interpretable machine learning: it offers a concrete rule-list model with accuracy close to random forests, markedly higher stability than existing rule learners, and a freely available implementation. The paper is also careful to define a theoretical counterpart of the algorithm and to prove convergence in probability to a distribution-dependent rule set, which is a non-circular route to a stability guarantee. The experimental evaluation is broad, covering multiple rule-learning families and reporting model size, stability, and 1-AUC. The provided software is a practical asset. However, the strength of the contribution is tempered by a mismatch between the theory and the implemented algorithm, as detailed in the major comments.
major comments (3)
- [Section 4, Assumption (A1); Algorithm 1; Section 5.1] Theorem 1 and Corollary 1 are proved under Assumption (A1), which requires a_n/n -> 0 and is explicitly tied to subsampling without replacement (Section 4, paragraph before (4.1)). However, Algorithm 1 and the reported experiments use sampling with replacement at the default random-forest settings (a_n = n) via ranger, and the released sirus package inherits this. The proof depends on a_n/n -> 0 in the U-statistic variance bound V[p_n(P)] <= (a_n/n) xi_{a_n,a_n} and in Lemmas 1-2, where sqrt(a_n) times O_p(1/sqrt(n)) vanishes only if a_n/n -> 0; with bootstrap a_n = n these terms are O_p(1) and do not vanish. The central asymptotic stability guarantee therefore applies to a subsampling variant of SIRUS, not to the implemented and benchmarked algorithm, and the manuscript does not state this limitation or provide an alternative analysis.
- [Section 3, definition of Pi; Section 4, proof of Corollary 1] The set Pi of all possible paths is defined without a depth bound (Section 3), so for fixed p and q it is infinite. Corollary 1's proof bounds P(hat P_{M_n,n,p0} != P*_{p0}) by a sum over P in Pi, which requires a finite union bound or a uniform convergence result over Pi. Pointwise convergence of hat p_{M_n,n}(P) for each fixed P (Theorem 1) does not control paths of growing depth, and the assertion that the proof 'can be easily extended to a path of any depth d by recursion' does not supply the needed uniformity. The theoretical statement should be restricted to bounded-depth paths (as the software does with max.depth=2) or a uniform argument must be provided.
- [Section 5.1 and Corollary 1] The empirical stability metric used in Section 5.1 is the average proportion of shared rules between models built on the 10 folds of a single cross-validation, whose training sets overlap, whereas Corollary 1 and the stability index (3.4) concern two independent samples of size n. These are different quantities, so the reported stability values do not directly evaluate the theoretical guarantee. The paper should either report a stability estimate based on truly independent samples or explain why the overlapping-fold proxy is expected to behave like the independent quantity.
minor comments (4)
- [Section 5, Tables 2-4] Standard deviations are omitted from Tables 2-4 with the claim that they are negligible; please report them, at least in an appendix, because several stability differences between SIRUS and its competitors are modest and the reader cannot assess their significance without variability estimates.
- [Section 3, paragraph on Pi] The statement that Pi 'is in fact a deterministic quantity, which only depends upon the dimension p and the order q of the quantiles' is inaccurate if paths of unbounded depth are allowed; it should be qualified with a depth bound or a bounded-depth definition.
- [Section 5.1 and Section 5.4] There are several typographical errors ('evalute', 'missclassification', '2st or 3nd class') and the notation M_n in the theory is not explicitly reconciled with the experiment-specific M determined by the stopping criterion (5.1); clarifying this would help the reader connect the asymptotic statements to the reported results.
- [Section 5.4, stopping criterion] The connection between the stopping criterion (5.1) and Theorem 2 is presented heuristically; a more precise explanation of why the average of epsilon_{M,n,p0} over an estimated set of p0 values controls the stability at the final, data-dependent p0 would be helpful.
Circularity Check
No load-bearing circularity: the stability theorem is self-contained and the empirical estimator is compared to an independently defined population target.
full rationale
The paper's derivation chain is self-contained. Theorem 1 proves that the empirical path-frequency estimator \hat p_{M_n,n}(P) converges in probability to p*(P), a population quantity defined from the theoretical CART criterion, theoretical quantiles, and the distribution P_{X,Y} alone. The limiting set P*_{p0} is therefore not fitted from the data or defined in terms of the empirical stability index; Corollary 1 is a standard continuous-mapping consequence of this consistency. The Dice-Sorensen index \hat S_{M,n,p0} is the object of study (overlap between two independent fits), not an input to the proof. The only author self-citations are peripheral: Biau and Scornet (2016) is cited for the standard equivalence between the CART criterion and Gini impurity, and Benard and Wright (2020) for the software package itself; neither is load-bearing for the asymptotic result. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via citation. The subsampling-without-replacement assumption (A1) being stricter than the bootstrap used in the implementation is a scope or correctness concern, not a circularity, since the theorem explicitly states its assumptions and does not fit constants to the target conclusion.
Assumptions & free parameters
free parameters (4)
- p0 =
tuned per dataset (e.g., 0.04 for SECOM)
- q =
10
- max tree depth =
2
- alpha =
0.05
assumptions (5)
- domain assumption Subsampling rate a_n satisfies a_n -> infinity and a_n/n -> 0 (Assumption A1).
- domain assumption Number of trees M_n satisfies M_n -> infinity (Assumption A2).
- domain assumption X has a strictly positive density with continuous, bounded, strictly positive marginals (Assumption A3).
- standard math The sample is i.i.d. from an unknown distribution P_{X,Y}.
- ad hoc to paper Ties in the theoretical CART criterion are broken according to a limit law defined in Definitions 1 and 2.
Cite this review
Pith. "Pith review of SIRUS: Stable and Interpretable RUle Set for Classification." pith.science (2026). https://pith.science/paper/KCE6LEEN
@misc{pith2026190806852,
author = {Pith},
title = {Pith review of: SIRUS: Stable and Interpretable RUle Set for Classification},
year = {2026},
howpublished = {\url{https://pith.science/paper/KCE6LEEN}},
note = {Machine review of arXiv:1908.06852}
}
read the original abstract
State-of-the-art learning algorithms, such as random forests or neural networks, are often qualified as "black-boxes" because of the high number and complexity of operations involved in their prediction mechanism. This lack of interpretability is a strong limitation for applications involving critical decisions, typically the analysis of production processes in the manufacturing industry. In such critical contexts, models have to be interpretable, i.e., simple, stable, and predictive. To address this issue, we design SIRUS (Stable and Interpretable RUle Set), a new classification algorithm based on random forests, which takes the form of a short list of rules. While simple models are usually unstable with respect to data perturbation, SIRUS achieves a remarkable stability improvement over cutting-edge methods. Furthermore, SIRUS inherits a predictive accuracy close to random forests, combined with the simplicity of decision trees. These properties are assessed both from a theoretical and empirical point of view, through extensive numerical experiments based on our R/C++ software implementation sirus available from CRAN.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[1]
S. Alelyani, Z. Zhao, and H. Liu. A dilemma in assessing stability of feature selection algorithms. In 13th IEEE International Conference on High Performance Computing & Communication, pages 701--707, Piscataway, 2011. IEEE
work page 2011
-
[2]
G. Biau and E. Scornet. A random forest guided tour. Test, 25: 0 197--227, 2016
work page 2016
-
[3]
A.-L. Boulesteix and M. Slawski. Stability and aggregation of ranked gene lists. Briefings in Bioinformatics, 10: 0 556--568, 2009
work page 2009
-
[4]
O. Bousquet and A. Elisseeff. Stability and generalization. Journal of Machine Learning Research, 2: 0 499--526, 2002
work page 2002
-
[5]
L. Breiman. Bagging predictors. Machine Learning, 24: 0 123--140, 1996
work page 1996
-
[6]
L. Breiman. Random forests. Machine Learning, 45: 0 5--32, 2001 a
work page 2001
-
[7]
L. Breiman. Statistical modeling: The two cultures (with comments and a rejoinder by the author). Statistical Science, 16: 0 199--231, 2001 b
work page 2001
-
[8]
L. Breiman. Setting up, using, and understanding random forests v3.1. 2003a. URL https://www.stat.berkeley.edu/ breiman/Using_random_forests_V3.1.pdf
Show all 50 references
-
[9]
Breiman, J.H
L. Breiman, J.H. Friedman, R.A. Olshen, and C.J. Stone. Classification and Regression Trees. Chapman & Hall/CRC, Boca Raton, 1984
1984
-
[10]
Chao, R.L
A. Chao, R.L. Chazdon, R.K. Colwell, and T.-J. Shen. Abundance-based similarity indices and their estimation when there are unseen species in samples. Biometrics, 62: 0 361--371, 2006
2006
-
[11]
Clark and T
P. Clark and T. Niblett. The cn2 induction algorithm. Machine Learning, 3: 0 261--283, 1989
1989
-
[12]
W.W. Cohen. Fast effective rule induction. In Proceedings of the Twelfth International Conference on Machine Learning, pages 115--123. Morgan Kaufmann Publishers Inc., San Francisco, 1995
1995
-
[13]
Cohen and Y
W.W. Cohen and Y. Singer. A simple, fast, and effective rule learner. In Proceedings of the Sixteenth National Conference on Artificial Intelligence and Eleventh Conference on Innovative Applications of Artificial Intelligence, pages 335--342, Palo Alto, 1999. AAAI Press
1999
-
[14]
Dembczy \'n ski, W
K. Dembczy \'n ski, W. Kot owski, and R. S owi \'n ski. Ender: A statistical framework for boosting decision rules. Data Mining and Knowledge Discovery, 21: 0 52--90, 2010
2010
-
[15]
Devroye and T
L. Devroye and T. Wagner. Distribution-free inequalities for the deleted and holdout error estimates. IEEE Transactions on Information Theory, 25: 0 202--207, 1979
1979
-
[16]
Doshi-Velez and B
F. Doshi-Velez and B. Kim. Towards a rigorous science of interpretable machine learning. arXiv:1702.08608, 2017
2017 arXiv
-
[17]
UCI machine learning repository, 2017
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http://archive.ics.uci.edu/ml
2017
-
[18]
M. Fokkema. Pre: An r package for fitting prediction rule ensembles. arXiv:1707.07149, 2017
2017 arXiv
-
[19]
Generating accurate rule sets without global optimization
Eibe Frank and Ian H Witten. Generating accurate rule sets without global optimization. In Proceedings of the Fifteenth International Conference on Machine Learning, pages 144--151, San Francisco, 1998. Morgan Kaufmann Publishers Inc
1998
-
[20]
A.A. Freitas. Comprehensible classification models: A position paper. ACM SIGKDD Explorations Newsletter, 15: 0 1--10, 2014
2014
-
[21]
Friedman, B.E
J.H. Friedman, B.E. Popescu, et al. Predictive learning via rule ensembles. The Annals of Applied Statistics, 2: 0 916--954, 2008
2008
-
[22]
F \"u rnkranz and G
J. F \"u rnkranz and G. Widmer. Incremental reduced error pruning. In Proceedings of the 11th International Conference on Machine Learning, pages 70--77, San Francisco, 1994. Morgan Kaufmann Publishers Inc
1994
-
[23]
He and W
Z. He and W. Yu. Stable feature selection for biomarker discovery. Computational Biology and Chemistry, 34: 0 215--225, 2010
2010
-
[24]
Kumbier, S
K. Kumbier, S. Basu, J.B. Brown, S. Celniker, and B. Yu. Refining interaction search through signed iterative random forests. arXiv:1810.07287, 2018
2018 arXiv
-
[25]
B. Letham. Statistical learning for decision making: Interpretability, uncertainty, and inference. PhD thesis, Massachusetts Institute of Technology, 2015
2015
-
[26]
Letham, C
B. Letham, C. Rudin, T.H. McCormick, and D. Madigan. Interpretable classifiers using rules and bayesian analysis: Building a better stroke prediction model. The Annals of Applied Statistics, 9: 0 1350--1371, 2015
2015
-
[27]
Z.C. Lipton. The mythos of model interpretability. arXiv:1606.03490, 2016
2016 arXiv
-
[28]
Meinshausen
N. Meinshausen. Node harvest. The Annals of Applied Statistics, 4: 0 2049--2072, 2010
2010
-
[29]
Meinshausen
N. Meinshausen. Package ‘nodeharvest’, 2015
2015
-
[30]
Mentch and G
L. Mentch and G. Hooker. Quantifying uncertainty in random forests via confidence intervals and hypothesis tests. Journal of Machine Learning Research, 17: 0 841--881, 2016
2016
-
[31]
Michalski
R.S. Michalski. On the quasi-minimal solution of the general covering problem. In Proceedings of the Fifth International Symposium on Information Processing, pages 125--128, New York, 1969. ACM
1969
-
[32]
Murdoch, C
W.J. Murdoch, C. Singh, K. Kumbier, R. Abbasi-Asl, and B. Yu. Interpretable machine learning: Definitions, methods, and applications. arXiv:1901.04592, 2019
1901 arXiv
-
[33]
J.R. Quinlan. C4.5: Programs for Machine Learning. Morgan Kaufmann, San Mateo, 1992
1992
-
[34]
Ribeiro, S
M.T. Ribeiro, S. Singh, and C. Guestrin. Why should i trust you? explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1135--1144, New York, 2016. ACM
2016
-
[35]
R.L. Rivest. Learning decision lists. Machine Learning, 2: 0 229--246, 1987
1987
-
[36]
Rogers and T.J
W.H. Rogers and T.J. Wagner. A finite sample distribution-free performance bound for local discrimination rules. The Annals of Statistics, 6: 0 506--514, 1978
1978
-
[37]
C. Rudin. Please stop explaining black box models for high stakes decisions. arXiv:1811.10154, 2018
2018 arXiv
-
[38]
u ping. Learning interpretable models. PhD thesis, Universit \
S. R \"u ping. Learning interpretable models. PhD thesis, Universit \"a t Dortmund, 2006
2006
-
[39]
Therneau, B
T. Therneau, B. Atkinson, B. Ripley, and Maintainer B. Ripley. Package ‘rpart’, 2018
2018
-
[40]
V. Vapnik. Statistical Learning Theory. 1998, volume 3. Wiley, New York, 1998
1998
-
[41]
Weiss and N
S.M. Weiss and N. Indurkhya. Lightweight rule induction. In Proceedings of the Seventeenth International Conference on Machine Learning, pages 1135--1142, San Francisco, 2000. Morgan Kaufmann Publishers Inc
2000
-
[42]
ranger: A fast implementation of random forests for high dimensional data in c++ and r
Marvin N Wright and Andreas Ziegler. ranger: A fast implementation of random forests for high dimensional data in c++ and r. Journal of Statistical Software, 77: 0 1--17, 2017
2017
-
[43]
H. Yang, C. Rudin, and M. Seltzer. Scalable bayesian rule lists. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pages 3921--3930. Proceedings of Machine Learning Research, 2017
2017
-
[44]
B. Yu. Stability. Bernoulli, 19: 0 1484--1500, 2013
2013
-
[45]
Yu and K
B. Yu and K. Kumbier. Three principles of data science: Predictability, computability, and stability (pcs). arXiv:1901.08152, 2019
1901 arXiv
-
[46]
Zucknick, S
M. Zucknick, S. Richardson, and E.A. Stronach. Comparing the characteristics of gene expression profiles derived by univariate and multivariate classification methods. Statistical Applications in Genetics and Molecular Biology, 7: 0 1--34, 2008
2008
-
[47]
Cvitkovi \'c , A.-S
M. Cvitkovi \'c , A.-S. Smith, and J. Pande. Asymptotic expansions of the hypergeometric function with two large parameters application to the partition function of a lattice gas in a field of traps. Journal of Physics A: Mathematical and Theoretical, 50: 0 265206, 2017
2017
-
[48]
Hoeffding
W. Hoeffding. A class of statistics with asymptotically normal distribution. The Annals of Mathematical Statistics, 19: 0 293--325, 09 1948. doi:10.1214/aoms/1177730196. URL https://doi.org/10.1214/aoms/1177730196
1948
-
[49]
Olver, D.W
F.W.J. Olver, D.W. Lozier, R.F. Boisvert, and C.W. Clark. NIST Handbook of Mathematical Functions Hardback and CD-ROM. Cambridge University Press, 2010
2010
-
[50]
Serfling
R.J. Serfling. Approximation Theorems of Mathematical Statistics, volume 162. John Wiley & Sons, 2009
2009
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.