{"id":"df8a1ca2-fda8-4588-9d52-d53180fcc74b","arxiv_id":"1908.06852","paper_version":5,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SIRUS grows a random forest whose splits are limited to fixed quantiles, counts how often each rule appears across the forest, and retains only the most frequent rules to produce a stable and readable classifier.","lead":"SIRUS is a classification method that turns random forests into a short, stable list of if-then rules while keeping most of the forest's accuracy. It matters because simple models in critical settings, such as manufacturing quality control, are often too unstable to trust, and SIRUS offers a stable alternative.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Stability theorem is proved for subsampling with a_n/n -> 0, but the implemented and tested SIRUS uses bootstrap with replacement (a_n = n), so the central asymptotic guarantee does not apply to the evaluated algorithm.","rationale":"The reader's verdict is CONDITIONAL, and I agree with CONDITIONAL, so the verdict should remain unchanged. However, the reader's weakest_assumption (A3, continuous densities) is not the most load-bearing issue in my reading. The paper is transparent about the categorical-data heuristic ('we naturally extend...', 'it is best to discard...'), so the scope of the theory is honestly stated. The sampling mismatch is more serious: the theorem explicitly assumes subsampling without replacement with a_n = o(n), while the algorithm, the software, and the experiments use bootstrap with replacement and a_n = n (default random-forest settings). The proof of Theorem 1 fails at multiple steps when a_n/n does not vanish: Lemma 1-2 require sqrt(a_n) * P(q* <= X < hat_q) -> 0, which is only valid if sqrt(a_n/n) -> 0, and the variance bound V[p_n(P)] <= (a_n/n) xi is vacuous for a_n = n. Thus the central statement 'SIRUS is asymptotically stable' is not proven for the artifact being evaluated. The empirical evidence for stability is strong and may well survive the test, but the paper's explicit claim of a theoretical grounding is overstated. A CONDITIONAL acceptance requiring the authors to either (a) apply the theory to the actual bootstrap scheme, (b) use and benchmark the subsampling variant matching the assumptions, or (c) clearly delimit the theory to the subsampling variant, is the appropriate outcome. No issue here warrants rejection: the method is novel, the implementation is public, and the empirical stability gains are substantial and reproducible in principle.","tokens_in":58422,"tokens_out":14722,"duration_ms":158829,"concrete_test":"Run SIRUS on 3-5 UCI datasets (e.g., Diabetes, Credit German, Spambase) with two sampling schemes: (i) the current bootstrap with replacement, a_n = n; (ii) subsampling without replacement with a_n = floor(n^0.7), so a_n/n -> 0. Compare the Dice-Sorensen stability (3.4) and 1-AUC over repeated 10-fold CV. If (ii) gives materially lower stability than (i), then the reported stability of the implementation is not explained by Theorem 1. If (ii) matches (i), the practical impact of the mismatch is small, but a proof for the bootstrap case is still missing and should be supplied or explicitly flagged as out of scope.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The core theoretical claim (Theorem 1 and Corollary 1) is established under Assumption (A1), which requires a_n -> inf and a_n/n -> 0, and the text explicitly assumes subsampling without replacement (Section 4, paragraph before (4.1)). However, the SIRUS algorithm as described in Algorithm 1 and as implemented in the sirus package (based on ranger) uses sampling with replacement at the default random-forest settings, i.e., a_n = n. The proof of Theorem 1 depends critically on a_n/n -> 0 in two places: Lemma 1-2 show sqrt(a_n)(L_{a_n}(H, hat_q) - L_{a_n}(H, q*)) -> 0 using the fact that sqrt(a_n) * O_p(1/sqrt(n)) -> 0; if a_n = n, this difference is O_p(1) and does not vanish. Likewise, the variance bound V[p_n(P)] <= (a_n/n) xi_{a_n,a_n} is only useful when a_n/n -> 0. Consequently, the asymptotic stability guarantee is proven for a subsampling variant of SIRUS, not for the bootstrap version that is benchmarked and released. The paper does not state this limitation or provide an alternative analysis for the implemented sampling scheme. This is distinct from the (A3) continuity limitation, which the authors acknowledge as a heuristic extension; the sampling mismatch is not acknowledged and affects the central theorem even for perfectly continuous data.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SIRUS, a rule-based classification algorithm that extracts a short list of rules from a random forest by selecting paths that occur with frequency above a threshold p0 in a forest grown with split values restricted to empirical quantiles. The selected rules are averaged to produce a class probability estimate. The authors argue that SIRUS is interpretable under a simplicity-stability-predictivity triptych and support this with three types of evidence: (i) a theoretical analysis in Section 4 showing asymptotic stability under assumptions (A1)-(A3), i.e., that the estimated rule set converges in probability to a deterministic set P*_{p0} and that the Dice-Sorensen stability index converges to 1; (ii) extensive experiments on 19 UCI datasets plus the SECOM case study comparing SIRUS with CART, C5.0, RIPPER, PART, FOIL, CBA, BRL, RuleFit, Node harvest, and random forests; and (iii) an open-source R/C++ implementation available on CRAN. The theoretical proofs in Appendix C are detailed for one- and two-split paths, with the extension to arbitrary depth asserted by recursion.","tokens_in":58737,"tokens_out":6959,"duration_ms":72783,"significance":"If the central claims hold, SIRUS would be a valuable contribution to interpretable machine learning: it offers a concrete rule-list model with accuracy close to random forests, markedly higher stability than existing rule learners, and a freely available implementation. The paper is also careful to define a theoretical counterpart of the algorithm and to prove convergence in probability to a distribution-dependent rule set, which is a non-circular route to a stability guarantee. The experimental evaluation is broad, covering multiple rule-learning families and reporting model size, stability, and 1-AUC. The provided software is a practical asset. However, the strength of the contribution is tempered by a mismatch between the theory and the implemented algorithm, as detailed in the major comments.","major_comments":[{"comment":"Theorem 1 and Corollary 1 are proved under Assumption (A1), which requires a_n/n -> 0 and is explicitly tied to subsampling without replacement (Section 4, paragraph before (4.1)). However, Algorithm 1 and the reported experiments use sampling with replacement at the default random-forest settings (a_n = n) via ranger, and the released sirus package inherits this. The proof depends on a_n/n -> 0 in the U-statistic variance bound V[p_n(P)] <= (a_n/n) xi_{a_n,a_n} and in Lemmas 1-2, where sqrt(a_n) times O_p(1/sqrt(n)) vanishes only if a_n/n -> 0; with bootstrap a_n = n these terms are O_p(1) and do not vanish. The central asymptotic stability guarantee therefore applies to a subsampling variant of SIRUS, not to the implemented and benchmarked algorithm, and the manuscript does not state this limitation or provide an alternative analysis.","section":"Section 4, Assumption (A1); Algorithm 1; Section 5.1"},{"comment":"The set Pi of all possible paths is defined without a depth bound (Section 3), so for fixed p and q it is infinite. Corollary 1's proof bounds P(hat P_{M_n,n,p0} != P*_{p0}) by a sum over P in Pi, which requires a finite union bound or a uniform convergence result over Pi. Pointwise convergence of hat p_{M_n,n}(P) for each fixed P (Theorem 1) does not control paths of growing depth, and the assertion that the proof 'can be easily extended to a path of any depth d by recursion' does not supply the needed uniformity. The theoretical statement should be restricted to bounded-depth paths (as the software does with max.depth=2) or a uniform argument must be provided.","section":"Section 3, definition of Pi; Section 4, proof of Corollary 1"},{"comment":"The empirical stability metric used in Section 5.1 is the average proportion of shared rules between models built on the 10 folds of a single cross-validation, whose training sets overlap, whereas Corollary 1 and the stability index (3.4) concern two independent samples of size n. These are different quantities, so the reported stability values do not directly evaluate the theoretical guarantee. The paper should either report a stability estimate based on truly independent samples or explain why the overlapping-fold proxy is expected to behave like the independent quantity.","section":"Section 5.1 and Corollary 1"}],"minor_comments":[{"comment":"Standard deviations are omitted from Tables 2-4 with the claim that they are negligible; please report them, at least in an appendix, because several stability differences between SIRUS and its competitors are modest and the reader cannot assess their significance without variability estimates.","section":"Section 5, Tables 2-4"},{"comment":"The statement that Pi 'is in fact a deterministic quantity, which only depends upon the dimension p and the order q of the quantiles' is inaccurate if paths of unbounded depth are allowed; it should be qualified with a depth bound or a bounded-depth definition.","section":"Section 3, paragraph on Pi"},{"comment":"There are several typographical errors ('evalute', 'missclassification', '2st or 3nd class') and the notation M_n in the theory is not explicitly reconciled with the experiment-specific M determined by the stopping criterion (5.1); clarifying this would help the reader connect the asymptotic statements to the reported results.","section":"Section 5.1 and Section 5.4"},{"comment":"The connection between the stopping criterion (5.1) and Theorem 2 is presented heuristically; a more precise explanation of why the average of epsilon_{M,n,p0} over an estimated set of p0 values controls the stability at the final, data-dependent p0 would be helpful.","section":"Section 5.4, stopping criterion"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid algorithmic contribution with broad experiments and a usable package. The main issue is that the theory is proved for a subsampling-without-replacement, bounded-depth variant, while the algorithm and experiments use bootstrap with replacement and unbounded depth. This is a serious gap, but it appears fixable by either modifying the algorithm to match the theory or by carefully restricting the theoretical claims and adapting the experiments to the subsampling variant. I do not see a basis for rejection, provided the authors close or clearly scope the theory-algorithm gap."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. SIRUS is a genuinely useful rule-set method. The core idea—limit split candidates to a shared set of empirical quantiles, then keep rules by occurrence frequency across random forest trees—is simple and effective. On 19 UCI datasets plus the SECOM case, it produces much shorter and more stable rule lists than RuleFit and Node harvest, with accuracy close to random forests. That empirical claim looks credible. The released R/C++ package on CRAN is real reproducible work.\n\nThe main problem is the gap between the theorem and the implementation. Theorem 1 and Corollary 1 are proved under Assumption (A1), which requires subsampling without replacement with a_n/n -> 0. The proof actually depends on that rate: Lemmas 1–2 use sqrt(a_n) times an O_p(1/sqrt(n)) deviation, and the variance bound V[p_n(P)] <= (a_n/n) xi only helps when a_n/n -> 0. But Algorithm 1 says “Subsample with replacement”, and the experiments use standard random forest settings, i.e. bootstrap with a_n = n. The paper never flags that the theorem does not apply to the tested version. This is not an issue you can wave away; it affects the stability guarantee even for perfectly continuous data. For the bootstrap version, stability is supported only empirically.\n\nSecondary issues: the proof is written for paths of depth one and two, and the extension to arbitrary depth is asserted without details (probably fixable, but still a gap). Standard deviations are omitted from the main tables with a hand-wave, which makes it hard to judge whether the stability differences are significant. And p0 is tuned using the same cross-validation used for accuracy reporting, so the reported numbers are mildly selection-biased—a minor issue, not fatal.\n\nCredit where due: the quantile discretization is a clever way to make rule extraction stable, the Dice–Sorensen metric is the appropriate quantity to study, and the empirical comparisons are broader than what most interpretability papers offer. Who is this for? An applied practitioner gets a ready-to-use method that seems to do what it claims. A theory reader gets a nice stability result for a subsampled variant, but should not cite it as a proof for the released bootstrap implementation. I would send this to a serious referee. The right verdict is major revision: acknowledge the sampling mismatch, either extend the proof or explicitly restrict the theoretical claims, and show standard deviations. Rejection would be too harsh.","headline":"A genuinely useful rule-set method with strong empirical stability, but the headline theorem is proved for a subsampling variant that the released code does not use.","tokens_in":59255,"tokens_out":3372,"would_cite":true,"duration_ms":35534,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62H30","62G20","68T05"],"pacs":[],"model":"deepseek-v4-flash","headline":"SIRUS turns a random forest into a short rule list whose contents converge: with enough data, the same rules are selected on every fresh sample, with accuracy close to the forest.","keywords":["classification","interpretability","rule ensembles","random forests","stability","quantile discretization","asymptotic convergence"],"falsifier":"Simulate data satisfying the continuity assumptions, run SIRUS on two independent samples of increasing size, and measure the Dice–Sorensen index of the two rule lists; if the index does not approach 1 for a threshold away from the tie values, Corollary 1 is false. A sharper boundary check is to engineer a distribution where two splits tie in the population CART criterion and set $p_0$ exactly at their common probability, a case the theorem excludes; the rule lists should flip between samples at that threshold.","tokens_in":58268,"feed_emoji":"📏","tokens_out":10232,"duration_ms":99019,"temperature":0.7,"pith_summary":"The paper presents SIRUS, a classification algorithm that converts a random forest into a short list of if-then rules, and it argues that interpretability should be judged by the triad of simplicity, stability, and predictive accuracy. Its central claim is that SIRUS is asymptotically stable: given enough data, two independent runs on independent samples select essentially the same rule list, because the selected rule set converges in probability to a deterministic set that depends only on the data distribution. If that claim holds, the traditional weakness of rule models—their tendency to change drastically under small data perturbations—is removed, which matters for applications where a handful of rules must be trusted for critical decisions. The paper proves the convergence under continuous-distribution assumptions and supports it with experiments showing SIRUS's stability exceeds that of existing rule learners while its accuracy remains close to random forests.","feed_headline":"Same rule list every time: SIRUS stabilizes interpretable models","feed_subtitle":"Short readable rules that survive data perturbation, with accuracy near random forests and a convergence proof.","key_machinery":"The central object is the path space $\\Pi$, made finite by $q$-quantile discretization: a path is a sequence of triplets $(j_k,r_k,s_k)$ encoding the variable, quantile index, and side of each split. Finiteness is what gives each path a nonzero, well-defined occurrence probability in a randomized tree; without discretization, continuous cut points make almost every path unique to a single sample, which is exactly why ordinary rule extraction from trees is unstable. The argument is carried by two convergence results: the empirical CART splitting criterion computed at empirical quantiles converges to the theoretical criterion at population quantiles, and the bagged occurrence-frequency estimates have vanishing variance by a U-statistic bound. Together these show $\\hat p_{M,n}(P)\\to p^\\star(P)$ in probability, which by the threshold selection and deterministic post-treatment yields convergence of the whole rule set.","core_discovery":"To make rule extraction stable, SIRUS restricts every split of every tree to a fixed set of empirical $q$-quantiles of each input variable (typically $q=10$), so the universe of possible tree paths is finite and independent of the sample: a path is a sequence of triplets specifying the variable, the quantile index, and the side of each split. The forest then estimates, for each path $P$, the probability $\\hat p_{M,n}(P)$ that a randomized tree contains $P$, and the method keeps all paths whose estimated probability exceeds a threshold $p_0$. The paper proves that under assumptions (A1)–(A3) these estimated probabilities converge in probability to theoretical probabilities $p^\\star(P)$ defined from population quantiles and the population CART criterion; consequently the selected path set converges to the deterministic set $P^\\star_{p_0}$, and the Dice–Sorensen stability index converges to 1 in probability. The post-treatment that removes redundant rules and the final averaging of rules are deterministic functions of the selected paths, so they inherit the convergence. The net claim is that on continuous data the rule list itself is a reproducible estimate of a population-level object, not a sample-dependent artifact.","pith_inferences":["Beyond the paper: the same quantile-discretization device is portable to regression and to other tree ensembles such as boosting or extremely randomized trees, so a stable rule-surrogate method could be built for any tree black box; regression is flagged as future work in the conclusion.","Beyond the paper: because the stability guarantee is proven only for continuous inputs, a practitioner-facing diagnostic suggests itself—plot the Dice–Sorensen index against sample size on real data; if it fails to approach 1, the data are likely outside the continuous regime and the theoretical protection does not apply.","Beyond the paper: the theorem excludes thresholds $p_0$ that equal a theoretical path probability, and those boundary values are exactly where finite-sample rule lists should flip; users could avoid unstable outputs by inspecting the histogram of path frequencies and not choosing $p_0$ inside a dense cluster of near-tied probabilities."],"forward_implications":["On data satisfying the assumptions, repeated runs of SIRUS on independent samples converge to the same rule list, so the model's output can be read as a stable statement about the data-generating process.","Because deeper paths have exponentially more competitors in the finite path space, the selected rules are almost always one- or two-split rules, which keeps the final model readable.","On the UCI and manufacturing datasets tested, SIRUS keeps 1-AUC close to random forests while cutting the model to a handful of rules, so interpretability is not bought with a large accuracy loss.","The stability comparison indicates that competing tree-based rule methods such as RuleFit and Node harvest are unstable by design when splits are continuous; the quantile discretization is the step that makes convergence possible."],"supporting_citations":[{"why":"Supplies the random forest algorithm which SIRUS modifies by restricting splits to empirical q-quantiles.","marker":"Breiman, 2001a"},{"why":"Defines the CART splitting criterion used at each node to choose splits among the quantiles.","marker":"Breiman et al., 1984"},{"why":"Provides the U-statistic representation of bagged tree estimates used to bound the variance of path occurrence frequencies.","marker":"Mentch and Hooker, 2016"},{"why":"Supplies the U-statistic variance inequality controlling the sampling component of $\\hat p_{M,n}(P)$.","marker":"Hoeffding, 1948"},{"why":"Defines RuleFit, the main tree-ensemble rule competitor that SIRUS must beat in stability and simplicity.","marker":"Friedman and Popescu, 2008"},{"why":"Defines Node harvest, the other tree-ensemble rule competitor used in the experimental comparison.","marker":"Meinshausen, 2010"}],"fun_headline_variants":["Stable rules from random forests: SIRUS proves reproducibility","SIRUS: Interpretable rules that don’t change with the data","Finite quantiles make rule extraction reproducible: SIRUS","SIRUS: Rule lists with random-forest accuracy and proven stability"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The stability proof assumes every input variable is continuous, with no ties and no categories; for discrete or categorical data the theorem does not directly apply and the paper relies on practical workarounds.","fun_headline_variants_meta":{"raw":{"variants":["Stable rules from random forests: SIRUS proves reproducibility","SIRUS: Interpretable rules that don’t change with the data","Finite quantiles make rule extraction reproducible: SIRUS","SIRUS: Rule lists with random-forest accuracy and proven stability"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00018,"raw_usage":{"total_tokens":1318,"prompt_tokens":973,"completion_tokens":345,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":269}},"tokens_in":589,"tokens_out":345,"duration_ms":3943,"temperature":1.0,"reasoning_tokens":269,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:32:28.213382+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate data satisfying the continuity assumptions, run SIRUS on two independent samples of increasing size, and measure the Dice–Sorensen index of the two rule lists; if the index does not approach 1 for a threshold away from the tie values, Corollary 1 is false. A sharper boundary check is to engineer a distribution where two splits tie in the population CART criterion and set $p_0$ exactly at their common probability, a case the theorem excludes; the rule lists should flip between samples at that threshold.","supporting_citations":[],"review_version":1}