{"id":"50eb1e1d-c44b-4da8-93aa-af7270873d0e","arxiv_id":"2506.16965","paper_version":3,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A ten-level recursive stacking architecture with OOF-based pruning and periodic feature compression improves accuracy with depth on 33 tabular datasets and slightly beats TabNet and Deep Forest on selected configurations.","lead":"This paper introduces RocketStack, an ensemble method that stacks classifier predictions ten levels deep, pruning weak models and compressing features along the way. It reports accuracy gains with depth across 33 tabular datasets and small wins over deep tabular baselines like TabNet and Deep Forest.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The depth-gain claims rest on level-wise OOF probabilities whose nested cross-validation is unspecified; if level-ℓ features are in-sample, the trend and best-model results are artifacts.","rationale":"The reader's weakest_assumption identified exactly the same load-bearing concern: the level-wise OOF probabilities at levels 2 through 10 are not defined in a way that guarantees clean train/test separation for the meta-models. My reading of Algorithm 1 confirms the omission. The models M^(ℓ−1) are first trained on X^(ℓ−1) and then, at the next level, their predictions on X^(ℓ−1) are used as features; without a nested retraining loop, those predictions are in-sample. This would inflate the accuracy of every deeper level, making the LMM trend tests and the best-meta-model trajectories artifacts. The concrete test I propose is to implement the nested cross-validation explicitly and re-measure the key results; if the trends survive, the central claim is supported, and if they collapse, the published conclusions are invalid. I do not see a reason to change the reader's CONDITIONAL verdict: the paper should be accepted only with this check and, ideally, released code. I am not raising post-hoc model selection or missing error bars as the primary concern because those weaken the representative-model comparisons but do not by themselves undermine the average depth-trend claim that the leakage issue threatens.","tokens_in":27653,"tokens_out":5084,"duration_ms":50855,"concrete_test":"Re-run the pipeline on 3–5 representative datasets (one small, one large, one multi-class) with an explicitly nested 5-fold OOF at every level: inside each outer training fold, split the training meta-feature matrix X^(ℓ−1) into 5 inner folds; for each inner fold, retrain all M^(ℓ−1) models on the other 4 inner folds, predict the held-out inner fold, and concatenate these inner-fold predictions to form P^(ℓ). Train the level-ℓ meta-models only on this nested-OOF X^(ℓ), and evaluate on the untouched outer test fold as before. Then compare level-10 accuracies, LMM trend slopes, and Table A5's best-model trajectories against the published numbers. If the level-10 gains shrink substantially or the trend p-values become non-significant, the reported depth improvements are artifacts of in-sample level-wise features.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Algorithm 1's level-ℓ feature construction is the load-bearing point. The central depth-gain results (LMM trends, Table A4, Figure 4) and the best-meta-model-versus-baseline comparisons (Table A5, Figure 5) all depend on the OOF probabilities P^(ℓ) being genuinely out-of-sample for the meta-models trained on X^(ℓ). The pseudocode says only 'obtain OOF probabilities P^(ℓ) for M^(ℓ−1) on X^(ℓ−1)' (Algorithm 1, inside the level loop). But M^(ℓ−1) contains models that were trained on X^(ℓ−1) at the previous level; predicting X^(ℓ−1) with those models is in-sample, not out-of-fold. To make P^(ℓ) OOF, one must run a nested cross-validation inside each outer fold: split X^(ℓ−1), itself built from previous-level OOF features, into inner folds, retrain the M^(ℓ−1) models on inner training folds, and predict inner validation folds. The manuscript does not describe this nested protocol, nor how it interacts with the outer fold's test set X_te. Section 3.3's statement that 'every stacking level starts with OOF probabilities derived from a 5-fold cross-validation loop' is ambiguous: it could mean nested CV, but the algorithm and reported metrics do not confirm it. If the same training rows are used to build P^(ℓ) and to train the level-ℓ meta-models, the level-wise features leak the target, inflating every accuracy trajectory. This issue is more load-bearing than the post-hoc best-model selection or missing error bars, because it affects the average trend claims themselves, not just the representative-model comparisons.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes RocketStack, a recursive stacking architecture that iteratively builds meta-features from out-of-fold predictions of the previous level's model pool, with pruning based on OOF scores (optionally noise-perturbed) and optional feature compression (SFE, autoencoder, attention) applied periodically or at every level. The authors evaluate the architecture on 33 OpenML datasets up to depth 10, reporting positive accuracy trends via linear mixed models, favorable runtime and dimensionality scaling, and a final-level accuracy slightly above TabNet and Deep Forest.","tokens_in":27984,"tokens_out":7793,"duration_ms":76189,"significance":"If the evaluation protocol is sound, the paper is a useful empirical contribution: it demonstrates that deep recursive stacking up to depth 10 is feasible with controlled feature growth, and the ablation of pruning noise and compression frequency is informative. The breadth of the study (33 datasets, multiple compression mechanisms, pruning ablations, runtime and dimensionality tracking) is a strength. The paper does not provide code, and the combination of the underspecified OOF generation and the oracle-style model selection currently prevents verification of the central quantitative claims.","major_comments":[{"comment":"The construction of the level-l features P^(l) is not specified sufficiently to rule out leakage. In Algorithm 1, the line 'obtain OOF probabilities P^(l) for M^(l−1) on X^(l−1)' appears inside the outer-fold loop, but M^(l−1) contains models that, at the previous level, were trained on X^(l−1); predicting X^(l−1) with those trained models is in-sample. To produce genuinely out-of-sample probabilities, one must run an inner cross-validation that retrains the surviving models on inner folds of X^(l−1) and predicts the inner validation folds, and the pseudocode must describe how this inner CV is nested inside the outer fold and how the outer test set X_te remains untouched. Section 3.3's statement that 'every stacking level starts with OOF probabilities derived from a 5-fold cross-validation loop' is ambiguous because it is written from the perspective of the base learners at level 1. Because every headline result (LMM trends in Table A4, Figure 4, and the best-model comparisons in Table A5/Figure 5) depends on these features being OOF, this gap must be resolved before the empirical claims can be evaluated.","section":"Section 3.3 and Algorithm 1"},{"comment":"The 'best meta-model per level' accuracy is obtained by selecting, for each level, the single classifier with the highest test-set accuracy after averaging over folds and datasets. This is an oracle estimate: it uses the test labels to choose the model, so it does not represent the accuracy of any fixed algorithm that a practitioner could deploy, and it inflates the apparent gain from depth. The comparisons against the 'best standalone ensemble' (also selected on test) are therefore not a valid head-to-head benchmark. The authors should either report the accuracy of a predetermined model (e.g., the average of all surviving models, or a model chosen by an inner validation split) or explicitly label these curves as oracle upper bounds and soften the claims accordingly.","section":"Section 4.2, Table A5, Figure 5"},{"comment":"The reported comparison against TabNet and Deep Forest is under-specified. The reader cannot tell how many datasets and folds were used, whether the same train/test splits were used for all methods, how the Bayesian HPO was configured for these baselines, or whether the RocketStack numbers in Table 4 are the averaged accuracy of all level-10 meta-models (as suggested by the match with Table A2) or some other estimator. With margins of 0.51 and 1.29 percentage points and no uncertainty estimates, the claim of 'slightly surpasses' is not yet supported.","section":"Section 4.5, Table 4"},{"comment":"The conclusions about pruning noise rely on a small set of hand-picked constants (λ in {0.05, 0.1}, the 5 + 80·std(·)^2 percentile offset, the 75th-percentile attention threshold, and the compression ratio of 1/3) plus a post-hoc selection of the best configuration on the same 33 datasets. No sensitivity analysis is provided, so it is unclear whether the 'light randomization is best' finding is robust or an artifact of these particular values and of the exploratory comparison across 15 variants. At minimum, the authors should report the number of configurations examined and apply a multiple-comparisons correction across the variant selection step, or present the results as exploratory rather than confirmatory.","section":"Section 3.4 (Eqs. 6-7) and Section 5.1"}],"minor_comments":[{"comment":"The word 'Correspondance' should be 'Correspondence'.","section":"Author footnote"},{"comment":"Table 2 reports accuracy in percent while Table 3 reports confidence intervals in decimal proportions; please make the units consistent.","section":"Tables 2 and 3"},{"comment":"The SFE each-level multi-class accuracy drops sharply from 93.45% to 85.52%; consider discussing whether this is due to feature collapse or another identifiable cause.","section":"Section 4.1.2"},{"comment":"The broken y-axis in some multi-class panels makes the trends appear steeper than they are; a consistent scale would allow easier comparison across variants.","section":"Figure 4A"},{"comment":"The feature counts are averages over datasets and therefore non-integer; this is acceptable but should be stated explicitly in the table caption or the surrounding text.","section":"Section 4.4 and Table A6b"}],"recommendation":"major_revision","confidential_remarks":"The main concern is not that the authors are hiding a flawed implementation, but that the manuscript does not specify the evaluation protocol precisely enough for a reader to distinguish a sound nested-CV design from a leaky in-sample design. Given the central role of the depth-gain claim, I recommend requesting a revision that adds an explicit nested-loop pseudocode, a precise definition of the 'best model per level' estimator, and full details of the TabNet/Deep Forest setup. If the authors cannot provide those details, the paper may need to be rejected."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper stacks 20 (binary) or 14 (multi-class) classifiers to depth 10, with pruning based on OOF scores plus Gaussian perturbation, and periodic compression at levels 3, 6, 9. Across 33 OpenML datasets it reports consistent accuracy gains with depth, with several LMM trend tests significant. The best meta-model per level reportedly beats TabNet and Deep Forest at level 10.\n\nWhat is actually new: the specific combination — depth-10 recursive stacking, OOF-score perturbation for pruning, and periodic compression — is not in the cited literature. Each component is known, but this is a systematic configuration study with 33 datasets and multiple compression families (SFE, autoencoders, attention). That is a useful empirical contribution. The runtime and feature-count tables are genuinely informative and show that pruning and compression do control growth.\n\nThe soft spot is the OOF protocol. Algorithm 1 says \"obtain OOF probabilities P^(l) for M^(l−1) on X^(l−1)\" but does not say how. If this means taking the already-trained level l−1 models and predicting on X^(l−1), that is in-sample, and every trend result in Table A4 is inflated. If it means a nested 5-fold CV inside each outer fold, that needs to be stated explicitly, with details about how the outer test fold is excluded. Section 3.3's sentence about \"every stacking level starts with OOF probabilities derived from a 5-fold cross-validation loop\" could be read either way. This is load-bearing, not a cosmetic detail. The paper also selects the \"best-performing model\" per level from test results (Table A5, Figure 5), which is post-hoc selection, and the small margins vs. baselines come without error bars. The sublinear runtime claim is overreaching; the data show sublinear growth in some configurations, but the stated bound is loose and unmeasured. These are fixable: release code, specify nested CV, report uncertainty, and stop picking representatives from test performance.\n\nThe stress-test note is fair on the OOF ambiguity. I do not read the paper as intentionally deceptive — the prose suggests the author knows OOF is needed — but the pseudocode is too terse to rule out leakage. Who this is for: anyone working on deep ensembles or AutoML stacking. With the OOF protocol clarified and code released, the empirical claims would be worth taking seriously. Without that, treat the depth-gain numbers as conditional.\n\nI would send it to peer review. A good referee can pin down the OOF protocol and the selection-bias issue; the empirical scope is valuable enough to warrant that effort.","headline":"A plausible but under-specified deep-stacking configuration study; the depth-gain claims hinge on an OOF protocol that the paper never actually describes.","tokens_in":28520,"tokens_out":3835,"would_cite":false,"duration_ms":38044,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Across 33 tabular datasets, a ten-level recursive stacking ensemble keeps gaining accuracy with depth.","keywords":["stack ensemble","deep stacking","recursive ensemble learning","out-of-fold pruning","stochastic perturbation","feature compression","attention mechanism","tabular classification"],"falsifier":"On any of the 33 datasets, instrument the pipeline so the test fold used for evaluation at level l is excluded from all training and out-of-fold generation at level l+1; if the level-10 advantage over level-1 disappears under this strict nested holdout, the depth trend is a leakage artifact.","tokens_in":27413,"feed_emoji":"🚀","tokens_out":7798,"duration_ms":71295,"temperature":0.7,"pith_summary":"The paper tries to establish that stacking, combining many classifiers through a meta-model, does not have to stop at one or two levels. It introduces RocketStack, a recursive stacking architecture that runs to ten levels by pruning weak learners with out-of-fold scores, perturbing those scores with Gaussian noise to avoid premature convergence, and compressing the growing feature matrix at fixed checkpoints. Across 33 datasets, the authors report that accuracy rises with depth, that the best meta-model at each level increasingly beats the best standalone ensemble, and that at the deepest level RocketStack slightly edges established deep tabular baselines while keeping runtime growth sublinear. If true, this makes deep recursive stacking a practical, scalable option for tabular machine learning rather than a curiosity.","feed_headline":"Deep stacking gains accuracy through ten levels on 33 datasets","feed_subtitle":"Recursive pruning and periodic compression let a ten-level ensemble beat deep tabular baselines on 33 datasets.","key_machinery":"The load-bearing mechanism is a level-aware recursive stacking loop. At level 1, out-of-fold (OOF) probabilities from a pool of 20 binary or 14 multi-class classifiers are concatenated with the original features; at each later level, new OOF probabilities from the surviving models are blended with the previous level's feature matrix, optionally compressed by SFE, autoencoder, or attention-based selection at levels 3, 6, and 9, and then used to train the next meta-models. Before propagation, models are pruned by keeping only those whose OOF scores exceed an adaptive percentile threshold, optionally after adding Gaussian noise at scale lambda times the score range. A final stack-of-stacking layer concatenates selected outputs from all levels for one last estimator. This loop is what lets the architecture trade model-pool size and feature width for depth.","core_discovery":"RocketStack's central claim is that stacking can be pushed to depth 10 with controlled pruning and periodic feature compression, and that this yields accuracy gains over shallower stacking and over the best single base learner. Across 23 binary and 10 multi-class datasets, linear mixed-effects trend tests find statistically significant positive accuracy trends with depth for the uncompressed baseline and for periodic SFE in binary and periodic attention in multi-class settings, while per-level compression and autoencoder variants fail to show consistent trends. Light Gaussian perturbation of out-of-fold scores before pruning (lambda = 0.05) improves stability and late-level gains relative to deterministic pruning, and periodic compression at levels 3, 6, and 9 gives a better accuracy-runtime trade-off than compressing at every level. The best meta-model per level increasingly outperforms the best standalone ensemble, and at level 10 untuned RocketStack reaches 88.90 percent accuracy on the binary benchmark and 94.82 percent on the multi-class benchmark, slightly above hyperparameter-tuned TabNet and Deep Forest.","pith_inferences":["If clean out-of-fold separation holds, the level-wise accuracy curves suggest RocketStack's meta-features behave like a learned feature hierarchy; a testable extension is whether features selected at deep levels transfer across datasets.","The pruning-noise result mirrors Dropout for model selection; a natural next experiment is a noise schedule that anneals lambda with depth or selection by Thompson sampling over model scores, to see whether hyper-deep stacking beyond ten levels remains stable.","The advantage of periodic over per-level compression suggests a general accumulate-then-compress design pattern for layered ensembles that could be tested in other recursive pipelines.","If base-level HPO gains truly vanish by depth 10, compute budgets for deep ensembles should shift from tuning base learners toward increasing depth and pruning diversity."],"forward_implications":["Stacking can be extended well beyond one or two meta-layers; on these 33 datasets the best meta-model at each level keeps improving through level 10 rather than saturating early.","Applying feature compression at fixed checkpoints rather than every level preserves accuracy while controlling feature count and runtime, with periodic SFE in binary and periodic attention in multi-class being the strongest compressed configurations.","Perturbing out-of-fold scores with light Gaussian noise before pruning improves late-level accuracy over deterministic pruning, suggesting that controlled randomness in model selection acts as a regularizer.","Hyperparameter optimization of base learners gives an early lead that deep recursion erases; untuned RocketStack at level 10 matches or slightly exceeds HPO-tuned versions and outperforms HPO-tuned TabNet and Deep Forest.","With pruning monotonically shrinking the model pool and compression bounding feature width, runtime growth with depth can remain sublinear under RocketStack's controls."],"supporting_citations":[{"why":"Foundational definition of stacked generalization that RocketStack extends to depth.","marker":"Wolpert (1992)"},{"why":"Formalizes stacked regressions, establishing the meta-learning formulation used at each level.","marker":"Breiman (1996b)"},{"why":"Direct precursor supplying OOF-based weighting and pruning plus the Bayesian HPO pipeline built upon.","marker":"Demirel et al. (2023)"},{"why":"Defines the SFE utility-based feature selection used for periodic compression.","marker":"Ahadzadeh et al. (2023)"},{"why":"Provides the autoencoder reconstruction framework for one compression family.","marker":"Baldi (2012)"},{"why":"Supplies the attention scoring whose 75th-percentile masking performs feature selection.","marker":"Vaswani et al. (2017)"},{"why":"TabNet is one of the deep tabular baselines RocketStack compares against at level 10.","marker":"Arik and Pfister (2021)"},{"why":"Deep Forest is the other deep tabular baseline RocketStack slightly outperforms.","marker":"Zhou and Feng (2018)"},{"why":"Dropout regularization is the analogy used to interpret OOF-score perturbation.","marker":"Srivastava et al. (2014)"}],"fun_headline_variants":["Ten-level stacking outperforms tuned tabular baselines","RocketStack: deep recursive stacking with pruning and compression","Stacking depth to 10 with pruning boosts accuracy","Level-aware recursive stacking beats standalone ensembles"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire depth-gain result depends on the level-wise out-of-fold predictions being computed with a clean train/test split that the pseudocode never spells out, so if a test fold is reused to build the next level's features, the reported gains could be artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Ten-level stacking outperforms tuned tabular baselines","RocketStack: deep recursive stacking with pruning and compression","Stacking depth to 10 with pruning boosts accuracy","Level-aware recursive stacking beats standalone ensembles"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000267,"raw_usage":{"total_tokens":1672,"prompt_tokens":1060,"completion_tokens":612,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":676,"completion_tokens_details":{"reasoning_tokens":551}},"tokens_in":676,"tokens_out":612,"duration_ms":6319,"temperature":1.0,"reasoning_tokens":551,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:15:31.877023+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On any of the 33 datasets, instrument the pipeline so the test fold used for evaluation at level l is excluded from all training and out-of-fold generation at level l+1; if the level-10 advantage over level-1 disappears under this strict nested holdout, the depth trend is a leakage artifact.","supporting_citations":[],"review_version":2}