{"id":"fa7340ee-fa0c-4dca-845a-e71f09c5387e","arxiv_id":"2412.01120","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A warm-start plus early-stopping estimator gives variable importance close to full retraining for wide neural networks and gradient-boosted trees, with lower compute.","lead":"This paper estimates variable importance by taking a fully trained model, removing a feature, and running only a few extra gradient steps instead of retraining from scratch. The warm-start plus early stopping method matches retraining accuracy in simulations and comes with convergence rates for wide neural networks and gradient-boosted trees.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The O(N^{-1/2}) guarantee rests on Assumption 4.1, requiring the full model and the reduced target to lie in the dropped-feature RKHS; real networks are not constrained to that space, and the paper's verification simulation is deliberately built to satisfy it.","rationale":"The reader's weakest-assumption analysis points to Assumption 4.1, and I agree that this is the most load-bearing condition for the theoretical claim. The GBDT corollaries also require β values (N^{5/4}, N^{15/4}) far above the β=10,000 used in experiments, and the practical stopping rule is a validation heuristic rather than the oracle bTop, but those are additional mismatches rather than the deepest one: the misspecification issue would affect even the neural-network branch, which is the paper's primary theoretical showcase. The empirical comparisons against retrain and dropout provide useful evidence that the method can work in practice, so I do not think the paper should be rejected outright. However, the stated O(N^{−1/2}) guarantee is conditional on a data-dependent RKHS-span assumption that realistic targets and warm-start models will generally violate, and the paper's own verification does not exercise the violation. The correct disposition is therefore to keep the reader's CONDITIONAL verdict: the paper is promising but needs either a misspecification-robust analysis, or an explicit relaxation of Assumption 4.1 to the shifted target f0,−I−f_c_N with verification that this condition is the one needed and is credible in applications.","tokens_in":37853,"tokens_out":16908,"duration_ms":158986,"concrete_test":"Generate data with X1 independent of X_−I, with E[sin X1]=0 but mean μ=E[X1] satisfying sin μ≠0. Choose φ in the RKHS of the dropped-feature kernel and ψ outside it (e.g., a threshold indicator), and set Y = φ(X_−I) + ψ(X_−I) sin(X1). Then f0,−I = φ ∈ H, but the warm-start shift f0,−I−f_c_N(·, μ) ≈ −ψ(·) sin μ ∉ H, violating the assumption in exactly the way the proof needs it. Run the warm-start early-stopping procedure as in Section 5.1 and plot ∥f_bTop−f0,−I∥_N^2 versus N on a log-log scale. If the curve plateaus or decays with exponent substantially worse than 1/2, Assumption 4.1 is load-bearing; if it still tracks N^{−1/2}, the concern is not practically decisive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central guarantee (Theorem 1, via Lemma 1) bounds ∥f_bTop−f0,−I∥_N^2 by a bias term plus variance and kernel-drift terms. The early-stopping theory is invoked for the shifted target f0,−I−f_c_N, and Assumption 4.1 asserts that both f_c_N and f0,−I lie in span{K(I)(·, X_−I)}, which the paper calls 'purely theoretical convenience.' In the implemented algorithm, f_c_N is a full-feature neural network evaluated with features I replaced by their mean; there is no reason for this function, or for a real regression target, to belong to the RKHS of the dropped-feature kernel. When it does not, the null-space bias term in Lemma 1 is nonzero and need not decay at the claimed O(N^{−1/2}) rate. Section 5.1 explicitly constructs f0 so that Assumption 4.1 holds, and Section D.3 computes C_H by assuming the membership, so the only simulation that checks the rate lives inside the assumption. The proof of Theorem 2 also assumes, rather than derives, the bound ∥f^δ_bTop(·)∥_2^2 = O(N^{−1/2}), compounding the gap. The method may still work empirically, but the stated reliability guarantee does not cover the misspecified case the method targets.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a warm-start plus early-stopping method for estimating variable importance without retraining a model for every dropped-feature subset. Starting from the fully trained model, the method runs gradient descent or gradient boosting on the feature-dropped data for a short time, using a stopping rule that is motivated by early-stopping theory for kernel methods. The main theoretical claims are Theorem 1, an O(N^{-1/2}) empirical-norm error bound for the stopped model under Assumptions 4.1-4.5, and Theorem 2, an analogous population-norm bound. The results are specialized to wide neural networks through the neural tangent kernel and to a simplified gradient-boosted decision tree algorithm, and an asymptotic normality result is claimed for VI estimation with neural networks. Simulations and a gas-turbine data example compare the method with dropout and retraining baselines.","tokens_in":38275,"tokens_out":8281,"duration_ms":74241,"significance":"The proposed computational strategy is valuable: if the theoretical guarantee held in realistic settings, variable importance could be estimated at the cost of one full training run plus short warm-start runs, with accuracy close to full retraining. The paper has concrete strengths: Lemma 1 gives a clean error decomposition into bias, variance, and kernel-drift terms; the connection to the early-stopping framework of Raskutti et al. and to the GBDT kernel of Ustimenko et al. is explicit; and the authors provide code and reproducible experiments. However, the significance is conditional because the main theorems require the target and the full model to lie in the dropped-feature RKHS, rely on an oracle stopping time that uses unknown quantities, and leave the width requirement for neural networks unquantified. These gaps mean the practical reliability claim is not yet established by the theory.","major_comments":[{"comment":"The O(N^{-1/2}) bound in Theorem 1 requires both f_N^c and f0,-I to lie in span{K(I)(·, X_-I)}. This assumption is exactly what makes the null-space bias term 2∑_{j=r+1}^N (ζ*_jj)^2 in Lemma 1 vanish; the paper calls it 'purely theoretical convenience' and Section 5.1/D.3 deliberately constructs f0 and computes C_H under this membership. For a real regression function outside this span the null-space bias need not decay, so the stated guarantee does not cover the misspecified setting that the method is intended for. The manuscript should either prove a misspecification bias bound or explicitly restrict the reliability claim to the RKHS-membership case.","section":"§4.2, Assumption 4.1 and Lemma 1"},{"comment":"The population-norm proof contains the sentence 'Here we assume that ∥f^δ_bTop(·)∥_2^2 can be upper bounded by O(1/√N)' after Lemma 4. This is the key bound needed to control the varying-kernel term in the random-design case, and it is assumed rather than proved in the general framework. The later Lemmas 5 and 7 supply such bounds only for the NN and GBDT examples, so Theorem 2 as stated for general kernel-based gradient methods is not established by the proof. This is load-bearing for the random-design claim.","section":"§6.1.2, proof of Theorem 2"},{"comment":"The theoretical stopping rule bTop/bTmax depends on C_H and σ, which are unknown in practice (eqs. (21)-(23)), while Algorithm 1 uses a validation-patience rule. No theorem or quantitative analysis connects the patience-based stopping time to the oracle bTop, so the empirical accuracy claims in Sections 5.2-5.7 do not follow from Theorem 1. The manuscript should add an explicit approximation result for the practical stopping rule or clearly label the empirical results as heuristic.","section":"§5, Algorithm 1, and Theorem 1"},{"comment":"The width m is only shown to exist, with no stated rate relative to N; the text says 'We do not specify a particular rate for m in relation to N.' To use the corollaries one needs a checkable overparameterization condition (e.g., m ≳ N^a) under which the linearization and NTK stability bounds hold; as written, the result is consistent with m depending on N in a way that the finite-width networks in the experiments (m=2048) do not satisfy. This makes the theorem's applicability to the implemented networks unclear.","section":"§4.3, Corollaries 1-2"},{"comment":"The asymptotic normality result is proved only for the 0-bias ReLU network setting of Corollary 2, but the text then says 'For general networks, even though we do not provide rigorous theoretical results, we should expect the same asymptotical normality to hold,' and the CI experiments (Section 5.6) use general networks and GBDT. This is an extrapolation beyond the proven statement; either prove the extension or restrict the CI claim accordingly.","section":"§4.5, Corollary 5"}],"minor_comments":[{"comment":"The keyword list contains 'wart-start'; this should be 'warm-start'.","section":"Keywords"},{"comment":"The display defining val(S∪{j})−val(S) is malformed ('V ISj'); fix the formatting so the difference is clearly equal to VI_S^j.","section":"§3.1"},{"comment":"The recursion in eq. (58) uses K(I)_τ(·, X(I)), while the induction display in eq. (84) uses K(I)(·, X(I)); reconcile the two definitions because the claimed induction depends on which kernel is used.","section":"§6.1.2, Lemma 4"},{"comment":"The text says the observed population bound is 'closer to or even faster than O(N^{-1})' while Theorem 1/2 state O(N^{-1/2}); either report the observed slope as a heuristic or align the claim with the proven rate.","section":"§5.1, Figure 1"},{"comment":"The cross-reference to 'Section 5.4' should point to the gas-turbine application (Section 5.7), since Section 5.4 is the LazyVI comparison.","section":"Appendix D.5"}],"recommendation":"major_revision","confidential_remarks":"I do not see a circularity problem: the VI result is derived from the early-stopping framework rather than assumed. The main risk is that the abstract and conclusion overstate the reliability guarantee relative to Assumption 4.1, the oracle stopping rule, and the unquantified width requirement. A major revision that adds a misspecification analysis, an explicit m-N trade-off, and a treatment of the practical stopping rule would make the paper's claims match its theorems."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper proposes a warm-start early-stopping scheme for variable importance where you start from the full model, zero out features, and run a few gradient steps. That is a genuinely practical idea, and the paper does real work trying to back it with NTK and GBDT kernel theory. I think the method is worth using, and the empirical results are decent. But the headline O(N^{-1/2}) guarantee rests on the target living in the dropped-feature RKHS (Assumption 4.1), which the paper itself calls a convenience, and the one simulation built to verify the rate constructs the data to satisfy that assumption. The proof of Theorem 2 also 'assumes' a population bound on the evolving-kernel term instead of proving it in the general case. Those are addressable, but they mean the theory as written does not cover the algorithm people would actually run.\n\nWhat is actually new: the GBDT kernel analysis for symmetric trees, extending Ustimenko et al. to the warm-start VI setting, is a real step beyond LazyVI, and the NTK linearization under warm-start is a useful side result. They also give a Wald-type CI using Williamson et al. for the neural network case. The experiments compare against dropout and retrain, show the early-stopping estimate tracks retrain, and the code and data are public. That is real evidence, and I believe the method will be useful in practice.\n\nSoft spots in proportion: the misspecification gap is the biggest one. Assumption 4.1 requires f_c^N and f0,-I to lie in the span of the dropped-feature kernel, which a real network or regression function will not satisfy exactly. The paper handwaves via universal approximation, but that does not put the target in the RKHS, it says you can approximate it, which would introduce an additional bias term not accounted for. The oracle stopping time uses unknown CH and sigma; the practical validation stopping is standard but not covered. The GBDT corollaries need random strength β ≥ N^{5/4} or N^{15/4}, while experiments use β=10,000 with N=5000, so the theory is in a different regime. These are not fatal to the method, but they do undercut the word 'reliable' in the title.\n\nWho this is for: anyone working on variable importance, Shapley approximations, or early-stopping theory for neural networks and GBDTs. It deserves a serious referee, not a desk reject. I would send it out, but with the specific request to either weaken the misspecification assumption or clearly scope the claims, and to address the oracle stopping rule gap.","headline":"A practical warm-start early-stopping scheme for variable importance with a serious theoretical attempt, but the headline guarantee only holds under an RKHS membership assumption that the verification simulation is built to satisfy.","tokens_in":38707,"tokens_out":2728,"would_cite":true,"duration_ms":24474,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that variable importance can be estimated from one full model fit plus short warm-started, early-stopped updates, without retraining per feature subset.","keywords":["variable importance","early stopping","warm-start","Shapley values","neural tangent kernel","gradient boosting decision trees","iterative kernel update","kernel methods"],"falsifier":"Construct a regression where the reduced target $f_{0,-I}$ is deliberately outside the span of the reduced-feature kernel, for example a function with an interaction between a dropped and a kept feature, and measure $\\|f_{\\widehat{T}} - f_{0,-I}\\|_N^2$ as $N$ grows; if the error fails to decay at the claimed $O(N^{-1/2})$ rate, the scope of Theorem 1 is falsified.","tokens_in":37651,"feed_emoji":"⚡","tokens_out":8954,"duration_ms":77231,"temperature":0.7,"pith_summary":"Variable importance asks how much a set of features contributes to prediction, but answering it usually means retraining the model once per feature subset, which is prohibitive when there are many variables. This paper claims that for any gradient-based learner whose updates can be written as an iterative kernel update equation, covering neural networks and gradient-boosted decision trees, one can train the full model once and then, for each subset, initialize from that full model with the features dropped and stop after a few gradient steps. The authors prove that the early-stopped model tracks the model that would have been trained on the reduced data, with squared error $O(N^{-1/2})$ in sample size, matching the precision of full retraining while using far fewer iterations. If the claim holds, variable importance and Shapley values become computationally feasible in high dimensions without sacrificing accuracy.","feed_headline":"One full fit plus early stops estimates feature importance","feed_subtitle":"Warm-started, early-stopped gradient runs match retrain accuracy for neural nets and boosted trees at a fraction of the cost.","key_machinery":"The load-bearing object is the iterative kernel update equation (14), $f_{\\tau+1}(X^{(I)}) = (I - \\epsilon K_\\tau^{(I)}) f_\\tau(X^{(I)}) + \\epsilon K_\\tau^{(I)} Y$, which rewrites both neural-network gradient descent (via the neural tangent kernel) and gradient-boosted trees (via a tree-induced kernel) as a linear recursion on the empirical kernel matrix. Lemma 1 decomposes the squared error into three terms: bias from not running long enough, variance from noise, and a difference term $D_\\tau^2$ caused by the kernel evolving during training. The stopping rule is formalized through the local empirical Rademacher complexity $\\widehat{R}_K(\\varrho)$, which sets the maximal horizon $\\widehat{T}_{\\max}$; the optimum $\\widehat{T}_{\\mathrm{op}}$ balances the decreasing bias and variance with the non-decreasing difference term. Warm-start enters by reparameterizing the recursion around the dropout error $e^{(I)} := f_{0,-I}(X^{(I)}) - f_N^c(X^{(I)})$, so the analysis reduces to early stopping from zero on the shifted target $f_{0,-I} - f_N^c$.","core_discovery":"On the paper's own terms, the central claim is Theorem 1: under Assumptions 4.1–4.5, a gradient descent or gradient boosting model started from the full-model fit $f_N^c$ and stopped at a data-dependent time $\\widehat{T}_{\\mathrm{op}}$ satisfies $\\|f_{\\widehat{T}_{\\mathrm{op}}} - f_{0,-I}\\|_N^2 \\le O(N^{-1/2})$ with high probability, with an analogous $L^2(P)$ bound in Theorem 2. The estimator built from this reduced-model fit, $\\widehat{VII}$, therefore achieves the same accuracy as retraining while starting from the dropout solution and moving only a few steps toward the dropped-feature target. For neural networks the same bound follows by bounding kernel drift and linearization error in terms of width, and for gradient-boosted trees by taking the random-strength parameter large enough; in the neural-network case the VI estimator is additionally asymptotically normal when the truth is nonzero, yielding Wald-type confidence intervals.","pith_inferences":["A natural testable extension is to compare the practical hold-out stopping rule used in the experiments with the theoretically derived $\\widehat{T}_{\\mathrm{op}}$ on the same problems; the theory motivates the rule, but the paper does not quantify the gap between them.","The same warm-start-plus-early-stopping mechanism should transfer to any iterative predictor with a tractable tangent kernel, such as convolutional or attention-based networks, since the proof only requires the update to be kernel-like and the kernel drift to be controllable; this transfer is an inference, not a claim in the paper.","Because dropout overestimates variable importance under feature correlation in the paper's simulations, replacing dropout with one or two warm-started gradient steps may offer a cheap correction in existing interpretation pipelines."],"forward_implications":["Variable importance for every feature subset can be obtained from one full training run plus a short warm-started run per subset, reducing the cost of model-agnostic interpretability from many retrainings to roughly one training run plus short updates.","Shapley-value estimation inherits the same speedup: the paper reports a 2–3 times faster runtime than retraining on its logistic-model simulation while matching retrain accuracy.","For sufficiently wide ReLU networks, $\\widehat{VII}$ is asymptotically normal when the true variable importance is nonzero, so Wald-type confidence intervals can be constructed without additional model fits.","For gradient-boosted trees with large random strength, the same guarantees hold, so tree ensembles can use the accelerated estimator rather than per-subset retraining."],"supporting_citations":[{"why":"Supplies the early-stopping theory for kernel methods: the error decomposition, the Rademacher-complexity stopping rule, and the proof pattern that Theorems 1 and 2 extend.","marker":"Raskutti et al. [2014]"},{"why":"Prior LazyVI method and simulation setup for variable importance with wide neural networks; the paper's notation and VI definition follow it, and it serves as the comparison baseline.","marker":"Gao et al. [2022]"},{"why":"Introduces the neural tangent kernel and its constancy in the infinite-width limit, which identifies the stationary kernel $K^{(I)}$ for neural networks.","marker":"Jacot et al. [2018]"},{"why":"Shows that wide networks evolve as linear models under gradient descent; the linearization theorem is extended to warm-start initialization to control neural-network error.","marker":"Lee et al. [2019]"},{"why":"Defines the kernel for gradient-boosted decision trees with symmetric trees and random strength, and gives the iterative update used in Lemma 3 and the GBDT corollaries.","marker":"Ustimenko et al. [2023]"},{"why":"The inference framework for nonparametric variable importance; its Theorem 1 supplies the asymptotic normality and Wald confidence intervals used in Corollary 5.","marker":"Williamson et al. [2023]"},{"why":"Provides eigenvalue decay rates for the NTK of two-layer ReLU networks on the sphere, used to turn the general bound into the population rate in Corollary 2.","marker":"Bietti et al. [2019]"},{"why":"Subset-sampling scheme for efficient Shapley-value estimation, used in the paper's Shapley experiments.","marker":"Williamson and Feng [2020]"}],"fun_headline_variants":["Warm-start early stopping speeds up variable importance","Skip full retraining: early stops match importance accuracy","Fast feature importance via warm-start and early stopping","One fit plus early stops: scalable importance for NN and trees"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof assumes that both the full model fit and the true reduced-model regression function lie exactly in the span of the kernel built from the remaining features; the paper states this is purely for theoretical convenience, but if the true function has components outside that span, the claimed $O(N^{-1/2})$ rates need not follow.","fun_headline_variants_meta":{"raw":{"variants":["Warm-start early stopping speeds up variable importance","Skip full retraining: early stops match importance accuracy","Fast feature importance via warm-start and early stopping","One fit plus early stops: scalable importance for NN and trees"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000229,"raw_usage":{"total_tokens":1494,"prompt_tokens":977,"completion_tokens":517,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":454}},"tokens_in":593,"tokens_out":517,"duration_ms":4730,"temperature":1.0,"reasoning_tokens":454,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:40:48.629086+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a regression where the reduced target $f_{0,-I}$ is deliberately outside the span of the reduced-feature kernel, for example a function with an interaction between a dropped and a kept feature, and measure $\\|f_{\\widehat{T}} - f_{0,-I}\\|_N^2$ as $N$ grows; if the error fails to decay at the claimed $O(N^{-1/2})$ rate, the scope of Theorem 1 is falsified.","supporting_citations":[{"cited_title":"Wainwright, and Bin Yu","cited_arxiv_id":null,"evidence_quote":"Supplies the early-stopping theory for kernel methods: the error decomposition, the Rademacher-complexity stopping rule, and the proof pattern that Theorems 1 and 2 extend."},{"cited_title":"Lazy Estimation of VI for Large NNs","cited_arxiv_id":null,"evidence_quote":"Prior LazyVI method and simulation setup for variable importance with wide neural networks; the paper's notation and VI definition follow it, and it serves as the comparison baseline."},{"cited_title":"Neural tangent kernel: Convergence and generalization in neural networks","cited_arxiv_id":null,"evidence_quote":"Introduces the neural tangent kernel and its constancy in the infinite-width limit, which identifies the stationary kernel $K^{(I)}$ for neural networks."},{"cited_title":"Gradient boosting performs gaussian process inference","cited_arxiv_id":null,"evidence_quote":"Defines the kernel for gradient-boosted decision trees with symmetric trees and random strength, and gives the iterative update used in Lemma 3 and the GBDT corollaries."},{"cited_title":"On the inductive bias of neural tangent kernels","cited_arxiv_id":null,"evidence_quote":"Provides eigenvalue decay rates for the NTK of two-layer ReLU networks on the sphere, used to turn the general bound into the population rate in Corollary 2."}],"review_version":1}