{"id":"8aae16b5-ed4c-4cea-b382-c6b436e5dbab","arxiv_id":"2506.08928","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Local MDI+ computes sample-specific feature importances for random forests and boosted trees by combining tree split structure with regularized linear models, and it outperforms LIME, TreeSHAP, and Local MDI at identifying predictive features.","lead":"A new method, Local MDI+, gives per-prediction feature importance scores for tree-based models by reading the scores off a linear re-expression of each tree. In tests on twelve benchmarks it identifies the features that matter for each prediction more accurately and more stably than LIME, TreeSHAP, and Local MDI.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Instance-specific claim rests on an unvalidated remove-and-retrain protocol; synthetic tests use globally fixed signal features, and the abstract's 10% figure is absent from the body.","rationale":"The reader's weakest_assumption correctly identifies the remove-and-retrain protocol as the soft spot. My read agrees and sharpens it: the synthetic experiments that do have ground truth do not test instance-specificity at all, because the signal-feature set is constant across samples. Thus the entire 'instance-specific' part of the abstract's claim rests on a protocol whose validity is assumed rather than demonstrated. This is load-bearing because the strongest claim is precisely about instance-specific identification, not global signal-feature ranking. The missing 10% figure compounds the problem: even if the protocol were valid, the headline number cannot currently be verified from the manuscript. I do not see an internal inconsistency in the method itself; the LMDI+ construction is clearly specified and the ablation study (Section 6) is a genuine strength. The condition for acceptance should be a validation of the evaluation protocol against synthetic instance-level ground truth, or direct instance-specific synthetic experiments. Since the reader already set the verdict to CONDITIONAL, my assessment does not move the verdict.","tokens_in":28102,"tokens_out":7138,"duration_ms":72190,"concrete_test":"Generate a synthetic DGP with known per-instance signal sets: for p=20 covariates, draw for each sample a random subset S_i of size 5, set E[Y|X] = sum_{j in S_i} beta_j X_ij (optionally with pairwise interactions within S_i), and add noise at PVE levels from Section 4.1. Train an RF, compute LMDI+, LIME, TreeSHAP, and Local MDI importances, and evaluate (a) AUROC for recovering S_i per sample and (b) the Section 4.3 remove-and-retrain protocol. If the protocol's method ranking does not match the AUROC ranking, the real-data comparisons cannot support instance-specific claims. In the same run, report the average percent R2/AUROC improvement at each retention level to substantiate or correct the abstract's 10% figure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that LMDI+ identifies instance-specific predictive features. The ground-truth experiments (Section 4.1, Figure 1; Section 4.2, Figure 2) use a fixed set of signal features (X_1,...,X_5 or X_1,...,X_6) for every sample, with the same response function applied to all rows; AUROC is computed by comparing per-sample importance scores against this same global label vector. These experiments therefore validate global signal-feature recovery, not instance-specificity. The only instance-specific evaluation is the remove-and-retrain protocol in Section 4.3, which masks each training row by replacing features outside that row's top-k with their column mean, retrains an RF, and evaluates on unmasked test data. No evidence is given that this protocol measures whether a method recovered the features that actually drive each individual prediction; it can reward methods that select globally predictive features because those features are retained in most rows and dominate the retrained model. The abstract's headline number ('average 10% improvement') does not appear in any body table or figure; Table 1 reports average ranks and Figures 3/12/13 report R2/AUROC curves, so the quantitative claim is not auditable. These gaps leave the instance-specific component of the central claim unsupported, even though the global signal-recovery evidence is solid.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Local MDI+ (LMDI+), a sample-specific extension of the global MDI+ feature importance framework. The method augments the stump basis of a decision tree with the raw feature values, fits a regularized GLM on this transformed representation (using out-of-bag samples), and defines the local importance of feature k for an instance x as the inner product of the transformed vector with the fitted GLM coefficients. The authors claim that LMDI+ identifies instance-specific predictive features better than LIME, TreeSHAP, and Local MDI, yields an average 10% improvement in predictive performance on twelve real-world datasets, produces more stable rankings across repeated random-forest fits, generalizes to gradient boosting, and enables counterfactual and subgroup-discovery use cases. The paper includes synthetic experiments with known signal features, correlation-robustness experiments, real-data remove-and-retrain evaluations, stability experiments, ablations, runtime comparisons, and two application case studies.","tokens_in":28450,"tokens_out":5351,"duration_ms":61527,"significance":"If the central empirical claim is established, LMDI+ would be a practically useful structure-based local importance method that avoids perturbation-based artifacts and offers improved stability over LIME and TreeSHAP. The paper's strengths include a clear algorithmic recipe, extensive experiments across multiple datasets and ensemble settings, an ablation study that isolates the contribution of each component, and a companion repository with implementation code. However, the significance is conditional: the synthetic experiments support global signal-feature recovery, while the instance-specific claim rests on an unvalidated remove-and-retrain protocol. The abstract's headline 10% figure is not substantiated by any reported number in the body, and the attribution formula itself is heuristic. These gaps are fixable but currently leave the central claim partially unsupported.","major_comments":[{"comment":"The synthetic experiments do not actually test instance-specificity. In Section 4.1, a fixed subset of signal features is randomly selected once per simulation, and the same response function is applied to every sample; the AUROC is then computed for each test sample against the same global binary signal-feature label vector. Section 4.2 likewise fixes signal features X1,...,X6 for all samples. These experiments therefore measure how well a method recovers globally signal features, not whether the importance scores vary correctly with the features that drive each individual prediction. To support the 'instance-specific' wording in the abstract and Section 1, the authors need either synthetic DGPs in which the signal features vary per sample, or another evaluation that isolates per-instance variation against a known oracle.","section":"Section 4.1/4.2"},{"comment":"The remove-and-retrain protocol is load-bearing for the claim that LMDI+ identifies instance-specific predictive features, but it is not validated for that purpose. Replacing features outside a sample's top-k with their column mean and retraining an RF can reward methods that select globally predictive features, since those features will be retained for most rows and will dominate the retrained model; the protocol does not establish that the retained set varies meaningfully across instances or that it recovers the features driving each individual prediction. A concrete test would be to construct synthetic data with per-instance signal features and verify that the protocol's performance tracks per-instance ground truth, and to compare LMDI+ against a global-only ranking baseline (e.g., global MDI+ or mean marginal importance) to show that the local component adds information beyond global feature ranks.","section":"Section 4.3, Setup"},{"comment":"The abstract claims an 'average 10% improvement in predictive performance when using only the selected features,' but no quantitative 10% figure appears anywhere in the body. Table 1 reports average ranks, Figures 3/12/13 report R2/AUROC curves, and Appendix D reports full-dataset numbers without a 10% average. The authors should either add the exact computation supporting the 10% claim (for example, average relative improvement across datasets and feature-retention levels) or qualify/remove the quantitative claim from the abstract.","section":"Abstract"},{"comment":"The LMDI+ attribution formula is introduced as an analogy to linear models, but no formal property is established for the resulting score. In a linear model, beta_k * x_k is the exact additive contribution to the prediction, but here the GLM is fitted on an augmented basis containing stump functions and the raw feature; the inner product of the transformed vector with the GLM coefficients is not shown to satisfy local accuracy, additivity, or any consistency property relative to the tree ensemble's prediction. This is load-bearing because the empirical claims are about the quality of these scores. The authors should provide a formal statement of what the score represents, or at minimum a sanity check on synthetic data with known per-instance local contributions.","section":"Section 3.3, Step 3"}],"minor_comments":[{"comment":"The notation in Step 3 is dense and ambiguous: it is not immediately clear whether a separate GLM is fit for each feature k on the augmented basis [Psi(S^(k)), x_k], or whether a single GLM is fit on the full concatenated basis and then restricted. Please clarify the dimension of beta^(k)_lambda and the fitting procedure.","section":"Section 3.3"},{"comment":"The 'unique feature ratio' measures overlap between top-k sets across five RF fits, but it does not directly measure rank-order stability. Reporting an additional rank-correlation metric (e.g., average pairwise Kendall tau across fits) would strengthen the stability claim.","section":"Section 5"},{"comment":"The counterfactual comparison reports mean L1 distances without standard errors or significance tests; the gaps between methods (e.g., 7.6 versus 9.5 on House 16H) may be within noise. Please add variability measures or statistical tests.","section":"Table 4"},{"comment":"The statement that LMDI+ performs '40% better than the Global model' is undefined; specify the relative MSE improvement and the baseline to which it is compared.","section":"Section 8"},{"comment":"The ablation tables report only average ranks; providing the underlying metric values (or a plot of the full curves) for at least a few datasets would make the ablation results more transparent.","section":"Section 6 / Appendix F"},{"comment":"There is a typo in the Future Work paragraph: 'also supports 0choices of generalized linear models' should read 'also supports choices of generalized linear models.'","section":"Section 10"}],"recommendation":"major_revision","confidential_remarks":"The paper is methodologically solid in its global signal-recovery experiments, and the code release is a plus. The main risk is that the real-data evaluation protocol in Section 4.3 has not been validated as a measure of instance-specificity, and the abstract's 10% figure is not backed by a body number; both are fixable with additional analysis. The self-citation to the authors' own MDI+ paper is appropriate and not circular. Scope fits TMLR-style empirical machine learning venues, provided the instance-specific claim is supported by a credible evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read on the LMDI+ paper. The headline is that this is a real extension of MDI+ to local attributions, with a broad and carefully run empirical sweep, but the instance-specific story is softer than the abstract claims. The 10% improvement figure is not in the body — Table 1 gives ranks, the figures give curves, but nowhere is that number derivable. That's a concrete reporting problem.\n\nWhat's genuinely new: they plug raw features into the stump basis and fit a per-feature regularized GLM; the inner product gives a local attribution. That's a clean, non-obvious extension of the MDI+ logic. The stability experiments are well designed — measuring unique top-k features across repeated model fits is a sensible metric. The ablation showing each component (OOB, raw features, GLM) adds value is good. The correlation robustness experiment is convincing. I also appreciate the runtime tables and the fact that code is referenced.\n\nWhere it gets soft. The synthetic experiments in 4.1 and 4.2 use the same signal features for every sample. So they validate global signal-feature recovery, not instance-specificity — the per-sample AUROC is over the same label vector. The only instance-specific evaluation is the remove-and-retrain protocol in 4.3. That protocol is known to be tricky: replacing non-top features with their mean and retraining can reward methods that simply keep globally predictive features. The paper doesn't validate that it measures what it claims. The stress-test note on this is on point.\n\nAlso, the attribution formula is heuristic. There's no axiomatic or formal grounding for why the GLM coefficient times transformed feature value is the right local attribution, beyond appealing to linear-model analogy. That's acceptable for a methods paper, but it means the theoretical contribution is modest.\n\nBottom line: as a method paper, it's useful. The global signal-recovery evidence is solid, the stability gains are real, and the runtime is acceptable. The instance-specific claims should be read with caution. The paper is already published in TMLR, but if I were refereeing the original submission, I would send it out — it deserves serious review. The missing 10% and the unvalidated protocol are fixable in revision, and the core method is worth engaging with.\n\nFor your reading group, I'd say maybe. It's a good case study in how hard it is to validate local feature importance, and the gap between global and local claims is instructive.\n\nYours, [name]","headline":"Solid global feature-importance extension with a shaky instance-specific validation and an unbacked headline number.","tokens_in":28916,"tokens_out":2475,"would_cite":true,"duration_ms":34005,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Local MDI+ extends the MDI+ global-importance framework to individual predictions, claiming that its per-sample attributions find predictive features more faithfully and more stably than LIME, TreeSHAP, and Local MDI across twelve…","keywords":["local feature importance","tree-based models","random forests","MDI+","model interpretability","counterfactual explanations","stability","gradient boosting"],"falsifier":"On a synthetic dataset where each instance has a known, instance-specific set of signal features, compute LMDI+ and baseline rankings and compare them against the per-instance ground truth; if the remove-and-retrain ordering ever ranks a method that selects globally predictive features above one that selects the true local signal, the protocol is measuring global fidelity rather than local fidelity.","tokens_in":27959,"feed_emoji":"🎯","tokens_out":7158,"duration_ms":76593,"temperature":0.7,"pith_summary":"Local MDI+ (LMDI+) is a method for saying which features drove a single prediction made by a random forest or gradient-boosted model. It extends the global MDI+ framework, which repairs the biases of mean decrease in impurity by reinterpreting decision trees as linear regressions on a transformed node basis, into a per-sample attribution: each feature's importance for an observation is the contribution of that feature in the linearized tree representation, averaged across trees. The paper reports that LMDI+ identifies instance-specific predictive features better than LIME, TreeSHAP, and Local MDI, with an average 10% improvement in predictive performance when only top-ranked features are retained, and that its rankings are more stable across repeated model fits. It also shows the method supports counterfactual explanations with smaller feature changes and discovers homogeneous subgroups in a housing case study. If these claims hold, practitioners get local explanations for tree ensembles that are more faithful and more reproducible without perturbation-based approximations.","feed_headline":"Local MDI+ beats LIME and TreeSHAP at picking per-prediction features","feed_subtitle":"Tree-model explanations that stay stable across random seeds and find the features that actually drive each prediction.","key_machinery":"The stump function maps each split into a three-valued basis element, and concatenating these over all splits gives the node basis. A known result, stated as Proposition 1 from MDI+, connects the mean decrease in impurity of feature k to the $R^{2}$ from regressing the response on the node-basis columns corresponding to splits on that feature, which explains MDI's biases. LMDI+ uses the same basis to turn the tree into a linear model: it appends raw features to the node basis, fits a regularized GLM on in-bag and out-of-bag samples, and reads off the per-observation attribution as the inner product of the transformed features with the fitted coefficients. The GLM shrinkage and out-of-bag evaluation are what correct the biases; the inner-product readout is what makes the importance local and stable.","core_discovery":"The paper proposes to compute sample-specific feature importances by exploiting the known equivalence between a fitted decision tree and ordinary least squares on the basis of stump functions defined by its splits. For each tree in the ensemble, LMDI+ builds a transformed feature map that concatenates these stump functions with the raw covariates, evaluates it on both in-bag and out-of-bag samples, fits a regularized generalized linear model to the response on that map, and defines the local importance of feature k for observation x as the inner product of the transformed feature vector with the corresponding GLM coefficients; features that never split are assigned zero. Scores are averaged over trees and ranked by absolute value. The paper claims this inherits MDI+'s corrections for overfitting, correlation bias, and poor handling of smooth or additive structure while adding sample specificity, and presents experiments—synthetic DGPs, correlation robustness, twelve real-world datasets, stability, ablations, gradient boosting, counterfactuals, and a housing subgroup case study—that it says support the method's predictive-feature identification, stability, and practical utility.","pith_inferences":["The linearized view suggests LMDI+ attributions are additive and therefore could be aggregated across samples or groups into global or subgroup-level importance statements, something the paper only hints at in its clustering use case.","The remove-and-retrain protocol may reward methods that find globally predictive features rather than features that drive the particular instance; a synthetic study with known per-instance ground truth would separate these two notions.","Because the GLM and link function are plug-in choices, the same construction could be tested with other regularized estimators for non-Gaussian responses or with interaction features appended to the node basis.","The stability gain may come partly from GLM shrinkage and averaging over trees; an ablation that varies only shrinkage while holding the basis fixed would isolate how much stability is due to regularization."],"forward_implications":["If LMDI+ identifies predictive features as claimed, local explanations for tree ensembles can be obtained without the perturbations used by LIME or the subset averaging used by TreeSHAP, reducing a known source of instability.","Using only the features LMDI+ ranks highest for each sample preserves more predictive performance after retraining than the same protocol with baseline methods, which matters when a practitioner must act on a short list of reasons.","Stable per-sample rankings across random seeds mean repeated fits of the same model are less likely to contradict each other in high-stakes settings.","The framework transfers to gradient-boosting ensembles, so the method is not tied to random forests.","Counterfactuals selected in LMDI+ importance space require smaller feature changes, and clustering on LMDI+ scores yields subgroups on which simple models fit better."],"supporting_citations":[{"why":"Supplies the MDI+ framework that LMDI+ extends, including the OOB/raw-feature/GLM recipe and Proposition 1 connecting MDI to R^2.","marker":"Agarwal et al. (2023)"},{"why":"Establishes that a fitted decision tree's predictions equal those of OLS on the stump-function basis, the equivalence LMDI+ relies on for linear attributions.","marker":"Klusowski and Tian (2023)"},{"why":"Defines Local MDI, the local-importance baseline derived from MDI that LMDI+ aims to improve, and supplies the decision-path attribution idea.","marker":"Sutera et al. (2021)"},{"why":"LIME is a primary perturbation-based baseline LMDI+ is compared against.","marker":"Ribeiro et al. (2016)"},{"why":"TreeSHAP is the primary Shapley-value baseline LMDI+ is compared against.","marker":"Lundberg et al. (2020)"},{"why":"Documents the entropy and correlation biases of MDI that motivate the OOB and regularized-GLM corrections inherited by LMDI+.","marker":"Strobl et al. (2007)"},{"why":"Supplies the remove-and-retrain evaluation protocol used for real-data experiments in Section 4.3.","marker":"Hooker et al. (2019)"}],"fun_headline_variants":["Local MDI+ outperforms LIME and TreeSHAP on instance-specific features","LMDI+ gives per-sample feature importance that beats LIME and TreeSHAP","Tree-model explainer LMDI+ beats LIME and TreeSHAP at local features","Per-prediction feature importance from Local MDI+ tops LIME and TreeSHAP"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The real-data comparisons assume that the remove-and-retrain protocol—replacing all but a sample's top-ranked features with their average and retraining a random forest—actually measures how well a method identifies the features driving that sample's prediction, an assumption the paper does not independently validate.","fun_headline_variants_meta":{"raw":{"variants":["Local MDI+ outperforms LIME and TreeSHAP on instance-specific features","LMDI+ gives per-sample feature importance that beats LIME and TreeSHAP","Tree-model explainer LMDI+ beats LIME and TreeSHAP at local features","Per-prediction feature importance from Local MDI+ tops LIME and TreeSHAP"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000943,"raw_usage":{"total_tokens":4084,"prompt_tokens":1058,"completion_tokens":3026,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":674,"completion_tokens_details":{"reasoning_tokens":2934}},"tokens_in":674,"tokens_out":3026,"duration_ms":22962,"temperature":1.0,"reasoning_tokens":2934,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:58:55.933669+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a synthetic dataset where each instance has a known, instance-specific set of signal features, compute LMDI+ and baseline rankings and compare them against the per-instance ground truth; if the remove-and-retrain ordering ever ranks a method that selects globally predictive features above one that selects the true local signal, the protocol is measuring global fidelity rather than local fidelity.","supporting_citations":[],"review_version":1}