{"id":"98b51625-dd5e-443f-a23d-96e8d1694ac3","arxiv_id":"2507.23665","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"A SHAP entropy and stability regularization for LightGBM is proposed, with small aggregate accuracy gains but no algorithm details or error bars.","lead":"This paper adds two SHAP-based penalty terms to the training loss of a LightGBM model, one that pushes its feature attributions to be sparse and one that pushes them to be stable across samples. The authors report small accuracy gains and lower SHAP entropy on ten tabular datasets, but they do not specify how the penalties are optimized inside the gradient booster.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim is not falsifiable from the text because no optimization procedure for Eq. (1) is given; the reported results cannot be reproduced or checked.","rationale":"The reader's weakest_assumption identifies exactly the gap I find most load-bearing: Eq. (1) is never shown to be minimizable inside LightGBM's boosting loop. The paper's stated training procedure is only a high-level list, gives no gradients, surrogate loss, pseudo-code, or code release, and does not reconcile the piecewise-constant nature of TreeSHAP values with gradient-based boosting. Without such an optimization recipe, the main claim that this method 'improves generalization performance while ensuring robust and interpretable feature attributions' is not demonstrable from the text. The aggregated tables also lack error bars and per-dataset breakdowns, so the claimed improvements are not statistically supported. Additionally, the stability term in Eq. (3) computes average over all pairs, not 'similar samples' as claimed, which is a further textual inconsistency that undermines the stated mechanism. These are fundamental gaps, not mere refinements. I therefore agree with the reader's REJECT verdict, driven by the unverifiable optimization procedure.","tokens_in":6339,"tokens_out":1333,"duration_ms":11201,"concrete_test":"Ask the authors for the full training procedure: the exact objective used inside each LightGBM boosting step, how they computed gradients of the SHAP-based penalties (or a surrogate objective), and runnable code. If no code or precise mathematical update rule is provided, the paper is unverifiable. If code is provided, re-run the reported experiments (e.g., Airfoil and Mushroom) with the stated hyperparameters and check that (1) the loss is actually minimized each iteration and (2) the reported RMSE/F1, entropy, and stability numbers are reproduced within rounding.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's central claim is that minimizing L_total = L_task + lambda_1 * L_entropy + lambda_2 * L_stability inside LightGBM training improves generalization and interpretability. However, the training procedure in Section 4.1.1 is only a three-line list that says the total loss is computed after each boosting iteration and — the model parameters are then updated to minimize this total loss function — without specifying gradients, a surrogate objective, or any implementation detail. This is a load-bearing gap because LightGBM's boosting procedure optimizes a second-order Taylor approximation of L_task, not an arbitrary loss; SHAP values are not model parameters, and TreeSHAP values are piecewise-constant functions of the model output, so the entropy and stability penalties are not standard differentiable losses. The paper provides no pseudo-code, no gradient derivation, no perturbation scheme, and no released code. Also, Eq. (3) is over all N(N-1) sample pairs, not 'similar' samples, so minimizing it conflicts with the stated goal. Given these gaps, the reported improvements in Tables 2 and 3 cannot be attributed to the claimed method, nor can the method be reimplemented or independently verified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a SHAP-guided regularization framework for tree-based models, focusing on LightGBM. The total loss in Eq. (1) combines the task loss with an entropy penalty on normalized SHAP values (Eq. (2)) and a stability penalty that averages pairwise absolute SHAP differences (Eq. (3)). The authors report experiments on 10 regression and classification datasets, observing that SHAP-guided LightGBM performs comparably or slightly better than baselines while improving SHAP entropy and top-k concentration. They argue that the method improves generalization and interpretability.","tokens_in":6557,"tokens_out":7152,"duration_ms":63193,"significance":"If the claims were fully supported, the paper would address an interesting and timely problem: integrating explanation-based constraints into the training of gradient-boosted tree models. The two regularization terms are compactly defined, and the empirical study covers a reasonable number of datasets. However, the contribution is not yet established: the optimization procedure is unspecified, the interpretability evaluation is circular, and the generalization gains are not statistically validated. The work therefore does not currently meet the standard for publication.","major_comments":[{"comment":"The training procedure does not specify how the combined loss L_total in Eq. (1) is minimized inside LightGBM. LightGBM's boosting algorithm optimizes a second-order Taylor approximation of the task loss, and TreeSHAP values are piecewise-constant functions of the ensemble, not differentiable model parameters. The paper gives no gradient derivation, no surrogate objective, no perturbation scheme, and no pseudo-code for incorporating L_entropy and L_stability into the boosting loop. Without this, the results in Tables 2 and 3 cannot be reproduced or attributed to the claimed regularized loss, so the central claim is unsupported.","section":"Section 4.1.1"},{"comment":"The stability loss in Eq. (3) averages the pairwise absolute SHAP differences over all N(N-1)/2 sample pairs, yet the surrounding text consistently describes the goal as enforcing consistency across similar samples and between different but similar data points. Minimizing the all-pair average encourages globally constant attributions rather than locally stable ones. The manuscript does not reconcile this discrepancy, and the definition should either be restricted to a neighborhood of similar inputs or the textual claim should be changed accordingly.","section":"Section 3.2.2, Eq. (3)"},{"comment":"The interpretability metrics reported as benefits—SHAP Entropy, Stability, and Top-k Concentration—are not independent evidence: SHAP Entropy and Stability are directly minimized by L_entropy and L_stability, and Top-k Concentration is a deterministic consequence of entropy minimization. The only non-circular evidence is the predictive accuracy comparison, but the gains are small (e.g., RMSE 11.45 vs. 11.78; F1 0.9207 vs. 0.9141) and no per-dataset variance, confidence intervals, or statistical significance tests are reported. It therefore remains unclear whether the generalization improvements are reliable.","section":"Section 4.2, Tables 2 and 3"}],"minor_comments":[{"comment":"The normalized SHAP probabilities p_hat_ij are not defined formally; the paper should state p_hat_ij = |phi_ij| / sum_{j'} |phi_ij'| and explain how zero SHAP values are handled in the logarithm.","section":"Eq. (2)"},{"comment":"There is a notation mismatch: the text introduces phi_ik for feature k of sample i, but the equation uses phi_ij and phi_i'j. Please harmonize the notation and define all indices explicitly.","section":"Eq. (3)"},{"comment":"The experiments only use LightGBM, although the introduction mentions XGBoost and CatBoost as potential models. Please either include these models or remove the broader claim.","section":"Section 4.1.1 / Tables 2 and 3"},{"comment":"The figures are described qualitatively; Figure 2 in particular lacks axis labels and units. Quantitative per-feature variance values or a clear caption would make the stability claims more interpretable.","section":"Figures 1 and 2"},{"comment":"The related work section mentions X-SHIELD and SHAPNN but does not provide a concrete comparison; a short comparison of the objective formulations or experimental differences would strengthen the novelty argument.","section":"Section 2.2"},{"comment":"The paper contains typographical and formatting errors, such as 'compromising similar feature importance' in Section 4.2 and the malformed email address in the author block. Please proofread the manuscript.","section":"General"}],"recommendation":"reject","confidential_remarks":"This manuscript reads as an early workshop draft rather than a complete journal paper. The missing optimization procedure and the circular evaluation are fundamental issues; even if the authors supply a detailed algorithm, the experiments would need to be redone with proper ablations and statistical tests. I would not consider the paper suitable for publication in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper in one line: the idea is worth a look, but the write-up leaves out the load-bearing part. It applies entropy and stability penalties built from SHAP values to LightGBM training via TreeSHAP, which is a reasonable extension of existing SHAP-in-the-loop work (SHAPNN, X-SHIELD) to gradient-boosted trees. The framing is honest: the authors call it a first exploration, and the evaluation uses ten standard datasets against five baselines. That is the good part.\n\nThe soft spot is central. Section 4.1.1 says only that after each boosting iteration the total loss is computed and the model parameters are updated to minimize it. No gradients, no surrogate objective, no pseudo-code, no code release. SHAP values are piecewise-constant functions of the tree ensemble, not standard differentiable losses, and the stability term in Eq. (3) is O(N^2) and not obviously optimizable inside LightGBM's second-order boosting loop. Without an optimization recipe, the method cannot be reimplemented and the reported RMSE/F1 improvements cannot be attributed to the claimed mechanism. There is also a real inconsistency: the text says the stability penalty acts across similar samples, but Eq. (3) averages over all sample pairs. That matters because the stated goal is smoothness over similar inputs.\n\nThe other issues are real but secondary. The aggregated results have no per-dataset breakdown, no error bars, and no statistical tests. The interpretability metrics (SHAP entropy, top-k concentration, stability) are the same quantities the regularizer minimizes, so reporting them as benefits is partly circular. The accuracy gains are small (RMSE 11.45 vs 11.78; F1 0.9207 vs 0.9141), so the practical impact hinges on whether the optimization story can be made concrete.\n\nOverall, the paper is a speculative late-breaking report, not a full-width rigorous study. The underlying idea has merit, and a serious referee could push the authors to specify the optimization scheme, add code, and fix the similar-samples mismatch. So I would not desk reject it, but I would send it back with a request for major revision. Whoever reads it should treat the reported numbers as illustrative, not verified.\n\nFor peer review: yes, it deserves referee time, mostly because the direction is interesting and the literature engagement is honest. I would not cite it in my own work until the method is actually specified and validated.","headline":"Plausible idea, under-specified implementation: the paper never shows how the SHAP-based loss is optimized inside LightGBM, so the reported gains cannot be verified.","tokens_in":7083,"tokens_out":2210,"would_cite":false,"duration_ms":23014,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training a LightGBM model with SHAP entropy and stability penalties yields sparser and more stable feature attributions without losing predictive accuracy.","keywords":["SHAP","regularization","entropy penalty","stability penalty","gradient-boosted trees","LightGBM","interpretability","feature attribution"],"falsifier":"A decisive check is to run the same LightGBM pipeline with $\\lambda_1=\\lambda_2=0$ and the same hyperparameter search: if the entropy, top-k concentration, and stability gains disappear, the SHAP penalties are not driving the reported improvement.","tokens_in":6131,"feed_emoji":"🎯","tokens_out":10177,"duration_ms":94283,"temperature":0.7,"pith_summary":"This paper proposes adding two SHAP-based penalties to the training loss of a gradient-boosted tree model (LightGBM): an entropy penalty that pushes normalized absolute SHAP values toward a sparse, concentrated distribution, and a stability penalty that penalizes pairwise differences in SHAP attributions across samples. The claim is that minimizing the combined loss improves generalization while yielding sparser and more stable feature attributions, without sacrificing predictive accuracy. If true, it would let practitioners regularize models directly on their explanations, making tree ensembles more interpretable by construction rather than only through post hoc analysis. The paper reports experiments on ten benchmark regression and classification datasets, comparing a SHAP-guided LightGBM against standard tree baselines and finding lower SHAP entropy and higher top-k concentration with comparable or slightly better RMSE, $R^2$, F1, and AUC.","feed_headline":"SHAP penalties sharpen tree models without losing accuracy","feed_subtitle":"A SHAP-guided LightGBM focuses explanations on the top features while keeping accuracy on ten benchmarks.","key_machinery":"The machinery is the regularized loss in Eq. (1), with two added terms. The entropy penalty (Eq. 2), $L_{\\text{entropy}} = -\\frac{1}{N}\\sum_{i=1}^{N}\\sum_{j=1}^{M} \\hat{p}_{ij}\\log(\\hat{p}_{ij})$, uses the Shannon entropy of normalized absolute SHAP values to reward sparse, concentrated attributions. The stability penalty (Eq. 3), $L_{\\text{stability}} = \\frac{2}{N(N-1)M}\\sum_{i\\ne i'}\\sum_j |\\varphi_{ij} - \\varphi_{i'j}|$, averages pairwise absolute SHAP differences across all sample pairs to reward smooth explanations. TreeSHAP computes the attributions $\\varphi_{ij}$ inside LightGBM's boosting loop, and the hyperparameters $\\lambda_1,\\lambda_2$ control how strongly the two interpretability objectives push the training.","core_discovery":"The central claim is that the regularized objective $L_{\\text{total}} = L_{\\text{task}} + \\lambda_1 L_{\\text{entropy}} + \\lambda_2 L_{\\text{stability}}$, in which $L_{\\text{entropy}}$ is the mean Shannon entropy of normalized absolute SHAP values and $L_{\\text{stability}}$ is the mean pairwise absolute SHAP difference, can steer LightGBM during boosting toward accurate models with sparse, stable explanations. On the paper's aggregated experiments, the SHAP-guided LightGBM reaches an RMSE of 11.45 and $R^2$ of 0.83 on regression data, against 11.78 and 0.83 for standard LightGBM; on classification data it reaches F1 0.9207 and AUC 0.9641, against 0.9141 and 0.9592 for LightGBM. SHAP entropy falls from 1.17 to 1.12 (regression) and from 1.8261 to 1.6542 (classification), while top-k concentration rises to 0.89 and 0.8905. The paper reads these results as evidence that explainability-driven regularization reduces overfitting to spurious correlations and concentrates attribution on the most relevant predictors.","pith_inferences":["A natural test the paper does not run is whether the entropy and stability gains survive when the same hyperparameter search is applied to an unregularized LightGBM; the reported comparison does not isolate the penalties from tuning effects.","The all-pairs stability term is $O(N^2)$ in samples, so a mini-batch or nearest-neighbor version would be needed for large datasets and would also match the paper's wording that stability is wanted 'across similar samples.'","If the optimization is given an explicit surrogate, the same penalties could be ported to neural models with differentiable SHAP approximations, extending the framework beyond tree ensembles."],"forward_implications":["A LightGBM trained with the SHAP-guided loss should produce lower SHAP entropy and higher top-k concentration than the same model without the penalties, on both regression and classification benchmarks.","The regularized model should remain at least competitive with standard LightGBM, XGBoost, CatBoost, Random Forest, and Decision Tree on RMSE, $R^2$, F1, and AUC, so interpretability gains do not come at the price of accuracy.","Feature attributions of the regularized model should be more stable across similar samples, so nearby data points receive similar explanations.","Because the loss is task-agnostic and the SHAP values are computed by TreeSHAP, the same two penalties can be applied to other gradient-boosted tree models and to any model with computable SHAP values."],"supporting_citations":[{"why":"Defines SHAP values, the attribution quantities used to build both regularization penalties.","marker":"[13]"},{"why":"Supplies the LightGBM gradient-boosting framework in which the regularized loss is implemented and tested.","marker":"[17]"},{"why":"Prior work on stable explanations across similar samples that the stability penalty extends.","marker":"[12]"},{"why":"The paper's citation for SHAP as a widely used attribution method in its related work.","marker":"[7]"},{"why":"Prior SHAP-integrated learning architecture that the paper positions as related work to distinguish its contribution.","marker":"[14]"},{"why":"Prior explanation-based regularization method that motivates the paper's approach.","marker":"[15]"}],"fun_headline_variants":["SHAP-guided regularization lifts LightGBM accuracy and interpretability","Entropy-based SHAP penalties yield sparser, more stable models","Explainability-driven training: SHAP constraints improve model reliability","SHAP focus as a regularizer: better scores, clearer explanations","Tree models sharpened by SHAP entropy penalties"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach hinges on the assumption that the combined loss in Eq. (1), including the SHAP entropy and stability penalties, can actually be minimized inside LightGBM's boosting loop, but the paper gives no gradients, surrogate objective, or update rule for those penalties.","fun_headline_variants_meta":{"raw":{"variants":["SHAP-guided regularization lifts LightGBM accuracy and interpretability","Entropy-based SHAP penalties yield sparser, more stable models","Explainability-driven training: SHAP constraints improve model reliability","SHAP focus as a regularizer: better scores, clearer explanations","Tree models sharpened by SHAP entropy penalties"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1626,"prompt_tokens":944,"completion_tokens":682,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":598}},"tokens_in":560,"tokens_out":682,"duration_ms":7546,"temperature":1.0,"reasoning_tokens":598,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:28:25.636289+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive check is to run the same LightGBM pipeline with $\\lambda_1=\\lambda_2=0$ and the same hyperparameter search: if the entropy, top-k concentration, and stability gains disappear, the SHAP penalties are not driving the reported improvement.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the LightGBM gradient-boosting framework in which the regularized loss is implemented and tested."},{"cited_title":"Alvarez Melis, T","cited_arxiv_id":null,"evidence_quote":"Prior work on stable explanations across similar samples that the stability penalty extends."},{"cited_title":"Antwarg, R","cited_arxiv_id":null,"evidence_quote":"The paper's citation for SHAP as a widely used attribution method in its related work."},{"cited_title":"X-SHIELD: Regularization for eXplainable Artificial Intelligence","cited_arxiv_id":"2404.02611","evidence_quote":"Prior explanation-based regularization method that motivates the paper's approach."}],"review_version":1}