{"id":"fe68aaa6-4010-4b99-8e2e-3b93dc8043b9","arxiv_id":"2509.01409","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Cost-sensitive credit models improve cost efficiency but produce less stable SHAP and LIME explanations, especially when the training data are imbalanced.","lead":"This paper compares standard credit-scoring models with cost-sensitive versions that tune each loan's default cost, and finds the cost-sensitive models save money but give less stable explanations. The trade-off matters for banks that must explain credit decisions to regulators and customers.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CoV stability metric on signed SHAP/LIME values with near-zero means may drive the reported IDCS instability gap; SRA support is partial.","rationale":"The reader's weakest_assumption correctly identifies the loaded CoV metric. This is the most load-bearing concern because the central new claim is that IDCS models produce 'significantly less stable explanations,' and the CoV analysis is the first and most visually emphasized evidence. If the metric is biased by signed values with near-zero means, the value-stability gap could be an artifact. The SRA results weaken the concern but do not eliminate it: SRA is rank-based and therefore not subject to the same problem, yet the paper itself notes exceptions (e.g., LIME SRA in SGCS), and the interaction with class imbalance is asserted rather than statistically demonstrated. Other potential issues, such as the ambiguous resampling description or lack of code, are secondary because they do not directly threaten the central comparison as much as the validity of the primary stability statistic. A concrete reanalysis with absolute values or a robust metric would settle whether the concern lands; until then, the conditional verdict is appropriate. I do not see grounds to move the verdict to accept or reject on the basis of this review.","tokens_in":25407,"tokens_out":7587,"duration_ms":91980,"concrete_test":"Recompute the CoV stability analysis of Section 4.2.1 using absolute SHAP/LIME importances (or a robust dispersion measure such as median absolute deviation divided by median absolute importance), and separately after excluding features whose mean absolute importance falls below a small threshold (e.g., 1e-6). If the IDCS-vs-traditional gap and the KS-test significance persist at all default rates, the concern is resolved. If the gap shrinks or disappears, the headline trade-off must be reworded to rely on SRA alone, and the 'particularly as class imbalance increases' claim should be tested with an explicit model-type-by-imbalance interaction test.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central trade-off claim rests on the stability comparison, and the primary value-based metric, CoV (Eq. 6), is defined as the average of sigma_ip/mu_ip over features. The paper does not state whether raw or absolute SHAP/LIME importances were used. SHAP values and LIME coefficients are signed and can be close to zero; if raw values are used, mu_ip near zero makes the ratio arbitrarily large, so a model whose explanations have small, sign-changing attributions will appear highly unstable even when its absolute importance is consistent. This is not a benign scaling issue: if IDCS models tend to produce more near-zero or sign-flipping attributions than traditional models, the gap in CoV is an artifact of the metric, not evidence of instability. The rank-based SRA metric is not affected by this problem and provides some independent support, but it has exceptions (e.g., LIME SRA is not significantly higher for SGCS, and SHAP SRA is non-monotonic for VUB/SGCS), and the claim that instability is 'particularly' stronger under class imbalance is not backed by an interaction test. Thus the headline conclusion is conditional on the CoV definition and on whether the value-stability gap survives a signed-mean-robust analysis.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates whether instance-dependent cost-sensitive (IDCS) classifiers in credit scoring yield less stable post-hoc explanations than standard classifiers. The authors benchmark eight models (XGBoost, logistic regression, random forest, neural network, and their IDCS variants) on four public credit-scoring datasets, measuring performance with AUC, AP, Brier, savings, and a newly proposed relAEC metric. Explanation stability is assessed for SHAP and LIME via the coefficient of variation (CoV) and sequential rank agreement (SRA) under controlled resampling to training default rates between 1% and 30%. The central claim is that IDCS models improve cost-efficiency but produce significantly less stable explanations, and that the stability gap grows as class imbalance increases.","tokens_in":25706,"tokens_out":5536,"duration_ms":60259,"significance":"If the result holds, it identifies a concrete, regulation-relevant trade-off between profit-driven loss functions and explanation stability, which is especially important in credit scoring under GDPR and the EU AI Act. The study is carefully designed on the whole: nested cross-validation for performance, a fixed test set, 25 resampling iterations per imbalance level, two complementary stability metrics, and four model families. The SRA results provide independent support for the main stability claim in most settings, and the relAEC metric is a useful normalization. The main caveat is that the value-based CoV metric, as defined in Eq. (6), is sensitive to near-zero signed means; the manuscript does not address this issue, and the interaction claim that instability increases 'particularly' for IDCS under class imbalance is not backed by a statistical test.","major_comments":[{"comment":"The manuscript does not state whether the CoV is computed on raw SHAP values/LIME coefficients or on their absolute values. Section 3.5 says that feature importances are derived from SHAP values and LIME coefficients, with 'higher absolute values indicating greater influence,' but Eq. (6) defines the CoV as sigma_ip/mu_ip. SHAP values and LIME coefficients are signed; if raw values are used, mu_ip near zero can inflate CoV arbitrarily, and a model with small, sign-changing attributions would appear unstable even when its absolute importance is consistent. Since the CoV gap between IDCS and traditional models is a headline result, please state explicitly which values were used. If raw values were used, repeat the analysis with absolute importances and report whether the gap persists; also state how zero or near-zero means were handled.","section":"Section 3.6, Eq. (6), and Section 3.5"},{"comment":"The abstract and conclusions state that IDCS explanations become less stable 'particularly as class imbalance increases,' but the paper reports no statistical test for an interaction between model type and imbalance level. The figures show higher CoV/SRA for IDCS at most levels, but the stronger-imbalance-effect claim requires e.g. a regression of stability on imbalance and model type with an interaction term, or a permutation test. This is especially important because SHAP SRA is explicitly non-monotonic for VUB and SGCS, and LIME SRA is not significantly higher for SGCS. The abstract should qualify the claim by these exceptions or report an appropriate interaction test.","section":"Section 4.2 (Figs. 4–7) and Section 6"},{"comment":"The text states that a two-sample Kolmogorov-Smirnov test 'confirms that the CoV distribution of IDCS classifiers is significantly higher than that of traditional classifiers.' The standard two-sample KS test is two-sided and tests for any distributional difference, not specifically for stochastic dominance. Please report a one-sided test, or an effect size and direction, to support the directional claim of 'significantly higher.'","section":"Section 4.2.1"}],"minor_comments":[{"comment":"Typo: 'analaysis' should be 'analysis'.","section":"Section 2"},{"comment":"Typo: 'defualter' should be 'defaulter'.","section":"Figure 8 caption"},{"comment":"Typo: 'optimizning' should be 'optimizing'.","section":"Section 4.1"},{"comment":"In the provided version, axis labels and panel labels appear as garbled character sequences (e.g., '/uni00000014...'), making the figures hard to read. Please ensure the final PDF rendering is clean.","section":"Figures 4–7"},{"comment":"The line 'Set tuning_round To T rue' should read 'Set tuning_round To True'.","section":"Algorithm 2"},{"comment":"Please state explicitly how ties in feature importance rankings are handled when computing SRA, and whether the 300 test observations are held fixed across all resampling iterations (as opposed to resampled per iteration).","section":"Section 3.8"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be the accepted version of an EJOR article; the provided arXiv text has corrupted figure labels, which should be fixed in the final version. The main revision should focus on the CoV signed-mean issue and the missing interaction test, since both directly affect the headline stability claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, honest empirical paper that documents a real operational trade-off. It is the first to measure SHAP/LIME stability for IDCS classifiers, and it does so with a careful protocol: nested CV for performance, controlled resampling for imbalance, two complementary stability metrics. The relAEC metric is a small but genuine contribution for cross-dataset cost comparison.\n\nThe performance results are banal but clean: IDCS models gain on cost metrics, lose on discrimination and calibration. The stability finding—IDCS explanations are less stable, especially under imbalance—is plausible and supported by the rank-based SRA as well as CoV. The KS tests they refer to are only in the supplementary materials, so I couldn't verify those p-values, but the boxplots tell a fairly consistent story.\n\nThe real soft spot is the CoV definition. Equation 6 averages sigma_ip / mu_ip over features. SHAP and LIME coefficients are signed and can be close to zero. A feature whose attribution flips sign or sits near zero will produce a huge ratio regardless of whether the absolute magnitude is stable. The paper never states whether raw or absolute importances were used. If raw values were used, part of the IDCS-vs-standard gap could be metric artifact, not genuine instability. The SRA results provide independent support in most settings, but they have exceptions (LIME SRA isn't significantly higher for SGCS; SHAP SRA is non-monotonic for VUB/SGCS). So the broad claim that IDCS explanations are 'significantly less stable' is not quite as uniform as the abstract implies.\n\nAlso, no code is provided, and the hyperparameter tuning for all 25 resampling iterations uses the same theta* from the first iteration—reasonable for computational reasons, but worth stating as a limitation.\n\nWho is this for? Anyone working in credit scoring or profit-driven XAI. It's a useful benchmark paper, not a theoretical breakthrough. It deserves a serious referee, and likely publication after the CoV issue is fixed or explicitly discussed. I'd recommend asking for an appendix showing CoV computed on absolute importances and a robustness check with a signed-mean-robust metric.","headline":"A useful empirical study of explanation stability for cost-sensitive credit models; the main trade-off claim is credible but rests on a stability metric with a known signed-value weakness.","tokens_in":26173,"tokens_out":2204,"would_cite":true,"duration_ms":23956,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Cost-sensitive credit models win on savings but lose on explanation stability.","keywords":["cost-sensitive learning","credit scoring","explanation stability","SHAP","LIME","class imbalance","Average Expected Cost","instance-dependent costs"],"falsifier":"Recompute the stability comparison using absolute importance values or a signed-mean-adjusted dispersion measure, such as median absolute deviation or variance of ranks without the CoV ratio, on the same 25-iteration resamples; if IDCS and traditional models show similar dispersion under that metric, the claimed stability gap is an artifact of the CoV formula, while if the gap persists the claim survives.","tokens_in":25329,"feed_emoji":"⚖️","tokens_out":4763,"duration_ms":52561,"temperature":0.7,"pith_summary":"This paper tries to establish that instance-dependent cost-sensitive (IDCS) classifiers—models that tailor misclassification penalties to each loan's individual cost—improve cost-efficiency but produce systematically less stable explanations than traditional cost-insensitive models. Using SHAP and LIME, the two most widely used explanation tools, the authors compare eight classifiers across four public credit datasets and measure how feature importances vary across repeated training runs. They find that IDCS models yield higher dispersion in feature-importance values and rankings, and that this instability grows sharply as class imbalance increases. If true, this reveals a concrete trade-off between profit-driven model design and regulatory demands for transparent, trustworthy explanations.","feed_headline":"Cost-optimized credit models yield shakier explanations","feed_subtitle":"SHAP and LIME feature rankings vary more for instance-dependent cost-sensitive classifiers, especially on imbalanced loans.","key_machinery":"The load-bearing object is the instance-dependent cost matrix—a false negative costs the loan amount times a loss-given-default parameter, while a false positive costs lost revenue plus the expected cost of lending to an alternative borrower—folded into an Average Expected Cost (AEC) loss that IDCS classifiers directly minimize. The stability comparison is carried by two metrics: the Coefficient of Variation (CoV), the average across features of the standard deviation divided by the mean of importance values over repetitions, and Sequential Rank Agreement (SRA), the pooled variance of feature ranks across repeated importance lists. The newly introduced relAEC metric normalizes AEC against a","core_discovery":"The paper's central claim is that IDCS classifiers achieve better cost-sensitive performance but create significantly less stable local explanations. Across four datasets and four model families (boosted trees, logistic regression, random forest, and neural networks) paired with their IDCS variants, SHAP and LIME feature importances from IDCS models show higher Coefficient of Variation (CoV) values and higher Sequential Rank Agreement (SRA) values than their cost-insensitive twins, indicating more variability across 25 resampling iterations. This instability becomes more pronounced as the training default rate is resampled toward extreme imbalance (e.g., 1% and 3%). The paper also introduces","pith_inferences":["If the CoV metric is sensitive to near-zero and sign-changing SHAP values, part of the observed stability gap may reflect IDCS models concentrating importance in small signed values; a follow-up using absolute values or rank-based dispersion would separate genuine instability from a ratio artifact.","The paper's proposed mechanism—costs entangle features, labels, and loan amounts—suggests that datasets with narrower false-negative-to-false-positive cost ratio distributions might show smaller stability gaps; this is testable with synthetic data.","The relAEC normalization could generalize beyond credit scoring to any instance-dependent cost setting, though its usefulness depends on the prior default rate being a sensible 'no model' baseline.","Class-imbalance-driven instability in IDCS explanations suggests that cost-aware stabilization techniques, such as smoothing or constrained optimization on feature attributions, could be developed to preserve cost efficiency without sacrificing explainability."],"forward_implications":["Banks adopting IDCS credit models should expect explanation instability as a systematic side effect, not a tuning artifact.","Under imbalanced loan portfolios, explanations for why a loan was denied may vary substantially between otherwise similar applications, complicating customer-facing justification.","Model selection for credit scoring should weigh cost-efficiency against explanation stability; the financially best model may not be the most explainable one.","Evaluation protocols for explanation methods in credit should include class-imbalance sweeps, because stability degrades nonlinearly at extreme default rates.","Repeated-retraining stability checks, like the 25-iteration resampling used in this paper, belong in model risk management for regulated credit models."],"supporting_citations":[{"why":"Supplies the instance-dependent cost matrix, the Average Expected Cost (AEC) formulation, and the foundational IDCS logistic regression model.","marker":"Bahnsen et al. (2014)"},{"why":"Provides the IDCS decision-tree ensemble approach used for the csforest model.","marker":"Bahnsen et al. (2015a)"},{"why":"Provides the IDCS XGBoost and logistic regression implementations used for csboost and cslogit.","marker":"Höppner et al. (2022)"},{"why":"Supplies the IDCS neural network implementation and prior empirical evidence that IDCS classifiers improve cost savings.","marker":"Vanderschueren et al. (2022a)"},{"why":"Establishes the resampling protocol and the baseline finding that class imbalance destabilizes SHAP and LIME explanations in cost-insensitive models.","marker":"Chen et al. (2024)"},{"why":"Defines SHAP, the Shapley-value-based explanation technique whose stability is measured.","marker":"Lundberg and Lee (2017)"},{"why":"Defines LIME, the local surrogate explanation technique whose stability is measured.","marker":"Ribeiro et al. (2016)"},{"why":"Defines Sequential Rank Agreement, one of the two stability metrics used to compare feature importance rankings.","marker":"Ekstrøm et al. (2018)"}],"fun_headline_variants":["Cost-aware credit models trade stability for savings","IDCS credit models: better costs, shakier explanations","Cost-efficient credit scoring undermines LIME and SHAP stability","Imbalanced loans worsen explanation instability in cost-sensitive models"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The headline comparison treats the coefficient of variation (standard deviation divided by mean) of feature importances as a valid stability measure, even though SHAP values and LIME coefficients can be negative or near zero, where a large ratio can reflect a small denominator rather than genuine instability.","fun_headline_variants_meta":{"raw":{"variants":["Cost-aware credit models trade stability for savings","IDCS credit models: better costs, shakier explanations","Cost-efficient credit scoring undermines LIME and SHAP stability","Imbalanced loans worsen explanation instability in cost-sensitive models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000704,"raw_usage":{"total_tokens":3006,"prompt_tokens":735,"completion_tokens":2271,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":2206}},"tokens_in":479,"tokens_out":2271,"duration_ms":16211,"temperature":1.0,"reasoning_tokens":2206,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T12:34:11.058882+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the stability comparison using absolute importance values or a signed-mean-adjusted dispersion measure, such as median absolute deviation or variance of ranks without the CoV ratio, on the same 25-iteration resamples; if IDCS and traditional models show similar dispersion under that metric, the claimed stability gap is an artifact of the CoV formula, while if the gap persists the claim survives.","supporting_citations":[],"review_version":1}