{"id":"4dc474b3-c1f6-418a-898b-1ad43f3d70e6","arxiv_id":"2504.19027","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"DiCE-Extended modifies the DiCE counterfactual explainer by adding a Dice-Sørensen-based robustness loss, claiming improved stability and proximity across four tabular datasets.","lead":"This paper adds a robustness penalty based on the Dice-Sørensen coefficient to the DiCE counterfactual explanation method, aiming to make explanations stable under small input changes. The authors report improved stability and proximity on four benchmark datasets, but the main robustness gain is measured with the same metric the method is trained to optimize.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The robustness loss in §2.3 is computed on binarized vectors, making it piecewise constant; with no surrogate or expectation smoothing described, the claimed gradient-based minimization is not well-defined, so the central mechanism is unsupported.","rationale":"The reader identifies the non-differentiability of the binarized Dice loss as the load-bearing assumption. I agree. Reading §2.3 carefully, the robustness loss is defined on binarized counterfactuals and inserted into a loss minimized with gradient-based optimizers (Adam, backpropagation). Binarization is a hard threshold, so the resulting loss has zero gradient almost everywhere. No relaxation or surrogate is mentioned anywhere in the paper, including in the experimental setup. The only way the reported learning curves could show a monotone decrease is if a surrogate or smoothing was used silently, or if the robustness term was not actually optimized and the curves are post-hoc evaluations. Either way, the central mechanism as described is not well-defined. The secondary circularity compounds this: the evaluation metric appears identical to the training objective, and λr was selected by grid search on that metric. This makes the +0.05 to +0.13 improvement an artifact of fitting the evaluator rather than evidence of genuinely more robust explanations. I also note an internal inconsistency in the sign of the loss term, though I do not rest the verdict on it. The conclusion's statement that proximity gains indicate adherence to decision boundaries is a logical non sequitur, but this is secondary. On balance, the paper's central claim is not supported as written. A revision could potentially remedy the issue by defining a differentiable surrogate, releasing code, and evaluating with an independently defined robustness metric; absent that, REJECT is appropriate. Thus I keep the reader's verdict unchanged.","tokens_in":9221,"tokens_out":6250,"duration_ms":65963,"concrete_test":"Run the Adult/PyTorch configuration with λr = 0 (robustness term removed) and identical random seeds and hyperparameters; if the reported robustness metric still improves by 0.05 or more, the robustness loss is not responsible for the gains. If it does not improve, then the binarized loss described in §2.3 must have supplied a nonzero gradient, which is impossible without an undocumented surrogate; inspect the implementation to identify how the gradient was computed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.3 defines the only novel component as LRobustness = λr · robustness_loss, where the robustness_loss is the Dice-Sørensen coefficient computed on binarized versions of c and c′ = c + δ. Binarization is a step function, so this loss is piecewise constant: its gradient with respect to c is zero almost everywhere (and undefined at thresholds). The paper explicitly states that optimization uses gradient-based updates (Adam, backpropagation, §2.5) but provides no straight-through estimator, Gumbel-softmax, continuous relaxation, or expectation over a smooth perturbation distribution. Without one, the term cannot be minimized by the stated optimizer, and the decreasing robustness-loss curves in Figure 1 are unexplained by the written method. Since this term is the paper's core contribution, the reported robustness gains (+0.05 to +0.13) cannot be attributed to the proposed mechanism. Compounding this, the reported robustness metric appears to be the same binarized Dice loss used as the training objective and tuned via grid search on λr; thus the headline improvement is partly circular, measuring the objective being optimized rather than an independent property. The sign convention also conflicts: the total loss subtracts λrLRobustness while calling robustness_loss a 'distance', so even the intended direction of optimization is ambiguous.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DiCE-Extended, an extension of DiCE that adds a robustness regularization term to the counterfactual-generation objective. The term is computed as a Dice–Sørensen distance between binarized original and perturbed counterfactual vectors, and the total loss combines proximity, diversity, and robustness weighted by λp, λd, and λr. The authors evaluate the method on COMPAS, Lending Club, German Credit, and Adult Income with Random Forest, PyTorch, and TensorFlow backends, and report that DiCE-Extended improves robustness, proximity, sparsity, diversity, and local fidelity while maintaining near-perfect validity.","tokens_in":9463,"tokens_out":7775,"duration_ms":69009,"significance":"The proposed goal—making counterfactuals robust to small perturbations—addresses a real limitation of DiCE, and the experimental design has strengths: four datasets, three backends, paired per-instance t-tests with reported p-values, and 1-NN fidelity checks. The paper also clearly separates quality dimensions in the loss. However, the central contribution is not currently supported. The robustness loss is piecewise constant and non-differentiable as written, the optimization direction is ambiguous because of the sign convention, the evaluation metric appears to coincide with the training loss (with λr tuned on the evaluation datasets), and the main quantitative improvements are not reported in any table. These issues prevent the claims from being attributed to the proposed mechanism.","major_comments":[{"comment":"The novel robustness loss is computed on binarized versions of c and c′ = c + δ. Binarization is a step function, making the loss piecewise constant with zero gradient almost everywhere, yet §2.5 states that optimization is gradient-based (backpropagation, Adam). No straight-through estimator, continuous relaxation, or other surrogate is described. Therefore the minimization of LRobustness is not well-defined as written, and the decreasing robustness-loss curves in Figure 1 and the robustness gains in §3.3 are unexplained by the stated method.","section":"§2.3, §2.5, Fig. 1"},{"comment":"The total loss is written as L = Ly_loss + λpLProximity − λdLDiversity − λrLRobustness, with LRobustness = λr · robustness_loss, and robustness_loss is called the Dice–Sørensen distance between the original and perturbed CFs. If it is a distance, subtracting it encourages the optimizer to increase it, i.e., to make CFs less robust; if it is instead a similarity, the term is misnamed. The intended optimization direction is ambiguous, and this ambiguity affects all reported results.","section":"§2.3"},{"comment":"The robustness metric used for evaluation appears to be the same binarized Dice–Sørensen quantity that is minimized as the robustness loss, and λr is chosen by grid search on the evaluation datasets themselves. The reported +0.05 to +0.13 robustness improvements are therefore largely expected from optimizing the evaluation criterion, and the experiments do not demonstrate robustness gains under an independent measure.","section":"§2.3 and §3.3"},{"comment":"The main empirical claims in §3.3—proximity reductions of 30–95%, sparsity improvements up to 50%, near-doubled diversity, and robustness gains of +0.05 to +0.13—are not accompanied by any table or figure reporting these metrics. Tables 2 and 3 contain only 1-NN fidelity scores, and Figure 1 shows training loss curves, not evaluation results. The validity rate of 100% and the statistical comparisons for these claims are also not shown in a data table. The central results are therefore not verifiable from the manuscript.","section":"§3.2, Tables 2–3"}],"minor_comments":[{"comment":"The sentence 'Following the DiCE framework [19], λd = 1.0 is set to 1.0' is redundant; it should read 'λd is set to 1.0.'","section":"§2.3"},{"comment":"In Eq. (1), n is described as the number of features, but the input is x ∈ R^d, so n is the hidden-layer width; d is the feature dimension.","section":"§2.5, Eq. (1)"},{"comment":"In the grid-search description, 'the robustness loss was found to peak at λr = 0.4' is confusing, since a loss peaking is not a desirable selection criterion; the following sentence says the total loss continued to decrease. Please clarify which quantity is maximized or minimized.","section":"§2.3"},{"comment":"Each table note says the highest fidelity is highlighted in bold, but the tables contain no bold entries as printed; this should be corrected.","section":"Tables 2 and 3"}],"recommendation":"reject","confidential_remarks":"The paper has an undefined optimization objective as its core novelty and a circular evaluation; these are not local presentation issues. The idea of robustness-aware counterfactual generation is worth pursuing, but the current manuscript does not establish the method or its results. I would encourage a resubmission only if the loss is redefined with a proper surrogate and the evaluation uses an independent robustness metric on held-out instances."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this is a straightforward extension of DiCE that adds a Dice-Sørensen robustness term, and the empirical setup is broad (four datasets, three backends). But the core loss is not actually optimizable as written, because it is computed on binarized vectors and no gradient surrogate is provided.\n\nWhat is genuinely new: the specific robustness regularizer, based on the Dice-Sørensen coefficient, is not in the prior work they cite. The paper also does a reasonable job of situating itself, and the 1-NN fidelity evaluation is a concrete, useful check on local decision-boundary alignment.\n\nThe soft spots are not minor. In Section 2.3, the robustness loss uses binarized c and c′ = c + δ. Binarization is a step function, so the loss is piecewise constant and its gradient is zero almost everywhere. The paper says they use Adam and backpropagation, but never explains how a gradient flows through the binarization. Without a straight-through estimator or a relaxation, the term cannot be minimized by the stated optimizer. The decreasing robustness-loss curves in Figure 1 have no supporting mechanism.\n\nThe sign convention adds to the confusion: the total loss subtracts λ_r LRobustness. If robustness_loss is a distance to be minimized, subtracting it would push the distance up. The paper does not address this.\n\nThe evaluation is partly circular. The robustness gains are measured with what looks like the same binarized Dice-Sørensen distance used as the training objective, and λ_r is tuned by grid search on the evaluation datasets. So the reported +0.05 to +0.13 improvements are partly an artifact of optimizing for the metric. The paper also omits the actual quality tables (proximity, sparsity, diversity, robustness numbers); only the 1-NN fidelity tables are given.\n\nThere is real work here, and the prose is not misleading. But the central mechanism is not well-defined, and the headline result is not independently verifiable. I would desk-reject in current form, with an encouragement to resubmit after fixing the loss, the sign, and the evaluation. For a reading group, it is a useful example of non-differentiable losses and circular metrics in XAI papers.","headline":"A DiCE extension with a non-differentiable robustness loss and circular evaluation; the central claim is unsupported as written.","tokens_in":10039,"tokens_out":3462,"would_cite":false,"duration_ms":33404,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding a Dice–Sørensen robustness term to the DiCE counterfactual explainer produces explanations that are closer, sparser, more diverse, and more robust while keeping validity near 100 percent.","keywords":["counterfactual explanations","explainable AI","DiCE","Dice-Sørensen coefficient","robustness","proximity","diversity","multi-objective optimization"],"falsifier":"Run a gradient check on the robustness term: compute $L_{\\text{Robustness}}$ after binarizing $c$ and $c + \\delta$, and evaluate whether small continuous steps in $c$ change the loss. Because the binarized loss is piecewise constant, the expected gradient is zero almost everywhere; if no surrogate gradient is implemented, the reported robustness-loss decline cannot come from minimizing that term. A simpler experimental falsifier: set $\\lambda_r = 0$ and keep all other settings identical; if the robustness scores are statistically unchanged, the added term is not doing the work attributed to it.","tokens_in":8944,"feed_emoji":"🛡️","tokens_out":9278,"duration_ms":82096,"temperature":0.7,"pith_summary":"DiCE-Extended seeks to fix a weakness in the widely used DiCE (Diverse Counterfactual Explanations) framework: DiCE generates multiple diverse alternative explanations but does not keep them stable when inputs or model details shift slightly. The paper's central claim is that adding a robustness loss built on the Dice–Sørensen coefficient to DiCE's proximity and diversity objectives yields counterfactuals that are at once closer to the original instance, sparser in features changed, more diverse, and more stable under small perturbations. Across four tabular datasets and three ML backends, the authors report that validity stays near 100 percent, median proximity shrinks by 30–95 percent, diversity and sparsity improve, and neural-network backends gain 0.05–0.13 in robustness scores, while 1-nearest-neighbour fidelity is preserved or improved. The practical motivation is that high-stakes decisions—loans, credit, recidivism—need explanations users can act on and that do not flip under minor variations. If the claim holds, robustness can be added to existing DiCE-style pipelines with only modest generation-time overhead.","feed_headline":"Robustness term yields closer, sparser, sturdier counterfactuals","feed_subtitle":"On four tabular benchmarks, the extended DiCE keeps validity near 100 percent while cutting proximity 30–95 percent.","key_machinery":"The load-bearing machinery is the robustness regularization term. The paper defines it by binarizing the generated counterfactual vector $c$ and its perturbed version $c' = c + \\delta$, then computing the Dice–Sørensen distance $2|c \\cap c'|/(|c| + |c'|)$ between the two binary vectors. This term is inserted into DiCE's objective with weight $\\lambda_r$, next to the proximity term (weight $\\lambda_p = 0.5$) and the diversity term based on the determinant of a similarity kernel (weight $\\lambda_d = 1.0$). The paper treats the resulting combined loss as minimized by gradient-based optimization; the robustness term is the new component that is supposed to pull generated counterfactuals toward regions where small perturbations do not change the explanation.","core_discovery":"The paper proposes to replace DiCE's two-term objective with a weighted three-term objective: minimize $L_{y\\_loss} + \\lambda_p L_{\\text{Proximity}} - \\lambda_d L_{\\text{Diversity}} - \\lambda_r L_{\\text{Robustness}}$, where $L_{\\text{Robustness}}$ is the Dice–Sørensen distance between binarized counterfactuals and counterfactuals perturbed by $\\delta$. With $\\lambda_p = 0.5$, $\\lambda_d = 1.0$, and $\\lambda_r = 0.4$ (selected by grid search), the paper reports that, across four datasets and three backends, DiCE-Extended consistently matches or beats standard DiCE: proximity decreases substantially with validity near 100 percent, sparsity and diversity rise, robustness improves by 0.05–0.13 for neural-network backends, 1-NN fidelity is usually preserved or improved, and generation time grows by only about 1.2×. The paper interprets these results as evidence that the added robustness term does not trade away interpretability or local fidelity but makes counterfactuals adhere more faithfully to the model's local decision boundary.","pith_inferences":["Editorial extension: substituting a differentiable surrogate for the binarized Dice–Sørensen term (for example, a soft-thresholded version) and rerunning the benchmarks would test whether the reported robustness gains come from the robustness term itself or from the re-weighted $\\lambda_p$ and $\\lambda_d$ alone.","Editorial extension: the robustness metric is computed by perturbing the generated counterfactuals $c$, not the original query $x$; a claim of stability under 'small input variations' would need a second metric that perturbs the query and checks whether the recommended actions stay the same.","Editorial extension: this approach connects naturally to adversarial robustness; one could test whether DiCE-Extended's counterfactuals become harder to flip when the underlying model is adversarially trained, which would make explanation robustness and model robustness complementary."],"forward_implications":["Standard DiCE users can expect the robustness term to shrink median proximity by 30–95 percent without sacrificing validity, which directly improves the actionability of the explanations.","Neural-network-based explainability pipelines gain the most stability, with absolute robustness scores rising 0.05–0.13, suggesting the term is most valuable exactly where counterfactual instability is worst.","Because 1-NN fidelity is preserved or improved in most configurations, adding the robustness term does not pull counterfactuals away from the model's local decision boundary.","The reported 1.2× generation-time overhead means robustness can be added to existing DiCE-style generators without redesigning the pipeline."],"supporting_citations":[{"why":"Defines the DiCE framework and the proximity, diversity, and sparsity metrics this paper extends; standard DiCE is the baseline.","marker":"[19]"},{"why":"Supplies the Dice–Sørensen coefficient used as the basis of the new robustness metric.","marker":"[20]"},{"why":"Introduces multi-objective counterfactual generation, the template for weighting proximity, diversity, and robustness.","marker":"[3]"},{"why":"Documents robustness of counterfactual explanations over time, motivating the stability requirement.","marker":"[8]"},{"why":"Provides the Adult Income dataset used in the benchmarks.","marker":"[2]"},{"why":"Provides the German Credit dataset used in the benchmarks.","marker":"[15]"},{"why":"Provides the COMPAS recidivism dataset used in the benchmarks.","marker":"[11]"},{"why":"Provides the Lending Club loan dataset used in the benchmarks.","marker":"[1]"}],"fun_headline_variants":["Robustness term makes counterfactuals sturdier","Dice-based robustness sharpens counterfactuals","Three-term objective boosts counterfactual stability","Counterfactuals get a robustness boost","New robustness metric yields closer, stable counterfactuals"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument depends on the assumption that the binarized Dice–Sørensen robustness loss can actually be minimized by the optimizer; after binarization the loss is flat almost everywhere, so gradient descent has no slope to follow unless a differentiator surrogate is used.","fun_headline_variants_meta":{"raw":{"variants":["Robustness term makes counterfactuals sturdier","Dice-based robustness sharpens counterfactuals","Three-term objective boosts counterfactual stability","Counterfactuals get a robustness boost","New robustness metric yields closer, stable counterfactuals"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00017,"raw_usage":{"total_tokens":1338,"prompt_tokens":1088,"completion_tokens":250,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":704,"completion_tokens_details":{"reasoning_tokens":179}},"tokens_in":704,"tokens_out":250,"duration_ms":2863,"temperature":1.0,"reasoning_tokens":179,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:03:50.440592+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a gradient check on the robustness term: compute $L_{\\text{Robustness}}$ after binarizing $c$ and $c + \\delta$, and evaluate whether small continuous steps in $c$ change the loss. Because the binarized loss is piecewise constant, the expected gradient is zero almost everywhere; if no surrogate gradient is implemented, the reported robustness-loss decline cannot come from minimizing that term. A simpler experimental falsifier: set $\\lambda_r = 0$ and keep all other settings identical; if the robustness scores are statistically unchanged, the added term is not doing the work attributed to it.","supporting_citations":[{"cited_title":"OpenML Dataset #4 5039 (2023), https://api.openml.org/d/45039","cited_arxiv_id":null,"evidence_quote":"Provides the COMPAS recidivism dataset used in the benchmarks."},{"cited_title":"IEEE Transactions on Dependable and S ecure Computing (2023)","cited_arxiv_id":null,"evidence_quote":"Supplies the Dice–Sørensen coefficient used as the basis of the new robustness metric."},{"cited_title":"OpenIntro Dataset (Unk nown), https://www.openintro.org/data/index.php?data=loans_full_schema","cited_arxiv_id":null,"evidence_quote":"Provides the Lending Club loan dataset used in the benchmarks."}],"review_version":1}