{"id":"52972562-d298-455b-bd8a-7427fd6762bd","arxiv_id":"1908.03020","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"CLEAR combines counterfactual searches with local regression to produce explanations of any classifier, and measures their fidelity against the classifier's decision boundary.","lead":"This paper introduces CLEAR, a tool that explains a single prediction by showing both a local regression equation and the smallest feature changes that would flip the classifier's decision. It argues that explanations should come with a measurable fidelity score, and reports that CLEAR's regressions are more faithful than LIME's on five tabular datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fidelity improvement from adding b-counterfactuals is computed in-sample: the same actual b-counterfactual points are used both to fit the regression and as fidelity targets, so the headline gain is partly circular.","rationale":"The central claim is that CLEAR's regressions have measurably higher counterfactual fidelity than LIME's, and that adding b-counterfactuals improves that fidelity. The first part has independent support in the 'not using b-counterfactuals' row of Table 1 and Figure 3, where CLEAR's balanced neighbourhood alone beats LIME. The second part, which is an explicit contribution and produces the largest numbers, is not independently supported because the evaluation and the training use the same b-counterfactual points. This is not a disagreement with consensus or an ad hominem; it is an internal inconsistency between the stated goal of measuring fidelity to the underlying classifier and a protocol that fits the regression to the very targets used for measurement. The reader identified the balanced-neighbourhood parameters as the weakest assumption, but the circular step is more load-bearing: whatever b1 and b2 are, the 'using b-counterfactuals' fidelity numbers are not out-of-sample. The non-circular comparison still favours CLEAR, so the appropriate outcome is a conditional acceptance with mandatory recomputation under a hold-out protocol, plus documentation of b1, b2, and the search step. If the hold-out fidelity advantage disappears, the headline should be reduced to the non-circular comparison only.","tokens_in":10497,"tokens_out":5111,"duration_ms":54987,"concrete_test":"Re-run Table 1's 'CLEAR using b-counterfactuals' condition with a leave-one-out protocol: for each observation x and each feature f, fit the regression on the neighbourhood dataset augmented with all actual b-counterfactual points except min_f(x), then compute fidelity for feature f using the held-out min_f(x). If the average fidelity collapses toward the 'not using b-counterfactuals' row (for example, not above 60% for Pima or Breast), the reported improvement is an in-sample artifact. Also report the b1 and b2 values used for the balanced neighbourhood and the search step size so the experiment can be exactly replicated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's third contribution and the headline row 'CLEAR using b-counterfactuals' in Table 1 are undermined by an in-sample evaluation loop. In Algorithm 1, after the balanced neighbourhood dataset N_tc is built, the optional step executes N_tc <- N_tc ∪ w, where w are the actual b-counterfactual points found in step 1. The regression r is then fit on this augmented dataset, with each b-counterfactual weighted 10. The fidelity error e is calculated by comparing the estimated b-perturbations w' derived from r with the same w. Since r is fit to w and is constrained to pass through x, solving r for the feature value that reaches the decision boundary will typically reproduce the training target w; the reported 'sharp improvement' from 57% to 77% (Pima) or 54% to 81% (Breast) largely reflects interpolation rather than explanatory generalization. The non-circular row 'CLEAR not using b-counterfactuals' does still beat LIME, so the broad qualitative claim may survive, but the quantitative claim that adding b-counterfactuals improves fidelity is not established by the reported protocol. A hold-out protocol is needed where the b-counterfactual points used for evaluation are excluded from the regression fit, or are evaluated by cross-validation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CLEAR, a local explanation method for arbitrary classifiers. CLEAR finds, for a prediction x, actual 'b-counterfactual' feature changes that flip the classifier's decision, builds a locally weighted regression (through x) on a balanced synthetic neighbourhood, and then uses that regression to estimate b-perturbations. Fidelity is defined as the proportion of estimated b-perturbations whose error against actual b-perturbations is below a threshold. The paper claims three contributions: counterfactual explanations with regression coefficients and interactions, a measurable fidelity definition, and the use of b-counterfactuals to improve regression fidelity. Experiments on five UCI datasets compare CLEAR with LIME and report substantially higher fidelity, with further gains when b-counterfactuals are added to the neighbourhood dataset.","tokens_in":10760,"tokens_out":5424,"duration_ms":58841,"significance":"The paper addresses an important gap in local explanation methods: the lack of a quantitative fidelity measure tied to counterfactual behaviour. If the evaluation were sound, the proposed metric and the idea of using boundary-counterfactual points to constrain local regressions would be a useful contribution. The non-circular comparison (CLEAR without b-counterfactuals versus LIME) is interesting and appears to favour CLEAR, and the authors provide prototype code for reproducibility. However, the headline quantitative claim that adding b-counterfactuals sharply improves fidelity is undermined by an in-sample evaluation loop, so the significance of the third contribution is not currently established.","major_comments":[{"comment":"The headline row 'CLEAR using b-counterfactuals' is evaluated in-sample. Step 8 appends the actual b-counterfactual points w to the neighbourhood dataset N_tc, each with weight 10, and the regression r is then fit on this augmented dataset. The fidelity error e is computed by comparing the estimated b-perturbations w' (derived from r) with the same w. Because r has been fit to w, solving the regression for the boundary will largely reproduce these training targets; the reported increases (e.g., 57% to 77% on Pima, 54% to 81% on Breast) therefore largely reflect interpolation rather than explanatory generalization. The claim that adding b-counterfactuals improves fidelity (the paper's third contribution) is not established by this protocol. Please report a hold-out or cross-validated evaluation in which the actual b-counterfactual points used as evaluation targets are excluded from the regression fit, or use the w points only for evaluation and not for training. The non-circular row 'CLEAR not using b-counterfactuals' can remain and supports the weaker comparison with LIME.","section":"3 (Algorithm 1) and Table 1"},{"comment":"The balanced-neighbourhood construction depends on the margins b1 and b2 around the decision boundary, used to split the selected points into N1, N2, and N3. The paper never specifies the values of b1 and b2 for any dataset, nor does it provide a sensitivity analysis. Since Section 4 and Figure 3 argue that balanced neighbourhood selection is a key source of CLEAR's advantage, the absence of these parameter values makes the experiments unreproducible and leaves open whether the particular choice of margins drives the reported results. Please state the values used for each dataset and show how fidelity varies with b1 and b2.","section":"3 (Algorithm 2)"},{"comment":"The paper repeatedly states that CLEAR's fidelity is 'significantly' higher than LIME's, but no statistical significance tests are reported. Although the gaps in Table 1 are large (e.g., 20% vs 57% on Pima), a formal test (or at least confidence intervals with a stated test procedure) is needed to support the word 'significantly'. In addition, LIME is evaluated on a metric it was not designed to optimize; this may be defensible given the paper's argument from Woodward's theory of explanation, but the comparison should be framed explicitly as an evaluation of LIME on CLEAR's counterfactual-fidelity metric rather than as a measure of LIME's overall quality.","section":"4 (Table 1 and LIME comparison)"}],"minor_comments":[{"comment":"The abstract states 'averaging over 45% higher in this paper's four case studies', while the introduction and full-text abstract say 'averaging over 40% higher in this paper's five case studies' and Table 1 lists five datasets; please harmonize the number of case studies and the reported average improvement.","section":"Abstract and Introduction"},{"comment":"The definitions are numbered as 'Definition 5.1' and 'Definition 5.3', but these are the first and third definitions in Section 3; the numbering should be corrected.","section":"Section 3"},{"comment":"There are spelling errors in proper names: 'Woodard' should be 'Woodward', and 'Riberio' should be 'Ribeiro'.","section":"Throughout"},{"comment":"In the pseudocode, the inner loop 'for each feature f do w<- Find Counterfactuals(x,m)' does not pass f to the function, and it is not clear how w accumulates the per-feature b-counterfactuals; please make the pseudocode explicit.","section":"Algorithm 1"},{"comment":"The one-dimensional search for actual b-perturbations uses 'regular amounts' but the step size is never specified, and the error threshold T is only mentioned as 0.25 in the text; please report these implementation details for reproducibility.","section":"3 (Step 1) and 4"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid core idea, and the non-circular comparison with LIME is worth pursuing. The main obstacle is the in-sample fidelity evaluation for the 'using b-counterfactuals' condition; this should be fixed with a hold-out protocol before the paper can be accepted. I would also ask the authors to disclose b1 and b2 and to add statistical tests."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CLEAR is a reasonable new entry in the local-explainability space: it adds counterfactual fidelity measurement to LIME-style regression, and on the non-circular comparison it beats LIME. But the paper's headline 'using b-counterfactuals' numbers are inflated by an in-sample evaluation loop. The stress-test note is right.\n\nWhat's actually new: the paper defines b-perturbations, estimates them from a locally weighted regression, and reports a fidelity error against the actual classifier. That combination—measuring how well a local regression reproduces the classifier's own minimum-change counterfactuals—is genuinely useful and not in the cited LIME, SHAP, or Anchor work. The balanced-neighborhood selection and constrained regression through x are sensible choices. The GitHub code and honest discussion of cases where fidelity is poor (e.g., the Adult example with R² 0.97 but fidelity 59%) are to their credit.\n\nSoft spots: Step 8 is the big one. Adding the actual b-counterfactual points to the regression training set with weight 10, then computing fidelity against those same points, means the regression is essentially interpolating its own training targets. That explains the sharp jumps from 57% to 77% (Pima) and 54% to 81% (Breast). The non-circular row (CLEAR not using b-counterfactuals) still beats LIME by a wide margin—80% vs 26% on Adult, for instance—so the method itself has value. But the specific claim that adding b-counterfactuals improves fidelity is not established by the reported protocol. A hold-out or cross-validation split is needed.\n\nMinor issues: the margins b1 and b2 in Algorithm 2 are never specified; the step size in the one-dimensional search for actual b-perturbations is unclear; and configuration choices (kernel width for LIME, number of synthetic points, number of independent variables) appear to be tuned on the test set without a clear validation split. None of these are fatal, but they should be documented.\n\nOverall: a solid idea with a flawed evaluation protocol. The paper is worth a serious referee—the fidelity metric and the balanced-neighborhood idea are contributions—but it needs revision before the quantitative claims can be trusted.\n\nRecommendation: send to peer review, request an out-of-sample fidelity protocol and full parameter disclosure.","headline":"CLEAR is a useful new local explanation method with a genuine counterfactual fidelity metric, but the headline fidelity gains from adding b-counterfactuals are circular and overstated; the non-circular version still beats LIME.","tokens_in":11319,"tokens_out":1864,"would_cite":false,"duration_ms":19286,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CLEAR generates local explanations that state the minimum changes to flip a prediction and measure their own fidelity, beating LIME by over 40% on average.","keywords":["counterfactual explanations","local explanations","model-agnostic","fidelity measurement","b-perturbations","regression","decision boundary","LIME"],"falsifier":"On a synthetic classifier with a known decision boundary, compute the true minimum feature changes by exhaustive grid search, run CLEAR with different values of the neighborhood margins b1 and b2, and compare its estimated changes with the true ones; if its fidelity changes materially across reasonable margin choices, especially when the regression's R-squared stays high, then the balanced-neighborhood assumption is not carrying the reported accuracy.","tokens_in":10262,"feed_emoji":"🔍","tokens_out":9860,"duration_ms":93544,"temperature":0.7,"pith_summary":"This paper argues that a satisfactory local explanation of a machine learning prediction has to do two things at once: answer counterfactual what-if questions about how the prediction would change if features were different, and state how accurately the explanation captures the classifier's actual behavior. To that end it introduces CLEAR, which first finds, by querying the classifier, the minimum change to each feature that would flip the prediction to another class, and then fits a local regression that goes through the original point. The regression is inverted to estimate those same minimum changes, and the gap between the estimated and actual changes defines a counterfactual fidelity error. Across five datasets, CLEAR's regressions achieve substantially higher fidelity than LIME's, averaging over 40% higher, because CLEAR uses a balanced neighborhood spanning the decision boundary and can include quadratic and interaction terms.","feed_headline":"Counterfactual explanation method beats LIME by over 40%","feed_subtitle":"CLEAR measures how well each local explanation matches the classifier's real behavior, so users can reject bad ones.","key_machinery":"The machinery has three parts. The b-perturbation is the unit of counterfactual truth: for each feature, a one-dimensional search moves away from x until the classifier's output crosses the decision boundary, yielding the minimal change required for a class flip. The balanced neighborhood dataset is constructed by sampling synthetic points and keeping about 200 distributed roughly evenly across probability bands between x and the classifier's decision boundary, so that the regression sees the region where counterfactual claims apply. Finally, the regression is constrained to pass through x, may include quadratic and interaction terms, and is inverted to estimate b-perturbations; the difference between those estimates and the queried values is the fidelity measure that lets CLEAR report when it does not know.","core_discovery":"The central claim is that explanation quality should be measured by counterfactual fidelity, not by how well a regression fits its training neighborhood or how often it reproduces the predicted class. The paper introduces the b-perturbation: the change in one feature's value, with all other features held fixed, that is just sufficient to move the classifier's output to a different class. CLEAR computes actual b-perturbations by one-dimensional searches that query the classifier, builds a local regression on a balanced neighborhood dataset, and estimates each b-perturbation by solving the regression equation at the decision boundary. The absolute difference between the estimated and actual b-perturbation is the fidelity error; a b-perturbation counts as faithful when that error falls below a user-chosen threshold. The paper reports that on its five case studies CLEAR's percentage fidelity is higher than LIME's in every configuration, with an average improvement of over 40%.","pith_inferences":["The paper leaves implicit that the same fidelity measure could select among explanation configurations: choose the regression family, neighborhood size, or term set with the lowest average counterfactual fidelity error, turning the interpretability/fidelity trade-off into an empirical search.","Because the balanced-neighborhood selection depends on two unspecified margins, b1 and b2, a robustness test would sweep these margins and report how fidelity changes; the reported averages would be stronger if they hold across a range of margin choices.","The single-feature search for b-perturbations could be extended to joint counterfactuals involving two or more features by inverting the regression along a shortest path to the decision boundary, with fidelity measured against multi-feature searches; the paper lists this as future work.","The fidelity protocol suggests a general benchmark for any local explainer: generate a test set, compute actual b-perturbations by querying the classifier, and compare each explainer's estimates, making explanation accuracy comparable across methods."],"forward_implications":["Explanation reports can include a per-explanation fidelity error, so a user can see whether a local explanation is trustworthy rather than assume it is.","LIME-style local regressions can be badly misleading about how much a feature must change to alter a prediction, even when their adjusted R-squared and classification accuracy look high.","Neighborhood design matters: data balanced across probability bands up to and beyond the decision boundary supports counterfactual estimation much better than points weighted tightly around x.","Including the actual b-counterfactual points as weighted constraints in the regression improves fidelity on most of the tested datasets.","Fidelity to b-perturbations is a stricter test than reproducing predicted probabilities, so it can serve as a common metric for comparing local explainers."],"supporting_citations":[{"why":"Provides LIME, the baseline local-regression method that CLEAR compares against and whose lack of fidelity measurement motivates the paper's criterion.","marker":"[17]"},{"why":"Introduces b-counterfactuals as minimal feature changes needed to flip a classification, the quantity CLEAR estimates and measures.","marker":"[20]"},{"why":"Supplies the requirement that an explanation contain a generalization relating input changes to output changes, justifying the regression component.","marker":"[22]"},{"why":"Establishes from cognitive science that explanations are sought as counterfactual contrasts, the design principle behind CLEAR.","marker":"[13]"},{"why":"Shows that the authors of LIME concede LIME does not measure its own fidelity, supporting the critique that motivates CLEAR.","marker":"[18]"},{"why":"Provides the manipulationist account of causation on which the counterfactual analysis of explanations is built.","marker":"[16]"}],"fun_headline_variants":["CLEAR measures counterfactual fidelity, beats LIME by 45%","CLEAR vs LIME: 45% higher fidelity via counterfactuals","CLEAR: counterfactual explanations with fidelity checks","Measure explanation quality: CLEAR flips features"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the balanced neighborhood of synthetic points, selected in probability bands between x and the decision boundary, is representative enough of the counterfactual region that regressions fit on it yield trustworthy estimates of the minimum changes that flip a prediction.","fun_headline_variants_meta":{"raw":{"variants":["CLEAR measures counterfactual fidelity, beats LIME by 45%","CLEAR vs LIME: 45% higher fidelity via counterfactuals","CLEAR: counterfactual explanations with fidelity checks","Measure explanation quality: CLEAR flips features"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000822,"raw_usage":{"total_tokens":3587,"prompt_tokens":928,"completion_tokens":2659,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":2584}},"tokens_in":544,"tokens_out":2659,"duration_ms":20980,"temperature":1.0,"reasoning_tokens":2584,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:27:35.736165+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a synthetic classifier with a known decision boundary, compute the true minimum feature changes by exhaustive grid search, run CLEAR with different values of the neighborhood margins b1 and b2, and compare its estimated changes with the true ones; if its fidelity changes materially across reasonable margin choices, especially when the regression's R-squared stays high, then the balanced-neighborhood assumption is not carrying the reported accuracy.","supporting_citations":[{"cited_title":"ACM SIGKDD 2016, KDD ’16, pp","cited_arxiv_id":null,"evidence_quote":"Provides LIME, the baseline local-regression method that CLEAR compares against and whose lack of fidelity measurement motivates the paper's criterion."},{"cited_title":"Woodward, Making things happen: a theory of causal expla- nation, Oxford University Press, Oxford, England, 2003","cited_arxiv_id":null,"evidence_quote":"Supplies the requirement that an explanation contain a generalization relating input changes to output changes, justifying the regression component."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes from cognitive science that explanations are sought as counterfactual contrasts, the design principle behind CLEAR."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows that the authors of LIME concede LIME does not measure its own fidelity, supporting the critique that motivates CLEAR."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the manipulationist account of causation on which the counterfactual analysis of explanations is built."}],"review_version":1}