{"id":"8e3bd6fb-47d6-422c-a8ea-49495d7e6697","arxiv_id":"2505.22326","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Counterfactuals placed where an individual's own model has wide conformal prediction intervals improve that individual's local model accuracy more than distance-only or random counterfactuals.","lead":"This paper proposes a way to choose counterfactual explanations that account for what a specific person already knows about a model. Its method selects counterfactuals in regions where the person's own model is most uncertain, as measured by conformal prediction intervals.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core heuristic—that wide conformal prediction intervals identify the most informative counterfactual locations—is only supported by a binary-set thought experiment and indirect experiments; a direct rank-correlation test between interval width and actual local model improvement is needed.","rationale":"The reader's weakest assumption identifies the same load-bearing issue: the conformal interval width is taken as a proxy for how informative a counterfactual will be, without a theoretical or direct empirical justification. My reading of the paper confirms this is the central unproven step. The thought experiment in Sec. 3.2 concerns binary prediction set size and does not transfer to the LWCP regression intervals actually used; the objective effectively maximizes rho(X'), the estimated absolute deviation of the individual's model from the entity's probabilities, and there is no evidence that this quantity predicts the local error reduction from adding a single labeled point. The experiments provide indirect support, but because they compare a full objective (width plus distance) against baselines, they do not test the width proxy in isolation. A direct rank-correlation test between width and actual improvement would settle the question. I do not see grounds to reject the paper; the heuristic may hold, but the evidence is currently conditional on this unvalidated proxy. The reader's CONDITIONAL verdict and request for stronger validation are appropriate, so I recommend no change to the verdict.","tokens_in":13921,"tokens_out":4085,"duration_ms":50430,"concrete_test":"On the hypercube dataset with a known oracle probability p_theta, fix an individual training subset T^(k) (e.g., 100 points). For a query X, generate a grid of candidate counterfactual points X' satisfying h_theta(X') != h_theta(X), and compute the LWCP width W(X') using Algorithm 1. For each candidate, add the labeled point (X', Y') to T^(k), retrain the individual's probability model, and measure the reduction in mean squared error of predicted probabilities on a held-out grid around X (or globally). Compute the Spearman rank correlation between W(X') and the actual error reduction across all candidates, repeated for 100 random queries and multiple T^(k) draws, with bootstrap confidence intervals. If the correlation is not consistently positive and significantly different from zero, the width proxy is not validated and the central claim loses its foundation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim relies on the assumption that minimizing L_info(X') = 1/C_alpha(X') selects counterfactuals whose addition best improves the individual's local model. In the LWCP implementation, the interval width is C_alpha(X) = 2 rho(X) d_alpha (Eq. 3), where d_alpha is a global calibration quantile, so maximizing width is equivalent to maximizing the estimated mean absolute deviation rho(X') of the individual's model from the entity's probability predictions. The only justification offered is the thought experiment in Sec. 3.2, which concerns a binary conformal prediction set at an already-observed point and does not apply to the LWCP regression intervals used in Algorithm 1. There is no analytical argument or direct measurement showing that adding a single labeled counterfactual at a maximum-width location reduces local predictive error more than at other locations. Wide intervals could reflect irreducible heteroscedastic noise, boundary effects, or regions where the individual's model is uniformly wrong in a way that one binary label cannot correct. The empirical evaluations on the hypercube and Tabformer datasets show that some lambda values help, but they do not isolate whether the width term is driving the improvement; the same results could arise from the distance term or from random beneficial augmentation. Thus the load-bearing condition—that interval width is a valid proxy for information gain—is unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CPICF, a method for generating individualised counterfactual explanations for a black-box binary classifier. An individual's knowledge is modelled by retraining the same classifier class on a small personal subset T^(k); locally weighted conformal prediction (LWCP) intervals, Eq. (3), are used to define an information score L_info(X') = 1/C_alpha(X'), and the counterfactual is chosen by minimising L_info(X') + lambda * L_dist(X, X') subject to the constraint that the entity's classifier flips its label. The method is demonstrated on a synthetic hypercube dataset and the Tabformer fraud dataset. The experiments evaluate, first, the change in absolute deviation between the individual's predicted probabilities and the entity's oracle probabilities after adding one counterfactual, Eq. (9), and second, data-augmentation gains measured by average precision, F1 score, and ROC AUC. The central claim is that wide conformal prediction intervals identify locations where an additional counterfactual will most improve the individual's local model, and that CPICFs therefore outperform distance-only, uncertainty-only, and unconstrained random counterfactuals.","tokens_in":14247,"tokens_out":6172,"duration_ms":65266,"significance":"If the central heuristic were established, the paper would make a useful contribution to the counterfactual-explanation literature by explicitly modelling the recipient's limited knowledge and connecting counterfactual generation to conformal uncertainty quantification. The paper has clear strengths: it ships code, uses a synthetic setup that permits direct visualisation of decision boundaries and intervals, addresses mixed continuous/categorical tabular data via a weighted Gower distance, and provides falsifiable experimental claims about local probability improvement and augmentation gains. The Tabformer experiments are a plausible real-world stress test. However, the main premise--that interval width is a valid proxy for information gain--is currently an unproven heuristic. The paper's own Conclusion correctly notes that the conformal coverage guarantee is lost once the counterfactual is added, and the experiments require a post hoc, alpha-dependent choice of lambda. These caveats are acknowledged honestly, but they are in tension with the abstract's and conclusion's wording that the approach is validated.","major_comments":[{"comment":"The objective L_info(X') = 1/C_alpha(X') is justified only by the thought experiment in Sec. 3.2, which concerns binary conformal prediction sets at an already-observed point. In the LWCP implementation used in Algorithm 1, the interval width is C_alpha(X') = 2 rho(X') d_alpha, so minimising L_info is equivalent to maximising the estimated mean absolute deviation rho(X') of the individual's model at X'. No analytical argument or direct empirical measurement is provided showing that adding a single labelled counterfactual at a maximum-rho location reduces local predictive error more than at other locations. Wide intervals could reflect irreducible heteroscedastic noise, boundary effects, or regions where one binary label cannot correct a uniformly wrong local model. Because this assumption enters the objective directly, the paper's central claim is currently unsupported.","section":"Sec. 3.4, Eq. (6), and Eq. (3)"},{"comment":"The experiments do not isolate the contribution of the width term. In the local-improvement evaluation, the baselines are lambda=0 (pure width), a very large lambda (mostly distance), and unconstrained random counterfactuals; an intermediate lambda is compared, but there is no distance-only counterfactual baseline matched for distance scale, nor a random-augmentation baseline matched for number of added points. In Table 1, CPICF augmentation is compared only to no augmentation, so the observed gains in average precision and F1 score could be produced by adding any reasonable near-boundary examples, not specifically by the conformal-width selection. A direct comparison against random and distance-only augmentation is needed to attribute the improvement to the information term.","section":"Sec. 4.1.1, Fig. 6, and Table 1"},{"comment":"The best value of lambda depends on alpha and is selected post hoc: for alpha=0.1, improvements are reported at lambda=1 or 100, while for alpha=0.2, improvement is reported at lambda=10. The paper acknowledges this and calls for careful calibration of lambda, but the claim that CPICFs improve knowledge is therefore conditional on a tuning parameter that the method does not predict. A principled selection rule, or at least a sensitivity analysis over a wider grid of lambda values with error bars, is necessary before the empirical evidence can be read as supporting the general claim.","section":"Sec. 4.1.1 and Appendix B"},{"comment":"The paper correctly states in the Conclusion that once the counterfactual is added, the conformal prediction guarantees no longer hold because the augmented data are not exchangeable. This is an important limitation: the method's name and justification lean on conformal prediction, but the final procedure has no finite-sample coverage or validity guarantee. The abstract and conclusion nevertheless describe the approach as 'validated' by the experiments. The presentation should clearly label the method as a heuristic whose conformal component provides an uncertainty estimate but no post-hoc guarantee, and the concluding claims should be softened accordingly.","section":"Sec. 5 (Conclusion) and Sec. 3.4"}],"minor_comments":[{"comment":"In the weighted Gower distance, the categorical sum runs from j=m to p, but if the first m features are continuous, the categorical features are j=m+1,...,p; the index in the second sum should start at m+1.","section":"Eq. (4)"},{"comment":"The expression inside the sum uses Delta_k(X), but the quantity defined in Eq. (7) is Delta_k(X; omega), so the notation should be Delta_k(X; omega) to make the dependence on the perturbation explicit.","section":"Eq. (9)"},{"comment":"The caption refers to 'blue, orange and green bars', but the figure shows boxplots rather than bars; additionally, the legend for the side lengths 0.1, 0.5, and 1.0 should be stated in the caption.","section":"Fig. 6 caption"},{"comment":"The text says 100 sample points are selected without replacement for T^(k) and then 100 CPICFs are generated using sampling with replacement; please clarify whether query points are drawn from T^(k) or from the wider training set, and how the shared starting classifier h_theta_k interacts with the different genetic algorithm seeds.","section":"Sec. 4.1.1"},{"comment":"The description of the individual's knowledge is ambiguous: Sec. 3.1 says the individual has access to T^(k) and the probabilities or scores of these points, while Algorithm 1 computes the regression targets Y_i = p_theta(X_i) for X_i in T^(k) using the entity's classifier. Please clarify whether the entity computes and reveals these targets or whether the individual already possesses them.","section":"Sec. 3.1 and Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline case for a statistics-oriented venue. The contribution is an algorithmic heuristic with promising but incomplete empirical support; the central assumption that interval width tracks information gain is not tested directly, and the evaluation baselines are too weak to isolate the effect. I recommend major revision rather than rejection because the paper is clearly written, the code is available, and the proposed direct test (rank-correlation between interval width and local error reduction, plus matched random/distance-only augmentation baselines) is feasible within the manuscript's scope. The authors' own Conclusion already concedes the loss of conformal guarantees, which is good scientific honesty, but the framing in the abstract should be brought in line with that caveat."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: CPICF is a genuinely new twist on counterfactual selection—it models the individual's knowledge as a small training set, builds a locally weighted conformal interval for the entity's probability scores from that set, and picks a counterfactual that minimizes 1/interval_width plus lambda*distance. That is a clean, practical objective, and I don't know another paper that personalizes counterfactuals this way. The hypercube experiments are well designed: they isolate the effect of a single added counterfactual on the individual's local probability accuracy, and the improvement for intermediate lambda is visible and consistent across seeds. The paper is also honest about its main limitation: the conformal coverage guarantee is gone after augmentation, and lambda needs tuning per dataset and alpha.\n\nThe soft spot is exactly where the stress test points. The idea that wide conformal intervals mark the most informative locations is supported only by a thought experiment about a binary conformal set at an already-observed point. That does not carry over to the LWCP regression intervals used in Algorithm 1, which are symmetric and width = 2*rho*d_alpha, a globally scaled estimate of local MAD. Wide intervals can reflect irreducible heteroscedasticity or boundary curvature, not necessarily a reducible local error. Nothing in the paper directly measures the rank correlation between interval width and actual reduction in the individual's local loss after adding a labeled point there. So the central mechanism remains a heuristic.\n\nThe experiments don't fully isolate the width term either. On Tabformer the baselines are only unconstrained random counterfactuals, not distance-matched or diversity-controlled alternatives; the genetic algorithm is light (50 evaluations, population 20), and the lambda sensitivity is acknowledged but not quantified. That said, for a COPA/CPPA type venue this is a reasonable empirical study: the code is available, the synthetic setting is transparent, and the paper's own appendix shows that the best lambda changes with alpha, which is a fair admission of fragility.\n\nWho is this for? Researchers working on explainability, active learning, or conformal-prediction-based data acquisition. It is a useful, citable formulation even before the mechanism is nailed down, but I would not treat the informativeness claim as established yet.\n\nMy recommendation: send it to peer review, with a clear request for a direct validation of the core proxy—e.g., rank-correlating interval width at candidate points with the actual change in local MAD after adding each candidate—and stronger baselines that control for distance and class balance. If they can show that width predicts improvement beyond distance alone, this is a solid contribution.","headline":"Genuinely novel personalization of counterfactuals via conformal interval width, but the core proxy—wide interval equals informative—is asserted, not established.","tokens_in":14726,"tokens_out":2421,"would_cite":true,"duration_ms":28117,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that counterfactuals placed where an individual's conformal prediction interval is widest improve that individual's local model accuracy more than distance-only or random counterfactuals.","keywords":["counterfactual explanations","conformal prediction","locally weighted conformal prediction","data augmentation","prediction interval width","individualised explanation","binary classification","transaction fraud detection"],"falsifier":"Run the paper's single-counterfactual experiment on a dataset with a known true decision boundary, generate counterfactuals at locations of different conformal-interval widths, and compare the local reduction in prediction error; if points from the widest intervals do not reduce the individual's local error more than points from medium-width intervals, the link between interval width and information gain is broken.","tokens_in":13724,"feed_emoji":"🎯","tokens_out":11699,"duration_ms":104617,"temperature":0.7,"pith_summary":"The paper argues that when an organisation explains a decision, the single most useful counterfactual to show a person is the one that lies where that person's mental model of the classifier is least certain. It formalises 'least certain' as the width of a conformal prediction interval produced from the person's own small set of known cases, and then selects a counterfactual that flips the predicted class while balancing closeness to the original case against that uncertainty. If this is right, an individual who receives such an explanation builds a more accurate local picture of the decision boundary than someone given a counterfactual chosen purely for proximity, purely for uncertainty, or at random. The experiments confirm the improvement for small training sets on a synthetic hypercube problem and on a realistic transaction-fraud dataset.","feed_headline":"Widest-uncertainty counterfactuals teach individuals fastest","feed_subtitle":"A new rule picks opposite-class points where an individual's model is least certain, shrinking local prediction error fastest.","key_machinery":"The load-bearing object is the objective function for the CPICF, $$\\arg\\min_{h_\\$\\theta$(X) \\neq h_\\$\\theta$(X')} \\left( L_{\\mathrm{info}}^{($T^{{(k)}}$)}(X') + \\$\\lambda$ L_{\\mathrm{dist}}(X,X') \\right),$$ where $L_{\\mathrm{info}}^{(T^{(k)})}(X') = 1/C_\\alpha(X')$ is the inverse width of the individual's locally weighted conformal prediction interval (a prediction band for the class probability built from locally weighted residuals and a dispersion estimate) and $L_{\\mathrm{dist}}$ is a weighted Gower distance for mixed continuous and categorical features. The interval $C_\\alpha(X')$ is built from a regression model that the individual would fit to their own subset $T^{(k)}$ of the training data, so the same query can yield different counterfactuals for different individuals. The machinery carries the argument by turning 'informative explanation' into a concrete, computable quantity: a wide interval marks a place where the individual's model is unsure, and the optimisation chooses a nearby opposite-class point in such a place.","core_discovery":"On the paper's own terms, the central claim is that counterfactual explanations can be personalised by explicitly modelling what a particular individual already knows and by using the width of a locally weighted conformal prediction interval as a measure of what the individual would learn from a new point. Minimising $L_{\\mathrm{info}}^{(T^{(k)})}(X') = 1/C_\\alpha(X')$ together with a weighted Gower distance $\\lambda L_{\\mathrm{dist}}(X,X')$, under the constraint $h_\\theta(X) \\neq h_\\theta(X')$, produces a conformal prediction interval counterfactual (CPICF). Adding a single CPICF to the individual's training data moves that individual's predicted probabilities closer to the full model's probabilities in a neighbourhood of the original query, and augmenting small training sets with CPICFs improves average precision and F1 score more than unconstrained counterfactual augmentation does.","pith_inferences":["The authors do not test this, but the same inverse-interval-width score is an active-learning acquisition function: an organisation could use $1/C_\\alpha$ to choose which unlabelled points to label for a particular individual, not just which counterfactual to display.","A natural extension is to add a diversity or repulsion term to the loss so that a batch of counterfactuals covers several uncertain regions rather than repeating the same one; the paper observes the diversity problem but does not solve it.","The observed link between $\\alpha$ and the best $\\lambda$ suggests a tuning shortcut the authors leave implicit: if interval width is the right informativeness proxy, then $\\lambda$ should be set relative to the scale of $1/C_\\alpha$, so one could fit a calibration rule mapping coverage error to distance weight instead of searching $\\lambda$ afresh for each dataset."],"forward_implications":["A recipient whose local model is improved by the counterfactual will predict the black box's behaviour near their own query more accurately after a single explanation than after a proximity-only or random explanation.","For small training sets, data augmentation with CPICFs raises average precision and F1 beyond the unaugmented model and beyond augmentation with unconstrained opposite-class points, which supports using the method when labelled data are scarce.","The relative weight $\\lambda$ controls the trade-off between usefulness and closeness; values that are too small (uncertainty only) or too large (distance only) fail to improve the individual's model, so the method requires choosing $\\lambda$ for the application.","Because the counterfactual is computed from the individual's own subset $T^{(k)}$, the same original query can produce different personalised counterfactuals for different individuals, rather than one global explanation."],"supporting_citations":[{"why":"supplies the distance-minimisation formulation for counterfactuals that the paper extends with an uncertainty term.","marker":"Wachter et al., 2017"},{"why":"founds conformal prediction and the coverage guarantee that motivates using prediction-set width as uncertainty.","marker":"Gammerman et al., 1998"},{"why":"provides the locally weighted conformal predictor used to build the intervals $C_\\alpha(X)$.","marker":"Lei et al. (2018)"},{"why":"provides conformalized quantile regression, the comparison interval method used alongside locally weighted conformal prediction.","marker":"Romano et al. (2019)"},{"why":"gives the energy-constrained conformal counterfactual baseline that also penalises prediction-set size but from a global model, which the paper contrasts with its individualised approach.","marker":"Altmeyer et al. (2024)"},{"why":"gives the diverse-counterfactual framework and the local k-nearest-neighbour evaluation idea that the paper generalises to probability deviations around the query.","marker":"Mothilal et al. (2020)"},{"why":"supplies the gradient-boosted tree implementation used as the classifier and as the regression models in the experiments.","marker":"Chen and Guestrin (2016)"},{"why":"supplies the synthetic transaction-fraud dataset used to test CPICFs on realistic tabular data with mixed feature types.","marker":"Padhi et al. (2021)"}],"fun_headline_variants":["Conformal intervals tailor counterfactuals to individual knowledge gaps","Counterfactuals that exploit your uncertainty teach you faster","Pick counterfactuals where your model is least sure to learn most","Individualised counterfactuals guided by conformal prediction uncertainty","Maximise learning by choosing counterfactuals near prediction boundaries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the width of an individual's uncertainty band is a faithful measure of how much a counterfactual placed there will teach that individual; the paper supports this with a thought experiment and indirect evaluations, not with a derivation from the band's coverage guarantee.","fun_headline_variants_meta":{"raw":{"variants":["Conformal intervals tailor counterfactuals to individual knowledge gaps","Counterfactuals that exploit your uncertainty teach you faster","Pick counterfactuals where your model is least sure to learn most","Individualised counterfactuals guided by conformal prediction uncertainty","Maximise learning by choosing counterfactuals near prediction boundaries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000484,"raw_usage":{"total_tokens":2429,"prompt_tokens":1022,"completion_tokens":1407,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":638,"completion_tokens_details":{"reasoning_tokens":1324}},"tokens_in":638,"tokens_out":1407,"duration_ms":11708,"temperature":1.0,"reasoning_tokens":1324,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:10:14.654916+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's single-counterfactual experiment on a dataset with a known true decision boundary, generate counterfactuals at locations of different conformal-interval widths, and compare the local reduction in prediction error; if points from the widest intervals do not reduce the individual's local error more than points from medium-width intervals, the link between interval width and information gain is broken.","supporting_citations":[{"cited_title":"Learning by transduction","cited_arxiv_id":null,"evidence_quote":"founds conformal prediction and the coverage guarantee that motivates using prediction-set width as uncertainty."},{"cited_title":"Tibshirani, and Larry Wasserman","cited_arxiv_id":null,"evidence_quote":"provides the locally weighted conformal predictor used to build the intervals $C_\\alpha(X)$."},{"cited_title":"Conformalized quantile regression","cited_arxiv_id":null,"evidence_quote":"provides conformalized quantile regression, the comparison interval method used alongside locally weighted conformal prediction."},{"cited_title":"Faithful model explanations through energy-constrained conformal counterfactuals","cited_arxiv_id":null,"evidence_quote":"gives the energy-constrained conformal counterfactual baseline that also penalises prediction-set size but from a global model, which the paper contrasts with its individualised approach."},{"cited_title":"Explaining Machine Learning classifiers through diverse counterfactual explanations","cited_arxiv_id":null,"evidence_quote":"gives the diverse-counterfactual framework and the local k-nearest-neighbour evaluation idea that the paper generalises to probability deviations around the query."},{"cited_title":"XGBoost : A scalable tree boosting system","cited_arxiv_id":null,"evidence_quote":"supplies the gradient-boosted tree implementation used as the classifier and as the regression models in the experiments."},{"cited_title":"Tabular transformers for modeling multivariate time series","cited_arxiv_id":null,"evidence_quote":"supplies the synthetic transaction-fraud dataset used to test CPICFs on realistic tabular data with mixed feature types."}],"review_version":1}