{"id":"6de75e72-71be-4e15-bae7-4cb8f5d38d4a","arxiv_id":"2608.09612","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"A local conformal prediction framework that aims to provide finite-sample valid intervals for individual causal effects by calibrating on causally similar units.","lead":"This paper proposes a method for building prediction intervals for the effect of a treatment on a single individual, rather than only on the average person. It selects similar calibration individuals, adds synthetic neighbors, and uses doubly robust scores, claiming finite-sample coverage guarantees for individual effects.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's coverage guarantee is unsupported: calibration-set selection uses Y (violating Assumption 3) and the model is fit on the same calibration set, so scores are in-sample and not exchangeable with the query score.","rationale":"The paper's central contribution is Theorem 1, a finite-sample coverage guarantee for an individual causal effect, relying on exchangeability of conformity scores in a local calibration set. The reader's rejection is well grounded: the implementation violates Assumption 3. I confirm this and add that even if Assumption 3 were fixed, the scoring protocol is not split-conformal. Section 3.2 fits the model on the same R* used for calibration; the residuals V_i are in-sample, with a distribution stochastically smaller than the out-of-sample query residual. The proof's 'treating mu_hat as fixed' is precisely the step that fails, since the model was fitted on the calibration outcomes. This is independent of the causal identification machinery and can be demonstrated in a purely predictive setting. The empirical results (96% coverage on synthetic, 100% on IHDP) are consistent with artifactually narrow score quantiles rather than valid finite-sample coverage. The local-calibration idea is plausible, and a corrected protocol (outcome-free selection plus proper split or full conformal) could restore validity, but the manuscript as written does not establish its headline claim. Verdict remains REJECT.","tokens_in":9819,"tokens_out":6716,"duration_ms":60724,"concrete_test":"Run the Section 5 synthetic DGP with an oracle-split variant: select R* using only the true causal covariates X1-X3 (no outcome-dependent scores), split R* randomly into a training half used to fit mu_hat and a calibration half used to compute V_i, and then construct the interval for each query. Compare empirical coverage of the true ITE to the paper's pipeline. If the oracle-split variant gives coverage near the nominal 90% while the paper's pipeline gives 96%, the gap shows that the reported coverage is inflated by outcome-dependent selection and in-sample residuals.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The finite-sample coverage guarantee in Theorem 1 requires (i) Assumption 3, i.e., the calibration set R* is selected without outcome information, and (ii) exchangeability of the conformity scores V_i with the query score V_q under a model fitted before seeing the calibration outcomes. Section 4 violates (i): Stage 1 uses Causal Forest variable importance omega_j estimated on observed Y in the cosine similarity, and Stage 2 imputes synthetic outcomes from a local linear model fitted on Y, so membership in R* depends on outcomes. The Shapley weights in Stage 3 also derive from a Causal Forest fit on Y, so the 'X-only' filter inherits outcome dependence. Section 3.2 violates (ii): the predictive model mu_hat is fitted on R*, and the non-conformity scores V_i are residuals on that same R*. These in-sample residuals are systematically smaller than the out-of-sample residual V_q, so the conformal quantile is biased low. The proof's statement 'treating mu_hat as fixed' is invalid because mu_hat is fitted on the same observations used for calibration scores. Thus the exchangeability argument fails independently on both grounds; the high empirical coverage in Tables 1-2 reflects these artifacts rather than a valid finite-sample guarantee.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an Individualized Causal Prediction (ICP) framework that uses conformal prediction to construct finite-sample prediction intervals for an individual causal effect tau_q = Y_q(1) - Y_q(0). The method localizes the calibration set to a neighborhood of the query unit using cosine similarity weighted by Causal Forest variable importance, augments that neighborhood with synthetic observations, filters via a composite \"super-relevance\" score, and calibrates with AIPW pseudo-outcome conformity scores. The central theoretical claim is Theorem 1, which asserts marginal coverage at level 1 - alpha under SUTVA, strong ignorability, and an outcome-independent calibration-set selection condition (Assumption 3). The paper reports experiments on a synthetic benchmark and the IHDP dataset, claiming that local strategies improve point accuracy over global baselines while maintaining nominal or above-nominal coverage.","tokens_in":10069,"tokens_out":5145,"duration_ms":50094,"significance":"If Theorem 1 were valid, the paper would address a genuinely important gap: standard CATE intervals cover conditional means but not individual counterfactual realizations, and local conformal calibration is a natural route to more conditional coverage. The paper also deserves credit for explicitly stating a selection measurability condition and for recognizing that outcome-dependent calibration-set construction would break exchangeability. The modular framing and the use of AIPW scores are reasonable building blocks. However, the manuscript's central guarantee is not established: the implementation violates the very assumption on which the proof relies, and the conformal scores are computed in-sample rather than in a split-conformal manner. Because the main theorem is the paper's core contribution, these are load-bearing flaws rather than presentation issues.","major_comments":[{"comment":"Assumption 3 is violated by the implementation. The cosine-similarity weights omega_j come from a global Causal Forest fitted on observed Y; the Shapley influence weights phi_i^Sh are derived from Causal Forest leaf weights of a forest fitted on Y; and the synthetic outcomes Ytilde_k are imputed from a local linear model fitted on Y. Since R* is formed by filtering the union of R_q and S_q on phi_i(q), membership in R* depends on outcome values. The proof's statement that 'calibration-set membership is independent of the outcome values' therefore does not hold for the algorithm as implemented, and the exchangeability step in Theorem 1 collapses.","section":"Section 4.1, Stages 1-3; Assumption 3"},{"comment":"The non-conformity scores V_i = |Y_i - muhat(X_i)| are computed on the same set R* that is used to fit muhat. The proof declares that muhat is treated as fixed, but muhat is fitted on R* before the scores are formed. In-sample residuals are not exchangeable with the out-of-sample query score V_q; they are systematically smaller, so the conformal quantile is biased downward. Split-conformal validity requires the score function to be fitted on a separate fold or to be out-of-fold, and the paper does neither.","section":"Section 3.2; proof of Theorem 1"},{"comment":"Theorem 1 is proved for residual scores |Y_i - muhat(X_i)|, but the implementation uses AIPW pseudo-outcome scores |Gamma_i - tauOOB(X_i)|. The paper provides no theorem showing that these AIPW scores, after local filtering and soft weighting, are exchangeable with the corresponding query score. The mismatch means the finite-sample coverage claim does not cover the actual procedure evaluated in the experiments.","section":"Section 3.3 vs. Section 4, Stage 4"},{"comment":"The synthetic augmentation step generates Xtilde by Gaussian perturbation and imputes Ytilde from a local linear model fitted on observed Y. These synthetic points are not draws from the true conditional distribution of Y given X and T, so they cannot be treated as exchangeable calibration observations. The paper appeals to 'local ergodicity of the SCM' but does not formalize this assumption or prove that imputed points preserve the rank condition underlying conformal coverage. Synthetic augmentation is therefore not a valid device for restoring small local sample sizes within the stated theorem.","section":"Section 4.1, Stage 2; Section 3.2, stage B"}],"minor_comments":[{"comment":"The abstract contains a typo: 'SUTV A' should be 'SUTVA'.","section":"Abstract; Section 1"},{"comment":"The headings '3.2 Conformalization of the Individual Causal Effect' and '3.3 Coverage Guarantee' each appear twice, which disrupts the numbering and makes the logical structure harder to follow.","section":"Section 3"},{"comment":"The Discussion states that any valid implementation must select the calibration set without using outcomes, but Stage 2's synthetic outcome imputation uses Y directly. The text should explicitly acknowledge this conflict with the formal condition rather than presenting it as automatically satisfied.","section":"Section 7"},{"comment":"The sentence in Section 5.2 that coverage 'confirms the finite-sample validity guarantee' overstates what an empirical coverage proportion can show, particularly without standard errors or repeated-seed variation; at best these tables illustrate that the intervals are conservative in the studied settings.","section":"Tables 1-2"},{"comment":"The Cunningham (2026) citation is to a Substack blog post; it is not a peer-reviewed source and should be replaced or removed if a citable reference is needed for the epigraph.","section":"References"}],"recommendation":"reject","confidential_remarks":"The central theorem is internally inconsistent with the implementation: the outcome-dependent variable importance, Shapley weights, and synthetic imputation violate Assumption 3, and the in-sample computation of conformity scores invalidates the exchangeability argument. These are not peripheral issues; they are the mechanism by which the claimed finite-sample guarantee would have to operate. I do not see a small revision that would fix the proof while keeping the method as described. The underlying question is timely, and a revised approach based on honest split-conformal calibration with genuinely outcome-independent selection could be worth considering as a new submission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThis paper is worth a look for the problem it tackles, but the main guarantee doesn't survive contact with the implementation. The idea is to build conformal prediction intervals for individual treatment effects by calibrating on a local, causally relevant neighborhood rather than the whole dataset. That's a legitimate gap, and the local weighting, synthetic augmentation, and AIPW scores form a plausible recipe. The writing is clear and the experiments, such as they are, show modest gains in point prediction.\n\nThe soft spot is load-bearing. Theorem 1 requires Assumption 3, that the calibration set be selected without using outcomes. The text says the assumption is satisfied by construction, but it is not. Stage 1 computes variable importance omega_j from a Causal Forest fitted on Y, and that importance re-weights the cosine similarity that defines the relevant neighborhood. Stage 2 imputes synthetic outcomes from a local model fitted on Y, so the synthetic calibration points carry outcome information. Either way, membership in R* depends on Y, and exchangeability of the conformity scores fails.\n\nThere's a second, independent problem. The predictive model mu_hat is fitted on the same calibration set R* used to form the non-conformity scores. Those in-sample residuals are systematically smaller than the query's out-of-sample residual, so the conformal quantile is biased low. The proof treats mu_hat as fixed, but it is estimated on the calibration data. This is not a minor technicality; it is the difference between a valid finite-sample guarantee and an empirical heuristic.\n\nThe empirical coverage numbers, which are all at or above nominal, are consistent with these artifacts rather than evidence for the theorem. The paper itself acknowledges the modular requirements at the end of Section 7: selection without outcomes and exchangeable scores. It just doesn't meet them.\n\nWhat's genuinely useful is the local-calibration framing. I could see a fixed version working: choose the neighborhood with pre-outcome variable importance (e.g., unsupervised or split-sample), set aside a clean calibration fold, and use proper split conformal with separate training. The current manuscript doesn't do that.\n\nI'd send this to a referee for a major revision, not because the current proof works but because the problem is real and the fix is identifiable. If the authors can re-do the selection and calibration to satisfy their own assumptions, there's a publishable paper here. As is, it's a heuristic with an unproven guarantee.","headline":"The paper's core coverage theorem rests on a selection measurability assumption that the implementation violates twice over, so the headline guarantee is unsupported.","tokens_in":10608,"tokens_out":2505,"would_cite":false,"duration_ms":23260,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62G15","62D20","62G08"],"pacs":[],"model":"deepseek-v4-flash","headline":"Local conformal intervals can cover individual causal effects with finite-sample guarantees.","keywords":["conformal prediction","individual treatment effects","causal inference","Causal Forests","heterogeneous treatment effects","doubly robust estimation","local calibration","finite-sample coverage"],"falsifier":"Permute the observed outcomes $Y$ among units while keeping $(X,T)$ fixed, rerun the full pipeline's Stage 1 (variable importance) and Stage 2 (synthetic imputation), and compute leave-one-out coverage at the nominal level: if calibration selection truly ignores outcomes, coverage stays at $1-\\alpha$, whereas any significant drop demonstrates outcome dependence in the calibration set and directly violates Assumption 3.","tokens_in":9545,"feed_emoji":"🎯","tokens_out":8837,"duration_ms":73987,"temperature":0.7,"pith_summary":"This paper claims that a prediction interval for the causal effect on one specific individual—the difference between that person's treated and untreated outcomes—can be made finite-sample valid, not just asymptotically valid for a conditional average. The method builds a calibration set from units close to the query in the causally relevant covariate subspace, adds synthetic local units, filters by a score that depends only on covariates and treatment, and applies split-conformal calibration with doubly robust AIPW scores. If the selection condition holds, Theorem 1 gives $P(\\tau_q \\in \\hat{C}(\\tau_q)) \\ge 1-\\alpha$, and the local design moves coverage from merely marginal toward conditional at the query point. The empirical sections show local calibration beating a global baseline on point accuracy while keeping coverage at or above nominal, which matters for personalized decisions under strong treatment-effect heterogeneity.","feed_headline":"Local conformal prediction covers individual causal effects","feed_subtitle":"Intervals are finite-sample valid for a single unit's treatment effect, not just the conditional average.","key_machinery":"The object carrying the argument is the super-relevant calibration set $R_q^{*(t)}$, obtained by taking the top fraction of treated (or control) units by Causal-Forest-weighted cosine similarity to $x_q$, adding Gaussian-perturbed synthetic copies with local linear imputation, and retaining the upper half by the composite score $\\phi_i(q) = (1-\\lambda)\\phi_i^{\\mathrm{Sh}}(q) + \\lambda \\phi_i^{\\mathrm{PS}}(q)$, where $\\phi_i^{\\mathrm{Sh}}$ is a Shapley influence weight from Causal Forest leaf structures and $\\phi_i^{\\mathrm{PS}} = 1-|\\hat{e}(X_i)-\\hat{e}(x_q)|$ is propensity-score proximity. The design intends $\\phi_i(q)$ to be a function of $(X_i,T_i,X_q)$ only, so membership in the calibration set is outcome-independent; split-conformal exchangeability of AIPW conformity scores $V_i = |\\hat{\\Gamma}_i - \\hat{\\tau}_{\\mathrm{OOB}}(X_i)|$ then yields the finite-sample quantile $Q_{1-\\alpha}$ that defines the interval. The AIPW pseudo-outcome $\\hat{\\Gamma}_i$ is doubly robust and Neyman-orthogonal, so first-order nuisance errors do not shift the scores to leading order.","core_discovery":"On the paper's own terms, the central discovery is that individual causal effects admit locally calibrated conformal intervals: under SUTVA, strong ignorability, and a selection measurability condition (the composite score $\\phi_i(q)$ used to enter $R_q^{*(t)}$ depends only on $(X_i,T_i,X_q)$, not on any outcome), the interval $\\hat{C}(\\tau_q)$ from Section 3.2 satisfies $P(\\tau_q \\in \\hat{C}(\\tau_q)) \\ge 1-\\alpha$ in finite samples. The same conformal mechanism that covers the counterfactual $Y_q(1)$ shifts by the observed $Y_q(0)$ to cover the individual effect $\\tau_q$. Because calibration is restricted to a causally relevant neighborhood instead of the full sample, the paper argues the marginal guarantee becomes approximately conditional at $x_q$, a strictly stronger statement than global reweighting approaches provide.","pith_inferences":["Extension (not the paper's claim): the current implementation's Stage 1 variable importance and Stage 2 synthetic outcomes are fitted on $Y$, so a direct test is to rerun the pipeline with outcome-permuted data and compare leave-one-out coverage; if selection truly ignored outcomes, coverage would be unchanged.","Extension (not the paper's claim): the selection-measurability principle suggests that any outcome-blind comparator rule—propensity matching, representation learning without labels, or instrument-based similarity—could be conformalized for individual effects, including instrumental-variable settings the paper mentions only as future work.","Extension (not the paper's claim): the approximate-conditional-coverage claim is testable by stratifying queries into covariate regions and checking coverage within each region; the gap between region-level and nominal coverage should shrink as the local mechanism becomes more homogeneous.","Extension (not the paper's claim): synthetic augmentation's benefit is tied to local positivity, so a diagnostic is to vary the augmentation noise scale and the number of synthetic points and observe whether coverage and width respond; if the local sample is already large, augmentation should be neutral."],"forward_implications":["A practitioner can report an interval for the effect of a treatment on the particular individual in front of them, with a finite-sample $1-\\alpha$ coverage guarantee, rather than an interval for the population conditional mean.","Local calibration gives approximately conditional coverage at the query point, so the reported uncertainty is more relevant to that unit than a global marginal interval.","The doubly robust AIPW scores let the method keep coverage while centering the interval on a semiparametrically efficient point estimate.","On the synthetic and IHDP benchmarks, local strategies achieve lower RMSE and MAE than the global baseline while maintaining nominal or above-nominal coverage, including 100% coverage on IHDP.","The framework is modular: any calibration-set selector that is outcome-independent and any exchangeable conformity scores can replace the specific Causal Forest and AIPW implementation."],"supporting_citations":[{"why":"Supplies the split-conformal exchangeability result that yields finite-sample coverage from exchangeable scores.","marker":"Vovk et al., 2005"},{"why":"Closest predecessor; proves marginal conformal coverage for counterfactuals and individual treatment effects and defines the global baseline the paper improves to local.","marker":"Lei and Candès, 2021"},{"why":"Provides the Causal Forest base estimator, variable importance, and out-of-bag AIPW pseudo-outcomes used as conformity scores.","marker":"Wager and Athey, 2018"},{"why":"Establishes Neyman orthogonality and double debiasing that justifies AIPW conformity scores and efficient centering.","marker":"Chernozhukov et al., 2018"},{"why":"Provides the structural causal model formalism that defines the query and the local mechanism.","marker":"Peters et al., 2017"},{"why":"Invariant causal prediction motivates the super-relevance filter as an outcome-free test of causal invariance.","marker":"Peters et al., 2016"},{"why":"Supplies the IHDP semi-synthetic benchmark with known individual treatment effects used in evaluation.","marker":"Hill, 2011"},{"why":"Provides the real observational earnings data used as an illustrative application without individual-treatment-effect ground truth.","marker":"LaLonde, 1986"}],"fun_headline_variants":["Individual causal effects get finite-sample conformal intervals","Local calibration covers single-unit treatment effects","Neighborhood-weighted sets for one effect, not the average","Conformal intervals that target the individual, not the population"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The coverage guarantee collapses if the rule for choosing the calibration set uses outcome values; the paper's Assumption 3 forbids this, but the implementation computes Causal Forest variable importance and imputed synthetic outcomes from $Y$, so the guarantee holds only insofar as those outcome-dependent quantities do not actually break exchangeability of the conformity scores.","fun_headline_variants_meta":{"raw":{"variants":["Individual causal effects get finite-sample conformal intervals","Local calibration covers single-unit treatment effects","Neighborhood-weighted sets for one effect, not the average","Conformal intervals that target the individual, not the population"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000486,"raw_usage":{"total_tokens":2366,"prompt_tokens":887,"completion_tokens":1479,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":1418}},"tokens_in":503,"tokens_out":1479,"duration_ms":11537,"temperature":1.0,"reasoning_tokens":1418,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:05:06.574412+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Permute the observed outcomes $Y$ among units while keeping $(X,T)$ fixed, rerun the full pipeline's Stage 1 (variable importance) and Stage 2 (synthetic imputation), and compute leave-one-out coverage at the nominal level: if calibration selection truly ignores outcomes, coverage stays at $1-\\alpha$, whereas any significant drop demonstrates outcome dependence in the calibration set and directly violates Assumption 3.","supporting_citations":[{"cited_title":"Elements of Causal Inference: Foundations and Learning Algorithms , publisher =","cited_arxiv_id":null,"evidence_quote":"Provides the structural causal model formalism that defines the query and the local mechanism."},{"cited_title":"Causal Inference by Using Invariant Prediction: Identification and Confidence Intervals , journal =","cited_arxiv_id":null,"evidence_quote":"Invariant causal prediction motivates the super-relevance filter as an outcome-free test of causal invariance."}],"review_version":1}