{"id":"aeccfd8b-8d53-47d8-9d6d-7e910e7a7c0b","arxiv_id":"2504.15386","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"The authors define and estimate a patient-level proportion of treatment effect explained in observational data using T-learners, enabling assessment of surrogate heterogeneity.","lead":"This paper proposes a framework to measure how well a surrogate marker substitutes for a primary outcome across different patient groups in non-randomized, real-world data. It matters because surrogate markers are widely used in public health and social science, where randomization is often impractical, and knowing when a surrogate is valid for a given person could change how studies are designed and where resources go.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Plug-in estimator for ΔS(x) is not generally consistent under the stated assumptions; the simulations only exercise cases where μ1(s,x)−μ0(s,x) is independent of s, leaving the central consistency claim untested.","rationale":"The reader's weakest-assumption analysis identifies exactly the load-bearing gap: the consistency of the plug-in estimator for ΔS(x) is asserted under conditions that do not imply consistency. The paper's Condition (C6) ('Var(S(0)|X) sufficiently small') is not a formal asymptotic requirement, and Assumption (C5) only supplies a Lipschitz bound. A second-order Taylor expansion shows the plug-in error is proportional to Var(S(0)|X) times the curvature of μg in s, which need not vanish as n grows. This is not a disagreement with prevailing consensus; it is an internal soundness issue in the central estimation claim. The simulations are constructed so that μ1−μ0 does not depend on s, making the plug-in exact and therefore providing no evidence about the general case. The paper's own caveat in Section 3.3—that numerical integration may be needed if the conditions do not hold—underscores that the stated assumptions are not sufficient. A concrete simulation with treatment-by-surrogate interaction, or an independent re-derivation of the bias term, would settle whether the proposed estimator is generally consistent. Because the framework could be repaired by either proving consistency under explicit affine-in-s conditions or replacing the plug-in with numerical integration, the appropriate verdict is conditional acceptance rather than outright rejection.","tokens_in":16310,"tokens_out":2625,"duration_ms":25556,"concrete_test":"Run a simulation setting in which the surrogate enters the outcome differently by treatment group, for example Y = G + 2S + 1.5·G·S + f(X) + ε, so that μ1(s,x)−μ0(s,x) = 1 + 1.5s and the plug-in is inexact. Compute the proposed plug-in estimate of ΔS(x) and compare it with a numerical-integration estimate (or the analytic true value) at n = 2000 and n = 20000. If the bias of bΔS(x) does not shrink toward zero as n increases, the consistency claim in Section 3.3 fails outside the affine-in-s setting. Additionally, report the plug-in bias for settings with a nonlinear S term, such as Y = G + 2S + G·S².","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central statistical claim is that bΔS(x) = bμ1(bζ0(x),x) − bμ0(bζ0(x),x) consistently estimates ΔS(x) = ∫ μ1(s,x)dF_{S|X,G=0}(s) − ∫ μ0(s,x)dF_{S|X,G=0}(s). Section 3.3 justifies this plug-in using Assumption (C5) (Lipschitz continuity of μg) and Condition (C6) (Var(S(0)|X) sufficiently small). This justification is not sufficient. Lipschitz continuity gives |E[μg(S,x)|X=x] − μg(E[S|x],x)| = O(sqrt(Var(S|x))), which does not vanish as n grows because Var(S(0)|X) is a fixed feature of the data-generating process, not a quantity that shrinks with sample size. Condition (C6) is vague rather than asymptotic; unless the conditional variance is actually zero, or μg is affine in s, the plug-in has a non-vanishing bias of the form (1/2)∂²μg/∂s²(ζ,x)·Var(S|x) plus higher-order terms. The paper does not state or prove an affine-in-s condition. The simulations in Appendix A never exercise the nonlinear case: in every setting the outcome contains the term +2S, with the same coefficient in both treatment groups, so μ1(s,x)−μ0(s,x) is independent of s and the plug-in equals the integral exactly. Thus the reported simulation evidence cannot distinguish the proposed estimator from a biased approximation, and the consistency claim in Section 3.3 is not supported for general nonlinear surrogate–outcome relationships. The paper itself notes in Section 3.3 that if these conditions do not hold, one may use numerical integration, but it does not provide a condition under which the plug-in error actually vanishes.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a framework for assessing heterogeneity in the proportion of treatment effect explained (PTE) by a surrogate marker in observational (non-randomized) data. It defines a conditional PTE RS(x) = 1 − ΔS(x)/Δ(x), where Δ(x) is the conditional average treatment effect and ΔS(x) is a residual treatment effect after accounting for the surrogate, and estimates these quantities with T-learners using linear models, GAMs, and regression forests as base learners. The residual effect is estimated by a plug-in: bΔS(x) = bμ1(bζ0(x), x) − bμ0(bζ0(x), x), where ζ0(x) = E(S(0)|X=x). The paper also proposes an individual-level identification procedure that flags subjects for whom RS(x) exceeds a threshold κ, and evaluates the methods via simulations and an NHANES application with HbA1c as a surrogate for fasting plasma glucose in the obesity comparison.","tokens_in":16769,"tokens_out":6880,"duration_ms":61543,"significance":"If the consistency claim were supported, the framework would fill a genuine gap in the surrogate literature by allowing heterogeneity in PTE to be estimated in real-world data with flexible machine-learning tools. The paper is clearly written, provides reproducible R code and an R package, and its application is relevant. However, the central theoretical claim for the plug-in estimator is not justified under the stated assumptions, and the simulation design does not exercise the case where the plug-in approximation is nontrivial. These issues are load-bearing for the paper's main message.","major_comments":[{"comment":"The consistency argument for bΔS(x) relies on (C5) and (C6), but these conditions do not imply that the plug-in error vanishes. Under Lipschitz continuity (C5), |∫ μg(s,x)dF_{S|X,G=0}(s) − μg(ζ0(x),x)| = O(sqrt(Var(S(0)|X))), and Var(S(0)|X) is a fixed feature of the data-generating process, not a quantity that shrinks with n. Condition (C6) ('sufficiently small') is not an asymptotic condition, and the manuscript does not provide a sequence of data-generating processes under which the error tends to zero. The plug-in is exact only if μ1(s,x) − μ0(s,x) is affine in s (or if Var(S(0)|X)=0), a condition that is not stated, proved, or cited. Consequently, the claim that bΔS(x) consistently estimates ΔS(x) for arbitrary Lipschitz outcome models is not justified.","section":"Section 3.3, Eq. (1)"},{"comment":"In every simulation setting (Settings 1–4), the outcome models contain the term +2S with the same coefficient in both treatment groups, so μ1(s,x) − μ0(s,x) is independent of s. Under this structure, the plug-in estimator equals the integral over S(0)|X exactly (apart from estimation error in ζ0 and the μg), so the simulations provide no evidence about the magnitude of the plug-in bias when the difference is nonlinear in s. At least one simulation with an S×X interaction or a quadratic term in S in the outcome model is needed to test the central consistency claim.","section":"Appendix A"},{"comment":"The asymptotic discussion is informal: there is no theorem stating conditions under which bΔS(x) converges to ΔS(x), and no rates are given for the base learners. Separate consistency of bλg, bμg, and bζ0 does not automatically yield consistency of the combination bμ1(bζ0,x) − bμ0(bζ0,x), because that also requires uniform convergence of bμg and control of the error in bζ0. The paper should either state a formal result with explicit conditions (e.g., affineness of the difference and uniform convergence rates) or replace the plug-in with an estimator for which consistency follows from existing theory, such as an integration step over an estimated conditional distribution of S(0)|X.","section":"Section 3.3"}],"minor_comments":[{"comment":"In the sentence introducing the residual treatment effect, 'ΔS(w)' appears to be a typo and should read 'ΔS(x)'.","section":"Section 2.3"},{"comment":"The claim that Monte Carlo integration 'becomes infeasible as the dimension of X grows' is vague; weighted resampling from the observed control group is a standard approach that does not necessarily require kernel density estimation in X. This is a presentation issue, not a substantive one.","section":"Section 3.2"},{"comment":"The caption says 'Bias reflects the absolute value of the difference between the estimate and the truth,' but the column is simply labeled 'Bias.' Renaming the column to 'Abs. Bias' or 'Median Abs. Bias' would improve clarity.","section":"Table 1 caption"},{"comment":"The abstract says the method examines 'heterogeneity in the surrogacy of hemoglobin A1c as a surrogate for fasting plasma glucose,' but the application uses obesity as the exposure and glycemic measures as surrogate/outcome. Consider clarifying the causal contrast in the abstract and in the example description.","section":"Abstract and Section 6"}],"recommendation":"major_revision","confidential_remarks":"The central consistency issue in Section 3.3 is serious: the stated conditions do not make the plug-in bias vanish, and the simulations avoid the problematic case by construction. If the authors can add an appropriate affineness condition and revise the theory accordingly, or provide a genuinely consistent alternative estimator for the integral, the paper may become acceptable. The novelty claim about the absence of methods for surrogate heterogeneity in non-randomized settings should be verified with a more systematic literature search. The paper is otherwise well organized and the applied example is interesting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper before deciding what to do with it. First, it fills a real gap: no prior method assesses heterogeneity in the proportion of treatment effect explained (PTE) by a surrogate in non-randomized data. Second, the main statistical justification for its central estimator does not hold as written, and the simulation design never exercises the questionable case.\n\nWhat's genuinely new is the combination: defining a conditional PTE, RS(x), for observational data, and estimating it with T-learners while accommodating confounders. That is a reasonable and useful assembly of existing building blocks (conditional PTE from the randomized-setting work, observational PTE from Han et al. and Agniel et al.). The paper also does some things well. The identification procedure with kappa thresholds and bootstrap-based decisions is sensible. The simulations are careful in many respects, with three base learners, honest sample-splitting, and coverage reported. The NHANES example is a nice illustration, and the code appears to be available.\n\nThe soft spot is the consistency claim for the plug-in estimator of Delta_S(x). The paper replaces the integral of mu_g(s,x) over F_{S|X,G=0} with mu_g(zeta_0(x), x). Condition (C6) just says Var(S(0)|X) is \"sufficiently small.\" That is not an asymptotic condition, and Lipschitz continuity (C5) only gives an error bounded by a constant times sqrt(Var(S(0)|X)) — which does not vanish as n grows. Unless the regression functions are affine in s, or the conditional variance actually shrinks, the plug-in has non-vanishing bias. The paper does not state or prove an affine condition. The simulations never test the approximation: in every setting, the outcome contains the term +2S with the same coefficient in both groups, so mu_1(s,x) - mu_0(s,x) is independent of s and the plug-in equals the integral exactly. So the reported simulation evidence cannot detect the bias that the theory leaves open.\n\nThere are also minor issues: sensitivity in the identification procedure is very low for GAMs and trees, and the paper mentions this but doesn't dwell on it. The causal assumptions (C3) and (C4) are strong, but the authors are upfront about that.\n\nOverall, the framework is worth engaging with, but the central consistency argument needs an explicit affine-in-s condition or a switch to numerical integration in the general case, plus simulation settings where the plug-in is inexact. The reader's stress-test concern lands; I don't see a way around it.\n\nFor a journal, this deserves peer review — an editor should send it out. A competent referee can push the authors to fix the theory and strengthen the simulations, and the practical framing is useful for public health and social science researchers. I would not cite it as-is for the consistency claim, but I would keep an eye on a revised version.","headline":"A useful framework for surrogate heterogeneity in observational data, but the plug-in estimator's consistency is not actually established and the simulations never stress it.","tokens_in":17234,"tokens_out":718,"would_cite":false,"duration_ms":7709,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62G05","62G08","62F40"],"pacs":[],"model":"deepseek-v4-flash","headline":"A framework to estimate per-patient surrogate strength in non-randomized data, with a procedure to identify individuals for whom the surrogate can replace the primary outcome.","keywords":["surrogate markers","proportion of treatment effect explained","heterogeneity","observational data","meta-learners","T-learner","causal inference"],"falsifier":"Run the paper's simulation machinery with a substantial conditional variance $\\operatorname{Var}(S(0)\\mid X)$ and with an outcome model in which $\\mu_1(s,x)-\\mu_0(s,x)$ genuinely changes with $s$; if the plug-in estimator's bias fails to vanish as the sample size grows, the central consistency claim for $\\hat{\\Delta}_S(x)$ fails in that regime.","tokens_in":16085,"feed_emoji":"📊","tokens_out":10860,"duration_ms":85465,"temperature":0.7,"pith_summary":"The paper aims to show that surrogate strength can be estimated at the individual level in non-randomized, observational data, where treatment is confounded rather than randomly assigned. It defines a covariate-conditional version of the proportion of treatment effect explained, $R_S(x) = 1 - \\Delta_S(x)/\\Delta(x)$, and proposes a T-learner meta-learner framework that estimates each ingredient with flexible base learners. If the framework works, researchers can identify which patients have a surrogate strong enough to replace the primary outcome, and which do not, in studies where randomization is impossible. The paper supports the claim through simulations with three base learners and an application evaluating hemoglobin A1c as a surrogate for fasting plasma glucose in a national health survey.","feed_headline":"Surrogate strength gets a per-patient score in real-world data","feed_subtitle":"A T-learner framework estimates the proportion of treatment effect explained by a surrogate for each person, despite confounding.","key_machinery":"The central object is the conditional proportion of treatment effect explained, $R_S(x) = 1 - \\Delta_S(x)/\\Delta(x)$, where $\\Delta(x)$ is the conditional average treatment effect and $\\Delta_S(x)$ is the treatment effect left after the surrogate is accounted for, integrated over the control distribution of $S$. The machinery that carries the argument is the T-learner meta-learner: instead of one model with treatment as a covariate, separate base learners are fitted for the treatment and control groups, estimating $\\lambda_g(x)$, $\\mu_g(s,x)$, and $\\zeta_0(x)$. The residual effect is computed by plugging the fitted control-surrogate mean $\\hat{\\zeta}_0(x)$ into the difference of the two fitted outcome regressions, which turns a difficult integral into a standard prediction task. Three base learners are used, linear models, generalized additive models, and regression forests, giving practitioners a bias-variance spectrum, and bootstrap resampling provides confidence intervals for $R_S(x)$.","core_discovery":"On its own terms, the paper claims that surrogate heterogeneity in real-world data is identifiable and estimable under the stated causal assumptions. The conditional treatment effect $\\Delta(x)$ and residual treatment effect $\\Delta_S(x)$ are written as supervised regressions, and the plug-in estimator $\\hat{\\Delta}_S(x) = \\hat{\\mu}_1(\\hat{\\zeta}_0(x), x) - \\hat{\\mu}_0(\\hat{\\zeta}_0(x), x)$ evaluates the fitted outcome-on-surrogate regressions at the predicted control-surrogate mean. The resulting $\\hat{R}_S(x) = 1 - \\hat{\\Delta}_S(x)/\\hat{\\Delta}(x)$ is used in a bootstrap-based identification procedure: if the lower confidence bound for $R_S(x_i)$ exceeds a prespecified threshold $\\kappa$, the surrogate is declared strong enough for that individual. The paper states that this is the first method for surrogate heterogeneity in non-randomized settings, and illustrates it with an obesity-versus-non-obesity comparison in which hemoglobin A1c is evaluated as a surrogate for fasting plasma glucose.","pith_inferences":["Editorial inference: the plug-in approximation is exact in the paper's simulation settings because the outcome models are linear in the surrogate and $\\mu_1(s,x) - \\mu_0(s,x)$ is independent of $s$; a simulation with an interaction between the surrogate and treatment in the outcome would stress the method beyond its current validation.","Editorial inference: the same conditional-PTE machinery could be combined with cross-fitting or an adaptive threshold $\\kappa$ tied to the cost of measuring the primary outcome, which the paper mentions as an open practical question.","Editorial inference: the framework's logic extends naturally to longitudinal or time-to-event surrogates in observational settings, where the residual-effect integral would require estimating conditional distributions of the surrogate trajectory."],"forward_implications":["A researcher with observational data and a candidate surrogate can estimate a per-patient curve of surrogate strength instead of a single global number.","With a prespecified threshold $\\kappa$, the framework can classify each future patient as eligible or ineligible for surrogate-only outcome measurement.","Flexible base learners allow the method to track nonlinear and non-additive confounding, with linear models available when the relationships are close to linear.","The bootstrap confidence intervals, combined with a multiple-testing correction, make the identification procedure output a statistically supported set of patients."],"supporting_citations":[{"why":"Supplies the PTE definition $R_S = 1 - \\Delta_S/\\Delta$ that the paper generalizes to conditional and non-randomized settings.","marker":"Wang and Taylor (2002)"},{"why":"Introduces the notion of proportion of treatment effect explained, the estimand at the paper's core.","marker":"Freedman et al. (1992)"},{"why":"Provides an existing real-data PTE estimator for non-randomized treatment that the paper extends to heterogeneity.","marker":"Han et al. (2022)"},{"why":"Gives a doubly robust high-dimensional surrogate PTE method in observational data, another baseline this framework builds on.","marker":"Agniel et al. (2023)"},{"why":"Establishes flexible conditional PTE estimation in randomized settings, which the paper reworks for observational data.","marker":"Knowlton et al. (2025)"},{"why":"Formalizes surrogate heterogeneity and tests for it under randomization, the gap this paper targets without randomization.","marker":"Parast et al. (2023a)"},{"why":"Defines T-learner meta-learners and their properties, the estimation strategy the paper adopts.","marker":"Künzel et al. (2019)"},{"why":"Provides the honest regression forest theory used to establish consistency of the forest base learner.","marker":"Athey et al. (2019)"},{"why":"Underpins the plug-in approximation of the residual-effect integral under smoothness and small-variance conditions.","marker":"Newey and McFadden (1994)"}],"fun_headline_variants":["First meta-learner framework for per-patient surrogate strength in real-world data","Meta-learners map surrogate validity for each individual in real-world data","Per-patient surrogate strength estimated despite confounding in non-randomized data","Surrogate heterogeneity in real-world data: a meta-learner approach","New framework scores surrogate validity per person in non-randomized settings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the plug-in step $\\hat{\\Delta}_S(x) = \\hat{\\mu}_1(\\hat{\\zeta}_0(x),x) - \\hat{\\mu}_0(\\hat{\\zeta}_0(x),x)$ faithfully approximates the integral over the distribution of $S(0)$ given $X$, an approximation the paper justifies only when the conditional variance of $S(0)$ is small or when the outcome regressions are smooth enough that perturbations in $S$ do not change the expectation.","fun_headline_variants_meta":{"raw":{"variants":["First meta-learner framework for per-patient surrogate strength in real-world data","Meta-learners map surrogate validity for each individual in real-world data","Per-patient surrogate strength estimated despite confounding in non-randomized data","Surrogate heterogeneity in real-world data: a meta-learner approach","New framework scores surrogate validity per person in non-randomized settings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000298,"raw_usage":{"total_tokens":1746,"prompt_tokens":985,"completion_tokens":761,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":601,"completion_tokens_details":{"reasoning_tokens":669}},"tokens_in":601,"tokens_out":761,"duration_ms":7045,"temperature":1.0,"reasoning_tokens":669,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:27:49.087154+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's simulation machinery with a substantial conditional variance $\\operatorname{Var}(S(0)\\mid X)$ and with an outcome model in which $\\mu_1(s,x)-\\mu_0(s,x)$ genuinely changes with $s$; if the plug-in estimator's bias fails to vanish as the sample size grows, the central consistency claim for $\\hat{\\Delta}_S(x)$ fails in that regime.","supporting_citations":[{"cited_title":"S., Graubard, B","cited_arxiv_id":null,"evidence_quote":"Introduces the notion of proportion of treatment effect explained, the estimand at the paper's core."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides an existing real-data PTE estimator for non-randomized treatment that the paper extends to heterogeneity."},{"cited_title":"P., Thi \\'e baut, R., and Parast, L","cited_arxiv_id":null,"evidence_quote":"Gives a doubly robust high-dimensional surrogate PTE method in observational data, another baseline this framework builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Underpins the plug-in approximation of the residual-effect integral under smoothness and small-variance conditions."}],"review_version":1}