Pith. sign in

REVIEW 2 major objections 5 minor 25 references

Treatment effect estimation by comparing observed and predicted outcomes: conditions for valid inference and practical illustration

T0 review · 2 major / 5 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper establishes that comparing observed outcomes under a new treatment with model-predicted outcomes under standard care estimates the average treatment effect among the treated, provided five explicit sufficient conditions hold.

desk verdict A clear, honest translation of standard g-formula assumptions to an applied radiotherapy setting; not new theory, but a genuinely useful conditions checklist for a method already in use. read the letter →

arxiv 2511.21266 v2 pith:QFKFR2VI submitted 2025-11-26 stat.ME

Lotta M. Meijerink , Artuur M. Leeuwenberg , Jungyeon Choi , Bas B. L. Penning de Vries , Johannes A. Langendijk , Judith G.M. van Loon , Remi A. Nout , Karel G.M. Moons
show 1 more author
Ewoud Schuit
This is my paper · ORCID
classification stat.ME MSC 62D20
keywords averagetreatmenteffectamongthetreatedpotentialoutcomescounterfactualpredictionmodel-basedclinicalevaluationtransportabilityignorabilitypositivitymodelmisspecification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper formalizes an approach to estimating the effect of a newly introduced treatment when randomized trials are not yet available: take patients who received the new treatment, predict what their outcome would have been under the standard treatment using a model built on historical standard-treatment patients, and average the observed-minus-predicted differences. The paper argues that this average is a valid estimate of the average treatment effect among the treated (ATT) when five conditions hold: transportability of the outcome model, ignorability of treatment assignment conditional on covariates, consistency, positivity, and correct model specification. These conditions are presented as sufficient, not necessary. The formalization matters because an intuitive but previously informal method—model-based clinical evaluation in radiotherapy—now has explicit, auditable assumptions. The paper illustrates the framework with a synthetic case study comparing proton therapy and photon therapy for head and neck cancer.

What carries the argument

The central object is the pre-introduction prediction model m_pre(X, P(0); β), fitted on a cohort where everyone received the standard treatment. It is used to generate counterfactual predictions of the outcome under standard treatment for patients who actually received the new treatment. The estimator is the mean difference between observed outcomes and these predicted counterfactual outcomes. The argument that this difference identifies the ATT is carried by the potential outcomes framework together with the five stated conditions, which jointly ensure that the model predictions are unbiased for the unobserved Y(0) and that the observed outcomes equal the potential outcomes under the deliv

What would settle it

The decisive check is simulation: generate data that satisfy all five conditions by construction, including a correctly specified outcome model, then compute the observed-minus-predicted estimator's bias against the known true ATT over many replications; any systematic nonzero bias would falsify the sufficiency claim. A complementary check is to use the paper's synthetic case-study setup, violate only Condition 5 by replacing the true nonlinear dose-response with a simpler misspecified model, and show that the estimate becomes biased.

Watch

Extended reading notes

Core claim

The central claim is that the estimator (1/N) Σ (Y_i − m_pre(X_i, P(0)_i; β̂)), computed over patients treated with the new treatment, equals the causal ATT E[Y(1) − Y(0) | T = 1] when the five conditions of Section 3.3 hold. Here m_pre is a model fitted in a pre-introduction population where everyone received the standard treatment; it predicts the counterfactual risk of the outcome under standard treatment from patient characteristics X and standard-treatment plan variables P(0). The model is used to impute the unobserved potential outcome Y(0) for each treated patient. The paper derives the sufficiency of these conditions in Appendix C and does not claim they are necessary, noting that we

Load-bearing premise

The load-bearing premise is Condition 5, correct model specification: the model family must contain the true conditional risk function, and the paper concedes even its case-study model is 'somewhat imperfectly specified,' so any misspecification biases the ATT estimate regardless of the other four conditions.

Editorial extensions

If this is right

  • Model-based clinical evaluation becomes a formal causal inference method with a known set of sufficient conditions, allowing researchers to audit whether those conditions hold in a given application.
  • The estimator can produce early evidence on the effectiveness of newly introduced treatments in settings where RCTs are not feasible or not yet available, using only a published prediction model and observed outcomes in the treated group.
  • If the five conditions hold, the same procedure yields ATT estimates for risk differences and can be adapted to other effect measures such as risk ratios or odds ratios.
  • When some post-introduction patients received the standard treatment, the model's mean calibration in that subgroup provides supportive—though not definitive—evidence that the assumptions are plausible.
  • Because the estimator is not doubly robust, model misspecification (Condition 5) directly biases the ATT estimate even if all other conditions hold; sensitivity analyses are therefore advisable, as the paper demonstrates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Read as a practical checklist, the five conditions are demanding; the paper's own case study admits the model is 'somewhat imperfectly specified,' suggesting real-world applications will often rely on the hope that prediction errors average out—a hope the paper does not quantify.
  • The estimator is not doubly robust: unlike methods that model both outcome and treatment assignment, there is no second source of protection if the outcome model is wrong. A natural extension would be a doubly robust variant that combines the outcome model with a treatment-selection model.
  • The same design could transfer beyond radiotherapy to any setting where a prediction model was developed before a treatment or protocol change, but transportability and ignorability would need to be re-argued case by case for those settings.
  • The paper's mean-calibration check in standard-treated post-introduction patients is a form of negative control; using an outcome that the treatment cannot plausibly affect would make that check sharper, because any observed difference would then trace to assumption violations rather than to a true treatment effect.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper formalizes an estimator of the average treatment effect among the treated (ATT) that compares observed outcomes under a newly introduced treatment with model-based predictions of the counterfactual outcome under standard treatment. The estimator is introduced in Section 3.2 as the average of Y_i - m_pre(X_i, P^(0)_i; beta_hat) over post-introduction patients treated with the new treatment. Section 3.3 lists five sufficient conditions: transportability, ignorability of treatment assignment, consistency, positivity, and correct model specification. The authors state that these conditions are sufficient for the estimator to be valid, deferring the formal derivation to Appendix C. The paper also discusses auxiliary validation strategies, including a negative-control comparison among post-introduction patients who received standard treatment, and illustrates the method with a synthetic radiotherapy case study involving proton versus photon therapy for head and neck cancer.

Significance. If the proof in Appendix C is supplied, the central identification claim is correct: the estimator is a direct g-computation identity for the ATT under the stated conditions. The paper's contribution is not a new estimator or a new identification result, but rather a clear, explicit checklist of sufficient conditions and their interpretation in a specific applied domain (model-based clinical evaluation in radiotherapy). The authors are appropriately candid about the strength and untestability of the conditions, especially correct model specification, and they discuss the possibility of negative-control validation. The manuscript also appears to include reproducible R code and synthetic-data appendices, which are useful for practitioners. The main limitation is that the formal derivation is not present in the provided text, and the treatment of statistical uncertainty in the estimated model parameters is only sketched. Overall, the paper is a useful methodological clarification for its target audience.

major comments (2)
  1. [Section 3.3 / Appendix C] The central claim of the paper is that Conditions 1-5 are sufficient for the estimator to identify the ATT, but the actual derivation is only referenced as 'Appendix C' and is not included in the submitted text. Because the paper's main contribution is this formalization, the proof should appear in full, either in the main text or in an appendix that is part of the submission. I independently verified the standard g-computation identity, so I do not believe the result is wrong; this is a completeness concern rather than a correctness error, but it is load-bearing for the paper's stated contribution.
  2. [Section 3.2 and Section 4] The estimator uses beta_hat, the fitted parameter vector from the pre-introduction model, but the main-text formula treats beta_hat as fixed. The paper acknowledges in Section 4 that propagating the sampling variance from model development is difficult, and the case study apparently uses bootstrapping in Appendix A.4. However, 'valid inference' in the title requires a clear statement of how the two-stage uncertainty (model fitting and outcome sampling) is handled. The identification argument is unaffected, but the paper should either describe the bootstrap procedure in the main text or explicitly justify a fixed-beta approximation. This is especially relevant because the case-study model is acknowledged to be imperfectly specified.
minor comments (5)
  1. [Section 3.2, Eq. (1)] The displayed equation for the estimator has typesetting issues ('AT T=' and the fraction are garbled). Please correct the notation so that the estimator is unambiguous.
  2. [Section 3.3.2, Condition 2] The independence statement 'Y(0) ⊥ T | X, P(0)' should explicitly indicate that it is required in the post-introduction population. As written, it could be misread as a global independence assumption.
  3. [Section 3.3.3, Condition 3] The consistency condition is written as 'Yi = Y_i(t) if T_i = t'; the subscript on Y_i(t) is redundant but harmless. More importantly, the text might clarify that this condition applies to both t=0 and t=1, although only Y(0) is used in the predictions.
  4. [Appendices] The main text repeatedly refers to Appendices A, B, C, and D, but these are not included in the provided version. Please ensure that all appendices are part of the submission package, since the case-study illustration, sensitivity analysis, and the central derivation all depend on them.
  5. [Section 3.4] The negative-control validation is a useful idea. It may be worth citing the broader literature on calibration-in-the-large more explicitly when discussing the comparison between observed and predicted outcomes in the standard-treatment subgroup.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the estimator is a genuine out-of-sample counterfactual prediction under explicitly sufficient conditions.

full rationale

The derivation is self-contained. The paper defines the ATT, proposes the observed-minus-predicted estimator, and lists five sufficient conditions: transportability, ignorability, consistency, positivity, and correct model specification. The outcome model is fitted on pre-introduction standard-treatment data and then applied out-of-sample to post-introduction treated patients; no ATT parameter is fitted or reverse-engineered from the target quantity. The sufficiency claim follows from the standard g-computation identity E[Y|T=1] - E[m_pre(X,P(0); beta0)|T=1] = ATT under those conditions. The paper explicitly frames the conditions as sufficient, not necessary, and acknowledges in Section 3.3.5 that the case-study model is 'somewhat imperfectly specified' and that misspecification will bias estimates; this is an admitted limitation, not a circular step. The negative-control calibration check is an independent validation: it applies the pre-fitted model to a post-introduction group that received the standard treatment, so a nonzero average difference is possible and informative. Self-citations (e.g., Langendijk et al. [4,12]) provide clinical and application context and do not carry the identification argument. The main text refers to Appendix C for the derivation; although the appendix is not reproduced in the provided excerpt, the result is a well-known g-formula/model-based standardization identity and can be verified independently. No circular step, fitted input renamed as prediction, or load-bearing self-citation chain was identified.

Assumptions & free parameters 0 free parameters · 6 assumptions · 0 invented entities

The central claim rests entirely on the five explicit conditions; none are ad hoc to this paper, all are familiar causal inference assumptions. No free parameters in the theory; the case study's fitted logistic coefficients are illustrative, not part of the general claim. No invented entities.

assumptions (6)
  • domain assumption Consistency / SUTVA: observed outcome equals the potential outcome under the received treatment.
    Stated as Condition 3 (§3.3.3); required to connect observed data to potential outcomes.
  • domain assumption Transportability: P_pre(Y(0)=1|X,P(0)) = P_post(Y(0)=1|X,P(0)).
    Stated as Condition 1 (§3.3.1); allows a model fit on pre-introduction data to make valid counterfactual predictions in the post-introduction population.
  • domain assumption Conditional ignorability of treatment assignment: Y(0) ⊥ T | X, P(0).
    Stated as Condition 2 (§3.3.2); required to apply the model to the non-random subset treated with the new treatment.
  • domain assumption Positivity: support of (X,P(0)) in the target-treated post-introduction population is contained in the support of (X,P(0)) in the pre-introduction population.
    Stated as Condition 4 (§3.3.4); prevents extrapolation beyond the model-development sample.
  • domain assumption Correct model specification: there exists β0 with m_pre(X,P(0);β0) = P_pre(Y=1|X,P(0)).
    Stated as Condition 5 (§3.3.5); misspecification directly biases the ATT estimator.
  • standard math Standard probability theory and the potential outcomes framework.
    Used throughout to define the ATT and derive the estimator; no non-standard mathematics invoked.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Treatment effect estimation by comparing observed and predicted outcomes: conditions for valid inference and practical illustration." pith.science (2026). https://pith.science/paper/QFKFR2VI

@misc{pith2026251121266,
  author       = {Pith},
  title        = {Pith review of: Treatment effect estimation by comparing observed and predicted outcomes: conditions for valid inference and practical illustration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QFKFR2VI}},
  note         = {Machine review of arXiv:2511.21266}
}
read the original abstract

Prediction models developed before the introduction of a new treatment may be used to estimate treatment effects of newly introduced treatments. One approach, known as model-based clinical evaluation in radiotherapy, does this by comparing observed outcomes under a new treatment with predicted outcomes had these patients received the standard treatment. This article clarifies the relevant conditions needed for valid average treatment effect estimation using this approach, using the potential outcomes framework and a practical case study.

Figures

Figures reproduced from arXiv: 2511.21266 by the authors.

Figure 1
Figure 1. Illustration of the approach as applied in the case study. (A) The patients that were treated before [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

25 extracted references · 11 canonical work pages

  1. [1]

    Randomised controlled trials—the gold standard for effectiveness research

    Eduardo Hariton and Joseph J. Locascio. “Randomised controlled trials—the gold standard for effectiveness research”. In:BJOG : an international journal of obstetrics and gynaecology 125.13 (Dec. 2018), p. 1716.issn: 1470-0328.doi:10.1111/1471-0528.15199

    arXiv 2018

  2. [2]

    Defining the role of real-world data in cancer clinical research: The position of the European Organisation for Research and Treatment of Cancer

    Robbe Saesen et al. “Defining the role of real-world data in cancer clinical research: The position of the European Organisation for Research and Treatment of Cancer”. In:European Journal of Cancer186 (June 2023), pp. 52–61.issn: 0959-8049.doi:10.1016/j.ejca. 2023.03.013. [3]Evaluation of new technology in health care: in need for guidance for relevant ev...

    work page doi:10.1016/j.ejca 2023

  3. [4]

    Clinical Trial Strategies to Compare Protons With Pho- tons

    Johannes A. Langendijk et al. “Clinical Trial Strategies to Compare Protons With Pho- tons”. In:Seminars in Radiation Oncology. Proton Radiation Therapy 28.2 (Apr. 2018), pp. 79–87.issn: 1053-4296.doi:10.1016/j.semradonc.2017.11.008

    work page doi:10.1016/j.semradonc.2017.11.008 2018

  4. [5]

    Boca Raton: Chapman & Hall/CRC, 2020.isbn: 1-4200-7616-7

    MA Hern´ an and JM Robins.Causal Inference: What if. Boca Raton: Chapman & Hall/CRC, 2020.isbn: 1-4200-7616-7

    2020

  5. [6]

    Generalizing causal inferences from individuals in randomized trials to all trial-eligible individuals

    Issa J. Dahabreh et al. “Generalizing causal inferences from individuals in randomized trials to all trial-eligible individuals”. eng. In:Biometrics75.2 (June 2019), pp. 685–694.issn: 1541-0420.doi:10.1111/biom.13009

    work page doi:10.1111/biom.13009 2019

  6. [7]

    Timothy L Lash et al.Modern Epidemiology. 4th ed. Wolters Kluwer, 2021.isbn: 978-1- 4511-9328-2. 11

    2021

  7. [8]

    Reduced radiation- induced toxicity by using proton therapy for the treatment of oropharyngeal cancer

    Tineke W.H. Meijer, Dan Scandurra, and Johannes A. Langendijk. “Reduced radiation- induced toxicity by using proton therapy for the treatment of oropharyngeal cancer”. In: British Journal of Radiology93.1107 (Mar. 2020), p. 20190955.issn: 0007-1285.doi:10. 1259/bjr.20190955

    2020

  8. [9]

    Modeling the Potential Benefits of Proton Therapy for Patients With Oropharyngeal Head and Neck Cancer

    Nataniel H. Lester-Coll and Danielle N. Margalit. “Modeling the Potential Benefits of Proton Therapy for Patients With Oropharyngeal Head and Neck Cancer”. In:International Journal of Radiation Oncology*Biology*Physics104.3 (July 2019), pp. 563–566.issn: 0360- 3016.doi:10.1016/j.ijrobp.2019.03.040

    work page doi:10.1016/j.ijrobp.2019.03.040 2019

Show all 25 references
  1. [10]

    A Model-Based Approach to Predict Short-Term Toxicity Benefits With Proton Therapy for Oropharyngeal Cancer

    Jean-Claude M. Rwigema et al. “A Model-Based Approach to Predict Short-Term Toxicity Benefits With Proton Therapy for Oropharyngeal Cancer”. In:International Journal of Radiation Oncology*Biology*Physics104.3 (July 2019), pp. 553–562.issn: 0360-3016.doi: 10.1016/j.ijrobp.2018.12.055

    2019 doi

  2. [11]

    Swallowing sparing intensity modulated radiother- apy (SW-IMRT) in head and neck cancer: Clinical validation according to the model- based approach

    Miranda E. M. C. Christianen et al. “Swallowing sparing intensity modulated radiother- apy (SW-IMRT) in head and neck cancer: Clinical validation according to the model- based approach”. eng. In:Radiotherapy and Oncology: Journal of the European Society for Therapeutic Radiolo...

    2016 doi

  3. [12]

    Selection of patients for radiotherapy with protons aiming at reduction of side effects: The model-based approach

    Johannes A. Langendijk et al. “Selection of patients for radiotherapy with protons aiming at reduction of side effects: The model-based approach”. In:Radiotherapy and Oncology 107.3 (June 2013), pp. 267–273.issn: 0167-8140.doi:10.1016/j.radonc.2013.05.007

    2013 doi

  4. [13]

    Causal Inference Using Potential Outcomes: Design, Modeling, De- cisions

    Donald B. Rubin. “Causal Inference Using Potential Outcomes: Design, Modeling, De- cisions”. In:Journal of the American Statistical Association100.469 (2005). Publisher: [American Statistical Association, Taylor & Francis, Ltd.], pp. 322–331.issn: 0162-1459. url:https://www.js...

    2005

  5. [14]

    Diagnosing and responding to violations in the positivity as- sumption

    Maya L Petersen et al. “Diagnosing and responding to violations in the positivity as- sumption”. In:Statistical methods in medical research21.1 (Feb. 2012), pp. 31–54.issn: 0962-2802.doi:10.1177/0962280210386207

    2012 doi

  6. [15]

    Zivich, Stephen R

    Paul N. Zivich, Stephen R. Cole, and Daniel Westreich.Positivity: Identifiability and Es- timability. arXiv:2207.05010 [stat]. July 2022.doi:10.48550/arXiv.2207.05010

    2022 doi arXiv

  7. [16]

    Using Negative Control Populations to Assess Unmeasured Confounding and Direct Effects

    Marco Piccininni and Mats Julius Stensrud. “Using Negative Control Populations to Assess Unmeasured Confounding and Direct Effects”. en-US. In:Epidemiology35.3 (May 2024), p. 313.issn: 1044-3983.doi:10.1097/EDE.0000000000001724

    2024 doi

  8. [17]

    Calibration: the Achilles heel of predictive analytics

    Ben Van Calster et al. “Calibration: the Achilles heel of predictive analytics”. In:BMC Medicine17.1 (Dec. 2019), p. 230.issn: 1741-7015.doi:10.1186/s12916-019-1466-7

    2019 doi

  9. [18]

    Bias Analysis

    “Bias Analysis”. In:Modern Epidemiology. 4th ed. Wolters Kluwer, 2021, pp. 1556–1616. isbn: 978-1-4511-9328-2

    2021

  10. [19]

    Sensitivity analysis for an unobserved moderator in RCT-to- target-population generalization of treatment effects

    Trang Quynh Nguyen et al. “Sensitivity analysis for an unobserved moderator in RCT-to- target-population generalization of treatment effects”. In:The Annals of Applied Statistics 11.1 (Mar. 2017). Publisher: Institute of Mathematical Statistics, pp. 225–247.issn: 1932- 6157, 1...

    2017 doi

  11. [20]

    Sensitivity analysis using bias functions for studies extending inferences from a randomized trial to a target population

    Issa J. Dahabreh et al. “Sensitivity analysis using bias functions for studies extending inferences from a randomized trial to a target population”. en. In:Statistics in Medicine 42.13 (2023), pp. 2029–2043.issn: 1097-0258.doi:10.1002/sim.9550

    2023 doi

  12. [21]

    arXiv:2307.10299 [stat]

    Xinwei Shen, Peter B¨ uhlmann, and Armeen Taeb.Causality-oriented robustness: exploiting general additive interventions. arXiv:2307.10299 [stat]. July 2023.doi:10.48550/arXiv. 2307.10299

    2023 doi arXiv

  13. [22]

    Anchor Regression: Heterogeneous Data Meet Causality

    Dominik Rothenh¨ ausler et al. “Anchor Regression: Heterogeneous Data Meet Causality”. In:Journal of the Royal Statistical Society Series B: Statistical Methodology83.2 (Apr. 2021), pp. 215–246.issn: 1369-7412.doi:10.1111/rssb.12398. 12

    2021 doi

  14. [23]

    Doubly Robust Estimation in Missing Data and Causal Inference Models

    Heejung Bang and James M. Robins. “Doubly Robust Estimation in Missing Data and Causal Inference Models”. In:Biometrics61.4 (Dec. 2005), pp. 962–973.issn: 0006-341X. doi:10.1111/j.1541-0420.2005.00377.x

    2005

  15. [24]

    Van Der Laan and Sherri Rose.Targeted Learning: Causal Inference for Observa- tional and Experimental Data

    Mark J. Van Der Laan and Sherri Rose.Targeted Learning: Causal Inference for Observa- tional and Experimental Data. en. Springer Series in Statistics. New York, NY: Springer, 2011.isbn: 978-1-4419-9781-4 978-1-4419-9782-1.doi:10.1007/978-1-4419-9782-1

    2011 doi

  16. [25]

    Estimation of Regression Co- efficients When Some Regressors Are Not Always Observed

    James M. Robins, Andrea Rotnitzky, and Lue Ping Zhao. “Estimation of Regression Co- efficients When Some Regressors Are Not Always Observed”. In:Journal of the American Statistical Association89.427 (1994). Publisher: [American Statistical Association, Taylor & Francis, Ltd.],...

    1994 doi

  17. [26]

    The combination of randomized and historical controls in clinical trials

    Stuart J. Pocock. “The combination of randomized and historical controls in clinical trials”. In:Journal of Chronic Diseases29.3 (Mar. 1976), pp. 175–188.issn: 0021-9681.doi:10. 1016/0021-9681(76)90044-8. 13

    1976

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.