REVIEW 1 major objections 6 minor 31 references
Which Regularized Propensity-Score and Doubly Robust Methods Are Best Calibrated When Exposures or Outcomes Are Rare? A Plasmode Study of Proxy-Based Confounding Adjustment
T0 review · 1 major / 6 minor · reviewed 2026-07-09 · glm-5.2
Pith's one-line read Rare exposures break standard confounding adjustment; targeted selection fixes them
desk verdict Useful benchmark of regularized confounding-adjustment pipelines, but the coverage ranking is confounded by inference method read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The comparison hinges on two axes: (1) five regularized variable-selection families—traditional LASSO, collaborative-controlled LASSO, outcome-adaptive LASSO (OAL), group LASSO/GLiDeR, and highly adaptive LASSO (HAL)—that determine which proxies enter the adjustment set, and (2) two downstream estimators—IPTW (weighting by inverse treatment probability) and TMLE (doubly robust targeted updating of an outcome regression). The plasmode simulation design preserves real covariate correlations from NHANES data while imposing a known null risk difference, so any deviation from zero is measurable confounding bias. Performance is assessed via bias, coverage of 95% intervals, relative error of model–
What would settle it
If outcome-aware selection (OAL, GLiDeR) or HAL-TMLE failed to maintain near-nominal coverage under rare exposure or rare outcome in an independent plasmode with a different anchor dataset and a misspecified propensity model, the central recommendation would not hold.
Extended reading notes
Core claim
The decisive stressor is not the number of variables or the outcome rarity alone but the combination of rare exposure with weighting-based estimation. Under rare exposure (9.4% prevalence), LASSO-IPTW bias reaches −5.2×10⁻³ with inflated standard errors and conservative over-coverage near 98%, while replacing IPTW with TMLE for the same nuisance models cuts bias to near zero and reduces empirical SE, though coverage drops to 91–93%. Outcome-aware selection strategies (OAL, GLiDeR) and HAL-TMLE maintain 94–95% coverage across all three scenarios, making the choice between selection strategy and downstream estimator the central lever for calibration under rarity.
Load-bearing premise
The simulation generates exposure from a correctly specified main-terms logistic model, so the true propensity score lies within the parametric class that all propensity-based pipelines fit. This removes propensity-model misspecification as a bias source, meaning the strong calibration of outcome-aware methods is conditional on a correctly specified exposure model—a condition unlikely to hold in real healthcare database studies.
Editorial extensions
If this is right
- Researchers using healthcare databases with rare exposures should avoid LASSO-IPTW and either adopt outcome-adaptive selection or pair their selection with a doubly robust TMLE step to avoid the bias and instability documented here.
- The six-order-of-magnitude runtime spread (sub-second to 16+ hours) means that method choice in applied pharmacoepidemiology is effectively a compute-budget decision as much as a statistical one, with HAL-TMLE offering the best calibration-per-compute trade-off.
- The finding that HAL G-computation produces near-unity coverage with 106–186% relative error suggests that bootstrap-based inference for highly flexible estimators can systematically misrepresent uncertainty even when point estimates are accurate.
- The rare-exposure arm exposed the largest methodological gaps, suggesting that future proxy-adjustment benchmarks should default to testing under rarity rather than balanced conditions.
Reading between the lines
- If the true propensity model were misspecified—as it likely is in real claims data where exposure drivers are poorly captured—the near-nominal calibration of OAL and GLiDeR might degrade, since the simulation's correctly specified PS gives all PS-based pipelines a common-mode advantage that would not hold in practice.
- The mild undercoverage of TMLE pipelines under rarity (91–93%) may reflect a general tension: influence-curve standard errors condition on the selected nuisance model and thus ignore selection uncertainty, whereas bootstrap-based methods propagate it—at the cost of hours of computation.
- The prescription-derived proxies in this anchor dataset are predominantly outcome predictors with weak treatment association, so the simulation may under-test scenarios where proxies are strong confounders of treatment rather than outcome.
- Extending the comparison to non-null effects would test whether the calibration ranking at the null transfers to power and bias under realistic effect sizes, which the authors flag as a one-argument change to their released pipeline.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript compares ten regularized confounding-adjustment pipelines—spanning traditional LASSO, collaborative-controlled LASSO, outcome-adaptive LASSO (OAL), group LASSO/GLiDeR, and highly adaptive LASSO (HAL), each paired with IPTW and/or TMLE—under a plasmode simulation anchored on NHANES 2013–2018 data. Three scenarios (frequent, rare-exposure, rare-outcome) are evaluated under a known null (RD = 0), with bias, coverage, relative error, and runtime reported. The central finding is that outcome-aware selection (OAL, GLiDeR) and doubly robust estimation (HAL-TMLE) are best calibrated, while LASSO-IPTW shows large bias under rare exposure and HAL G-computation over-covers with large relative error. The design is well-structured (ADEMP framework, 1000 replicates, common-replicate comparison, null truth anchor), and the runtime accounting is a valuable practical contribution.
Significance. The paper addresses a genuine gap: no prior single-design study has assembled this breadth of regularized selection families crossed with IPTW versus doubly robust estimation while varying exposure prevalence and reporting compute cost. The null-anchored plasmode design is a strength—it isolates residual-confounding bias cleanly. The runtime table (Table 4) and practical guidance table (Table 5) are useful for practitioners. The reproducible code repository and bundled anchor data are commendable. The finding that replacing IPTW with TMLE removes rare-exposure bias and instability is practically important and well-demonstrated.
major comments (1)
- §4.3 and Table 5: The calibration ranking identifying OAL (IPTW) and GLiDeR as 'best calibrated' (coverage 94–95%) while LASSO-TMLE, C-TMLE, and Standard TMLE 'under-cover modestly' (91–93%) is confounded by the inference method. OAL and GLiDeR use 200-replicate bootstrap SEs that propagate selection uncertainty through the full pipeline, while every under-covering TMLE method uses efficient influence-curve SEs that condition on the cross-validated nuisance model. The authors acknowledge this in §4.3 ('the calibration and runtime comparisons are therefore partly confounded by which error sources each SE captures'), but Table 5's practical recommendation treats calibration as a property of the selection/estimation strategy rather than the inference method. The systematic pattern is striking: all bootstrap-based methods (OAL, GLiDeR, HAL G-Comp) achieve near-nominal or conservative, while
minor comments (6)
- §2.5, Table 2: The realized exposure prevalence in the rare-exposure scenario is 9.4%, which is moderate rather than truly rare by pharmacoepidemiology standards. The authors should clarify how this maps to its intended severity.
- §2.3, Table 1: The traditional LASSO pipelines enforce inclusion of all investigator-specified covariates, whereas the other methods do not. This difference is mentioned only in the limitations (§4.5) and should be noted in the methods.
- §3.1, Figure 1: The caption notes that interval widths are not strictly comparable across methods due to different inference types. This is important context that should be stated more prominently.
- §4.5: The statement that higher HAL degrees were 'computationally infeasible' could quantify the attempted max_degree values and approximate runtime.
- Table 3: The Monte Carlo error is stated as ~0.7 pp for coverage near 95%, but the bolding threshold is |Δ| ≤ 1.5 pp. Clarify why 1.5 rather than ~1.4 pp was chosen.
- §4.2: The comparison with prior work is thorough but could note that the Franklin et al. (2017) plasmode also used a null, making the design choice less novel than implied.
Circularity Check
No significant circularity: the plasmode simulation's known null (RD=0) makes bias directly measurable, and all compared methods use independent published implementations.
full rationale
The paper's core evaluation logic is not circular. The plasmode simulation fixes the true risk difference at zero (RD=0) with engineered confounding, so any nonzero estimate is by construction measurable residual-confounding bias—this is a calibration design, not a self-fulfilling prediction. The ten compared pipelines (LASSO, C-TMLE, OAL, GLiDeR, HAL) each use independent published implementations from their respective origin studies (Shortreed 2017, Koch 2018, Benkeser 2016, Ju 2019, Wyss 2024), and the comparison results do not reduce to any self-cited theorem or fitted-then-predicted quantity. Self-citations exist (Karim 2018, 2025 for the LASSO baseline and the NHANES anchor dataset), but these are not load-bearing for the central comparative claims: the anchor dataset is a real public survey, and the LASSO baseline is one of ten pipelines evaluated on equal footing. The authors transparently acknowledge that the calibration comparison is partly confounded by inference method (bootstrap vs. influence-curve SEs, §4.3), and that the PS-based pipelines benefit from correct propensity specification (§4.5)—but these are validity caveats, not circularity. No step in the derivation chain reduces to its inputs by definition.
Assumptions & free parameters
free parameters (5)
- Proxy-burden count coefficient scaling =
5-fold scaling
- PS truncation threshold =
5/(sqrt(n) * log(n))
- OAL tuning-parameter range =
See Web Appendix B
- HAL max_degree =
1
- Bootstrap resamples =
200
assumptions (5)
- domain assumption Conditional exchangeability given (L, Z)
- domain assumption Positivity: 0 < P(A=1|L,Z) < 1
- domain assumption Correctly specified main-terms logistic propensity model
- ad hoc to paper Null effect (RD=0) as a valid calibration benchmark
- domain assumption Proxy variables capture unmeasured confounders
Cite this review
Pith. "Pith review of Which Regularized Propensity-Score and Doubly Robust Methods Are Best Calibrated When Exposures or Outcomes Are Rare? A Plasmode Study of Proxy-Based Confounding Adjustment." pith.science (2026). https://pith.science/paper/NQKPBO6U
@misc{pith2026260707065,
author = {Pith},
title = {Pith review of: Which Regularized Propensity-Score and Doubly Robust Methods Are Best Calibrated When Exposures or Outcomes Are Rare? A Plasmode Study of Proxy-Based Confounding Adjustment},
year = {2026},
howpublished = {\url{https://pith.science/paper/NQKPBO6U}},
note = {Machine review of arXiv:2607.07065}
}
read the original abstract
Purpose. Confounding adjustment in health-care database studies screens large proxy libraries where events per variable are low, straining standard propensity score (PS) methods, and many regularized variable-selection strategies exist (outcome-adaptive LASSO [OAL], group LASSO/GLiDeR, highly adaptive LASSO [HAL]). Yet few comparisons have varied exposure prevalence within such a selection menu, and none pairs it with doubly robust estimation, compute accounting, and a null-RD truth anchor. Methods. We conducted a plasmode simulation anchored on National Health and Nutrition Examination Survey data (2013-2018; 25 investigator-specified covariates, 142 prescription-derived proxies), comparing ten pipelines combining these strategies with inverse probability of treatment weighting (IPTW) and targeted maximum likelihood estimation (TMLE). Three scenarios were evaluated under a known null (true risk difference, RD = 0): frequent, rare-exposure, and rare-outcome. We report bias, standard error (SE), relative error, 95% coverage, and runtime. Results. HAL (G-Computation) had near-zero bias but highly concentrated estimates, giving near-unity coverage and large relative error (106-186%). OAL (IPTW), GLiDeR, and HAL (TMLE) were best calibrated, whereas the regularized-LASSO TMLE pipelines under-covered modestly (91-93%) in the rare scenarios. Under rare exposure, LASSO-IPTW had the largest bias and inflated SE and over-covered (conservatively), problems that TMLE removed. On real data, methods agreed (RD approximately 0.07-0.085). Runtimes spanned <1 s to >16 h. Conclusions. Under a null benchmark, pairing outcome-aware selection (OAL, GLiDeR) or doubly robust estimation (TMLE) with regularized models best balanced bias, calibration, and robustness to rarity. The rare-exposure arm exposed the largest gaps; method choice should weigh the prioritized metric against compute.
Figures
Reference graph
Works this paper leans on
-
[1]
Karim ME, Lei Y. How effective are machine learning and doubly robust estimators in incorporating high-dimensional proxies to reduce residual confounding? Pharmacoepidemiology and Drug Safety . 2025;34(5):e70155
work page 2025
-
[2]
Karim ME, Pang M, Platt R W. Can we train machine learning methods to outperform the high- dimensional propensity score algorithm? Epidemiology. 2018;29(2):191-198
work page 2018
-
[3]
Karim ME. High-dimensional propensity score and its machine learning extensions in residual con- founding control. The American Statistician . 2025;79(1):72-90
work page 2025
-
[4]
National Health and Nutrition Examination Survey Data, 2013–2018
Centers for Disease Control and Prevention, National Center for Health Statistics. National Health and Nutrition Examination Survey Data, 2013–2018. Published online 2018. Accessed July 7, 2026. https://wwwn.cdc.gov/nchs/nhanes/
work page 2013
-
[5]
Wyss R, Yanover C, El-Hay T, et al. Machine learning for improving high-dimensional proxy con- founder adjustment in healthcare database studies: An overview of the current literature. Pharma- coepidemiology and Drug Safety . 2022;31(9):932-943
work page 2022
-
[6]
Ju C, Wyss R, Franklin JM, Schneeweiss S, Häggström J, Laan MJ van der. Collaborative-controlled LASSO for constructing propensity score-based estimators in high-dimensional data. Statistical meth- ods in medical research . 2019;28(4):1044-1063
work page 2019
-
[7]
Wyss R, Laan M van der, Gruber S, et al. Targeted learning with an undersmoothed LASSO propen- sity score model for large-scale covariate adjustment in health-care database studies. American Jour- nal of Epidemiology . 2024;193(11):1632-1640
work page 2024
-
[8]
Outcome-adaptive lasso: Variable selection for causal inference
Shortreed SM, Ertefaie A. Outcome-adaptive lasso: Variable selection for causal inference. Biometrics. 2017;73(4):1111-1122
work page 2017
Show all 31 references
-
[9]
Covariate selection with group lasso and doubly robust estimation of causal effects
Koch B, Vock DM, Wolfson J. Covariate selection with group lasso and doubly robust estimation of causal effects. Biometrics. 2018;74(1):8-17
2018
-
[10]
The highly adaptive lasso estimator
Benkeser D, Van Der Laan M. The highly adaptive lasso estimator. In: 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA) . IEEE; 2016:689-696. 12
2016
-
[11]
Highly adaptive LASSO: Machine learning that provides valid nonparametric inference in realistic models
Butzin-Dozier Z, Qiu S, Hubbard AE, Shi JS, Laan MJ van der. Highly adaptive LASSO: Machine learning that provides valid nonparametric inference in realistic models. medRxiv. Published online 2024
2024
-
[12]
reader reaction to ‘outcome-adaptive lasso: Variable selection for causal inference’ by shortreed and ertefaie (2017)
Jones J, Ertefaie A, Shortreed SM. Rejoinder to “reader reaction to ‘outcome-adaptive lasso: Variable selection for causal inference’ by shortreed and ertefaie (2017). ” Biometrics. 2023;79(1):521-525. doi:10.1111/biom.13681
2017 doi
-
[13]
Flexible collaborative estimation of the average causal effect of a treatment using the outcome-highly-adaptive lasso
Ju C, Benkeser D, Laan M van der. Flexible collaborative estimation of the average causal effect of a treatment using the outcome-highly-adaptive lasso. arXiv preprint arXiv:180606784 . Published online 2018
2018
-
[14]
Robust inference on the average treatment effect using the outcome highly adaptive lasso
Ju C, Benkeser D, Der Laan MJ van. Robust inference on the average treatment effect using the outcome highly adaptive lasso. Biometrics. 2020;76(1):109-118
2020
-
[15]
Constructing inverse probability weights for marginal structural models
Cole SR, Hernán MA. Constructing inverse probability weights for marginal structural models. Amer- ican Journal of Epidemiology . 2008;168(6):656-664
2008
-
[16]
Dealing with limited overlap in estimation of average treatment effects
Crump RK, Hotz VJ, Imbens GW, Mitnik OA. Dealing with limited overlap in estimation of average treatment effects. Biometrika. 2009;96(1):187-199
2009
-
[17]
Diagnosing and responding to violations in the positivity assumption
Petersen ML, Porter KE, Gruber S, Wang Y, Laan MJ van der. Diagnosing and responding to violations in the positivity assumption. Statistical Methods in Medical Research . 2012;21(1):31-54
2012
-
[18]
Doubly robust estimation in missing data and causal inference models
Bang H, Robins JM. Doubly robust estimation in missing data and causal inference models. Biomet- rics. 2005;61(4):962-973
2005
-
[19]
Targeted maximum likelihood learning
Laan MJ van der, Rubin D. Targeted maximum likelihood learning. The International Journal of Biostatistics. 2006;2(1)
2006
-
[20]
Targeted Learning: Causal Inference for Observational and Experimental Data
Laan MJ van der, Rose S. Targeted Learning: Causal Inference for Observational and Experimental Data. Springer; 2011
2011
-
[21]
Data-adaptive selection of the propensity score truncation level for inverse-probability-weighted and targeted maximum likelihood estimators of marginal point treatment effects
Gruber S, Phillips R V, Lee H, Laan MJ van der. Data-adaptive selection of the propensity score truncation level for inverse-probability-weighted and targeted maximum likelihood estimators of marginal point treatment effects. American Journal of Epidemiology . 2022;191(9):1640...
2022 doi
-
[22]
Why does obesity cause diabetes? Cell metabolism
Klein S, Gastaldelli A, Yki-Järvinen H, Scherer PE. Why does obesity cause diabetes? Cell metabolism. 2022;34(1):11-20
2022
-
[23]
Plasmode simulation for the evaluation of pharmacoepidemiologic methods in complex healthcare databases
Franklin JM, Schneeweiss S, Polinski JM, Rassen JA. Plasmode simulation for the evaluation of pharmacoepidemiologic methods in complex healthcare databases. Computational Statistics & Data Analysis. 2014;72:219-226
2014
-
[24]
Regularized regression versus the high- dimensional propensity score for confounding adjustment in secondary database analyses
Franklin JM, Eddings W, Glynn RJ, Schneeweiss S. Regularized regression versus the high- dimensional propensity score for confounding adjustment in secondary database analyses. American Journal of Epidemiology . 2015;182(7):651-659. doi: 10.1093/aje/kwv108
2015 doi
-
[25]
Using simulation studies to evaluate statistical methods
Morris TP, White IR, Crowther MJ. Using simulation studies to evaluate statistical methods. Statis- tics in Medicine . 2019;38(11):2074-2102. 13
2019
-
[26]
A cautionary note for plasmode simulation studies in the setting of causal inference
Shaw PA, Gruber S, Williamson BD, et al. A cautionary note for plasmode simulation studies in the setting of causal inference. Published online 2025. https://arxiv.org/abs/2504.11740
2025 arXiv
-
[27]
Rsimsum: Summarise results from monte carlo simulation studies
Gasparini A. Rsimsum: Summarise results from monte carlo simulation studies. Journal of Open Source Software. 2018;3(26):739
2018
-
[28]
High-dimensional propen- sity score adjustment in studies of treatment effects using health care claims data
Schneeweiss S, Rassen JA, Glynn RJ, A vorn J, Mogun H, Brookhart MA. High-dimensional propen- sity score adjustment in studies of treatment effects using health care claims data. Epidemiology. 2009;20(4):512-522. doi: 10.1097/EDE.0b013e3181a663cc
2009 doi
-
[29]
Covariate selection in high-dimensional propen- sity score analyses of treatment effects in small samples
Rassen JA, Glynn RJ, Brookhart MA, Schneeweiss S. Covariate selection in high-dimensional propen- sity score analyses of treatment effects in small samples. American Journal of Epidemiology . 2011;173(12):1404-1413. doi: 10.1093/aje/kwr001
2011 doi
-
[30]
Comparing the performance of propensity score methods in healthcare database studies with rare outcomes
Franklin JM, Eddings W, Austin PC, Stuart EA, Schneeweiss S. Comparing the performance of propensity score methods in healthcare database studies with rare outcomes. Statistics in Medicine . 2017;36(12):1946-1963. doi: 10.1002/sim.7250
2017 doi
-
[31]
outcome-adaptive lasso: Variable selection for causal inference
Baldé I, Yang YA, Lefebvre G. Reader reaction to “outcome-adaptive lasso: Variable selection for causal inference” by shortreed and ertefaie (2017). Biometrics. 2023;79(1):514-520. 14
2017
Reviewed July 9, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.