REVIEW 3 major objections 6 minor 27 references
Targeted Learning Estimation of Sampling Variance for Improved Inference
T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read The paper proposes a one-step targeted variance estimator whose confidence intervals for the log causal risk ratio reach nominal 0.95 coverage where the standard influence-function variance underestimates.
desk verdict Plausible new EIF-based variance estimator for log(CRR), but the paper overclaims asymptotic efficiency for the algorithm it actually implements, and the formulas need a close check before the method can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the efficient influence function of the variance parameter $\Sigma^2(P) = P\{D^*_{\Psi,P}\}^2$, decomposed orthogonally into components in the tangent spaces of the covariate distribution, the outcome regression, and the propensity score (Theorem 1, Eq. 9). It is used twice: its clever covariates $H_{\bar{Q}}$ and $H_g$ define a one-dimensional universal least favorable parametric submodel along which both the outcome regression and propensity score are fluctuated in small logistic steps, and solving its estimating equation is what makes the resulting variance plug-in estimator asymptotically efficient. The one-step TMLE is the engine that carries the argument: one targeted update, rather than iterative refitting, corrects bias for the variance without the additional variance inflation that the paper attributes to iterative TMLE.
What would settle it
Numerically verify Theorem 1 by finite-difference Gateaux differentiation of $\Sigma^2(P)$ along one-parameter submodels for the outcome regression and propensity score on a small simulated dataset; if the numerical derivative disagrees with Eq. 9, especially in the $Q_W$ component, the targeting step is misaligned. Independently run the paper's $n=100$, $\beta_p=1$, $\beta_{\psi_0}=0$ simulation 1000 times: if Wald intervals using the one-step variance estimator cover the true log(CRR) less than 90% of the time, the claimed restoration of nominal coverage fails.
Extended reading notes
Core claim
The paper claims that the anti-conservatism of the standard influence-function variance estimator is not an inevitable small-sample artifact but a correctable bias: by defining the variance of the efficient influence function of log(CRR) as a statistical parameter and estimating it with a one-step TMLE, the variance estimator tracks the true sampling variance even when propensity scores approach 0 or 1. The central object is the analytic variance formula, which averages terms with 1/g(1|W) and 1/g(0|W) in the denominators, so rare treatment and outcome patterns are explicitly integrated rather than being missed through indicator functions. The one-step TMLE, built on the derived EIF, targets this variance directly using the universal least favorable submodel, avoiding the overfitting the authors attribute to iterative TMLE. Simulations across effect sizes 0, 0.5, and 2, sample sizes 100 to 1000, and six positivity levels show coverage near 0.95 and Type I error near 0.05, with the empirical estimator dropping below 0.85 coverage and inflating Type I error in extreme cases.
Load-bearing premise
The derived efficient influence function in Theorem 1 must be correct; the appendix obtains component influence functions from a book chapter and the printed equation contains apparent typographical slips (for example, a $\psi_1$ where $\psi_0$ is expected), so an error here would move the targeting update in the wrong direction and the claimed efficiency would collapse.
Editorial extensions
If this is right
- Point estimates of log(CRR) from TMLE will come with standard errors that are not systematically too small when treatment probabilities approach zero.
- Type I error rates for testing no causal effect will stay near 0.05 in small samples, instead of rising as positivity degrades.
- The same derivation template—derive the EIF of the variance of the influence function and target it with a universal least favorable submodel—applies to other causal parameters such as treatment-specific means, risk differences, and odds ratios.
- Because the estimator is a plug-in with an analytic form, it can be computed without iteratively refitting the data, avoiding the variance inflation associated with iterative TMLE.
- The derived influence function of the variance also opens the possibility of constructing confidence intervals for the variance itself, which could be used in adaptive experiments to bound Type I error.
Reading between the lines
- Extension: the gap between the empirical influence-function variance and the targeted variance could serve as a data-driven positivity diagnostic—a large gap flags samples where standard inference is unreliable.
- Extension: in rare-outcome longitudinal analyses where nonparametric bootstrap has been observed to beat the empirical influence-function variance, the one-step targeted variance may provide a computationally cheaper alternative without bootstrap resampling.
- Extension: simultaneously targeting the causal estimand and its variance in a joint TMLE is a direct next step; if realized, it would give jointly efficient estimation and inference.
- Extension: the ratio of the one-step TMLE variance to the empirical IF variance could be used to calibrate existing software outputs, giving applied users a finite-sample correction without changing their estimator.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a one-step targeted maximum likelihood estimator (TMLE) for the variance of the efficient influence function of the log causal risk ratio (log(CRR)). Using a one-dimensional universally least favorable submodel, the authors update initial estimates of the outcome regression and propensity score, then plug the updated estimates into an analytic variance formula. The manuscript derives the efficient influence function of this variance parameter, states an asymptotic efficiency theorem, and compares the proposed estimator with the empirical influence-function variance, a simple substitution estimator, and an iterative TMLE in simulations across sample sizes, effect sizes, and positivity violations. The reported simulations show that confidence intervals using the one-step TMLE achieve coverage closer to the nominal 0.95 and lower Type-I error than the empirical influence-function variance estimator, especially in small samples and under near-positivity violations. A real-data-based simulation using the WASH Benefits Bangladesh study illustrates the method.
Significance. If the theoretical claims were fully established, the paper would provide a principled way to improve inference for causal risk ratios in finite samples and near-positivity settings, where the standard influence-function variance is known to be anti-conservative. The numerical results are plausible and the code is released, which is a strength. However, the central theoretical contribution is an asymptotic efficiency claim for the proposed one-step TMLE, and that claim is not currently supported: the algorithm as described does not solve the EIF estimating equation, the printed EIF formulas contain apparent typos, and the conditions of the efficiency theorem are not met by the simulation implementation. The finite-sample coverage improvements may well be real, but the manuscript does not yet substantiate the stated efficiency result, and the practical value depends on correcting the targeting step and verifying the formulas against the code.
major comments (3)
- [Section 3.2.4, Step 5; Theorem 2; Appendix C] The stopping criterion in Step 5, |P_n D^{(i)}_{Σ2,P_n}| > σ̂_n √n log n, is dimensionally inconsistent with the requirement P_n D*_{Σ2,P_n} = o_p(n^{-1/2}) that is used in the von Mises expansion in Appendix C. For a mean-zero function, the sample mean P_n D* has typical scale σ̂_n/√n, so the threshold is larger than the typical scale by a factor of order n log n. As a result, the loop will stop after one or very few dε = 0.001 updates, leaving P_n D* at O_p(n^{-1/2}) rather than o_p(n^{-1/2}). The implemented estimator therefore does not solve the EIF estimating equation, and Theorem 2 (Eq. 15) does not apply to the procedure as described. The authors should replace the stopping rule with one that targets |P_n D*| ≤ tol·n^{-1/2} and re-run the simulations and code checks.
- [Eq. (10) and Corollary 1 (Eqs. 12-14)] The EIF formulas as printed cannot serve as the validated basis for the targeting step. In Eq. (10), the last term of D*_{Σ2,QW,P} contains (Q̄_1/ψ_1 − Q̄_0/ψ_1)², where the second denominator should be ψ_0. More importantly, the clever covariates H_{1,Q̄,P} and H_{0,Q̄,P} in Eq. (14) are written with P_0 expectations rather than expectations under the current distribution P_ε; the universal least favorable submodel property in Eq. (19) requires the current distribution, so an implementation that literally uses P_0 would break the identity in Eq. (19). These appear to be correctable typos, but they must be fixed and verified against the released code before the targeting update can be trusted. The appendix also defers the component EIFs to a book chapter references, so the manuscript alone does not allow an independent check of the derivation.
- [Appendix C and Section 4.1] The remainder analysis in Appendix C.1 concludes that asymptotic efficiency requires both Q̄ and g to converge at an n^{-1/4} L2 rate, achievable by HAL, and the paper explicitly notes the estimator is not doubly robust. However, the simulation implementation uses SuperLearner libraries (GLM, elastic net, BART, GAM) rather than HAL, and no rate condition is verified for these libraries in the simulation settings. Consequently, the coverage and Type-I error results in Section 4.2 are finite-sample empirical findings, not consequences of Theorem 2; the manuscript should state this distinction clearly and either add HAL-based simulations or temper the asymptotic efficiency claim.
minor comments (6)
- [Eq. (6) and Eq. (7)] The second term in Eq. (6) and the second term in Eq. (7) are missing the square on ψ_0 in the denominator; compare with Eq. (4) where the factor is 1/(ψ_0)². In Eq. (7), the notation g*_n(0|W) should be g^{(**)}_n(0|W) for consistency.
- [Section 3.2.4] The step size dε = 0.001 is introduced without any sensitivity analysis or guidance on choosing it; the manuscript should report whether results are sensitive to this tuning choice.
- [References and citations] The introduction attributes the iterative TMLE for the variance of the treatment-specific mean to Coyle et al. and cites reference [9], but [9] is the van der Laan and Rose book; the specific chapter or paper by Coyle and colleagues should be cited directly.
- [Appendix B] The derivation says the component EIFs were obtained using the method in 'A.3 in the targeted learning book [10]', but reference [10] is the van der Laan and Gruber paper; the intended book reference should be corrected.
- [Table 1 and Section 4.1] Table 1 reports the proportion of observations with propensity scores truncated at 0.025, but the text in Section 4.1 says truncation is within [0.025, 0.975]; the table caption should refer to the estimated propensity scores rather than g_0 and should clarify the truncation interval.
- [Section 5] The real-data simulation says 10,000 repeated samples of size 500 are drawn, but the analysis is then described as 'based on 140 data samples'; the relationship between these numbers should be explained.
Circularity Check
No significant circularity: the variance estimator is a plug-in TMLE for a defined estimand, and the simulation comparisons use external Monte Carlo truth.
full rationale
The paper defines the target parameter Σ²(P) as the variance of the efficient influence function of log(CRR) (Eq. 4), derives the EIF of this parameter (Theorem 1), and constructs a one-step TMLE as a plug-in estimator with a targeting step (Eqs. 7, 16-19). Using the EIF of the target parameter to build an efficient estimator of that parameter is the standard TMLE construction, not a circular reduction: the EIF is the derivative of the same functional, and estimating a variance from its EIF is not 'predicting' the input. The claimed improved coverage and Type I error are evaluated in simulations relative to a Monte-Carlo approximation of Σ²(P0) (Section 4), which is external to the estimator's construction. The paper does rely on prior work by the same authors for the universal least favorable submodel [8] and for component EIFs [9,10], but those citations supply general methodology or previously derived EIFs for other parameters (treatment-specific means), not the paper's conclusion; they are not load-bearing in the sense of assuming the target result. The stopping-criterion mismatch with the Theorem 2 condition and apparent EIF typos are correctness risks, not circularity. No step reduces by definition, by fitted input, or by self-citation chain to its own inputs.
Assumptions & free parameters
free parameters (4)
- step_size_depsilon =
0.001
- propensity_truncation_sim =
0.025, 0.975
- outcome_truncation_sim =
0.001, 0.999
- real_data_truncation =
0.036 (5/(sqrt(n) log(n)))
assumptions (5)
- domain assumption The statistical model M is nonparametric, so the tangent space is the full Hilbert space of mean-zero functions.
- domain assumption Standard causal identification assumptions: no unmeasured confounding, positivity, and consistency.
- standard math TMLE regularity conditions: Donsker class, second-order remainder o_p(n^{-1/2}), and L2 consistency.
- ad hoc to paper Initial estimators of Q and g converge to their true values at L2 rate n^{-1/4} or faster (achieved by HAL).
- standard math The universal least favorable submodel theory of van der Laan and Gruber (2016) is valid.
Cite this review
Pith. "Pith review of Targeted Learning Estimation of Sampling Variance for Improved Inference." pith.science (2026). https://pith.science/paper/BZJ7X3OQ
@misc{pith2026250510624,
author = {Pith},
title = {Pith review of: Targeted Learning Estimation of Sampling Variance for Improved Inference},
year = {2026},
howpublished = {\url{https://pith.science/paper/BZJ7X3OQ}},
note = {Machine review of arXiv:2505.10624}
}
read the original abstract
For robust statistical inference it is crucial to obtain a good estimator of the variance of the proposed estimator of the statistical estimand. A commonly used estimator of the variance for an asymptotically linear estimator is the sample variance of the estimated influence function. This estimator has been shown to be anti-conservative in limited samples or in the presence of near-positivity violations, leading to elevated Type-I error rates and poor coverage. In this paper, capitalizing on earlier attempts at targeted variance estimators, we propose a one-step targeted variance estimator for the causal risk ratio (CRR) in scenarios involving treatment, outcome, and baseline covariates. While our primary focus is on the variance of log(CRR), our methodology can be extended to other causal effect parameters. Specifically, we focus on the variance of the IF for the log relative risk (log(CRR)) estimator, which requires deriving the efficient influence function for the variance of the IF as the basis for constructing the estimator. Several methods are available to develop efficient estimators of asymptotically linear parameters. In this paper, we concentrate on the so-called one-step targeted maximum likelihood estimator, which is a substitution estimator that utilizes a one-dimensional universal least favorable parametric submodel when updating the distribution. We conduct simulations with different effect sizes, sample sizes and levels of positivity to compare the estimator with existing methods in terms of coverage and Type-I error. Simulation results demonstrate that, especially with small samples and near-positivity violations, the proposed variance estimator offers improved performance, achieving coverage closer to the nominal level of 0.95 and a lower Type-I error rate.
Figures
Figures from the paper (10 more)
Reference graph
Works this paper leans on
-
[1]
The Highly Adaptive Lasso Estimator
David Benkeser and Mark van der Laan. “The Highly Adaptive Lasso Estimator”. In: IEEE International Conference on Data Science and Advanced Analytics 2016 (2016), pp. 689–
work page 2016
-
[2]
Peter J. Bickel et al. Efficient and Adaptive Inference in Semiparametric Models . Springer Series in Statistics. New York: Springer, 1998.isbn: 978-0-387-98518-3. doi: 10.1007/978- 1-4612-1842-5
doi:10.1007/978- 1998
-
[3]
One-step targeted maximum likelihood estimation for time-to-event outcomes
Weixin Cai and Mark J. van der Laan. “One-step targeted maximum likelihood estimation for time-to-event outcomes”. In: Biometrics 76.3 (2020), pp. 722–733. doi: 10.1111/biom. 13172. url: https://doi.org/10.1111/biom.13172
doi:10.1111/biom 2020
-
[4]
XGBoost: A Scalable Tree Boosting System
Tianqi Chen and Carlos Guestrin. “XGBoost: A Scalable Tree Boosting System”. In: Pro- ceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM. 2016, pp. 785–794
work page 2016
-
[5]
BART: Bayesian Additive Regression Trees
Hugh A Chipman, Edward I George, and Robert E McCulloch. “BART: Bayesian Additive Regression Trees”. In: The Annals of Applied Statistics 4.1 (2010), pp. 266–298
work page 2010
-
[6]
sl3: Modern Pipelines for Machine Learning and Super Learning
Jeremy R Coyle et al. sl3: Modern Pipelines for Machine Learning and Super Learning . R package version 1.4.2. 2021. doi: 10.5281/zenodo.1342293. url: https://doi.org/10. 5281/zenodo.1342293
-
[7]
tmle: An R Package for Targeted Maximum Like- lihood Estimation
Susan Gruber and Mark J. van der Laan. “tmle: An R Package for Targeted Maximum Like- lihood Estimation”. In: Journal of Statistical Software 51.13 (2012). doi:10.18637/jss.v051.i13, pp. 1–35. url: https://www.jstatsoft.org/v51/i13/
-
[8]
Mark van der Laan and Susan Gruber. “One-Step Targeted Minimum Loss-based Esti- mation Based on Universal Least Favorable One-Dimensional Submodels”. In: The in- ternational journal of biostatistics 12.1 (May 2016), pp. 351–378. issn: 2194-573X. doi: 10 . 1515 / ijb - 2015 - 0054. url: https : / / www . ncbi . nlm . nih . gov / pmc / articles / PMC4912007...
work page 2016
Show all 27 references
-
[9]
Targeted Learning in Data Science: Causal Inference for Complex Longitudinal Studies
Mark J van der Laan and Sherri Rose. Targeted Learning in Data Science: Causal Inference for Complex Longitudinal Studies . Springer Series in Statistics. New York: Springer, 2018. isbn: 978-3-319-65303-9. doi: 10.1007/978-3-319-65304-6
2018 doi
-
[10]
Targeted Minimum Loss Based Estimation of an Intervention Specific Mean Outcome
Mark J. van der Laan and Susan Gruber. “Targeted Minimum Loss Based Estimation of an Intervention Specific Mean Outcome”. In: The Berkeley Electronic Press 290 (2011). url: https://biostats.bepress.com/ucbbiostat/paper290/
2011
-
[11]
Super Learner
Mark J. van der Laan, Eric C. Polley, and Alan E. Hubbard. “Super Learner”. In: Statistical Applications in Genetics and Molecular Biology 6.1 (Sept. 2007). issn: 1544-6115. doi: 10.2202/1544-6115.1309
2007
-
[12]
van der Laan and Sherri Rose
Mark J. van der Laan and Sherri Rose. Targeted Learning: Causal Inference for Obser- vational and Experimental Data . Springer Series in Statistics. New York: Springer-Verlag,
-
[13]
Targeted maximum likelihood learning
Mark J. van der Laan and Daniel Rubin. “Targeted maximum likelihood learning”. In: The International Journal of Biostatistics 2.1 (2006), pp. 1–40. doi: 10.2202/1557-4679.1043
2006
-
[14]
Evaluating the robustness of targeted maximum likelihood estimators via realistic simulations in nutrition intervention trials
Haodong Li et al. “Evaluating the robustness of targeted maximum likelihood estimators via realistic simulations in nutrition intervention trials”. In: Statistics in medicine 41.12 (May 2022), pp. 2132–2165. issn: 0277-6715. doi: 10.1002/sim.9348. url: https://www. ncbi.nlm.ni...
2022 doi
-
[15]
Effects of Water Quality, Sanitation, Handwashing, and Nutritional Interventions on Diarrhoea and Child Growth in Rural Bangladesh: A Cluster Randomised Controlled Trial
Stephen P Luby et al. “Effects of Water Quality, Sanitation, Handwashing, and Nutritional Interventions on Diarrhoea and Child Growth in Rural Bangladesh: A Cluster Randomised Controlled Trial”. In: The Lancet Global Health 6.3 (2018), e302–e315
2018
-
[16]
On the asymptotic distribution of differentiable statistical functions
Richard von Mises. “On the asymptotic distribution of differentiable statistical functions”. In: The Annals of Mathematical Statistics 18.3 (1947), pp. 309–348. doi: 10.1214/aoms/ 1177730380. url: https://projecteuclid.org/euclid.aoms/1177730380
1947
-
[17]
Applying the causal roadmap to longitudinal national Danish registry data: a case study of second-line diabetes medication and dementia
Nerissa Nance et al. Applying the causal roadmap to longitudinal national Danish registry data: a case study of second-line diabetes medication and dementia. arXiv:2310.03235 [stat]. Oct. 2023. doi: 10.48550/arXiv.2310.03235. url: http://arxiv.org/abs/2310.03235 (visited on 03...
-
[18]
Causality: Models, Reasoning and Inference
Judea Pearl. Causality: Models, Reasoning and Inference . 1st. Vol. 19. 2. Cambridge, UK: Cambridge University Press, 2000, p. 3
2000
-
[19]
Diagnosing and responding to violations in the positivity as- sumption
Maya L Petersen et al. “Diagnosing and responding to violations in the positivity as- sumption”. In: Statistical methods in medical research 21.1 (Feb. 2012), pp. 31–54. issn: 0962-2802. doi: 10.1177/0962280210386207 . url: https://www.ncbi.nlm.nih.gov/ pmc/articles/PMC4107929...
2012 doi
-
[20]
Improving Propensity Score Estimators’ Robustness to Model Misspecification Using Super Learner
Romain Pirracchio, Maya L. Petersen, and Mark van der Laan. “Improving Propensity Score Estimators’ Robustness to Model Misspecification Using Super Learner”. In: Amer- ican Journal of Epidemiology 181.2 (Jan. 2015), pp. 108–119. issn: 0002-9262. doi: 10. 1093/aje/kwu253. url:...
2015
-
[21]
Polley and Alan E
Eric C. Polley and Alan E. Hubbard. SuperLearner: A prediction algorithm for combining models. R package version 2.0-26. 2019. url: https://CRAN.R- project.org/package= SuperLearner
2019
-
[22]
R: A Language and Environment for Statistical Computing
R Core Team. R: A Language and Environment for Statistical Computing . R Foundation for Statistical Computing. Vienna, Austria, 2017. url: https://www.R-project.org/
2017
-
[23]
One-step targeted maximum likelihood estimation for targeting cause-specific absolute risks and survival curves
H. C. W. Rytgaard and M. J. van der Laan. “One-step targeted maximum likelihood estimation for targeting cause-specific absolute risks and survival curves”. In: Biometrika 111.1 (2024), pp. 129–145. doi: 10.1093/biomet/asad033 . url: https://doi.org/10. 1093/biomet/asad033
2024 doi
-
[24]
Robust variance estimation and inference for causal effect estimation
Linh Tran et al. Robust variance estimation and inference for causal effect estimation . en. arXiv:1810.03030 [math, stat]. Oct. 2018. url: http : / / arxiv . org / abs / 1810 . 03030 (visited on 10/05/2022)
2018 arXiv
-
[25]
Anastasios A. Tsiatis. Semiparametric Theory and Missing Data. Springer Series in Statis- tics. New York: Springer, 2006. isbn: 978-0-387-29180-2. doi: 10.1007/0-387-36276-2 . 17 A Derivation of target parameter - variance of IC of log(CRR) We derive the analytic form of the v...
2006 doi
-
[696]
doi: 10.1109/DSAA.2016.93
2016 doi
- [2011]
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.