Pith. sign in

REVIEW 3 major objections 5 minor 13 references

This paper shows that in double machine learning, coverage of both analytical and bootstrap confidence intervals varies widely with the choice of nuisance machine learner, and often gets worse, not better, when the sample size grows.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 06:39 UTC pith:ENZANFAU

load-bearing objection The simulation benchmark is useful and worth refereeing; the real-data causal claim is overreached and the NN result is invalid due to full-data tuning. the 3 major comments →

arxiv 2607.29456 v1 pith:ENZANFAU submitted 2026-07-31 stat.ML cs.LGstat.AP

Analytical and Bootstrap Confidence Intervals of Double Machine Learning: Simulation studies and an application to rural-urban difference in obesity prevalence

Haozheng Xu , Siyuan Ma , Qingyan Xiang This is my paper
classification stat.ML cs.LGstat.AP MSC 62F4062G0962G20
keywords double machine learningconfidence interval coveragebootstrapnuisance learner choicepartially linear regressionsimulation studyrural-urban obesity disparity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks a practical question: when a researcher uses double machine learning (DML) to estimate a treatment effect, does the choice of machine learning algorithm for the nuisance models actually matter for the confidence intervals, and are bootstrap intervals more reliable than analytical ones? Through simulations under four data-generating processes and five learners (OLS, LASSO, random forest, LightGBM, neural networks), it establishes that coverage of nominal 95% intervals varies widely by learner, and that analytical and bootstrap intervals often fail together. Its counterintuitive central finding is that increasing sample size from 1,000 to 3,000 frequently lowers coverage for both interval types, because bias persists while variance estimates shrink. A county-level analysis of rurality and obesity illustrates the same learner dependence, with a positive rurality–obesity association whose size and significance vary by method. The paper argues learner choice is therefore not a nuisance detail but a primary determinant of whether DML inference is trustworthy.

Core claim

The paper's core claim is that in the partially linear DML model, the machine learner used for the nuisance functions is a first-order determinant of interval coverage, not a tuning detail. Across four DGPs (linear; nonlinear continuous; nonlinear binary; quasi-high-dimensional with noise covariates) and five learners, nominal 95% analytical and bootstrap intervals show coverage ranging from near-100% to 0%. The most striking pattern is that raising N from 1,000 to 3,000 often reduces coverage for both interval types, because the estimator's bias does not disappear while its estimated variance shrinks. Bootstrap percentile intervals do not systematically fix this; they sometimes have much wi

What carries the argument

The load-bearing mechanism is the partially linear DML estimator with cross-fitting: an orthogonal score function is built by residualizing the outcome and treatment on confounders with a chosen machine learner, and the treatment effect is estimated by regressing the outcome residual on the treatment residual. Confidence intervals come either from the asymptotic variance of this score or from bootstrap percentiles over repeated DML fits, and the paper varies the nuisance learner to see how the whole pipeline holds up.

Load-bearing premise

The paper's conclusions rest on the representativeness of its four data-generating processes—covariates are independent or weakly correlated and nuisance functions have fixed known forms—so real applications with stronger confounding, different misspecification, or more correlated covariates may not show the same coverage patterns.

What would settle it

Simulate the same five learners under a DGP with strong confounding and a treatment-assignment mechanism driven by all covariates, not independent margins, then plot analytical and bootstrap coverage at N=1000, 3000, and 10000; if coverage no longer declines with N or the learner ranking changes, the paper's conclusions are specific to its four DGPs.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • A larger sample is not a safety net for DML inference: in several simulation settings, coverage at N=3000 is below coverage at N=1000 for both analytical and bootstrap intervals.
  • Bootstrap confidence intervals are not a reliable alternative to analytical ones; they can be wider, but when bias dominates they reproduce or worsen the coverage failure.
  • Flexible machine learning learners do not automatically dominate simple ones; OLS/LASSO are best in the linear DGP, while random forest is the most stable nonlinear learner in these simulations.
  • Applied reports using DML should report learner-robustness checks, since the real-data conclusions (here, rurality increasing obesity) vary in magnitude and significance across learners.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the coverage decline with N reflects persistent bias rather than a variance artifact, then bias-corrected or debiased nuisance learners, or confidence intervals centered on a bias-corrected estimate, could restore nominal coverage at larger N; the paper does not test this.
  • The results suggest a simple diagnostic for any DML application: compute both analytical and bootstrap intervals across several learners and track the bias-to-standard-error ratio as N grows to flag settings where coverage is likely to degrade.
  • The quasi-high-dimensional scenario's near-zero coverage for most learners indicates that DML inference in high-dimensional nuisance settings may be even more fragile than existing theory suggests; a broader benchmark with structured sparse DGPs could test this.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper studies the finite-sample behavior of double machine learning (DML) confidence intervals in the partially linear regression model. Through simulations across four data-generating scenarios (linear, nonlinear continuous, nonlinear binary, and quasi-high-dimensional), five nuisance learners (OLS, LASSO, Random Forest, LightGBM, Neural Networks), and two sample sizes (N=1000, 3000), the authors compare analytical asymptotic CIs with bootstrap percentile CIs in terms of bias, standard error, interval width, and coverage probability. They report that learner choice substantially affects coverage, that bootstrap CIs are not uniformly better than analytical CIs, and that coverage sometimes decreases as sample size grows. They then apply DML to county-level U.S. data to estimate the effect of rurality (RUCC) on obesity prevalence, reporting a statistically significant increasing effect for both continuous and binary treatments across some learners.

Significance. If the conclusions hold, the paper provides useful practical evidence for applied DML users: flexible nuisance learners do not automatically deliver reliable inference, and analytical and bootstrap confidence intervals can diverge. The simulation design is broad and clearly described, and the use of the established DoubleML R package, a reproducible Quarto workflow, and reported hyperparameter grids (Appendix A) are strengths. The real-data analysis, however, currently supports a causal claim that is undermined by improper neural-network tuning and overbroad causal language. The simulation comparisons would also be more credible with Monte Carlo uncertainty attached to coverage estimates. With targeted revisions, the paper could be a valuable reference for practitioners choosing nuisance learners and inference methods in DML.

major comments (3)
  1. [§2.3, §3.2, §5.1, Appendix A] Nuisance-learner tuning is not described as nested inside the DML cross-fitting folds. In simulations, tuning is said to be performed 'from each data scenario' and selected by outcome/treatment RMSE, but Appendix A does not state whether tuning uses training folds only. In the real-data analysis, §5.1 explicitly says Neural Network hyperparameters were tuned 'over the entire observed dataset.' This violates the cross-fitting principle described in §2.2.1: the nuisance fits and hyperparameters have effectively seen test-fold outcomes and treatments, so the DML estimator and its standard errors are not valid. The NN row in Table 8 (0.358, p<2e-16 for continuous; 1.293, p<4e-13 for binary) is therefore not a reliable DML result. Please nest all tuning inside the cross-validation folds or use a separate validation split, and re-report; if that is not possible, remove Neural Networks from the
  2. [§5.2 / Abstract] The abstract's claim that 'greater rurality has a statistically significant increasing effect on county-level obesity prevalence' is stronger than the valid results support. Excluding the invalid NN results, Table 8 shows significance for continuous RUCC with OLS (p=0.027) and Random Forest (p=0.002), but for binary RUCC OLS is not significant (p=0.062) and only Random Forest is significant (p=0.003). Moreover, this is a non-randomized county-level comparison; DML controls only observed covariates, so unmeasured confounding cannot be ruled out. Recommend rephrasing as a positive conditional association, and explicitly noting that results depend on the treatment definition (continuous vs. binary).
  3. [§3.2 / Tables 2–7] Coverage probability is the paper's primary outcome, but it is reported as a point estimate without Monte Carlo uncertainty. With 500 replications, the binomial standard error is roughly 0.98–2.2 percentage points depending on the true coverage level; differences below about 2–4 percentage points are not distinguishable. For example, in Table 2 the OLS analytical coverage at N=1000 is 93.0 vs. 94.0 bootstrap, and Random Forest analytical vs. bootstrap coverage is 79.2 vs. 79.4. Statements such as 'bootstrap estimates are higher in all cases but Neural Networks' overstate small differences. Please report Wilson intervals or Monte Carlo standard errors for coverage and use them when comparing methods or sample sizes.
minor comments (5)
  1. [Abstract / §4.1] The sentence about coverage decreasing with sample size is repeated almost verbatim twice in the abstract and in bullet 5 of Section 4.1. Please consolidate and state the conditionality ('in many settings, but not uniformly').
  2. [§2.2.3 / §3.2 / Table 8] The number of bootstrap resamples B is only stated in Table 8 (B=100). State B explicitly in the simulation methods and in §2.2.3, since it affects bootstrap interval stability.
  3. [§2.4] The paper claims a reproducible Quarto pipeline, but no link to code or data is provided. Please include a repository URL or supplementary archive; this is especially important for the real-data analysis.
  4. [§3.1.4] Scenario 4 uses nearPD() to adjust the covariance matrix, which means the actual correlation structure deviates from the stated block form. Briefly describe the resulting covariance or note that the adjustment was minor to reassure readers.
  5. [Tables 2–7] Consider adding a small note to each table that coverage probabilities are based on 500 (or 300) replications, with the corresponding Monte Carlo standard error, rather than only mentioning it in the text.

Circularity Check

0 steps flagged

No circularity: the paper's simulation and real-data findings are empirical and rest on external DML theory, not on self-referential derivation.

full rationale

The paper does not claim to derive its main results from first principles in a way that could reduce to its own inputs. The analytical confidence intervals are taken from external DML asymptotic theory (Chernozhukov et al. 2018) as implemented in the DoubleML package, and the bootstrap intervals are standard percentile resampling (Efron and Tibshirani). The simulation coverage probabilities are Monte Carlo evaluations against explicitly specified DGPs, so conclusions such as 'coverage probability decreases with sample size' are empirical findings rather than identities forced by a fitted parameter. The real-data analysis estimates a treatment effect via the DML partialling-out score; the estimate is not a fitted parameter being renamed as a prediction. The one flagged methodological issue is in Section 5.1, where Neural Network hyperparameters were tuned 'over the entire observed dataset'; this can leak information and invalidate the NN inference, making the NN estimate (Table 8) unreliable. However, tuning hyperparameters to minimize nuisance RMSE does not make the subsequent effect estimate equal to the tuning criterion by construction, so this is a validity/leakage concern rather than circularity. There is no load-bearing self-citation: the cited bootstrap theory (Lin and Han) is external, and the cited DoubleML software is an independent implementation. No self-definitional, fitted-input-as-prediction, imported-uniqueness, or ansatz-smuggling pattern is present. The appropriate finding is therefore no significant circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 0 invented entities

The central claim is empirical and does not introduce new entities or parameters. It relies on standard DML theory, the correctness of the DoubleML implementation, and the assumption that the simulation DGP space is representative of real applications.

axioms (3)
  • domain assumption DML asymptotic normality and Neyman orthogonality hold for the PLR model across all learners and DGPs
    The analytical confidence intervals are valid only if the regularity conditions in Chernozhukov et al. (2018) hold. The paper does not verify these conditions for each learner; low coverage rates for some learners suggest violations.
  • ad hoc to paper The DoubleML package's analytical variance estimator correctly implements DML theory
    Section 2.4 states all analytical CIs come from the DoubleML package; the paper does not independently verify the implementation.
  • domain assumption Unconfoundedness and correct nuisance model specification hold in the real-data analysis
    Section 5.2 interprets DML estimates as causal effects of rurality on obesity, requiring that observed ACS covariates capture all confounders and that the PLR model is correctly specified; the paper does not discuss sensitivity to hidden confounding.

pith-pipeline@v1.3.0-daily-deepseek · 16749 in / 11039 out tokens · 116200 ms · 2026-08-03T06:39:37.554852+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Analytical and Bootstrap Confidence Intervals of Double Machine Learning: Simulation studies and an application to rural-urban difference in obesity prevalence." pith.science (2026). https://pith.science/paper/ENZANFAU

@misc{pith2026260729456,
  author       = {Pith},
  title        = {Pith review of: Analytical and Bootstrap Confidence Intervals of Double Machine Learning: Simulation studies and an application to rural-urban difference in obesity prevalence},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ENZANFAU}},
  note         = {Machine review of arXiv:2607.29456}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Double Machine Learning (DML) is a popular approach for treatment effect estimation in various settings, which allows a wide range of flexible machine learning methods to be used for nuisance parameter estimation while preserving valid inference. In practice, however, applied researchers must choose among many machine learning algorithms for nuisance models, and the impact of this choice on the variance estimation of DML is not well characterized. We conduct a comprehensive simulation study to compare the coverage probability of DML confidence intervals across different machine learning algorithms. In this study, we compare (1) analytical confidence intervals derived by DML theory versus (2) bootstrap confidence interval. We use a set of learners including ordinary least squares, LASSO, Random Forest, LightGBM, and Neural Networks under different data generation settings. We evaluate the performance across difference settings by bias, confidence interval width, and most importantly, coverage probability. Our results show substantial variability in coverage performance across analytical and bootstrap confidence intervals, highlighting that learner choice plays a critical role in reliable DML inference. Surprisingly, we find that in many settings, when sample size increases, the coverage probability of both DML analytical and bootstrap confidence interval decreases. We further investigate coverage probabilities using a real dataset on rural urban differences among U.S. counties. The real data analysis discovers that (1) the model performance still varies by the learner choices and (2) greater rurality has a statistically significant increasing effect on county level obesity prevalence.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

13 extracted references · 4 canonical work pages · 1 internal anchor

  1. [5]

    Estimating Causal Effects with Double Machine Learning -- A Method Evaluation

    “Estimating Causal Effects with Double Machine Learning -- A Method Evaluation.” doi:10.48550/ARXIV .2403.14385. Fuhr, Jonathan, and Dominik Papies

    Pith/arXiv arXiv doi:10.48550/arxiv.2403.14385

  2. [6]

    Double Machine Learning meets Panel Data -- Promises, Pitfalls, and Potential Solutions

    “Double Machine Learning Meets Panel Data -- Promises, Pitfalls, and Potential Solutions.” doi:10.48550/ARXIV .2409.01266. Higham, N. J

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2409.01266

  3. [9]

    Statistical and Machine Learning Methods for Evaluating Trends in Air Quality under Changing Meteorological Conditions

    “Statistical and Machine Learning Methods for Evaluating Trends in Air Quality under Changing Meteorological Conditions.” Atmospheric Chemistry and Physics 22(16):10551–66. doi:10.5194/acp- 22-10551-2022. Robinson, P. M

    doi:10.5194/acp- 2022

  4. [1988]

    Root -N-Consistent Semiparametric Regression

    “Root -N-Consistent Semiparametric Regression.” Econometrica 56(4):931. doi:10.2307/1912705. Saco, Gabriel

    doi:10.2307/1912705

  5. [1996]

    Regression Shrinkage and Selection Via the Lasso

    “Regression Shrinkage and Selection Via the Lasso.” Journal of the Royal Statistical Society Series B: Statistical Methodology 58(1):267–88. doi:10.1111/j.2517-6161.1996.tb02080.x. U.S. Census Bureau

    arXiv 1996

  6. [2001]

    Random Forests

    “Random Forests.” Machine Learning 45(1):5–32. doi:10.1023/A:1010933404324. CDC

    doi:10.1023/a:1010933404324

  7. [2002]

    Computing the Nearest Correlation Matrix --a Problem from Finance

    “Computing the Nearest Correlation Matrix --a Problem from Finance.” IMA Journal of Numerical Analysis 22(3):329–43. doi:10.1093/imanum/22.3.329. Ke, Guolin, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu

    doi:10.1093/imanum/22.3.329

  8. [2018]

    Double/Debiased Machine Learning for Treatment and Structural Parameters

    “Double/Debiased Machine Learning for Treatment and Structural Parameters.” The Econometrics Journal 21(1):C1–68. doi:10.1111/ectj.12097. Efron, Bradley, and R. J. Tibshirani

    doi:10.1111/ectj.12097

  9. [2019]

    Mlr3: A Modern Object-Oriented Machine Learning Framework in R

    “Mlr3: A Modern Object-Oriented Machine Learning Framework in R.” Journal of Open Source Software 4(44):1903. doi:10.21105/joss.01903. Lin, Ziming, and Fang Han

    doi:10.21105/joss.01903 1903

  10. [2020]

    Double Machine Learning with Gradient Boosting and Its Application to the Big N Audit Quality Effect

    “Double Machine Learning with Gradient Boosting and Its Application to the Big N Audit Quality Effect.” Journal of Econometrics 216(1):268–83. doi:10.1016/j.jeconom.2020.01.018. Appendix Appendix A. Simulation Grid Tuning results Learner Hyperparameter Candidate Values LightGBM Num_leaves 15, 31, 63 Learning_rate 0.05, 0.10, 0.15 Neural Networks Size 3, 8...

    work page doi:10.1016/j.jeconom.2020.01.018 2020

  11. [2022]

    Bootstrap vs Asymptotic Variance Estimation When Using Propensity Score Weighting with Continuous and Binary Outcomes

    “Bootstrap vs Asymptotic Variance Estimation When Using Propensity Score Weighting with Continuous and Binary Outcomes.” Statistics in Medicine 41(22):4426–43. doi:10.1002/sim.9519. Bach, Philipp, Victor Chernozhukov, Malte S. Kurz, Martin Spindler, and Klaassen Sven

    work page doi:10.1002/sim.9519

  12. [2024]

    Improving the Finite Sample Estimation of Average Treatment Effects Using Double/Debiased Machine Learning with Propensity Score Calibration

    “Improving the Finite Sample Estimation of Average Treatment Effects Using Double/Debiased Machine Learning with Propensity Score Calibration.” https://arxiv.org/abs/2409.04874v2. Breiman, Leo

    Pith/arXiv arXiv

  13. [2025]

    Double Robust Variance Estimation with Parametric Working Models

    “Double Robust Variance Estimation with Parametric Working Models.” Biometrics 81(2):ujaf054. doi:10.1093/biomtc/ujaf054. Tibshirani, Robert

    work page doi:10.1093/biomtc/ujaf054