Pith. sign in

REVIEW 5 major objections 8 minor 19 references

Finite Sample Hypothesis Tests for Stacked Estimating Equations

T0 review · 5 major / 8 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper introduces a bootstrap hypothesis test for stacked estimating equations that uses repeated sample splitting and averaged p-values to achieve exact or nearly exact finite-sample level, with power close to the uniformly most…

desk verdict Solid bootstrap test for stacked estimating equations, but the exactness claim only holds in a narrow case and the real-data analysis is missing key null-simulation details. read the letter →

arxiv 1908.03968 v1 pith:MGGHYICQ submitted 2019-08-11 stat.ME

Eli S. Kravitz , Raymond J. Carroll , David Ruppert This is my paper · ORCID
classification stat.ME MSC 62F0362F4062G0962J12
keywords stackedestimatingequationsbootstraphypothesistestsamplesplittingp-valuelotteryfinite-samplelevelM-estimationresidualresamplingphysicalactivityscore
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Stacked estimating equations solve two parameters jointly when the second estimating equation depends on the first parameter; Wald tests based on the asymptotic covariance matrix can have unreliable level in small samples. The paper proposes a bootstrap test: split the sample repeatedly, estimate the first parameter on one half, plug it into the second equation on the other half, take the mean of the resulting p-values, and compare that mean to an empirical null distribution obtained by simulating responses under the null. When the null response distribution is known, the test is exact up to simulation error; with residual resampling it has approximately the nominal level. In Gaussian linear simulations with $n=100$, the maximum power difference from the uniformly most powerful F-test is 0.015. The paper also cites a companion result that the sample-splitting estimator is asymptotically equivalent to the stacked estimating equation estimator in parametric models.

What carries the argument

The load-bearing object is the averaged split p-value $p_{H_1}=B^{-1}\sum_{b=1}^B p_b$, where each $p_b$ comes from a fresh Bernoulli(1/2) sample split. Its null distribution is not uniform even though each individual $p_b$ is, so the paper simulates the null distribution by bootstrap: draw responses $Y^*$ independent of the predictors, or residual-resample with replacement, then rerun the whole splitting and estimation procedure many times and set $p^*=\hat F(p_{H_1})$, the empirical fraction of null $p_{H_1}$ values below the observed value. This empirical CDF of null p-values carries the inference, replacing the asymptotic covariance matrix of the stacked estimator. For parametric models, the paper invokes the companion asymptotic expansion showing that the split estimator has the same limiting distribution as the stacked estimating equation estimator, which justifies the method while the bootstrap handles finite-sample calibration.

What would settle it

Simulate null data from the real-data logistic model: $Y$ is a binary outcome whose probability depends on covariates $Z$ only, with $\beta_0=0$ and with the activity predictors $X$ correlated with $Z$; run the paper's bootstrap algorithm exactly as described, regenerating $Y^*$ without $Z$. If the rejection rate over many null datasets exceeds the nominal level by more than simulation error, the claimed level fails for this null model.

Watch

Extended reading notes

Core claim

The central claim is that finite-sample inference for stacked estimating equations can be made exact, or nearly exact, by a bootstrap that operates on split-sample p-values rather than on asymptotic variance estimates. The method estimates $\theta$ from the training half using $\sum_i \delta_{ib}\Psi(W_i,\theta)=0$, then estimates $\beta$ from the test half using $\sum_i (1-\delta_{ib})K(Y_i,\hat\theta_b,\beta)=0$, producing an individual p-value $p_b$; the test statistic is $p_{H_1}=B^{-1}\sum_b p_b$. Under the null, responses are regenerated either from the known error distribution, which gives an exact test up to simulation error, or by residual resampling, which gives an approximately valid level. The paper reports simulated level near nominal for eight error distributions and power within 0.015 of the F-test, and it applies the procedure to test whether a physical-activity score predicts mortality.

Load-bearing premise

The load-bearing premise is that the bootstrap null distribution used to calibrate the test matches the real null model under which the data were generated; in the real-data example the null model includes additional adjustment variables, while the paper's described null simulation regenerates responses without those variables, so the stated p-value may not have the claimed exact level unless the null simulation is adapted.

Editorial extensions

If this is right

  • At samples as small as $n=100$, the bootstrap test keeps rejection rates near the nominal level across normal, $t$, and Laplace errors, so practitioners no longer need to rely on asymptotic standard errors in this setting.
  • The test's power tracks the uniformly most powerful F-test; the largest simulated gap is 0.015, and residual resampling costs only a small amount of power compared with knowing the error distribution.
  • Because separate p-values from single splits can range from 0 to 1, averaging over $B\approx 50$ splits removes the p-value lottery while retaining proper level.
  • For parametric models, the sample-splitting estimator and the stacked estimating equation estimator have the same limiting distribution, so the bootstrap output can be interpreted as inference for the stacked solution itself.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the null simulation is replaced by one that preserves the covariate structure $Z$ used in the real-data logistic model, the claimed level should be checked separately; the paper describes a null simulation without $Z$, so the reported $p=0.003$ would be on firmer ground with a $Z$-preserving null bootstrap.
  • Editorial inference: the same averaged-p-value bootstrap should extend to generalized estimating equations and other two-stage estimators where asymptotic variance estimates are unreliable, provided a credible null response distribution can be simulated; this is a natural next step the paper does not pursue.
  • Editorial inference: one can benchmark the method's calibration directly by comparing the empirical distribution of $p_{H_1}$ under a known null with the bootstrap's estimate $\hat F$; any gap would show exactly where finiteness bites.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 8 minor

Summary. This paper proposes a bootstrap-based hypothesis test for parameters in stacked estimating equations. The procedure repeatedly splits the sample, estimates the first-stage parameter θ on the training portion, then estimates the second-stage parameter β on the test portion, and averages the per-split p-values. The null distribution of this averaged p-value is obtained by simulating responses under the null and repeating the splitting procedure. The authors claim the test is exact up to simulation error when the null imposes no nuisance parameters and the error distribution is known, and they present simulations showing correct level and power close to the F-test. A data analysis builds a physical activity score and tests its association with mortality.

Significance. The idea of combining repeated sample splitting with a bootstrap null distribution is useful for small-sample inference in estimating-equation settings where Wald tests fail. The exactness argument for the strong null with known errors is logically sound, and the level/power simulations in Tables 2–5 are extensive and largely support the procedure's practical performance in the linear setting. However, the paper’s central claims are weakened by the absence of a derivation of the limiting distribution (deferred to a companion paper), a numerical mismatch in the reported power difference, and the lack of a specification of the null simulation for the real-data logistic regression application.

major comments (5)
  1. [Abstract and §4.3] The abstract states that 'we derive the limiting distribution of sample splitting estimator,' but Section 4.3 only summarizes results from Kravitz et al. (2019) and contains no derivation. The manuscript should either provide the derivation or revise the abstract to credit the companion paper.
  2. [§5.4, Tables 4–5] Section 5.4 reports that 'the maximum difference between the two tests is 0.015,' but the displayed tables contradict this. In Table 4, for β0=0.6 and B=10, the bootstrap power is 0.91 while the F-test power is 0.95, a difference of 0.04; Table 5 shows the same discrepancy. The reported 0.015 figure should be corrected, and the power comparison should be stated consistently with the tables.
  3. [§6 and Supplement S.1] Section 6 reports p*=0.003 for testing β0=0 in the logistic model (12), but the paper nowhere specifies how the null distribution of pH1 was simulated in this setting. Supplement S.1.1 and S.1.2 describe null simulations only for linear models; they do not state how covariates Z are handled under the null (in particular, the null model still depends on Z through β), how θ is re-estimated in null replications, or how residual resampling would be adapted to a binary outcome. Without this information, the reported p-value cannot be reproduced, and the finite-sample level properties from Section 5.3 do not automatically transfer.
  4. [§5.3, model (8)] Model (8) uses the same outcome Y in both estimating equations, so under the alternative β0≠0 the two equations cannot both be unbiased: if Y = β0 X^T θ + ε, then E[X(Y−X^T θ)] = (β0−1)E[X X^T θ] ≠ 0. Thus the first-stage estimator is inconsistent under the alternative, and the power simulations in Tables 4–5 are not simulations of a correctly specified stacked estimating equations model as defined in Section 4. The authors should either use two distinct outcomes (W and Y) as in Section 4, or explicitly justify the working-model interpretation and its implications for the test.
  5. [§5.1 and Supplement S.1] Section 5.1 (step 2) and Supplement S.1 leave unspecified how the per-split p-value p_b is computed from the second-stage estimating equation. Since pH1 is defined as the mean of the p_b, this is a central ingredient of the test statistic. In particular, it is not stated whether a Wald, t, or bootstrap p-value is used, and whether the second-stage regressor X^T θ̂_b is treated as fixed or whether the uncertainty from estimating θ̂_b is accounted for. The algorithm is therefore not fully defined as written.
minor comments (8)
  1. [§5.1, step 3] The notation β̂_b := B^{-1} Σ_b β̂_b is self-referential; use a distinct symbol (e.g., β̂_avg) for the averaged estimate.
  2. [§5.1, step 5] The expression I(~p_n) should be I(~p_i) or I(~p_b) to index the null p-values, not the sample size n.
  3. [§5.4] The text says simulations use B=50 and β0 from −1 to 1, but Table 4 shows B=10,...,250 and β0 from 0 to 1; please align the text with the tables.
  4. [§5.3] The distributions of X used in the level and power simulations are not stated; please specify whether X is fixed or random and from which distribution.
  5. [Table 4 caption] ‘uniformally’ should be ‘uniformly’, and ‘The variance assumed to be known’ should be ‘The error variance is assumed known’.
  6. [§5.5] ‘calucaltions’ should be ‘calculations’.
  7. [References] The citation to Efron (1977) for residual resampling is likely incorrect; residual resampling is usually attributed to Efron (1979) or Efron and Tibshirani (1994).
  8. [§6.3] ‘The p-values from the from splits’ has a duplicated phrase.

Circularity Check

0 steps flagged · score 1.0 of 10

No meaningful circularity: the bootstrap p-value is defined against an independently simulated null distribution and benchmarked against the UMP F-test; the one self-citation is not load-bearing.

full rationale

The central bootstrap procedure is not circular under the review rules. The test statistic p* is defined as the empirical null CDF of pH1, and the null distribution is simulated from the model under H0 (e.g., Y*_i ~ N(0,1) independent of X in Supplement S.1.1). This is an independent input rather than a restatement of the observed test statistic, so the final p-value is not forced by construction. Section 5.2's exactness claim is explicitly limited to 'no unknown nuisance parameters', and the paper itself concedes 'In these cases the bootstrap test is not exact', so the later use of residual resampling is an acknowledged approximation rather than a disguised identity. The power comparison is made against the uniformly most powerful invariant F-test, an external benchmark, and the level simulations use error distributions and null models specified independently of the data being tested. The only self-citation is Kravitz et al. (2019), used in Section 4.3 for asymptotic expansions of the sample-splitting estimator; that asymptotic theory is not load-bearing for the bootstrap test's validity, which is simulation-based. The Section 6 logistic-regression analysis reports p* = 0.003 without specifying a null simulation that preserves the covariate vector Z under H0; this is a reproducibility and validity gap, not a circular reduction, because no equation of the analysis is defined in terms of the p-value it claims to predict. Overall, the derivation chain is self-contained and the circularity burden is low.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The central test procedure has only tuning parameters, not fitted physical constants. The load-bearing assumptions are standard M-estimation regularity conditions, the ability to simulate the null, and the correctness of the authors' self-cited companion asymptotic theory, which is not independently supported here.

free parameters (2)
  • B (number of sample splits) = 10 to 250, recommended 50
    Variance reduction parameter studied in Section 5.5. The test's validity does not hinge on B, so it is a tuning parameter rather than a load-bearing fitted constant.
  • pi (split probability) = 0.5
    Set to 1/2 for approximately equal split sizes; not fitted to data.
assumptions (6)
  • domain assumption Standard M-estimation regularity conditions hold for the stacked estimating equations, including consistency and asymptotic normality of the estimators.
    Section 4.3 inherits the asymptotic theory from these conditions through Kravitz et al. (2019).
  • domain assumption Observations (Y_i, W_i, X_i, Z_i) are independent and identically distributed.
    Stated at the start of Section 4.
  • domain assumption The first estimating equation has a unique solution and the pair (theta, beta) is identifiable.
    Section 5.3 imposes ||theta|| = 1 for identifiability in simulations; uniqueness is assumed for the stacked system.
  • domain assumption Under the null hypothesis, Y can be simulated independently of X, or from a residual-resampling model, to reproduce the null distribution of the test statistic.
    The exactness claim in Section 5.2 uses Y = epsilon under H0; Supplement S.1 generates null samples this way.
  • domain assumption Residual resampling gives a valid approximation to the error distribution when it is unknown.
    Section 5.2 uses Efron and Tibshirani's residual bootstrap for the approximate null.
  • ad hoc to paper The asymptotic results in the companion paper Kravitz et al. (2019) are correct.
    Section 4.3 attributes the limiting distribution and equivalence to this self-cited companion paper; no derivation or independent verification appears in this manuscript.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Finite Sample Hypothesis Tests for Stacked Estimating Equations." pith.science (2026). https://pith.science/paper/MGGHYICQ

@misc{pith2026190803968,
  author       = {Pith},
  title        = {Pith review of: Finite Sample Hypothesis Tests for Stacked Estimating Equations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MGGHYICQ}},
  note         = {Machine review of arXiv:1908.03968}
}
read the original abstract

Suppose there are two unknown parameters, each parameter is the solution to an estimating equation, and the estimating equation of one parameter depends on the other parameter. The parameters can be jointly estimated by "stacking" their estimating equations and solving for both parameters simultaneously. Asymptotic confidence intervals are readily available for stacked estimating equations. We introduce a bootstrap-based hypothesis test for stacked estimating equations which does not rely on asymptotic approximations. Test statistics are constructed by splitting the sample in two, estimating the first parameter on a portion of the sample then plugging the result into the second estimating equation to solve for the next parameter using the remaining sample. To reduce simulation variability from a single split, we repeatedly split the sample and take the sample mean of all the estimates. For parametric models, we derive the limiting distribution of sample splitting estimator and show they are equivalent to stacked estimating equations.

Figures

Figures reproduced from arXiv: 1908.03968 by the authors.

Figure 1
Figure 1. Three of the 8 marginal model plots from binary regressio [PITH_FULL_IMAGE:figures/full_fig_p019_1.png] view at source ↗
Figure 2
Figure 2. Variance of a simple sample split estimator of the sample mean [PITH_FULL_IMAGE:figures/full_fig_p020_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 19 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

    work page

  2. [2]

    C., Patel, A., Hartge, P., De Gonzalez, A

    Arem, H., Moore, S. C., Patel, A., Hartge, P., De Gonzalez, A. B., Visvanathan, K., Campbell, P. T., Freedman, M., Weiderpass, E., Adami, H. O., et al. (2015). Leisure time physical activity and mortality: a detailed pooled analysis of the dose-response relationship. JAMA I nternal M edicine , 175, 959--967

    work page 2015

  3. [3]

    J., Ruppert, D., Crainiceanu, C

    Carroll, R. J., Ruppert, D., Crainiceanu, C. M., and Stefanski, L. A. (2006). Measurement E rror in N onlinear M odels . Chapman and Hall/CRC

    work page 2006

  4. [4]

    Dezeure, R., B \"u hlmann, P., Meier, L., and Meinshausen, N. (2015). High-dimensional inference: Confidence intervals, p -values and r-software hdi. Statistical Science , 30, 533--558

    work page 2015

  5. [5]

    Efron, B. (1977). The efficiency of C ox's likelihood function for censored data. Journal of the American statistical Association , 72, 557--565

    work page 1977

  6. [6]

    and Tibshirani, R

    Efron, B. and Tibshirani, R. J. (1994). An introduction to the bootstrap . CRC press

    work page 1994

  7. [7]

    and Hu, F

    Gr ntved, A. and Hu, F. B. (2011). Television viewing and risk of type 2 diabetes, cardiovascular disease, and all-cause mortality: a meta-analysis. Journal of the American Medical Association , 305, 2448--2455

    work page 2011

  8. [8]

    Huber, P. J. (1964). Robust estimation of a location parameter. Annals of M athematical S tatistics , 35, 73--101

    work page 1964

Show all 19 references
  1. [9]

    Huber, P. J. (1967). The behavior of maximum likelihood estimates under nonstandard conditions. In Proceedings of the F ifth B erkeley S ymposium on M athematical S tatistics and P robability , volume 1, pages 221--233. University of California Press

    1967

  2. [10]

    S., Carroll, R

    Kravitz, E. S., Carroll, R. J., and Ruppert, D. (2019). Sample splitting as an m-estimator with application to physical activity scoring

    2019

  3. [11]

    and Zeger, S

    Liang, K.-Y. and Zeger, S. L. (1986). Longitudinal data analysis using generalized linear models. Biometrika , 73, 13--22

    1986

  4. [12]

    Meinshausen, N., Meier, L., and B \"u hlmann, P. (2009). P-values for high-dimensional regression. Journal of the American Statistical Association , 104, 1671--1681

    2009

  5. [13]

    and Pearson, E

    Neyman, J. and Pearson, E. S. (1928). On the use and interpretation of certain test criteria for purposes of statistical inference: Part i. Biometrika , 20, 175--240

    1928

  6. [14]

    and Pearson, E

    Neyman, J. and Pearson, E. S. (1933). Ix. on the problem of the most efficient tests of statistical hypotheses. Philosophical Transactions of the Royal Society of London. Series A, Containing Papers of a Mathematical or Physical Character , 231, 289--337

    1933

  7. [15]

    Prince, S., Saunders, T., Gresty, K., and Reid, R. (2014). A comparison of the effectiveness of physical activity and sedentary behaviour interventions in reducing sedentary time in adults: a systematic review and meta-analysis of controlled trials. Obesity Reviews , 15, 905--919

    2014

  8. [16]

    F., Thompson, F

    Schatzkin, A., Subar, A. F., Thompson, F. E., Harlan, L. C., Tangrea, J., Hollenbeck, A. R., Hurwitz, P. E., Coyle, L., Schussler, N., Michaud, D. S., et al. (2001). Design and serendipity in establishing a large cohort with wide dietary intake distributions: the National Inst...

    2001

  9. [17]

    Stefanski, L. A. and Boos, D. D. (2002). The calculus of M -estimation. The American Statistician , 56, 29--38

    2002

  10. [18]

    and Roeder, K

    Wasserman, L. and Roeder, K. (2009). High dimensional variable selection. Annals of Statistics , 37, 2178

    2009

  11. [19]

    Yin, J., Jin, X., Shan, Z., Li, S., Huang, H., Li, P., Peng, X., Peng, Z., Yu, K., Bao, W., et al. (2017). Relationship of sleep duration with all-cause mortality and cardiovascular events: A systematic review and dose-response meta-analysis of prospective cohort studies. Jour...

    2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.