Pith. sign in

REVIEW 3 major objections 6 minor 24 references

Sample Splitting as an M-Estimator with Application to Physical Activity Scoring

T0 review · 3 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proves that averaging many sample splits yields the same asymptotic estimator as fitting both stages to the full data, so multi-split inference adds no information beyond the unsplit analysis.

desk verdict The paper proves a clean asymptotic equivalence between averaged sample splitting and stacked estimating equations, but the motivating physical activity score likely violates the smoothness assumptions the proof relies on. read the letter →

arxiv 1908.03967 v1 pith:GYCF7N56 submitted 2019-08-11 stat.ME

Eli S. Kravitz , Raymond J. Carroll , David Ruppert This is my paper · ORCID
classification stat.ME MSC 62F1262J1262P10
keywords samplesplittingM-estimationestimatingequationsstackedtwo-stagemodelingphysicalactivityscoreasymptotictheorymultiplesplits
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Sample splitting lets one data set serve two roles—here, building a physical-activity score on half of a cohort and then using that score to predict mortality on the other half—without letting the same observations do both jobs. The paper develops asymptotic theory for the split-sample estimator in a two-stage M-estimation setting and gives formulas for standard errors and confidence intervals. Its central result is that as the number of splits $B$ and the sample size $n$ grow, with $B$ growing more slowly than $\sqrt{n}$, the average of the split estimates is asymptotically the same as solving the stacked estimating equations on the whole data set. Thus when the split proportion is $\pi = 1/2$, the multi-split estimator carries no information beyond an ordinary full-data analysis, and confidence intervals can be obtained without splitting at all. The motivating application builds a physical-activity score from the NIH-AARP cohort and estimates its association with mortality: the log-odds coefficient is about $-0.026$, with a confidence interval excluding zero.

What carries the argument

The load-bearing object is the per-split asymptotic linear expansion of each M-estimator. Each split contributes an influence-function term whose average over $B$ splits, combined with the law-of-large-numbers limits $B^{-1}\sum_b \delta_{ib} \to \pi$ and $B^{-1}\sum_b (1-\delta_{ib}) \to 1-\pi$, reproduces the influence function of the stacked estimating equations estimator—the estimator obtained by solving the first-stage and second-stage score equations jointly on the entire data set. The technical condition that makes the collapse work is that the $o_P(1)$ remainder terms in the per-split expansions are uniformly $O_P(n^{-1/2+a})$ across all splits, so their average vanishes when $B$ grows slower than $n^{1/2-a}$.

What would settle it

Simulate the paper's logistic two-stage model at sample sizes $n = 250$ and $n = 1000$, compute the $B$-split average for $B$ proportional to $n^{0.6}$ and to $n^{1/2}$ with $\pi = 1/2$, and examine $\sqrt{n}$ times the difference between the split average and the stacked-estimating-equations estimate; if this difference does not shrink to zero as $n$ grows, the uniform remainder assumption fails.

Watch

Extended reading notes

Core claim

The paper sets up each sample split as two estimating equations: the first-stage parameter $\theta$ is estimated from the Bernoulli($\pi$) training subsample, and the second-stage parameter $\beta$ is estimated from the complementary subsample using the fitted score $f(X;\hat{\theta})$ as a covariate. A single split has a standard M-estimation expansion, and the paper obtains the joint limiting distribution and a covariance estimator for it. For a fixed number of splits $B$, the final estimators are the sample means of the per-split estimates, and their covariance is estimated from averaged influence functions. The main theorem in Section 4.3 lets $B$ grow with $n$ at rate $B = o(n^{1/2-a})$ for any $a>0$. Because the Bernoulli indicators average to their probabilities, $B^{-1}\sum_{b=1}^{B}\delta_{ib} \to \pi$, and the averaged influence functions collapse to those of the stacked estimating equations estimator that solves both score equations on the entire sample. The paper states the conclusion directly: when $\pi = 1/2$, the estimates from sample splitting become asymptotically equivalent to estimates from the entire data set without sample splitting.

Load-bearing premise

The proof relies on the approximation error from each split's statistical estimate being uniformly small across all B splits, so averaging the estimates does not accumulate error; the paper borrows this uniform bound from earlier work rather than proving it for its own two-stage logistic model.

Editorial extensions

If this is right

  • Averaging over many splits gives the same asymptotic estimator as stacking the estimating equations on the full data, so the multi-split average contains no extra information beyond an unsplit fit.
  • At $\pi = 1/2$, the split-sample estimator and the full-data stacked estimator share the same influence function, so their confidence intervals are asymptotically identical.
  • For one split or a fixed number of splits, the covariance formulas support Wald tests and normal-based confidence intervals, with simulation coverage close to 95 percent once $n \geq 250$.
  • In the physical-activity application, the score coefficient is about $-0.026$ with a 95 percent confidence interval excluding zero, so a higher score is associated with lower mortality risk even after accounting for score uncertainty.
  • Treating the score as estimated rather than fixed widens the intervals compared with a naive two-stage analysis that ignores score-building uncertainty.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: for estimation and confidence intervals, the theorem implies that once the sample is large, spending computation on many splits is unnecessary; the full-data stacked estimating equations give the same asymptotic answer. The paper itself continues to recommend the exact split-based test when testing $\beta_0 = 0$.
  • Editorial inference: this reframes the 'p-value lottery' critique of single splits. Averaging over many splits does not create a new inference procedure; it converges to the p-value of the unsplit full-data analysis, so the lottery disappears by becoming the standard analysis.
  • Editorial inference: the same equivalence should apply to other two-stage protocols that use a first-stage fit to construct a covariate for a second-stage model, such as propensity scores or polygenic risk scores, whenever the estimating functions meet the smoothness and uniform-remainder conditions.
  • Editorial inference: the theorem requires $B$ to grow slower than $\sqrt{n}$. At $B$ comparable to or larger than $\sqrt{n}$, the averaged remainders may not vanish, so the common practice of taking $B$ very large with moderate $n$ may behave differently from this limit; checking that boundary empirically is an open question.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper develops asymptotic theory for sample splitting in a two-stage estimation setting. In the first stage, a parameter θ is estimated from a training half of the data; in the second stage, a function f(X; θ) of that estimate is used as a covariate in a model for β estimated on the validation half. The authors provide asymptotic expansions and variance estimators for one split and for a fixed finite number of splits B, and then claim that when B grows with n at rate B = o(n^{1/2-a}), the split-averaged estimator is asymptotically equivalent to the stacked estimating equations estimator computed on the entire data set. The motivating application is a physical activity score for mortality risk in the NIH-AARP cohort; simulations for linear and logistic models examine finite-sample coverage of confidence intervals.

Significance. If the equivalence result holds under the stated conditions, it is a genuinely interesting conceptual contribution: it says that many-split averaging does not provide asymptotic information beyond full-data stacked estimating equations, and it gives practitioners a way to justify confidence intervals in split-sample analyses. The paper also provides useful simulation evidence and a realistic data application, with explicit asymptotic expansions that can be used directly. However, the proof is presented only as a sketch, and the key step that makes the averaging over B splits work is imported from Carroll et al. (1996) without verification for the specific two-stage estimating equations studied here.

major comments (3)
  1. [Section 4.3 and Appendix A.3] The central equivalence theorem depends on the assertion that the oP(1) remainder terms in expansions (A.2) and (A.3) are OP(n^{-1/2+a}) "as in Carroll et al. (1996)". When averaging over B splits, the proof needs this remainder bound to hold uniformly in b, because the step from the averaged expansions (A.4)-(A.5) to the limiting expansions (A.6) requires control of B^{-1} sum_{b=1}^B R_{nb}. A pointwise OP(n^{-1/2+a}) statement for each fixed b does not control the maximum or the average of the remainders, and the manuscript does not state or verify the required stochastic equicontinuity or uniform remainder condition. Since this uniformity is exactly what produces the rate restriction B = o(n^{1/2-a}), the main claim in Section 4.3 is not established as written.
  2. [Section 4.2] The variance estimator for a finite fixed number of splits is incorrectly defined. After defining A_i = B^{-1} sum_{b=1}^B A_{ib}, the text defines Â_i = sum_{b=1}^B Â_{ib}, omitting the factor B^{-1}. With this definition, the sample covariance S(Â_i) would be approximately B^2 times the target covariance, so the resulting standard errors for the averaged estimator would be wrong by a factor of B. The definition should be Â_i = B^{-1} sum_{b=1}^B Â_{ib}, and the left-hand sides of the two displays in this section, which read n^{1/2}(θ̂_b - θ) and n^{1/2}(β̂_b - β), should refer to the averaged estimators θ̂ and β̂, respectively.
  3. [Section 6.2 and Appendix A.1] The asymptotic theory assumes that f(X; θ) is differentiable with respect to θ (Appendix A.1). The physical activity score defined in Section 6.2 depends on θ through the location of the minimum of each fitted marginal model, the absolute value of that minimum, and the total T obtained by summing the maxima of the rescaled curves. These operations are not automatically differentiable in θ, and the manuscript does not provide conditions under which the score is smooth, such as uniqueness and interiority of the optimizing points together with an envelope theorem argument. Without such verification, the theorem of Section 4.3 is not directly applicable to the motivating data analysis, and the confidence intervals reported in Table 5 rely on an unverified regularity condition.
minor comments (6)
  1. [Section 4.3, Eq. (8)] The displayed expansion for n^{1/2}(β̂ - β) contains K(Y_i, β, θ̂_b), but the subscript b is not defined in the averaged expansion; this should be K(Y_i, β, θ), matching the appendix version in (A.6).
  2. [Appendix A.1] The assumption that E(θ̂_k) = E(θ̂_l) = θ and E(β̂_k) = E(β̂_l) = β is not generally satisfied by nonlinear M-estimators and is not used in the subsequent expansions; it would be cleaner to replace this with a consistency assumption or remove it.
  3. [Appendix A.3] The appendix is labeled a sketch, but the uniform remainder control is the load-bearing step of the paper's main theorem; a proof or a precise high-level condition should be supplied instead of a citation.
  4. [Abstract] "Motivating by problems" should be "Motivated by problems".
  5. [Section 1] The phrase "They create B copies of the data" is imprecise; the procedure creates B random partitions of the same dataset, not B copies.
  6. [Table 3] The word "logistc" should be "logistic".

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the multi-split estimator's equivalence to stacked estimating equations is derived from explicit M-estimator expansions and external standard asymptotics, not from a fitted input or self-referential definition.

full rationale

The central claim is the asymptotic equivalence of the mean-aggregated multi-split estimator to the stacked-estimating-equations estimator (Section 4.3 and Appendix A.4). The derivation does not define any input in terms of the output: the split estimator is defined by independent Bernoulli splits and per-split M-estimators (Section 3.3), and the stacked estimator is defined by the full-data equations (A.7). The proof proceeds by standard M-estimator Taylor expansions (A.2)-(A.5), an external remainder-rate bound cited to Carroll et al. (1996), and algebra showing that an averaged influence function simplifies to the full-data influence function. No parameter is fitted to a subset and then 'predicted' on a closely related quantity; the paper makes no empirical prediction claim. The only self-citations (Kravitz et al. 2019 for an exact test, Carroll et al. 1996/2006 for standard asymptotics) are not used to define the target result, and Carroll's remainder bound is an independent published technical result rather than a restatement of this paper's conclusion. The skeptic's concern that the Section 6.2 score uses minima, maxima, and absolute values, which may violate the differentiability assumption in A.1, is a correctness or regularity risk rather than circularity: even if the theorem does not cover the application, the equivalence is derived from assumptions rather than assumed. I find no step where a conclusion is identical by construction to an input.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central theorem rests on standard M-estimator regularity conditions and the random-split design; no new entities are postulated. The split fraction pi is a user-chosen design parameter, and the simulation constants 2 and 1.5 are hand-picked but do not enter the theory.

free parameters (2)
  • split fraction pi = 0.5
    The Bernoulli probability that an observation enters the training split is user-chosen and set to 1/2 in the applications and simulations; it enters the asymptotic weights in the theorem.
  • simulation constants 2 and 1.5 = 2 and 1.5
    In Section 5 the simulation model uses f(X, theta) = 2 + 1.5 (X^T theta)^2, with constants chosen by hand to produce a reasonable number of zeros and ones in Y. These are not fitted and do not affect the theory.
assumptions (5)
  • standard math Standard M-estimation regularity conditions: Psi and K are continuous, twice differentiable, have unique roots, and the derivative matrices are nonsingular.
    Invoked in Appendix A.1 as the basis for the asymptotic expansions in Equations (5) and (6).
  • domain assumption K depends on theta only through a differentiable function f(X, theta), and f is differentiable in theta.
    Introduced in Section 3.1 and used in the Taylor expansion in Appendix A.2, reflecting the composite-score construction.
  • domain assumption Split indicators delta_ib are independent and identically distributed Bernoulli(pi) across i and b, and are independent of all data.
    Defined in Sections 3.2 and 3.3; this independence drives the convergence of B^{-1} sum_b delta_ib to pi in Section 4.3.
  • domain assumption Per-split remainder terms are uniformly OP(n^{-1/2+a}) across B splits, and B = o(n^{1/2-a}) for any a > 0.
    Assumed in Appendix A.3 and Section 4.3, following Carroll et al. (1996); this uniform control is needed so the remainder does not blow up when averaged over B.
  • ad hoc to paper Estimators from different splits are exchangeable and unbiased with E(theta_hat_k) = E(theta_hat_l) = theta and E(beta_hat_k) = E(beta_hat_l) = beta.
    Stated in Appendix A.1 but not used in the expansions; likely a vestigial assumption rather than a load-bearing step.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sample Splitting as an M-Estimator with Application to Physical Activity Scoring." pith.science (2026). https://pith.science/paper/GYCF7N56

@misc{pith2026190803967,
  author       = {Pith},
  title        = {Pith review of: Sample Splitting as an M-Estimator with Application to Physical Activity Scoring},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GYCF7N56}},
  note         = {Machine review of arXiv:1908.03967}
}
read the original abstract

Sample splitting is widely used in statistical applications, including classically in classification and more recently for inference post model selection. Motivating by problems in the study of diet, physical activity, and health, we consider a new application of sample splitting. Physical activity researchers wanted to create a scoring system to quickly assess physical activity levels. A score is created using a large cohort study. Then, using the same data, this score serves as a covariate in a model for the risk of disease or mortality. Since the data are used twice in this way, standard errors and confidence intervals from fitting the second model are not valid. To allow for proper inference, sample splitting can be used. One builds the score with a random half of the data and then uses the score when fitting a model to the other half of the data. We derive the limiting distribution of the estimators. An obvious question is what happens if multiple sample splits are performed. We show that as the number of sample splits increases, the combination of multiple sample splits is effectively equivalent to solving a set of estimating equations.

Figures

Figures reproduced from arXiv: 1908.03967 by the authors.

Figure 1
Figure 1. Three of the 8 marginal model plots from binary regressio [PITH_FULL_IMAGE:figures/full_fig_p020_1.png] view at source ↗
Figure 2
Figure 2. The solid blue line denotes the estimate of [PITH_FULL_IMAGE:figures/full_fig_p021_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 10 canonical work pages

  1. [1]

    C., Patel, A., Hartge, P., De Gonzalez, A

    Arem, H., Moore, S. C., Patel, A., Hartge, P., De Gonzalez, A. B., Visvanathan, K., Campbell, P. T., Freedman, M., Weiderpass, E., Adami, H. O., et al. (2015). Leisure time physical activity and mortality: a detailed pooled analysis of the dose-response relationship. JAMA I nternal M edicine , 175, 959--967

    2015

  2. [2]

    Battey, H., Fan, J., Liu, H., Lu, J., and Zhu, Z. (2015). Distributed estimation and inference with statistical guarantees. arXiv preprint arXiv:1509.05457 ,

    arXiv 2015

  3. [3]

    J., K \"u chenhoff, H., Lombard, F., and Stefanski, L

    Carroll, R. J., K \"u chenhoff, H., Lombard, F., and Stefanski, L. A. (1996). Asymptotics for the SIMEX estimator in nonlinear measurement error models. Journal of the American Statistical Association , 91, 242--250

    work page 1996

  4. [4]

    J., Ruppert, D., Crainiceanu, C

    Carroll, R. J., Ruppert, D., Crainiceanu, C. M., and Stefanski, L. A. (2006). Measurement E rror in N onlinear M odels . Chapman and Hall/CRC

    2006

  5. [5]

    Chang, X., Lin, S.-B., Wang, Y., et al. (2017). Divide and conquer local average regression. Electronic Journal of Statistics , 11, 1326--1350

    work page 2017

  6. [6]

    Dezeure, R., B \"u hlmann, P., Meier, L., and Meinshausen, N. (2015). High-dimensional inference: Confidence intervals, p -values and r-software hdi. Statistical Science , 30, 533--558

    2015

  7. [7]

    Fieller, E. C. (1932). The distribution of the index in a normal bivariate population. Biometrika , 24, 428--440

    work page 1932

  8. [8]

    Fieller, E. C. (1954). Some problems in interval estimation. Journal of the Royal Statistical Society. Series B , 16, 175--185

    work page 1954

Show all 24 references
  1. [9]

    and Hu, F

    Gr ntved, A. and Hu, F. B. (2011). Television viewing and risk of type 2 diabetes, cardiovascular disease, and all-cause mortality: a meta-analysis. Journal of the American Medical Association , 305, 2448--2455

    2011

  2. [10]

    M., Reedy, J., Krebs-Smith, S

    Guenther, P. M., Reedy, J., Krebs-Smith, S. M., and Reeve, B. B. (2008). Evaluation of the healthy eating index-2005. Journal of the American Dietetic Association , 108, 1854--1864

    2008

  3. [11]

    Huber, P. J. (1964). Robust estimation of a location parameter. Annals of M athematical S tatistics , 35, 73--101

    1964

  4. [12]

    Huber, P. J. (1967). The behavior of maximum likelihood estimates under nonstandard conditions. In Proceedings of the F ifth B erkeley S ymposium on M athematical S tatistics and P robability , volume 1, pages 221--233. University of California Press

    1967

  5. [13]

    S., Carroll, R

    Kravitz, E. S., Carroll, R. J., and Ruppert, D. (2019). Finite-sample hypothesis tests for stacked estimating equations

    2019

  6. [14]

    K., and Li, B

    Li, R., Lin, D. K., and Li, B. (2013). Statistical inference in massive data sets. Applied Stochastic Models in Business and Industry , 29, 399--409

    2013

  7. [15]

    Meinshausen, N., Meier, L., and B \"u hlmann, P. (2009). P-values for high-dimensional regression. Journal of the American Statistical Association , 104, 1671--1681

    2009

  8. [16]

    Prince, S., Saunders, T., Gresty, K., and Reid, R. (2014). A comparison of the effectiveness of physical activity and sedentary behaviour interventions in reducing sedentary time in adults: a systematic review and meta-analysis of controlled trials. Obesity Reviews , 15, 905--919

    2014

  9. [17]

    F., Thompson, F

    Schatzkin, A., Subar, A. F., Thompson, F. E., Harlan, L. C., Tangrea, J., Hollenbeck, A. R., Hurwitz, P. E., Coyle, L., Schussler, N., Michaud, D. S., et al. (2001). Design and serendipity in establishing a large cohort with wide dietary intake distributions: the National Inst...

    2001

  10. [18]

    Stefanski, L. A. and Boos, D. D. (2002). The calculus of M -estimation. The American Statistician , 56, 29--38

    2002

  11. [19]

    Department of Health and Human Services and U.S

    U.S. Department of Health and Human Services and U.S. Department of Agriculture (2005). Dietary guidelines for A mericans. Technical report

    2005

  12. [20]

    Wang, Y., Wang, S., and Carroll, R. J. (2015). The direct integral method for confidence intervals for the ratio of two location parameters. Biometrics , 71, 704--713

    2015

  13. [21]

    and Roeder, K

    Wasserman, L. and Roeder, K. (2009). High dimensional variable selection. Annals of Statistics , 37, 2178

    2009

  14. [22]

    Yin, J., Jin, X., Shan, Z., Li, S., Huang, H., Li, P., Peng, X., Peng, Z., Yu, K., Bao, W., et al. (2017). Relationship of sleep duration with all-cause mortality and cardiovascular events: A systematic review and dose-response meta-analysis of prospective cohort studies. Jour...

    2017

  15. [23]

    J., and Duchi, J

    Zhang, Y., Wainwright, M. J., and Duchi, J. C. (2012). Communication-efficient algorithms for statistical optimization. In Advances in Neural Information Processing Systems , pages 1502--1510

    2012

  16. [24]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence '...

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.