Pith. sign in

REVIEW 3 major objections 4 minor 17 references

Subsampling-based Tests in Mediation Analysis

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Pith's one-line read Splitting the sample into subsamples and studentizing per-subsample Sobel statistics yields a t-distribution null that is the same for all three composite-null cases in mediation analysis.

desk verdict Correct and elegant subsample studentization for the mediation null, but the final Cauchy combination step is not fully justified and needs a theorem or a condition check before CSMT is presented as validated. read the letter →

arxiv 2411.10648 v2 pith:GMQGE6B3 submitted 2024-11-16 stat.ME

classification stat.ME MSC 62F0362F05
keywords mediationanalysiscompositenullhypothesisSobeltestsubsamplingstudentizationCauchycombinationsizecontrol
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Testing for a mediation effect is hard because the null hypothesis of no mediation is a union of three distinct cases: the exposure–mediator effect is zero, the mediator–outcome effect is zero, or both are zero. Classical tests such as Sobel's are calibrated for only one case and turn conservative when both effects vanish. The paper shows that splitting the sample into K disjoint subsamples, computing a Sobel statistic in each, and studentizing these values gives a statistic that converges to a t-distribution with K−1 degrees of freedom under every null case, making the null distribution pivotal. Repeating this across many random splits and combining p-values with a Cauchy combination test yields a procedure, called CSMT, that controls size accurately and shows competitive power in simulations.

What carries the argument

The central object is the studentized subsample Sobel statistic, built by partitioning the data into K disjoint subsamples, computing the classical Sobel statistic S_{G_i} within each, and applying a one-sample t-statistic to the K resulting values. Because each S_{G_i} is asymptotically normal under every null scenario but with a variance that depends on which null case holds, the studentization cancels the unknown scale and leaves a pivotal t_{K−1} limit. The second piece is the Cauchy combination test, which aggregates p-values from M random splits using random positive weights; the paper rejects when the combined statistic exceeds the standard Cauchy quantile.

What would settle it

Simulate data under H00 (both coefficients zero) with large n, fix one dataset, compute p-values from M = 500 random splits, form the Cauchy-combined statistic C_m, and repeat over many datasets; if the empirical distribution of C_m deviates substantially from the standard Cauchy in the tail, the pivotal rejection region is not valid.

Watch

Extended reading notes

Core claim

The central claim is Theorem 2: under the composite null H0: αβ = 0, the studentized subsample statistic T_n = $K^{{1/2}}$ \bar{S}_K / ( (1/(K−1)) \sum (S_{G_i} − \bar{S}_K)^2 )^{1/2} converges in distribution to a t random variable with K−1 degrees of freedom. Each subsample Sobel statistic S_{G_i} is asymptotically normal with mean zero and variance either 1 (when exactly one of α and β is zero) or 1/4 (when both are zero), so the sample variance across subsamples estimates and removes the scale. The null distribution is therefore the same in all three null cases, requiring a single cutoff. Repeating the split M times and combining the resulting p-values through a weighted Cauchy combination test gives CSMT, which the paper demonstrates by simulation to control size more accurately than the Sobel test, the MaxP test, and an adaptive bootstrap test while achieving higher power at stronger signals.

Load-bearing premise

The procedure's size control ultimately depends on the Cauchy combination of p-values from different random splits following the standard Cauchy distribution even though those p-values are dependent, and the paper does not prove this.

Editorial extensions

If this is right

  • A single t_{K−1} cutoff can be used for mediation testing regardless of which of the three null cases holds, eliminating the need to know the null type.
  • Combining p-values from repeated random splits stabilizes the test results and raises power, as shown in the numerical studies.
  • CSMT controls size more accurately than the classical Sobel and MaxP tests, which are conservative when both coefficients are zero.
  • At stronger signal strengths, CSMT outperforms the adaptive bootstrap test while avoiding that test's inflated size.
  • The method remains valid for the general structural equation framework covered by the assumptions, not only for linear models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the proof of Theorem 2 only uses asymptotic normality of the per-subsample statistics, so the same subsample-studentization device could pivotailize other test statistics whose normal limit has a null-dependent variance.
  • Editorial inference: the recommended choice K = ⌊0.5√n⌋ is based on empirical evidence; a data-driven or cross-validated choice of K has not been explored and could improve finite-sample behavior.
  • Editorial inference: because the Cauchy combination's null distribution is assumed rather than proved under the dependence across splits, practitioners using CSMT on smaller samples should verify the empirical null by permutation before trusting nominal p-values.
  • Editorial inference: the SPIRIT data analysis points to butyric, acetic, and valeric acids as candidate mediators of metformin's anti-inflammatory effect; a confirmatory study with multiple-testing adjustment would be needed to make this a robust scientific claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a subsampling-based test for the composite null of no mediation effect (alpha*beta = 0). The idea is to split the data into K disjoint subsamples, compute the Sobel statistic in each subsample, and then studentize their mean to obtain a statistic T_n whose asymptotic null distribution is claimed to be t with K-1 degrees of freedom under all three null configurations (H00, H01, H10). This is the content of Theorem 2. To reduce variability across different random splits and improve power, the paper then combines p-values from M random splits using the Cauchy combination test, calling the resulting procedure CSMT. Section 5 recommends K = floor(0.5 sqrt(n)), and Section 6 reports simulation comparisons with Sobel, MaxP, and an adaptive bootstrap test, followed by a real-data analysis of the SPIRIT trial.

Significance. If the main claim is correct, the paper offers a simple, pivotal null distribution for a composite-null mediation problem, which is a genuinely useful contribution because standard Sobel and MaxP tests are conservative under H00. Theorem 2 is elementary and self-contained, with no fitted quantities entering the null distribution, which is a strength. The paper also provides an R implementation and reports extensive simulations. However, the actual procedure recommended to users, CSMT, relies on a Cauchy combination of dependent p-values for which no theorem is stated or proved; this gap is load-bearing for the paper's headline claim of accurate size control. The fixed-K theory also does not directly cover the recommended growing K.

major comments (3)
  1. [Section 4] The claim that the combined statistic C_m follows a standard Cauchy distribution, and hence that the p-value is 0.5 - arctan(C_m)/pi, is not justified. The p-values p_m are computed from M random splits of the same data and are therefore dependent. Liu and Xie (2020) is cited, but the paper neither states the precise theorem being used nor verifies its conditions. In particular, Theorem 2 gives only marginal asymptotic t-calibration of each p_m, not finite-sample validity, and arbitrary dependence among valid p-values does not make the weighted sum of their Cauchy transforms exactly Cauchy. Unless the authors state the relevant Liu-Xie result and prove that its conditions hold (or prove an asymptotic version for their setting), the size control of CSMT is not established by the theoretical part of the manuscript.
  2. [Theorem 2 and Section 5] Theorem 2 is stated and proved only for fixed K, but Section 5 recommends K = floor(0.5 sqrt(n)), which grows with n. The manuscript does not provide a theorem that covers K = K_n tending to infinity, even though in this regime the subsample sizes n/K_n also grow. The authors should either extend the asymptotic result to K_n -> infinity with n/K_n -> infinity, or explicitly limit the theoretical claim to a fixed K chosen by the user and describe the recommended K as a finite-sample heuristic. As written, the recommended implementation is not covered by the stated theorem.
  3. [Proof of Theorem 2] The proof of Theorem 2 says the result follows by applying Theorem 1 to each subsample and the continuous mapping theorem, but this omits an essential step: one must establish joint convergence of (S_{G_1}, ..., S_{G_K}) to independent normal variables, and then verify that the sample variance of the S_{G_i} is consistent. Since the subsamples are disjoint and K is fixed, this is straightforward, but it should be stated explicitly. Additionally, in the proof of Theorem 1 under H01/H10, the variance estimator is written as \hat{\alpha}^2 s_1^2 + \hat{\beta}^2 s_2^2, whereas the Sobel statistic in equation (1) uses \hat{\alpha}^2 s_2^2 + \hat{\beta}^2 s_1^2; the subscripts appear to be swapped.
minor comments (4)
  1. [Section 4] The notation is inconsistent: the statistic is first written as C_m with index m ranging over splits, but later the same symbol C_m is used as the combined statistic; the index m is reused in both the denominator and the numerator of the weighted sum.
  2. [Section 4] The sentence 'where wms are non-negative weights satisfying that sum_{m=1}^M w_m = 1' should read 'non-negative weights w_m satisfying ...' to avoid a missing subscript.
  3. [Section 5] The recommendation K = floor(0.5 sqrt(n)) is based only on empirical evidence summarized in the supplementary materials; the main text should clarify whether this choice is meant to be applied as a deterministic function of n or as a practical guideline independent of the asymptotic theory.
  4. [Section 6] In Example 2, the text says 'the power of the ABtest decreases as the signal strength increases,' but the accompanying Figure 3 is not described in enough detail; a brief explanation of why this occurs would improve readability.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: Theorem 2's pivotal t_{K-1} null is derived from an in-paper proof of the Sobel limit under all three null cases; the only soft spot is Section 4's unverified application of the external Liu-Xie Cauchy combination to dependent split p-values, a correctness risk rather than a circular step.

full rationale

The derivation chain is self-contained and contains no circular step. Theorem 1 (the composite-null limiting law of the Sobel statistic) is proved in the paper from Assumptions 1-2, via the delta method under H01/H10 and a polar-transform argument under H00; the Glonek (1993) citation is attribution, not load-bearing. Theorem 2 then applies Theorem 1 to each of K disjoint subsamples and studentizes: each S_Gi converges to N(0,1) under H01/H10 and to N(0,1/4) under H00, and since T_n = K^{1/2} average(S_Gi)/sd is scale-invariant, its weak limit is t_{K-1} in all three cases by the continuous mapping theorem. No parameter is fitted to produce this limiting law, and K enters only as degrees of freedom. The p-value p = P(|t_{K-1}| > |T_n|) is a standard asymptotic calibration, not a fitted quantity; the tuning choices (K = floor(0.5 sqrt(n)), M splits, random Cauchy weights) are practical and do not enter the null derivation. The one soft spot is Section 4: the paper asserts 'C follows a standard Cauchy distribution' for the combination of M p-values from different splits and rejects at the standard Cauchy quantile, citing Liu and Xie (2020), but it does not restate or verify that theorem's conditions for these dependent, only asymptotically t-calibrated p_m. That is a missing-support and correctness risk for the final CSMT size, not circularity: the combined statistic is not equivalent to its own inputs, and the cited source is an independent external result. There are no load-bearing self-citations (the reference list contains no work by the present authors), no uniqueness theorem imported from the authors' prior work, and no fitted input renamed as a prediction. Hence the honest finding is no circularity, with a minor flag on Section 4.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central derivation uses standard asymptotic assumptions for the estimators and the independence of subsample statistics. Two tuning parameters, K and M, are chosen in practice but are not part of the null-distribution theory. No new physical or mathematical entities are introduced.

free parameters (2)
  • K (number of subsamples) = K = floor(0.5 * sqrt(n))
    Chosen from simulations in Section 5 and Supplementary Figure 6; affects finite-sample behavior and power, but the asymptotic validity of the single-split test holds for any fixed K.
  • M (number of repeated splits) = 500 in all simulations
    An arbitrary practical number of splits used in the numerical studies; not part of the asymptotic derivation.
assumptions (3)
  • domain assumption Assumptions 1 and 2: the estimators of the mediation coefficients are asymptotically normal with a diagonal covariance matrix and have consistent variance estimators.
    Invoked at the start of Section 2 to establish the asymptotic distribution of the Sobel statistic in Theorem 1.
  • domain assumption Sobel statistics from disjoint subsamples are asymptotically independent.
    Used before Theorem 2 to justify the studentized statistic converging to a t-distribution; holds for iid data and regular estimators.
  • domain assumption The Cauchy combination of p-values is a valid test when the p-values are dependent across random splits.
    Section 4 applies the Liu-Xie Cauchy combination to correlated p-values from repeated splits and rejects using the standard Cauchy quantile, but no theorem in this paper proves the combined statistic has the claimed null distribution under this dependence structure.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Subsampling-based Tests in Mediation Analysis." pith.science (2026). https://pith.science/paper/GMQGE6B3

@misc{pith2026241110648,
  author       = {Pith},
  title        = {Pith review of: Subsampling-based Tests in Mediation Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GMQGE6B3}},
  note         = {Machine review of arXiv:2411.10648}
}
read the original abstract

Testing for mediation effect poses a challenge since the null hypothesis (i.e., the absence of mediation effects) is composite, making most existing mediation tests quite conservative and often underpowered. In this work, we propose a subsampling-based procedure to construct a test statistic whose asymptotic null distribution is pivotal and remains the same regardless of the three null cases encountered in mediation analysis. The method, when combined with the popular Sobel test, leads to an accurate size control under the null. We further introduce a Cauchy combination test to construct p-values from different subsample splits, which reduces variability in the testing results and increases detection power. Through numerical studies, our approach has demonstrated a more accurate size and higher detection power than the competing classical and contemporary methods.

Figures

Figures reproduced from arXiv: 2411.10648 by the authors.

Figure 1
Figure 1. QQ plots of the p-values in log scale from the four competing [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Empirical sizes for the four competing methods under the sparse [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. The value of K has been determined based on the recommendation in Section 5. The Sobel and MaxP tests are outperformed by the ABtest and CSMT. Interestingly, the power of the ABtest decreases as the signal strength increases, but it exhibits the highest power compared to all other methods at lower signal strengths. Here we would like to set a reminder that this improved power comes at the cost of inflated size. In c… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: QQ plots of the p-values from the four competing methods under [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Empirical sizes for the four competing methods under the sparse [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Empirical sizes and powers for the four competing methods with [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 16 canonical work pages

  1. [1]

    Regulation of inflammation by short chain fatty acids

    Marco AR Vinolo, Hosana G Rodrigues, Renato T Nachbar, and Rui Curi. Regulation of inflammation by short chain fatty acids. Nutrients, 3 0 (10): 0 858--876, 2011

  2. [2]

    Metformin affects gut microbiome composition and function and circulating short-chain fatty acids: a randomized trial

    Noel T Mueller, Moira K Differding, Mingyu Zhang, Nisa M Maruthur, Stephen P Juraschek, Edgar R Miller III, Lawrence J Appel, and Hsin-Chieh Yeh. Metformin affects gut microbiome composition and function and circulating short-chain fatty acids: a randomized trial. Diabetes care, 44 0 (7): 0 1462--1471, 2021

  3. [3]

    A general approach to causal mediation analysis

    Kosuke Imai, Luke Keele, and Dustin Tingley. A general approach to causal mediation analysis. Psychological methods, 15 0 (4): 0 309, 2010

  4. [4]

    Mediation analysis for survival data using semiparametric probit models

    Yen-Tsung Huang and Tianxi Cai. Mediation analysis for survival data using semiparametric probit models. Biometrics, 72 0 (2): 0 563--574, 2016

  5. [5]

    Odds ratios for mediation analysis for a dichotomous outcome

    Tyler J VanderWeele and Stijn Vansteelandt. Odds ratios for mediation analysis for a dichotomous outcome. American journal of epidemiology, 172 0 (12): 0 1339--1348, 2010

  6. [6]

    Causal mediation analysis with survival data

    Tyler J VanderWeele. Causal mediation analysis with survival data. Epidemiology, 22 0 (4): 0 582--585, 2011

  7. [7]

    Asymptotic confidence intervals for indirect effects in structural equation models

    Michael E Sobel. Asymptotic confidence intervals for indirect effects in structural equation models. Sociological methodology, 13: 0 290--312, 1982

  8. [8]

    A comparison of methods to test mediation and other intervening variable effects

    David P MacKinnon, Chondra M Lockwood, Jeanne M Hoffman, Stephen G West, and Virgil Sheets. A comparison of methods to test mediation and other intervening variable effects. Psychological methods, 7 0 (1): 0 83, 2002

Show all 17 references
  1. [9]

    On the behaviour of wald statistics for the disjunction of two regular hypotheses

    GFV Glonek. On the behaviour of wald statistics for the disjunction of two regular hypotheses. Journal of the Royal Statistical Society: Series B (Methodological), 55 0 (3): 0 749--755, 1993

  2. [10]

    Cauchy combination test: a powerful test with analytic p-value calculation under arbitrary dependency structures

    Yaowu Liu and Jun Xie. Cauchy combination test: a powerful test with analytic p-value calculation under arbitrary dependency structures. Journal of the American Statistical Association, 115 0 (529): 0 393--402, 2020

  3. [11]

    Adaptive bootstrap tests for composite null hypotheses in the mediation pathway analysis

    Yinqiu He, Peter X K Song, and Gongjun Xu. Adaptive bootstrap tests for composite null hypotheses in the mediation pathway analysis . Journal of the Royal Statistical Society Series B: Statistical Methodology, 86 0 (2): 0 411--434, 11 2023. ISSN 1369-7412. doi:10.1093/jrsssb/q...

  4. [12]

    H.C. Yeh, N. Maruthur, N.Y. Wang, G. Jerome, A. Dalcin, E. Tseng, K. White, E. Miller, S. Juraschek, N. Mueller, et al. A randomized controlled trial of behavioral weight loss and metformin on IGFs in cancer survivors. Obesity, 28: 0 17, 2020

  5. [13]

    Yeh, N.M

    H.C. Yeh, N.M. Maruthur, N.Y. Wang, G.J. Jerome, A.T. Dalcin, E. Tseng, K. White, E.R. Miller, S.P. Juraschek, N.T. Mueller, et al. Effects of behavioral weight loss and metformin on insulin-like growth factors in cancer survivors: A randomized trial. J. Clin. Endocrinol. Meta...

  6. [14]

    Metformin is associated with higher relative abundance of mucin-degrading akkermansia muciniphila and several short-chain fatty acid--producing microbiota in the gut

    Jacobo De La Cuesta-Zuluaga, Noel T Mueller, Vanessa Corrales-Agudelo, Eliana P Vel \'a squez-Mej \' a, Jenny A Carmona, Jos \'e M Abad, and Juan S Escobar. Metformin is associated with higher relative abundance of mucin-degrading akkermansia muciniphila and several short-chai...

  7. [15]

    Anti-inflammatory properties of the short-chain fatty acids acetate and propionate: a study with relevance to inflammatory bowel disease

    Sofia Tedelind, Fredrik Westberg, Martin Kjerrulf, and Alexander Vidal. Anti-inflammatory properties of the short-chain fatty acids acetate and propionate: a study with relevance to inflammatory bowel disease. World journal of gastroenterology: WJG, 13 0 (20): 0 2826, 2007

  8. [16]

    Increases in circulating and fecal butyrate are associated with reduced blood pressure and hypertension: results from the spirit trial

    Curtis Tilves, Hsin-Chieh Yeh, Nisa Maruthur, Stephen P Juraschek, Edgar Miller, Karen White, Lawrence J Appel, and Noel T Mueller. Increases in circulating and fecal butyrate are associated with reduced blood pressure and hypertension: results from the spirit trial. Journal o...

  9. [17]

    Xiushuang Chen, Qinghui Kong, Xiaoxiao Zhao, Chenxi Zhao, Pin Hao, Irfan Irshad, Hongjun Lei, Muhammad Fakhar-e-Alam Kulyar, Zeeshan Ahmad Bhutta, Hassan Ashfaq, et al. Sodium acetate/sodium butyrate alleviates lipopolysaccharide-induced diarrhea in mice via regulating the gut...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.