Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Multiple Testing in Generalized Universal Inference

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Generalized universal inference e-values can be plugged into the e-BH procedure to control false discovery rates for risk minimizers, with finite-sample validity and no likelihood assumptions.

desk verdict A workmanlike extension of GUe-values to e-BH, correct under stated assumptions but with a real gap between theorem and implementation. read the letter →

arxiv 2412.01008 v1 pith:7NQRVKND submitted 2024-12-01 stat.ME

Neil Dey , Ryan Martin , Jonathan P. Williams This is my paper · ORCID
classification stat.ME
keywords e-valuefalsediscoveryrategeneralizeduniversalinferenceempiricalriskminimizationquantileregressione-BHproceduremultipletestinglearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to show that e-values built through generalized universal inference (GUe-values) can be plugged into the e-BH multiple-testing procedure, yielding finite-sample control of the false discovery rate when the hypotheses concern the minimizer of a risk function. The central theoretical result is that if each GUe-value uses a learning rate satisfying the strong central condition and all null hypotheses are true, the combined e-value $G_M$ has type I error at most $\alpha$; if at least one null is false and the empirical risks converge uniformly, the test is consistent. This matters because ordinary e-value constructions require a correctly specified statistical model, whereas quantile regression and other risk-minimization problems do not come with a trustworthy likelihood. The paper demonstrates the procedure on quantile regression, where it checks whether a covariate matters at any of several quantiles, and reports simulation power rising as the signal or sample size grows.

What carries the argument

The central object is the GUe-value, which converts a risk-minimization problem into an e-value by exponentiating the gap between the validation-set empirical risk at a candidate or null region and at the training-set empirical risk minimizer, scaled by a learning rate $\omega$. The proof engine is the strong central condition, $\mathbb{E}\exp[-\omega(\ell(\theta;Z)-\ell(\theta^*;Z))] \le 1$ for all $\theta\in\Theta$ and $\omega\in[0,\bar\omega)$, which is exactly the inequality that makes the expectation of each GUe-value at most one. On top of that, the e-BH transformation $e_m^* = m e_m/M$ and the averaged combination $G_M$ turn a set of possibly dependent GUe-values into a single test with the stated error control; in the simulations, the learning rates are selected by the bootstrap-calibration algorithm suggested in the GUe framework.

What would settle it

Run the implemented procedure with bootstrap-selected learning rates under a bounded-loss global null where one can verify a fixed learning rate satisfies the strong central condition, with $M=49$, $n=50$, and $\alpha=0.1$; if the empirical rejection frequency of $G_M$ exceeds $\alpha$ by more than Monte Carlo error, Theorem 1 does not cover the data-selected learning rate and the finite-sample claim needs an added argument.

Watch

Extended reading notes

Core claim

The paper's claim is that the GUe-value, $G_n(\theta) = \exp[-\omega |S_2|(\hat R_{S_2}(\hat\theta_1) - \hat R_{S_2}(\theta))]$, is a valid e-value (expectation at most one under the null) for a risk-minimizer null whenever the strong central condition holds, and that such e-values are exactly the right input for the e-BH procedure. Sorting the GUe-values and working with $e_m^* = m e_m / M$ therefore controls the false discovery rate under arbitrary dependence. For the meta-hypothesis that none of the individual nulls is false, the paper combines the sorted GUe-values into $G_M = M^{-1}\sum_{m=1}^M (m/M)\,G_{(m)}^{n_m}(\Theta_0^{(m)})$ and proves that under the complete null $\Pr(G_M \ge \alpha^{-1}) \le \alpha$, while under uniform convergence of empirical risk and at least one false null the rejection probability tends to 1. The intended application is quantile regression, where the target coefficients are minimizers of expected check loss rather than parameters of a likelihood.

Load-bearing premise

The argument depends on each learning rate being small enough that the strong central condition holds and on that rate being chosen before the validation data are seen, while the simulations instead pick the rate from the data by bootstrap calibration and no theorem shows the selected rate still satisfies the condition.

Editorial extensions

If this is right

  • Any ERM-based testing problem can inherit the recipe: if the strong central condition holds for each learning rate, a single combined GUe-value gives a finite-sample $\alpha$-level test without specifying a likelihood.
  • The e-BH step controls FDR at level $\alpha$ even when the individual GUe-values are arbitrarily dependent, so strongly correlated quantile-specific tests do not need an independence assumption.
  • In quantile regression, the meta-test 'this predictor matters at some quantile' is consistent: whenever the predictor has nonzero coefficient at a tested quantile, the rejection probability of $G_M$ tends to 1 as $n\to\infty$.
  • Because the construction only needs a loss function and the strong central condition, the same multiple-testing scheme applies beyond quantile regression to any target defined as a risk minimizer, such as robust M-estimates or the minimum clinically important difference.
  • The simulation results indicate that the combined test distinguishes all-null settings, where its type II error is near $1-\alpha$, from settings with genuine signals, where the type II error drops as the signal strength or sample size grows.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A consequence the paper leaves implicit is that the e-BH layer also protects mixtures of true and false nulls at the individual level; the reported FDR is only checked under the global null, so a simulation with a sparse set of false nulls would directly test that part of the claim.
  • The theorems cover fixed learning rates, but the implementation chooses learning rates from the same data via bootstrap calibration; proving that the selected rate still satisfies the strong central condition would close the gap between the theoretical guarantee and the simulated procedure.
  • The average-with-weight $(m/M)$ is one valid symmetric merge; other merging functions from the e-value literature could give higher power in particular signal patterns, and the paper does not compare them.
  • For the full quantile-regression question, a natural extension is to let the number of tested quantiles grow with $n$ or to cover all of $[0,1]$ using the online version of the GUe-value; the paper names this as future work, but it is a direct testable next step.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The manuscript proposes a multiple-testing method that combines generalized universal inference (GUe) e-values with the e-BH procedure. For hypotheses about minimizers of risk functions, each test uses a GUe-value G_n(Θ0)=inf_{θ∈Θ0} exp[-ω|S2|(R̂_S2(θ̂_S1)-R̂_S2(θ))]. The authors prove in Theorem 1 that when every GUe-value uses a learning rate satisfying the strong central condition and all nulls are true, the averaged e-BH-transformed e-value GM in Eq. (1) satisfies Pr(GM≥α^{-1})≤α; Theorem 2 gives consistency when at least one null is false, under a uniform-convergence condition on the empirical risk. Simulations for quantile regression assess Type II error and report FDR at the global null, with learning rates selected by Algorithm 1 of Dey et al. (2024).

Significance. If the guarantees are extended to the implemented learning-rate calibration, the paper offers a useful finite-sample-error-control framework for multiple testing on risk minimizers without likelihood assumptions, and it identifies quantile regression as a natural application. Feeding GUe-values into e-BH is a straightforward but potentially valuable combination because e-BH controls FDR under arbitrary dependence among e-values. The paper is also transparent about relying on bootstrap calibration for the learning rate; however, the current manuscript does not establish that the simulations are covered by the theorems, so the advertised finite-sample validity is not yet fully supported.

major comments (3)
  1. [Section 4 (Simulations), with Sections 2 and 3] The finite-sample e-value guarantee in Theorem 1 is conditional on the learning rate satisfying the strong central condition and, implicitly in the GUe construction, being chosen independently of the validation sample S2. In the simulations, learning rates are chosen by Algorithm 1 of Dey et al. (2024), but the manuscript neither states whether that algorithm uses only S1 nor proves that the bootstrap-selected ω satisfies the strong central condition or is independent of S2. Consequently, the implemented procedure is not covered by the stated guarantees, and the abstract's claim of finite-sample valid error control is too strong as written. Please either prove a preservation result for the calibrated learning rate, or cleanly separate the validity simulations using fixed ω known to satisfy the strong central condition from the heuristic calibrated-ω results.
  2. [Section 4 (Examples 1 and 2)] The empirical FDR evidence is limited to the all-null case: at Δ=0 and Γ=0 the FDR of the unmerged GUe-values is reported as 0.04, based on 100 Monte Carlo iterations. FDR control is a property that must hold under arbitrary mixtures of true and false nulls; the all-null FDR is only the FWER special case and cannot detect inflation caused by the learning-rate calibration in partial-null configurations. Please add partial-null FDR simulations with standard errors, and preferably report the distribution of calibrated learning rates to show that their data dependence does not invalidate the e-value property.
  3. [Theorem 1, proof] The proof of Theorem 1 is not 'virtually identical' to Theorem 2 of Wang and Ramdas (2022), which is a statement about FDR control of the e-BH procedure, not a probability bound for the particular average in Eq. (1). Because this is the paper's main finite-sample validity result, please include the short derivation; for example, for nonnegative e-values with expectation at most 1, E[Σ_{m=1}^M m G_(m)] ≤ M(M+1)/2, so E[GM] ≤ (M+1)/(2M), and Markov's inequality gives the stated bound.
minor comments (5)
  1. [Abstract and Introduction] The paper repeatedly claims FDR control, but Theorem 1 is a global Type I error bound for GM and no theorem explicitly states the e-BH FDR guarantee when applied to GUe-values; please state this as a corollary so that the headline claim is directly supported by a displayed result.
  2. [Section 4 (Examples 1 and 2)] The statement that 'the strong central condition easily holds because the data are bounded' is too quick: boundedness alone does not imply E exp[-ω(ℓ(θ;Z)-ℓ(θ*;Z))] ≤ 1 for all θ and all ω in [0,ω̄), which also requires identifiability or a suitable separation property for the risk minimizer. Please give the specific argument or cite the relevant condition from Dey et al. (2024).
  3. [Section 4 (Examples 1 and 2)] At 100 Monte Carlo iterations, the standard error of an estimated proportion near 0.05 is about 0.02, so the reported 0.04 FDR should be accompanied by binomial confidence intervals or standard errors; the same applies to the Type II error curves.
  4. [Theorem 2] The uniform-convergence assumption sup_{ϑ∈Θ_m}|R̂_n^{(m)}(ϑ)-R^{(m)}(ϑ)|=o_p(1) is stated but not verified for the quantile-regression loss; please state compactness and Lipschitz conditions under which it holds, or cite a standard uniform law of large numbers.
  5. [Eq. (1)] The superscripts (m) denote sorted GUe-values, but the null sets Θ0^m are then relabeled by the same sorted index; this relabeling should be stated explicitly to avoid ambiguity about which hypothesis corresponds to which entry of the sorted list.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the main theorem is a conditional corollary of GUe-value validity and e-BH; the calibration-algorithm gap in the simulations is a missing proof, not a circular reduction.

full rationale

The paper's derivation chain is not circular. Theorem 1 assumes that every sorted GUe-value uses a learning rate satisfying the strong central condition, which is exactly the prior theorem from Dey et al. (2024) making each G_n(Theta0) an e-value. Given that input, GM is an average of e-BH-transformed e-values and hence itself an e-value, so Markov's inequality yields the stated bound. This is a direct corollary of the cited e-BH and GUe-value theorems, not a self-fulfilling fit. Theorem 2 likewise applies the prior consistency theorem under the stated uniform-convergence assumption; no input is renamed as a prediction. The self-citation to Dey et al. (2024) is load-bearing for the GUe-value construction, but that prior theorem has stated assumptions that do not include the multiple-testing target, so under the stated rules it counts as independent support rather than circularity. The only substantive weakness, visible in Section 4, is that learning rates in the simulations are chosen by Algorithm 1 of Dey et al. (2024), and the paper provides no theorem showing that the selected omega satisfies the strong central condition or is independent of the validation split. Consequently, the reported empirical FDR of 0.04 at the global null is a heuristic check, not a guarantee inherited from Theorem 1. This is an omitted-support or validity gap, not an equation-level reduction of the prediction to the fitted input, so it does not warrant a circularity score above 1.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The paper imports the GUe-value validity theorem, the strong central condition, and the e-BH theorem from prior work, two of which are by the same authors. The only fitted constant is the learning rate, which is chosen by calibration rather than by a proof. No new entities are introduced.

free parameters (2)
  • Learning rate omega_m for each GUe-value = Varies per quantile and dataset; selected by Algorithm 1 of Dey et al. (2024)
    The GUe-value is exp(-omega |S2| (Rhat_S2(theta_1)-Rhat_S2(theta))). Its e-value property holds only for omega in (0, omega_bar) satisfying the strong central condition. The paper selects omega by bootstrap calibration without proving the chosen value preserves validity.
  • Sample split ratio n1/n2 = Not stated
    GUe-value validity and power depend on how S is split into training and validation sets; the paper does not report the split.
assumptions (4)
  • domain assumption Strong central condition holds for each null with some omega > 0
    Needed for each GUe-value to be a valid e-value (Section 2 and Theorem 1 of Dey et al. 2024). The paper asserts it holds because data are bounded in simulations, but does not verify it for general use.
  • domain assumption Uniform convergence of empirical risk: sup |Rhat_n - R| = op(1)
    Assumed in Theorem 2 to conclude consistency of the combined GUe-value; not proved for the quantile regression setup.
  • standard math e-BH procedure of Wang and Ramdas (2022) controls FDR under arbitrary dependence
    External theorem used as a black box in Section 3.
  • domain assumption Validity and consistency theorems for GUe-values from Dey et al. (2024)
    Theorems 1 and 2 of this paper rely directly on Theorem 1 and Theorem 2 of the authors' previous paper; no proof is reproduced here.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multiple Testing in Generalized Universal Inference." pith.science (2026). https://pith.science/paper/7NQRVKND

@misc{pith2026241201008,
  author       = {Pith},
  title        = {Pith review of: Multiple Testing in Generalized Universal Inference},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7NQRVKND}},
  note         = {Machine review of arXiv:2412.01008}
}
read the original abstract

Compared to p-values, e-values provably guarantee safe, valid inference. If the goal is to test multiple hypotheses simultaneously, one can construct e-values for each individual test and then use the recently developed e-BH procedure to properly correct for multiplicity. Standard e-value constructions, however, require distributional assumptions that may not be justifiable. This paper demonstrates that the generalized universal inference framework can be used along with the e-BH procedure to control frequentist error rates in multiple testing when the quantities of interest are minimizers of risk functions, thereby avoiding the need for distributional assumptions. We demonstrate the validity and power of this approach via a simulation study, testing the significance of a predictor in quantile regression.

Figures

Figures reproduced from arXiv: 2412.01008 by the authors.

Figure 1
Figure 1. Type II error rate of the combined GUe-value in Example 1, estimated using 100 [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Type II error rate of the combined GUe-value in Example 2, estimated using 100 [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. The average learning rate (over 100 Monte Carlo samples) chosen by Algorithm [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Sequential Scoring Rule Evaluation for Forecast Method Selection

    math.ST 2025-05 conditional novelty 6.0 of 10

    A sequential test based on ratios of scoring rules is shown to be a generalized e-value, yielding finite-sample error control for forecast method selection.

Reference graph

Works this paper leans on

28 extracted references · 12 canonical work pages · cited by 1 Pith paper

  1. [1]

    Benjamini and Y

    Y. Benjamini and Y. Hochberg. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society Series B: Statistical Methodology, 57: 0 289--300, 1995. doi:10.1111/J.2517-6161.1995.TB02031.X

    arXiv 1995

  2. [2]

    Benjamini and D

    Y. Benjamini and D. Yekutieli. The control of the false discovery rate in multiple testing under dependency. The Annals of Statistics, 29 0 (4): 0 1165--1188, 2001. ISSN 00905364, 21688966. URL http://www.jstor.org/stable/2674075

    arXiv 2001

  3. [3]

    P. G. Bissiri, C. Holmes, and S. G. Walker. A general framework for updating belief distributions. Journal of the Royal Statistical Society Series B: Statistical Methodology, 78 0 (5): 0 1103--1130, 2016. doi:10.1111/rssb.12158

    doi:10.1111/rssb.12158 2016

  4. [4]

    Cella and R

    L. Cella and R. Martin. Direct and approximately valid probabilistic inference on a class of statistical functionals. International Journal of Approximate Reasoning, 151: 0 205--224, 2022. doi:10.1016/j.ijar.2022.09.011

    work page doi:10.1016/j.ijar.2022.09.011 2022

  5. [5]

    W. T. Chung and K. C. Chung. The use of the e-value for sensitivity analysis. Journal of Clinical Epidemiology, 163: 0 92--94, 2023. doi:10.1016/j.jclinepi.2023.09.014

    work page doi:10.1016/j.jclinepi.2023.09.014 2023

  6. [6]

    de Heide, A

    R. de Heide, A. Kirichenko, P. Gr\" u nwald, and N. Mehta. Safe- B ayesian generalized linear regression. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 2623--2633. PMLR, 26--28 Aug 2020

    work page 2020

  7. [7]

    N. Dey, R. Martin, and J. P. Williams. Generalized universal inference on risk minimizers, 2024. URL https://arxiv.org/abs/2402.00202

    arXiv 2024

  8. [8]

    Gr \"u nwald and T

    P. Gr \"u nwald and T. van Ommen. Inconsistency of B ayesian inference for misspecified linear models, and a proposal for repairing it. Bayesian Analysis, 12 0 (4): 0 1069 -- 1103, 2017. doi:10.1214/17-BA1085

    doi:10.1214/17-ba1085 2017

Show all 28 references
  1. [9]

    Gr\"unwald, R

    P. Gr\"unwald, R. de Heide, and W. Koolen. Safe testing. Journal of the Royal Statistical Society Series B: Statistical Methodology, 86 0 (5): 0 1091--1128, 2024. doi:10.1093/jrsssb/qkae011

    2024 doi

  2. [10]

    P. D. Gr\" u nwald and N. A. Mehta. Fast rates for general unbounded loss functions: from ERM to generalized B ayes. Journal of Machine Learning Research, 21 0 (56): 0 1--80, 2020

    2020

  3. [11]

    Hedayat, J

    S. Hedayat, J. Wang, and T. Xu. Minimum clinically important difference in medical studies. Biometrics, 71: 0 33--41, 2015

    2015

  4. [12]

    C. C. Holmes and S. G. Walker. Assigning a value to a power likelihood in a general B ayesian model. Biometrika, 104 0 (2): 0 497--503, 03 2017. ISSN 0006-3444. doi:10.1093/biomet/asx010

    2017 doi

  5. [13]

    P. J. Huber. Robust Statistics. Wiley Series in Probability and Statistics. John Wiley & Sons, Inc, 1981

    1981

  6. [14]

    Hudson, M

    A. Hudson, M. Carone, and A. Shojaie. Inference on function-valued parameters using a restricted score test, 2021. URL https://arxiv.org/abs/2105.06646

    2021 arXiv

  7. [15]

    Ignatiadis, R

    N. Ignatiadis, R. Wang, and A. Ramdas. E-values as unnormalized weights in multiple testing, 2023. URL https://arxiv.org/pdf/2204.12447

    2023 arXiv

  8. [16]

    Koenker and G

    R. Koenker and G. Bassett, Jr. Regression quantiles. Econometrica, 46 0 (1): 0 33--50, 1978. ISSN 0012-9682,1468-0262

    1978

  9. [17]

    S. P. Lyddon, C. C. Holmes, and S. G. Walker. General Bayesian updating and the loss-likelihood bootstrap . Biometrika, 106 0 (2): 0 465--478, 03 2019. ISSN 0006-3444. doi:10.1093/biomet/asz006

    2019 doi

  10. [18]

    Martin and N

    R. Martin and N. Syring. Direct G ibbs posterior inference on risk minimizers: Construction, concentration, and calibration. In Advancements in Bayesian Methods and Implementation, volume 47 of Handbook of Statistics, pages 1--41. Elsevier, 2022. doi:10.1016/bs.host.2022.06.004

    2022 doi

  11. [19]

    H. D. Nguyen and M. Gupta. Finite sample inference for empirical B ayesian methods. Scandinavian Journal of Statistics, 50 0 (4): 0 1616--1640, 2023. doi:10.1111/sjos.12643

    2023 doi

  12. [20]

    Ramdas and R

    A. Ramdas and R. Wang. Hypothesis testing with e-values, 2024. URL https://arxiv.org/abs/2410.23614

    2024 arXiv

  13. [21]

    Shafer, A

    G. Shafer, A. Shen, N. Vereshchagin, and V. Vovk. Test martingales, B ayes factors and p-values. Statistical Science, 26 0 (1): 0 84--101, 2011. ISSN 08834237. URL http://www.jstor.org/stable/23059157

    2011

  14. [22]

    Syring and R

    N. Syring and R. Martin. Gibbs posterior inference on the minimum clinically important difference. J. Statist. Plann. Inference, 187: 0 67--77, 2017. ISSN 0378-3758

    2017

  15. [23]

    Syring and R

    N. Syring and R. Martin. Calibrating general posterior credible regions. Biometrika, 106 0 (2): 0 479--486, 2019. doi:10.1093/biomet/asy054

    2019 doi

  16. [24]

    van Erven, P

    T. van Erven, P. D. Gr\" u nwald, N. A. Mehta, M. D. Reid, and R. C. Williamson. Fast rates in statistical and online learning. Journal of Machine Learning Research, 16 0 (54): 0 1793--1861, 2015

    2015

  17. [25]

    Vovk and R

    V. Vovk and R. Wang. E-values: Calibration, combination and applications. Annals of Statistics, 49 0 (3), 2021. doi:10.1214/20-AOS2020

    2021 doi

  18. [26]

    Vovk and R

    V. Vovk and R. Wang. Confidence and discoveries with e-values. Statistical Science, 38 0 (2): 0 329 -- 354, 2023. doi:10.1214/22-STS874

    2023 doi

  19. [27]

    Wang and A

    R. Wang and A. Ramdas. False discovery rate control with e-values. Journal of the Royal Statistical Society Series B: Statistical Methodology, 84 0 (3): 0 822--852, 01 2022. ISSN 1369-7412. doi:10.1111/rssb.12489

    2022 doi

  20. [28]

    Wasserman, A

    L. Wasserman, A. Ramdas, and S. Balakrishnan. Universal inference. Proceedings of the National Academy of Sciences, 117 0 (29): 0 16880--16890, 2020

    2020

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.