{"id":"ad49d88d-750d-42b8-a5a9-d0b237249c82","arxiv_id":"2507.03815","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"The g-value aggregates p-values across multiple analysis strategies by rescaling the largest p-value with a null-distribution-derived correction, controlling type I error while preserving power.","lead":"Scientists often test one hypothesis many ways, then pick the answer they like. This paper introduces a 'g-value' that combines all the p-values from these multiple analyses into one test while keeping the false-positive rate at its nominal level. It is meant for multi-analyst and multi-method studies.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Example 1's joint-null covariance is not distribution-free: for non-Gaussian symmetric F, the t-test and signed-rank correlation differs from sqrt(3/pi), so the derived alpha* is not generally valid.","rationale":"The reader correctly identifies the general computation of alpha* as the weak point, but the specific analytic example contains a concrete mathematical error that is more immediately checkable than the resampling algorithm. The g-value framework itself controls size by definition once the joint null distribution is known, so the central idea is not false; however, the paper's only analytical demonstration of how to obtain that distribution is flawed for non-Gaussian symmetric errors, and the numerical illustration inherits that flaw. This supports keeping the conditional verdict: the paper needs either a corrected analytic example, an explicit distributional assumption, or a validated numerical procedure for alpha* before the general claims can be accepted. The concern is technical, not personal, and is settled by an elementary closed-form computation or a targeted simulation.","tokens_in":10730,"tokens_out":11399,"duration_ms":137943,"concrete_test":"Recompute the covariance in Eq. (3) for X ~ Uniform(-1,1): since G(x)=x and sigma=1/sqrt(3), (sqrt(3)/sigma) E[|X|G(|X|)] = 1 rather than sqrt(3/pi), directly contradicting the claimed joint distribution. Then estimate the actual t15 correlation by large-n simulation, recompute alpha* from Eq. (4) with that correlation, and repeat the Section 3.2 Monte Carlo; if the empirical type I error shifts away from 0.05, the paper's alpha* is not distribution-free and the example needs correction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The g-value's size control is by construction once the joint null distribution of the p-values is known, so the load-bearing question is whether the paper supplies a valid way to compute alpha*. The analytic example does not. In Eq. (3), T_n^(2) is approximated by (sqrt(3)/sqrt(n)) sum G(|X_i|) sign(X_i), where G is the CDF of |X_i|. The asymptotic covariance between T_n^(1) and T_n^(2) is then (sqrt(3)/sigma) E[|X_i| G(|X_i|)]. This quantity is not universal. For X ~ Uniform(-1,1), G(x)=x and sigma=1/sqrt(3), so the covariance equals 1 (the two tests are asymptotically identical); for X ~ N(0,sigma^2), integration by parts gives sqrt(3/pi) ~ 0.977, the value used in Eq. (4). Thus the covariance matrix [[1, sqrt(3/pi)], [sqrt(3/pi), 1]] and the resulting alpha* ~ 0.0601 are only correct for a specific error distribution, not for every symmetric F as the example claims. The Section 3.2 simulation uses t15 data with an alpha* derived from the normal correlation; the reported size close to 0.05 is therefore not backed by the derivation. This does not invalidate the g-value framework itself, but it removes the only analytical demonstration supporting 'rigorously controls type I error' for arbitrary multi-design sets, leaving the unvalidated resampling algorithm of Section 4.2 as the sole general route.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a 'g-value' for multi-design studies: for a pre-specified set A of analysis strategies, one computes the p-value P_n(A) for each strategy, takes the largest p-value, and rescales it by α/α*, where α* is the α-quantile of the null distribution of the maximum p-value. The authors argue that this construction controls the asymptotic type I error at α while avoiding the conservatism of simply reporting the largest p-value, and that it reduces selective reporting by forcing researchers to enumerate A. The method is illustrated with a two-test example involving the one-sample t-test and the Wilcoxon signed-rank test, supported by a small Monte Carlo study, and the paper discusses practical computation via a bootstrap-plus-permutation algorithm and a quantile-based robust variant.","tokens_in":11083,"tokens_out":26180,"duration_ms":282655,"significance":"Conceptually, the g-value is an elegant and appealing idea: instead of aggregating p-values through an arbitrary combination rule, one calibrates the maximum p-value against its true null distribution, thereby turning the most conservative multi-design summary into an exact-level test. If the joint null distribution of the p-values were available, the construction would deliver much of what the abstract promises. The paper also usefully distinguishes its setting from classical multiple testing and selective inference. However, the current manuscript does not establish the required null distribution in any general setting: the analytic example is valid only under additional distributional assumptions, the general resampling algorithm is unvalidated, and the quantile variant as written does not control size. The contribution is therefore a promising framework rather than a demonstrated method.","major_comments":[{"comment":"The covariance between the normalized t-statistic and the Wilcoxon signed-rank statistic is not universal. From the influence-function representation in Eq. (3), the covariance equals (√3/σ) E[|X_i| G(|X_i|)], which depends on F through G and σ. The value √(3/π) used in Eq. (4) is correct for Gaussian F but not for every symmetric F. For instance, if X_i ~ Uniform(-1,1), then σ=1/√3 and G(x)=x, so the covariance equals 1; the two tests are asymptotically equivalent and the null distribution of max P is Uniform(0,1), so α* should be 0.05, not 0.0601. Consequently, the claim that Example 1 demonstrates asymptotic size α for arbitrary symmetric F is false, and the Section 3.2 simulation, which feeds the normal-based α* into t15 data, does not provide the advertised evidence. The framework itself survives if α* is computed from the correct joint null distribution, but the paper's only analytical demonstration needs either a restriction to a specific F or a derivation of the F-dependent correction.","section":"Section 3.1, Eq. (3) and Eq. (4)"},{"comment":"The robust g-value G_n(A;γ) replaces max_A P_n(A) by Quantile_γ but keeps the same α* defined in Eq. (2) from the null distribution of the maximum. This does not control type I error: because Quantile_γ ≤ max_A P_n(A) pointwise for γ<1, the event {Quantile_γ ≤ α*} contains {max_A P_n(A) ≤ α*}, so the asymptotic rejection probability is at least α, and generically strictly larger. The quantile variant requires its own calibration constant computed from the null distribution of the quantile statistic.","section":"Section 4.1 (quantile variant)"},{"comment":"The bootstrap-with-replacement followed by permutation proposal is the only general route to α* for arbitrary A, but it is not accompanied by any formal result or simulation. The statement that permutation 'ensures that the null hypothesis holds for all resampled datasets' addresses the marginal validity of each test, not the joint null distribution of the entire vector of p-values under the original data-generating process; for analyses that differ in variable transformations, model specification, or preprocessing, the resampling scheme can alter the dependence structure. As written, the paper provides no evidence that the resampled maximum-p distribution consistently estimates the quantity needed in Eq. (2). A theorem with explicit conditions or a simulation study is required before the general claims in the abstract and conclusion can be accepted.","section":"Section 4.2 (computation)"},{"comment":"The paper states in the abstract and conclusion that the g-value 'rigorously controls type I error rates' and 'maintains strong statistical power.' These claims are not established for the general setting: no theorem is stated for the existence and uniqueness of α* under the stated regularity conditions, and the power statement is only illustrated in the two-test example. Please restate the guarantees as asymptotic and conditional on a correctly specified null distribution of max_A P_n(A), or prove them under explicit assumptions.","section":"Section 2 and Section 5 (scope of claims)"}],"minor_comments":[{"comment":"The one-sample procedure used in Example 1 is the Wilcoxon signed-rank test, not the Wilcoxon rank-sum test; the terminology should be corrected.","section":"Section 3.1 and Section 3.2"},{"comment":"The notation G_n(A) for the g-value collides with the CDF G(·) used in Eq. (3); consider renaming the CDF (e.g., F_{|X|}).","section":"Section 3.1"},{"comment":"The notation 'argzero' is non-standard; define it as the unique γ satisfying the displayed equation, and state conditions for existence and uniqueness.","section":"Section 3.1, Eq. (2)"},{"comment":"With 10^5 Monte Carlo replications, the Monte Carlo standard error of a rejection proportion near 0.05 is about 0.0007; reporting the estimated type I error with this precision or a confidence interval would make the 'close to 0.05' claim quantitative.","section":"Section 3.2"},{"comment":"The 'algorithm' is described only in prose; a step-by-step pseudocode and a discussion of how α* is estimated from the resampled distribution would improve reproducibility.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a short, well-written proposal, but the gap between the stated guarantees and the demonstrated results is substantial. I would encourage a major revision rather than rejection because the core calibration idea is sound and could be made correct with additional work; however, the current version should not be published without either restricting the claims or supplying the missing distributional results and simulations."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe g-value idea is genuinely useful: instead of taking the max p-value and being conservative, or the min and being liberal, they rescale the max by a factor derived from the joint null distribution of the test statistics. For a pair (t-test, Wilcoxon signed-rank) under normal errors, they get an alpha* of 0.0601 and show by simulation that the type I error is around 0.05. That two-test case is clean, and the central construction is not in the previous specification-curve or p-value combination literature I know.\n\nThe problem is that the paper sells the example as general for any symmetric F, and that's false. The asymptotic correlation between the t and Wilcoxon statistics is sqrt(3/pi) only under normality; for a uniform error distribution it's 1. So the derived alpha* does not control size for arbitrary symmetric F. The simulation uses t15 data with the normal-derived alpha*, so that's not evidence for the general claim. This matters because the analytic example is the only fully worked demonstration of the framework; the general route is the bootstrap+permutation algorithm in Section 4.2, and that has no theory and no simulation. The quantile variant and the power-preservation claim also rest on the same unproven foundation.\n\nNone of this kills the idea. If you know the joint null distribution, the construction works by definition; the issue is that computing that distribution is usually hard, and the paper doesn't yet provide a reliable general method or a worked non-normal example. The literature review is fine, and the connections to specification curves and many-analyst studies are honest.\n\nFor me this is a conditional paper: the framework deserves referee time, but it needs a corrected Example 1 (either restrict to normality or derive the distribution-dependent alpha*), proper simulations for the resampling algorithm, and a precise statement of what is proven versus conjectured. As is, I would not cite it as a general solution.\n\nI'd send it to a statistician who works on multiple testing or resampling; they'll see the gap immediately but also see the useful kernel.","headline":"A neat rescaling idea for aggregating p-values, but the flagship analytic example only works for a specific error distribution, and the general algorithm is unvalidated.","tokens_in":11631,"tokens_out":2515,"would_cite":false,"duration_ms":26142,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62F03","62F40","62G10"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes a global p-value, the g-value, that aggregates all p-values from a multi-design study by rescaling the maximum p-value by a calibration constant, so the test keeps asymptotic size α with power close to a single valid…","keywords":["g-value","global p-value","multi-design studies","multi-analyst studies","p-hacking","selective reporting","type I error control","p-value aggregation"],"falsifier":"Take a small sample, run two analyses on the same variable where one uses a t-test and the other uses a different model specification or a transformation of the outcome, and apply the proposed bootstrap-then-permutation algorithm to compute $\\alpha^*$. Under $H_0$ with $\\mu = 0$, simulate the empirical rejection rate of $G_n(\\mathcal{A}) \\le \\alpha$ over many replications; if the rate exceeds $\\alpha$ by more than simulation error, the claimed size control fails. A second check applies the same simulation to the quantile variant, since the paper does not prove that the max-based $\\alpha^*$ remains valid when the statistic is replaced by a quantile.","tokens_in":10548,"feed_emoji":"📊","tokens_out":7418,"duration_ms":80365,"temperature":0.7,"pith_summary":"The paper argues that the usual response to analytical flexibility—either pick one test or take the most conservative one—is unnecessary. It introduces a global p-value, or g-value, defined as the largest p-value across a pre-specified set of legitimate analysis strategies, rescaled by a constant that depends on the set and on the significance level α. The constant is chosen so that, under the null hypothesis, the asymptotic probability that the g-value falls below α is exactly α. If this holds, a multi-design study can be summarized by one number that controls the false-positive rate, keeps power close to a single well-chosen test, and removes the incentive to report only the smallest p-value.","feed_headline":"Global p-value merges many analyses, keeps type-I error at 5%","feed_subtitle":"Instead of cherry-picking the smallest p-value, researchers report a calibrated maximum across all pre-specified analyses.","key_machinery":"The load-bearing object is the calibration constant $\\alpha^*(\\alpha, \\mathcal{A})$, the value $\\gamma \\in [\\alpha, 1]$ at which the asymptotic null probability that all p-values in $\\mathcal{A}$ exceed $1-\\gamma$ equals $\\alpha$. Equivalently, it inverts the null distribution of the largest p-value. In the paper's worked example, the two test statistics have an asymptotic bivariate normal distribution with correlation $\\sqrt{3/\\pi}$, so the null distribution of the maximum p-value has a closed form and $\\alpha^*$ can be computed by a one-dimensional root search. For settings without a closed form, the paper proposes a resampling algorithm—bootstrap resampling of observations followed by permutation of the variable of interest—to approximate the null distribution of the maximum p-value. A quantile variant, replacing the maximum by a $\\gamma$-quantile of the p-values, is offered as a robustness adjustment.","core_discovery":"At the paper's center is the decision rule $G_n(\\mathcal{A}) = (\\alpha/\\alpha^*(\\alpha,\\mathcal{A}))\\,\\max_{A\\in\\mathcal{A}} P_n(A)$, where $\\alpha^*(\\alpha,\\mathcal{A})$ solves $\\lim_{n\\to\\infty} \\Pr(\\max_{A\\in\\mathcal{A}} P_n(A) \\le \\gamma \\mid H_0) = \\alpha$ for $\\gamma$. Because the unadjusted maximum p-value is conservative—it rejects only when all tests happen to be significant—the paper rescales it by $\\alpha/\\alpha^*$, which is equivalent to comparing the largest p-value against the larger threshold $\\alpha^*$. The paper proves in a two-test example (one-sample t-test and Wilcoxon rank-sum test) that the asymptotic joint distribution of the test statistics is bivariate normal with correlation $\\sqrt{3/\\pi}$, computes $\\alpha^*(0.05,\\mathcal{A}) \\approx 0.0601$, and shows by simulation that empirical rejection rates under $H_0$ sit near $0.05$ while power tracks the individual tests rather than the conservative bound.","pith_inferences":["A natural extension not developed in the paper is to apply the same calibration logic to any class of asymptotically normal test statistics, so the g-value could serve as a template for aggregating specification-level effect sizes or confidence bounds.","Because the paper deliberately leaves dependent observations out of scope, extending the bootstrap-permutation scheme to time series would require a block-bootstrap or sieve-based null, which is a next step the paper does not take.","The requirement to enumerate $\\mathcal{A}$ in advance suggests a practical workflow: researchers register the set of analyses before looking at data, then report the g-value; this could be tested as an intervention on reporting behavior.","The quantile variant may control size only under stronger assumptions than the maximum variant; a simulation comparing type I error across quantile levels would clarify when it is safe."],"forward_implications":["A multi-design study can report a single g-value; rejecting $H_0$ when $G_n(\\mathcal{A}) \\le \\alpha$ gives an asymptotic type I error of $\\alpha$ regardless of how many strategies are in $\\mathcal{A}$.","The g-value's power is close to that of the individual valid tests, so the method avoids the large power loss of the most conservative p-value.","Because the set $\\mathcal{A}$ must be stated in advance, selective reporting of only the most favorable p-value is structurally discouraged.","The framework extends to regression and maximum-likelihood settings whenever estimators admit an influence-function representation; for complex cases the bootstrap-permutation algorithm provides a numerical route.","The quantile version lets researchers downweight a few poorly fitting analysis choices while retaining the g-value interpretation."],"supporting_citations":[{"why":"Provides the specification-curve alternative that the g-value competes with; the paper contrasts operating on p-values rather than effect sizes.","marker":"Simonsohn et al. (2020)"},{"why":"Supplies the representation of the Wilcoxon statistic as a sum of transformed signs and the joint central limit theorem used to compute the null distribution of the maximum p-value.","marker":"Lehmann and Romano (2022)"},{"why":"Source of the calibration-by-rescaling idea that the g-value adapts to control size while avoiding conservatism.","marker":"Boulaguiem et al. (2024b,a)"},{"why":"Baseline p-value aggregation method whose averaging approach the g-value deliberately avoids in favor of a calibrated maximum.","marker":"Vovk and Wang (2020)"},{"why":"Baseline harmonic-mean p-value method for dependent tests that the paper cites as an existing aggregation approach with limitations.","marker":"Wilson (2019)"}],"fun_headline_variants":["Global p-value combines analyses, checks selective reporting","Calibrated max p-value unifies multi-design studies","One global p-value to rule all analyses, no cherry-picking","Multi-design studies get one calibrated p-value, no cherry-picks","Rescale max p-value to keep type I error at 5%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on being able to compute or reliably approximate the null distribution of the maximum p-value over the full set $\\mathcal{A}$; if the resampling scheme does not produce a valid null for every analysis strategy simultaneously (for example, when the strategies use different models or transformations), the calibrated constant $\\alpha^*$ will not deliver the claimed error control.","fun_headline_variants_meta":{"raw":{"variants":["Global p-value combines analyses, checks selective reporting","Calibrated max p-value unifies multi-design studies","One global p-value to rule all analyses, no cherry-picking","Multi-design studies get one calibrated p-value, no cherry-picks","Rescale max p-value to keep type I error at 5%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00076,"raw_usage":{"total_tokens":3361,"prompt_tokens":920,"completion_tokens":2441,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":2355}},"tokens_in":536,"tokens_out":2441,"duration_ms":20099,"temperature":1.0,"reasoning_tokens":2355,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:02:02.252620+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a small sample, run two analyses on the same variable where one uses a t-test and the other uses a different model specification or a transformation of the outcome, and apply the proposed bootstrap-then-permutation algorithm to compute $\\alpha^*$. Under $H_0$ with $\\mu = 0$, simulate the empirical rejection rate of $G_n(\\mathcal{A}) \\le \\alpha$ over many replications; if the rate exceeds $\\alpha$ by more than simulation error, the claimed size control fails. A second check applies the same simulation to the quantile variant, since the paper does not prove that the max-based $\\alpha^*$ remains valid when the statistic is replaced by a quantile.","supporting_citations":[{"cited_title":"P., and Nelson, L","cited_arxiv_id":null,"evidence_quote":"Provides the specification-curve alternative that the g-value competes with; the paper contrasts operating on p-values rather than effect sizes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the representation of the Wilcoxon statistic as a sum of transformed signs and the joint central limit theorem used to compute the null distribution of the maximum p-value."},{"cited_title":"and Wang, R","cited_arxiv_id":null,"evidence_quote":"Baseline p-value aggregation method whose averaging approach the g-value deliberately avoids in favor of a calibrated maximum."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Baseline harmonic-mean p-value method for dependent tests that the paper cites as an existing aggregation approach with limitations."}],"review_version":1}