Global p-Values in Multi-Design Studies

Christophe P\'erignon; Francesca Chiaromonte; Guillaume Coqueret; St\'ephane Guerrier; Yuming Zhang

REVIEW 4 major objections 5 minor 48 references

Global p-Values in Multi-Design Studies

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proposes a global p-value, the g-value, that aggregates all p-values from a multi-design study by rescaling the maximum p-value by a calibration constant, so the test keeps asymptotic size α with power close to a single valid…

desk verdict A neat rescaling idea for aggregating p-values, but the flagship analytic example only works for a specific error distribution, and the general algorithm is unvalidated. read the letter →

arxiv 2507.03815 v1 pith:PGIXWWGI submitted 2025-07-04 stat.ME

Guillaume Coqueret , Yuming Zhang , Christophe Pérignon , Francesca Chiaromonte , Stéphane Guerrier This is my paper · ORCID

classification stat.ME MSC 62F0362F4062G10

keywords g-valueglobalp-valuemulti-designstudiesmulti-analystp-hackingselectivereportingtypeIerrorcontrolaggregation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the usual response to analytical flexibility—either pick one test or take the most conservative one—is unnecessary. It introduces a global p-value, or g-value, defined as the largest p-value across a pre-specified set of legitimate analysis strategies, rescaled by a constant that depends on the set and on the significance level α. The constant is chosen so that, under the null hypothesis, the asymptotic probability that the g-value falls below α is exactly α. If this holds, a multi-design study can be summarized by one number that controls the false-positive rate, keeps power close to a single well-chosen test, and removes the incentive to report only the smallest p-value.

What carries the argument

The load-bearing object is the calibration constant $\alpha^*(\alpha, \mathcal{A})$, the value $\gamma \in [\alpha, 1]$ at which the asymptotic null probability that all p-values in $\mathcal{A}$ exceed $1-\gamma$ equals $\alpha$. Equivalently, it inverts the null distribution of the largest p-value. In the paper's worked example, the two test statistics have an asymptotic bivariate normal distribution with correlation $\sqrt{3/\pi}$, so the null distribution of the maximum p-value has a closed form and $\alpha^*$ can be computed by a one-dimensional root search. For settings without a closed form, the paper proposes a resampling algorithm—bootstrap resampling of observations followed by permutation of the variable of interest—to approximate the null distribution of the maximum p-value. A quantile variant, replacing the maximum by a $\gamma$-quantile of the p-values, is offered as a robustness adjustment.

What would settle it

Take a small sample, run two analyses on the same variable where one uses a t-test and the other uses a different model specification or a transformation of the outcome, and apply the proposed bootstrap-then-permutation algorithm to compute $\alpha^*$. Under $H_0$ with $\mu = 0$, simulate the empirical rejection rate of $G_n(\mathcal{A}) \le \alpha$ over many replications; if the rate exceeds $\alpha$ by more than simulation error, the claimed size control fails. A second check applies the same simulation to the quantile variant, since the paper does not prove that the max-based $\alpha^*$ remains valid when the statistic is replaced by a quantile.

Watch

Extended reading notes

Core claim

At the paper's center is the decision rule $G_n(\mathcal{A}) = (\alpha/\alpha^*(\alpha,\mathcal{A}))\,\max_{A\in\mathcal{A}} P_n(A)$, where $\alpha^*(\alpha,\mathcal{A})$ solves $\lim_{n\to\infty} \Pr(\max_{A\in\mathcal{A}} P_n(A) \le \gamma \mid H_0) = \alpha$ for $\gamma$. Because the unadjusted maximum p-value is conservative—it rejects only when all tests happen to be significant—the paper rescales it by $\alpha/\alpha^*$, which is equivalent to comparing the largest p-value against the larger threshold $\alpha^*$. The paper proves in a two-test example (one-sample t-test and Wilcoxon rank-sum test) that the asymptotic joint distribution of the test statistics is bivariate normal with correlation $\sqrt{3/\pi}$, computes $\alpha^*(0.05,\mathcal{A}) \approx 0.0601$, and shows by simulation that empirical rejection rates under $H_0$ sit near $0.05$ while power tracks the individual tests rather than the conservative bound.

Load-bearing premise

Everything rests on being able to compute or reliably approximate the null distribution of the maximum p-value over the full set $\mathcal{A}$; if the resampling scheme does not produce a valid null for every analysis strategy simultaneously (for example, when the strategies use different models or transformations), the calibrated constant $\alpha^*$ will not deliver the claimed error control.

Editorial extensions

If this is right

A multi-design study can report a single g-value; rejecting $H_0$ when $G_n(\mathcal{A}) \le \alpha$ gives an asymptotic type I error of $\alpha$ regardless of how many strategies are in $\mathcal{A}$.
The g-value's power is close to that of the individual valid tests, so the method avoids the large power loss of the most conservative p-value.
Because the set $\mathcal{A}$ must be stated in advance, selective reporting of only the most favorable p-value is structurally discouraged.
The framework extends to regression and maximum-likelihood settings whenever estimators admit an influence-function representation; for complex cases the bootstrap-permutation algorithm provides a numerical route.
The quantile version lets researchers downweight a few poorly fitting analysis choices while retaining the g-value interpretation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension not developed in the paper is to apply the same calibration logic to any class of asymptotically normal test statistics, so the g-value could serve as a template for aggregating specification-level effect sizes or confidence bounds.
Because the paper deliberately leaves dependent observations out of scope, extending the bootstrap-permutation scheme to time series would require a block-bootstrap or sieve-based null, which is a next step the paper does not take.
The requirement to enumerate $\mathcal{A}$ in advance suggests a practical workflow: researchers register the set of analyses before looking at data, then report the g-value; this could be tested as an intervention on reporting behavior.
The quantile variant may control size only under stronger assumptions than the maximum variant; a simulation comparing type I error across quantile levels would clarify when it is safe.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

The g-value idea is genuinely useful: instead of taking the max p-value and being conservative, or the min and being liberal, they rescale the max by a factor derived from the joint null distribution of the test statistics. For a pair (t-test, Wilcoxon signed-rank) under normal errors, they get an alpha* of 0.0601 and show by simulation that the type I error is around 0.05. That two-test case is clean, and the central construction is not in the previous specification-curve or p-value combination literature I know.

The problem is that the paper sells the example as general for any symmetric F, and that's false. The asymptotic correlation between the t and Wilcoxon statistics is sqrt(3/pi) only under normality; for a uniform error distribution it's 1. So the derived alpha* does not control size for arbitrary symmetric F. The simulation uses t15 data with the normal-derived alpha*, so that's not evidence for the general claim. This matters because the analytic example is the only fully worked demonstration of the framework; the general route is the bootstrap+permutation algorithm in Section 4.2, and that has no theory and no simulation. The quantile variant and the power-preservation claim also rest on the same unproven foundation.

None of this kills the idea. If you know the joint null distribution, the construction works by definition; the issue is that computing that distribution is usually hard, and the paper doesn't yet provide a reliable general method or a worked non-normal example. The literature review is fine, and the connections to specification curves and many-analyst studies are honest.

For me this is a conditional paper: the framework deserves referee time, but it needs a corrected Example 1 (either restrict to normality or derive the distribution-dependent alpha*), proper simulations for the resampling algorithm, and a precise statement of what is proven versus conjectured. As is, I would not cite it as a general solution.

I'd send it to a statistician who works on multiple testing or resampling; they'll see the gap immediately but also see the useful kernel.

Referee Report

4 major / 5 minor

Summary. The paper proposes a 'g-value' for multi-design studies: for a pre-specified set A of analysis strategies, one computes the p-value P_n(A) for each strategy, takes the largest p-value, and rescales it by α/α*, where α* is the α-quantile of the null distribution of the maximum p-value. The authors argue that this construction controls the asymptotic type I error at α while avoiding the conservatism of simply reporting the largest p-value, and that it reduces selective reporting by forcing researchers to enumerate A. The method is illustrated with a two-test example involving the one-sample t-test and the Wilcoxon signed-rank test, supported by a small Monte Carlo study, and the paper discusses practical computation via a bootstrap-plus-permutation algorithm and a quantile-based robust variant.

Significance. Conceptually, the g-value is an elegant and appealing idea: instead of aggregating p-values through an arbitrary combination rule, one calibrates the maximum p-value against its true null distribution, thereby turning the most conservative multi-design summary into an exact-level test. If the joint null distribution of the p-values were available, the construction would deliver much of what the abstract promises. The paper also usefully distinguishes its setting from classical multiple testing and selective inference. However, the current manuscript does not establish the required null distribution in any general setting: the analytic example is valid only under additional distributional assumptions, the general resampling algorithm is unvalidated, and the quantile variant as written does not control size. The contribution is therefore a promising framework rather than a demonstrated method.

major comments (4)

[Section 3.1, Eq. (3) and Eq. (4)] The covariance between the normalized t-statistic and the Wilcoxon signed-rank statistic is not universal. From the influence-function representation in Eq. (3), the covariance equals (√3/σ) E[|X_i| G(|X_i|)], which depends on F through G and σ. The value √(3/π) used in Eq. (4) is correct for Gaussian F but not for every symmetric F. For instance, if X_i ~ Uniform(-1,1), then σ=1/√3 and G(x)=x, so the covariance equals 1; the two tests are asymptotically equivalent and the null distribution of max P is Uniform(0,1), so α* should be 0.05, not 0.0601. Consequently, the claim that Example 1 demonstrates asymptotic size α for arbitrary symmetric F is false, and the Section 3.2 simulation, which feeds the normal-based α* into t15 data, does not provide the advertised evidence. The framework itself survives if α* is computed from the correct joint null distribution, but the paper's only analytical demonstration needs either a restriction to a specific F or a derivation of the F-dependent correction.
[Section 4.1 (quantile variant)] The robust g-value G_n(A;γ) replaces max_A P_n(A) by Quantile_γ but keeps the same α* defined in Eq. (2) from the null distribution of the maximum. This does not control type I error: because Quantile_γ ≤ max_A P_n(A) pointwise for γ<1, the event {Quantile_γ ≤ α*} contains {max_A P_n(A) ≤ α*}, so the asymptotic rejection probability is at least α, and generically strictly larger. The quantile variant requires its own calibration constant computed from the null distribution of the quantile statistic.
[Section 4.2 (computation)] The bootstrap-with-replacement followed by permutation proposal is the only general route to α* for arbitrary A, but it is not accompanied by any formal result or simulation. The statement that permutation 'ensures that the null hypothesis holds for all resampled datasets' addresses the marginal validity of each test, not the joint null distribution of the entire vector of p-values under the original data-generating process; for analyses that differ in variable transformations, model specification, or preprocessing, the resampling scheme can alter the dependence structure. As written, the paper provides no evidence that the resampled maximum-p distribution consistently estimates the quantity needed in Eq. (2). A theorem with explicit conditions or a simulation study is required before the general claims in the abstract and conclusion can be accepted.
[Section 2 and Section 5 (scope of claims)] The paper states in the abstract and conclusion that the g-value 'rigorously controls type I error rates' and 'maintains strong statistical power.' These claims are not established for the general setting: no theorem is stated for the existence and uniqueness of α* under the stated regularity conditions, and the power statement is only illustrated in the two-test example. Please restate the guarantees as asymptotic and conditional on a correctly specified null distribution of max_A P_n(A), or prove them under explicit assumptions.

minor comments (5)

[Section 3.1 and Section 3.2] The one-sample procedure used in Example 1 is the Wilcoxon signed-rank test, not the Wilcoxon rank-sum test; the terminology should be corrected.
[Section 3.1] The notation G_n(A) for the g-value collides with the CDF G(·) used in Eq. (3); consider renaming the CDF (e.g., F_{|X|}).
[Section 3.1, Eq. (2)] The notation 'argzero' is non-standard; define it as the unique γ satisfying the displayed equation, and state conditions for existence and uniqueness.
[Section 3.2] With 10^5 Monte Carlo replications, the Monte Carlo standard error of a rejection proportion near 0.05 is about 0.0007; reporting the estimated type I error with this precision or a confidence interval would make the 'close to 0.05' claim quantitative.
[Section 4.2] The 'algorithm' is described only in prose; a step-by-step pseudocode and a discussion of how α* is estimated from the resampled distribution would improve reproducibility.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the g-value is a calibration rule, and its size control follows from the definition of alpha* rather than from any fitted parameter or load-bearing self-citation.

full rationale

The g-value construction in Eq. (2) is a calibration rule: alpha* is defined as the null quantile of the maximum p-value, so the statement that the rule has asymptotic size alpha is true by construction rather than by an independent theorem. This is normal statistical calibration, not a fitted-input/prediction circularity, because alpha* is not estimated from data and the hard part—computing the joint null distribution of the p-values—is addressed separately (analytically in Example 1, via resampling in Section 4.2). The only self-reference is the citation of Boulaguiem et al. (2024a,b) for the general idea of adjusting conservative procedures; that citation is illustrative, not load-bearing, since the g-value's definition and calibration do not rest on any result from those papers. The analytic example's use of a covariance matrix that is not distribution-free for all symmetric F is a correctness concern about the example, not a circularity: the alpha* used in the simulation is computed from the stated null distribution and then evaluated on t15 data rather than being fit to the t15 simulation. The quantile variant and the permutation/bootstrapping algorithm are under-verified assumptions, but they are not circular: they do not assume the conclusion. Hence no step reduces to its own input in the sense of the circularity checklist.

Assumptions & free parameters 1 free parameters · 5 assumptions · 1 invented entities

The framework is parameter-light: the only user-chosen knob is the quantile level γ in the robust variant, which has no supporting theory. The main burdens are domain assumptions about the set A and the joint null distribution of p-values, with the bootstrap-plus-permutation validity being an ad hoc assertion.

free parameters (1)

quantile level γ in robust variant
Introduced in Section 4.1 for G_n(A;γ) without derivation of its null distribution; the same α* from the max distribution is reused.

assumptions (5)

domain assumption The set A of analysis strategies is fixed in advance and chosen independently of the data, and every test in A is asymptotically valid and powerful.
Stated in Section 2: the size computations assume Pr(A) is not data-dependent; the g-value does not correct for ex post selection of A.
domain assumption The limiting joint distribution of the p-values (equivalently test statistics) across A under H0 is known or can be approximated by resampling.
Required to compute α* in Eq. (2)-(4); for general A the paper proposes bootstrap+permutation in Section 4.2 without proof.
standard math The maximum p-value statistic has a continuous, strictly increasing null distribution, so the argzero defining α* exists and is unique.
Implicit in the definition of α* in Eq. (2); true for continuous p-variables but not stated or proved.
ad hoc to paper Bootstrap with replacement followed by permuting the variable of interest yields a valid null distribution for all A∈A simultaneously.
Section 4.2, last two paragraphs. This is the load-bearing premise for the general algorithm.
standard math Estimators in regression settings admit an influence function representation √n(β̂_j−β_j)=Σφ_j(Y_i,x_i,β)+o_p(1).
Section 4.2, used to justify extending the approach to regression via CLT; holds for MLE under regularity conditions.

invented entities (1)

g-value (global p-value)
purpose: A single summary statistic that aggregates p-values from multiple analyses A by rescaling the largest p-value with α/α*, enabling a size-α decision rule.
Introduced in Section 3.1, Eq. (2). It is a mathematical construct defined within the paper; its justification is the calibration equation, which is internal rather than external evidence.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Global p-Values in Multi-Design Studies." pith.science (2026). https://pith.science/paper/PGIXWWGI

@misc{pith2026250703815,
  author       = {Pith},
  title        = {Pith review of: Global p-Values in Multi-Design Studies},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PGIXWWGI}},
  note         = {Machine review of arXiv:2507.03815}
}

read the original abstract

Replicability issues -- referring to the difficulty or failure of independent researchers to corroborate the results of published studies -- have hindered the meaningful progression of science and eroded public trust in scientific findings. In response to the replicability crisis, one approach is the use of multi-design studies, which incorporate multiple analysis strategies to address a single research question. However, there remains a lack of methods for effectively combining outcomes in multi-design studies. In this paper, we propose a unified framework based on the g-value, for global p-value, which enables meaningful aggregation of outcomes from all the considered analysis strategies in multi-design studies. Our framework mitigates the risk of selective reporting while rigorously controlling type I error rates. At the same time, it maintains statistical power and reduces the likelihood of overlooking true positive effects. Importantly, our method is flexible and broadly applicable across various scientific domains and outcome results.

Figures

Figures reproduced from arXiv: 2507.03815 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 46 canonical work pages

[1]

Allison, P. D. (2009). Missing data. The SAGE handbook of quantitative methods in psychology , 23:72--89

work page 2009
[2]

C., Rogelberg, S

Banks, G. C., Rogelberg, S. G., Woznyj, H. M., Landis, R. S., and Rupp, D. E. (2016). Evidence on questionable research practices: The good, the bad, and the ugly. Journal of Business and Psychology , 31:323--338

work page 2016
[3]

J., Berger, J

Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., Bollen, K. A., Brembs, B., Brown, L., Camerer, C., et al. (2018). Redefine statistical significance. Nature Human Behaviour , 2(1):6--10

work page 2018
[4]

and Hochberg, Y

Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B , 57(1):289--300

1995
[5]

Y., Abaye, D

Boateng, E. Y., Abaye, D. A., et al. (2019). A review of the logistic regression model with emphasis on medical research. Journal of Data Analysis and Information Processing , 7(04):190

work page 2019
[6]

F., et al

Botvinik-Nezer, R., Holzmeister, F., Camerer, C. F., et al. (2020). Variability in the analysis of a single neuroimaging dataset by many teams. Nature , 582:84--88

work page 2020
[7]

Boukerche, A., Zheng, L., and Alfandi, O. (2020). Outlier detection: Methods, models, and classification. ACM Computing Surveys , 53(3):1--37

work page 2020
[8]

Boulaguiem, Y., Insolia, L., Victoria-Feser, M.-P., Couturier, D.-L., and Guerrier, S. (2024a). Multivariate adjustments for average equivalence testing. arXiv Preprint , (2411.16429)

work page Pith review arXiv 2024

Show all 48 references

[9]

N., Victoria-Feser, M.-P., Guerrier, S., and Couturier, D.-L

Boulaguiem, Y., Quartier, J., Lapteva, M., Kalia, Y. N., Victoria-Feser, M.-P., Guerrier, S., and Couturier, D.-L. (2024b). Finite sample corrections for average equivalence testing. Statistics in Medicine , 43(5):833--854

2024
[10]

Brodeur, A., Cook, N., and Heyes, A. (2020). Methods matter: P-hacking and publication bias in causal analysis in economics. American Economic Review , 110(11):3634--3660

2020
[11]

Brodeur, A., L \'e , M., Sangnier, M., and Zylberberg, Y. (2016). Star wars: T he empirics strike back. American Economic Journal: Applied Economics , 8(1):1--32

2016
[12]

F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al

Camerer, C. F., Dreber, A., Forsell, E., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., Almenberg, J., Altmejd, A., Chan, T., et al. (2016). Evaluating replicability of laboratory experiments in economics. Science , 351(6280):1433--1436

2016
[13]

F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., et al

Camerer, C. F., Dreber, A., Holzmeister, F., Ho, T.-H., Huber, J., Johannesson, M., Kirchler, M., et al. (2018). Evaluating the replicability of social science experiments in nature and science between 2010 and 2015. Nature Human Behavior , 2:637--644

2018
[14]

Christensen, G., Freese, J., and Miguel, E. (2019). Transparent and reproducible social science research: How to do open science . University of California Press

2019
[15]

J., Gopalan, R., Berlin, J., and Matthews, D

Easterbrook, P. J., Gopalan, R., Berlin, J., and Matthews, D. R. (1991). Publication bias in clinical research. The Lancet , 337(8746):867--872

1991
[16]

Elliott, G., Kudrin, N., and W \"u thrich, K. (2022). Detecting p-hacking. Econometrica , 90(2):887--906

2022
[17]

and Li, R

Fan, J. and Li, R. (2001). Variable selection via nonconcave penalized likelihood and its oracle properties. Journal of the American Statistical Association , 96(456):1348--1360

2001
[18]

Gasparin, M., Wang, R., and Ramdas, A. (2025). Combining exchangeable p-values. Proceedings of the National Academy of Sciences , 122(11):e2410849122

2025
[19]

S., Parker, T

Gould, E., Fraser, H. S., Parker, T. H., Nakagawa, S., Griffith, S. C., Vesk, P. A., Fidler, F., Hamilton, D. G., Abbey-Lee, R. N., Abbott, J. K., et al. (2025). Same data, different analysts: Variation in effect sizes due to analytical decisions in ecology and evolutionary bi...

2025
[20]

Hansen, L. P. (1982). Large sample properties of generalized method of moments estimators. Econometrica , 50(4):1029--1054

1982
[21]

R., Burli, P., Chen, N., Grieco, P., Ekpe, G., Pugatch, T., et al

Huntington-Klein, N., Arenas, A., Beam, E., Bertoni, M., Bloem, J. R., Burli, P., Chen, N., Grieco, P., Ekpe, G., Pugatch, T., et al. (2021). The influence of hidden researcher decisions in applied microeconomics. Economic Inquiry , 59(3):944--960

2021
[22]

Ioannidis, J. P. (2005). Why most published research findings are false. PLoS Medicine , 2(8):e124

2005
[23]

K., Loewenstein, G., and Prelec, D

John, L. K., Loewenstein, G., and Prelec, D. (2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science , 23(5):524--532

2012
[24]

Johnson, V. E. (2013). Revised standards for statistical evidence. Proceedings of the National Academy of Sciences , 110(48):19313--19317

2013
[25]

King, G. (1995). Replication, replication. PS: Political Science & Politics , 28(3):444--452

1995
[26]

Lehmann, E. L. and Romano, J. P. (2022). Testing S tatistical H ypotheses, 4th Edition . Springer

2022
[27]

and Sevestre, P

M \'a ty \'a s, L. and Sevestre, P. (2013). The econometrics of panel data: Handbook of theory and applications , volume 28. Springer Science & Business Media

2013
[28]

and Michaillat, P

McCloskey, A. and Michaillat, P. (2025). Critical values robust to p-hacking. Review of Economics and Statistics , Forthcoming:1--35

2025
[29]

J., Dreber, A., Holzmeister, F., Huber, J., Johannesson, M., Kirchler, M., Neus \"u ss, S., Razen, M., Weitzel, U., Abad-D \' az, D., et al

Menkveld, A. J., Dreber, A., Holzmeister, F., Huber, J., Johannesson, M., Kirchler, M., Neus \"u ss, S., Razen, M., Weitzel, U., Abad-D \' az, D., et al. (2024). Nonstandard errors. Journal of Finance , 79(3):2339--2390

2024
[30]

Miao, X., Wu, Y., Chen, L., Gao, Y., and Yin, J. (2022). An experimental survey of missing data imputation algorithms. IEEE Transactions on Knowledge and Data Engineering , 35(7):6630--6650

2022
[31]

M., Gerber, A., Glennerster, R., Green, D

Miguel, E., Camerer, C., Casey, K., Cohen, J., Esterling, K. M., Gerber, A., Glennerster, R., Green, D. P., Humphreys, M., Imbens, G., Laitin, D., Madon, T., Nelson, L., Nosek, B. A., Petersen, M., Sedlmayr, R., Simmons, J. P., Simonsohn, U., and der Laan, M. V. (2014). Promot...

2014
[32]

and Roehrkasse, A

Muller, C. and Roehrkasse, A. F. (2025). Falling racial inequality and rising educational inequality in us prison admissions for drug, violent, and property crimes. Proceedings of the National Academy of Sciences , 122(4):e2418077122

2025
[33]

A., Alter, G., Banks, G

Nosek, B. A., Alter, G., Banks, G. C., Borsboom, D., Bowman, S. D., et al. (2015). Promoting an open research culture. Science , 348(6242):1422--1425

2015
[34]

Shrout, P. E. and Rodgers, J. L. (2018). Psychology, science, and knowledge construction: Broadening perspectives from the replication crisis. Annual Review of Psychology , 69(1):487--510

2018
[35]

L., Martin, D

Silberzahn, R., Uhlmann, E. L., Martin, D. P., Anselmi, P., Aust, F., Awtrey, E., Bahn \' k, S ., Bai, F., Bannard, C., Bonnier, E., et al. (2018). Many analysts, one data set: Making transparent how variations in analytic choices affect results. Advances in Methods and Practi...

2018
[36]

P., Nelson, L

Simmons, J. P., Nelson, L. D., and Simonsohn, U. (2021). Pre-registration: Why and how. Journal of Consumer Psychology , 31(1):151--162

2021
[37]

D., and Simmons, J

Simonsohn, U., Nelson, L. D., and Simmons, J. P. (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General , 143(2):534

2014
[38]

P., and Nelson, L

Simonsohn, U., Simmons, J. P., and Nelson, L. D. (2020). Specification curve analysis. Nature Human Behaviour , 4(11):1208--1214

2020
[39]

Stanley, T. D. (2005). Beyond publication bias. Journal of Economic Surveys , 19(3):309--345

2005
[40]

and Tibshirani, R

Taylor, J. and Tibshirani, R. J. (2015). Statistical learning and selective inference. Proceedings of the National Academy of Sciences , 112(25):7629--7634

2015
[41]

Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B , 58(1):267--288

1996
[42]

and Wang, R

Vovk, V. and Wang, R. (2020). Combining p-values via averaging. Biometrika , 107(4):791--808

2020
[43]

J., and Hammad, M

Wang, H., Bah, M. J., and Hammad, M. (2019). Progress in outlier detection techniques: A survey. IEEE Access , 7:107964--108000

2019
[44]

Wilson, D. J. (2019). The harmonic mean p-value for combining dependent tests. Proceedings of the National Academy of Sciences , 116(4):1195--1200

2019
[45]

Xie, Y., Wang, K., and Kong, Y. (2021). Prevalence of research misconduct and questionable research practices: A systematic review and meta-analysis. Science and Engineering Ethics , 27(4):41

2021
[46]

Yoon, S., Baik, B., Park, T., and Nam, D. (2021). Powerful p-value combination methods to detect incomplete association. Scientific Reports , 11(1):6980

2021
[47]

and Kumbier, K

Yu, B. and Kumbier, K. (2020). Veridical data science. Proceedings of the National Academy of Science , 117(8):3920--3929

2020
[48]

Zou, H. (2006). The adaptive lasso and its oracle properties. Journal of the American Statistical Association , 101(476):1418--1429

2006

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Allison, P. D. (2009). Missing data. The SAGE handbook of quantitative methods in psychology , 23:72--89

work page 2009

[2] [2]

C., Rogelberg, S

Banks, G. C., Rogelberg, S. G., Woznyj, H. M., Landis, R. S., and Rupp, D. E. (2016). Evidence on questionable research practices: The good, the bad, and the ugly. Journal of Business and Psychology , 31:323--338

work page 2016

[3] [3]

J., Berger, J

Benjamin, D. J., Berger, J. O., Johannesson, M., Nosek, B. A., Wagenmakers, E.-J., Berk, R., Bollen, K. A., Brembs, B., Brown, L., Camerer, C., et al. (2018). Redefine statistical significance. Nature Human Behaviour , 2(1):6--10

work page 2018

[4] [4]

and Hochberg, Y

Benjamini, Y. and Hochberg, Y. (1995). Controlling the false discovery rate: a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B , 57(1):289--300

1995

[5] [5]

Y., Abaye, D

Boateng, E. Y., Abaye, D. A., et al. (2019). A review of the logistic regression model with emphasis on medical research. Journal of Data Analysis and Information Processing , 7(04):190

work page 2019

[6] [6]

F., et al

Botvinik-Nezer, R., Holzmeister, F., Camerer, C. F., et al. (2020). Variability in the analysis of a single neuroimaging dataset by many teams. Nature , 582:84--88

work page 2020

[7] [7]

Boukerche, A., Zheng, L., and Alfandi, O. (2020). Outlier detection: Methods, models, and classification. ACM Computing Surveys , 53(3):1--37

work page 2020

[8] [8]

Boulaguiem, Y., Insolia, L., Victoria-Feser, M.-P., Couturier, D.-L., and Guerrier, S. (2024a). Multivariate adjustments for average equivalence testing. arXiv Preprint , (2411.16429)

work page Pith review arXiv 2024