REVIEW 5 minor 43 references
Selective inference is easier with p-values
T0 review · 0 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A p-value with non-decreasing null density can be adjusted for any known selection procedure by one integral, making selective inference a plug-in operation.
desk verdict A genuinely new and mostly correct unification of selective inference via p-values, with an honest mapping of its limitations; deserves serious peer review. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the selection function $s(x,z)=P(S=1\mid p=x,Z=z)$, assumed known, which governs how the analyst decides to test a null after seeing the p-value. For a selectively dominant p-value—defined as one whose post-selection distribution stochastically dominates that of a uniform p-value subjected to the same selection—Theorem 1 says the selectively adjusted p-value $p_{\mathrm{sel}} = \int_0^p s(x,Z)\,dx/\int_0^1 s(x,Z)\,dx$ is stochastically uniform after selection, so rejecting when $p_{\mathrm{sel}}\le\alpha$ controls selective error. Theorem 2 identifies selective dominance with the condition that the null conditional density of $p$ given $Z$ is non-decreasing, which is verified for UMP and UMPU p-values, permutation p-values, and $F$-test p-values.
What would settle it
A decisive check: simulate any standard test statistic under its null, transform to a p-value, and look at the histogram conditional on the auxiliary variable $Z$. If the density decreases on $[0,1]$ for a set of $Z$ with positive probability, Theorem 2 says the p-value is not selectively dominant, so a single such example among commonly used tests would break the paper's blanket claim. A toy calculation shows the failure mode is real: with $p=1-\sqrt{1-U}$ (a valid decreasing-density p-value) and selection $s(x)=\mathbf{1}\{x\le0.1\}$, the corrected test rejects with probability $P(p\le0.01\mid p\le0.1)\approx0.105$, above the nominal $0.1$.
Extended reading notes
Core claim
The central discovery is that selective dominance holds exactly when the null conditional density of the p-value $p$ given $Z$ is non-decreasing, and that this condition turns the selection-adjusted p-value $p_{\mathrm{sel}} = \int_0^p s(x,Z)\,dx \,/\, \int_0^1 s(x,Z)\,dx$ into a valid p-value after selection, where $s(x,z)=P(S=1\mid p=x,Z=z)$ is the known selection function. The paper shows that all the p-values practitioners commonly use satisfy this condition, and then uses the corrected p-value to give short derivations of conditional and hybrid inference on winners, rank verification in exponential families, data carving, and selective variants of Fisher's combination test. It also shows that the correction is tight when the original p-value is exactly uniform under the null.
Load-bearing premise
The load-bearing assumption is that the selection function $s(x,z)=P(S=1\mid p=x,Z=z)$ is known exactly and depends on no unknown parameters; whenever selection probabilities must be estimated or depend on the effect size being tested, the adjusted p-value cannot be computed from the formula.
Editorial extensions
If this is right
- A researcher who sees a p-value only when $p\le\alpha$ can correct for this publication bias by rejecting when $p\le\alpha^2$, for any selectively dominant p-value.
- For independent selectively dominant p-values, the winning null is rejected when $p_{(1)}\le\alpha p_{(2)}$, and the closed version of this test makes sequential discoveries while controlling family-wise error.
- In the Gaussian rank-verification problem, a two-sided rejection at level $\alpha$ justifies the statement that the winner is strictly larger, while verifying that the winner is at least as large requires no selection correction at all.
- Fisher's top-$k$ and truncated combination tests remain valid even when some null p-values are conservative (super-uniform), whereas earlier versions required exact uniformity.
- Data carving becomes a general tool: when data fission or thinning makes the conditional distribution of the selection statistic given the p-value parameter-free, the selection function is known and the integral correction can be computed, at least numerically.
Reading between the lines
- The paper treats the selection function as known; a natural extension, not pursued in the paper, is a plugin or conservative-envelope version that uses an estimated or upper-bounded $s(x,z)$ and studies how misspecification degrades selective error.
- The explicit tie-handling correction in rank verification suggests that other selective methods that ignore ties in discrete or grouped data may be either conservative or anticonservative, and the same $1/N$ tie-breaking adjustment could be ported to them.
- Practically, the framework turns common heuristics like 'we only publish $p\le0.05$' into a transparent correction $p/0.05$ that applies to almost any standard p-value, without deriving a truncated-normal distribution for each new test statistic.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces selectively dominant p-values, defined as p-values whose conditional density under the null is non-decreasing on [0,1]. It proves (Theorem 2) that this density condition is equivalent to selective dominance: after any selection process with known selection probability s(x,z), the conditional distribution of the p-value given selection stochastically dominates that of a uniform random variable subjected to the same selection. Theorem 1 then shows that the adjusted p-value p_sel = ∫_0^p s(x,Z)dx / ∫_0^1 s(x,Z)dx is uniformly valid post-selection. The theory is applied to inference on winners (Section 3), rank verification in exponential families (Section 4), and combination tests including data carving and publication-bias-aware meta-analysis (Section 5). The paper also corrects a tie-handling error in Hung and Fithian (2019) and gives a counterexample to their claimed level inflation.
Significance. If the results hold, this is a substantial unification of selective inference: instead of deriving conditional procedures case by case, one only needs a valid p-value, a non-decreasing null density, and a known selection function. The appendices provide careful measure-theoretic proofs of the main theorems, a verified counterexample to the rank-verification level claim, and a transparent coupling argument for the data-carving example. The paper also makes its code available. The central assumption—that the selection function is known and parameter-free—is explicitly stated and scoped; it limits applicability but does not affect the internal validity of the theorems. My reading confirms the soundness of the main derivations.
minor comments (5)
- [Abstract and Section 1.2] The phrase 'all commonly used p-values' overstates the support provided by Examples 2–6, which cover important parametric families, permutation tests, and F-tests but not a mathematically defined class of 'commonly used' p-values; moreover, Theorem 2 requires a conditional density, so discrete p-values are excluded unless randomized. Suggest qualifying the wording.
- [Section 2.3, Example 9] The statement that the selection function is 'always accessible to us via extensive simulations' deserves a caveat: a Monte Carlo estimate of s(x,z) does not by itself provide exact finite-sample Type I error control unless the estimation error is incorporated; the paper currently treats s as known.
- [Section 5.1] The statement that pO/α remains valid under 'p-hacking' is heuristic, based on modeling null p-hacked p-values as having increasing density on [0,α]; this is a reasonable empirical model but should be labeled as a heuristic rather than a theorem.
- [Appendix A.5] The LASSO post-selection inference example assumes the noise level σ is known; the paper could note that unknown σ requires additional treatment (e.g., via the square-root LASSO) so that readers do not infer that the framework resolves all LASSO settings without further conditions.
- [Appendix A.12] In the displayed re-expression of psel, the denominator is written as q+(Z) + (1/N(Z))(q+(Z) - q(Z)), which has the sign reversed relative to the correct expression q+(Z) + (1/N(Z))(q(Z) - q+(Z)) shown in the main text around equation (23); the surrounding narrative and Example 12 use the correct form, so this appears to be a typographical error.
Circularity Check
No significant circularity: central theorems are proved from definitions and standard testing theory; applications are checked against external benchmarks.
full rationale
The paper's load-bearing claims are Theorem 1 and Theorem 2. Theorem 1 is a direct consequence of the definition of selective dominance: if p's post-selection distribution stochastically dominates that of a uniform under the same selection, then applying the conditional CDF of that uniform, psel = F_{U|Z,S'}(p), yields a post-selection p-value. This is a proof, not an assumption of the conclusion; the substantive content is the characterization in Theorem 2 and the demonstration in Appendix B that UMP/MLR and UMPU p-values satisfy the non-decreasing-density condition. The proof of Theorem 2 constructs the relevant density inequalities for arbitrary selection functions and gives an explicit violating selection function for the converse, so the characterization is not circular. The examples and corollaries are derived from these theorems rather than fitted to data, and the reproductions of conditional LCB, LASSO post-selection inference, and hybrid inference are validated against external references (Lee et al., Fithian et al., Andrews et al.). The correction to Hung and Fithian's tie handling is supported by an explicit selection-function computation and simulation. The known-selection-function assumption is stated openly as a scope condition, not smuggled in, and the paper contains no load-bearing self-citations or fitted parameters renamed as predictions. Overall the derivation chain is self-contained.
Assumptions & free parameters
assumptions (4)
- domain assumption Under the null, p-values considered have a conditional density given Z.
- domain assumption The selection function s(x,z) is known to the analyst and independent of unknown parameters.
- domain assumption All p-values in the main applications are independent and selectively dominant, verified for UMP, MLR, exponential, F, and permutation cases.
- standard math Standard Neyman-Pearson testing theory: UMP and UMPU tests exist and have the stated form.
Cite this review
Pith. "Pith review of Selective inference is easier with p-values." pith.science (2026). https://pith.science/paper/ZLV7SR7E
@misc{pith2026241113764,
author = {Pith},
title = {Pith review of: Selective inference is easier with p-values},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZLV7SR7E}},
note = {Machine review of arXiv:2411.13764}
}
abstract
Selective inference is a subfield of statistics that enables valid inference after selection of a data-dependent question. In this paper, we introduce selectively dominant p-values, a class of p-values that allow practitioners to easily perform inference after arbitrary selection procedures. Unlike a traditional p-value, whose distribution must stochastically dominate the uniform distribution under the null, a selectively dominant p-value must have a post-selection distribution that stochastically dominates that of a uniform having undergone the same selection process; moreover, this property must hold simultaneously for all possible selection processes. Despite the strength of this condition, we show that all commonly used p-values (e.g., p-values from two-sided testing in parametric families, one-sided testing in monotone likelihood ratio and exponential families, $F$-tests for linear regression, and permutation tests) are selectively dominant. By recasting two canonical selective inference problems-inference on winners and rank verification-in our selective dominance framework, we provide simpler derivations, a deeper conceptual understanding, and new generalizations and variations of these methods. Additionally, we use our insights to introduce selective variants of methods that combine p-values, such as Fisher's combination test.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
Identification of and correction for publication bias
Isaiah Andrews and Maximilian Kasy. Identification of and correction for publication bias. American Economic Review, 109 0 (8): 0 2766–94, August 2019. doi:10.1257/aer.20180310. URL https://www.aeaweb.org/articles?id=10.1257/aer.20180310
-
[2]
Isaiah Andrews, Toru Kitagawa, and Adam McCloskey. Inference on Winners* . The Quarterly Journal of Economics, 139 0 (1): 0 305--358, 09 2023. ISSN 0033-5533. doi:10.1093/qje/qjad043. URL https://doi.org/10.1093/qje/qjad043
-
[3]
Selection-corrected statistical inference for region detection with high-throughput assays
Yuval Benjamini, Jonathan Taylor, and Rafael A Irizarry. Selection-corrected statistical inference for region detection with high-throughput assays. Journal of the American Statistical Association, 114 0 (527): 0 1351--1365, 2019
work page 2019
-
[4]
Multiple comparisons and type iii errors
Eve Bofinger. Multiple comparisons and type iii errors. Journal of the American Statistical Association, 80 0 (390): 0 433--437, 1985
work page 1985
-
[5]
More powerful selective inference for the graph fused lasso
Yiqun Chen, Sean Jewell, and Daniela Witten. More powerful selective inference for the graph fused lasso. Journal of Computational and Graphical Statistics, 32 0 (2): 0 577--587, 2023
2023
-
[6]
Selective inference for k-means clustering
Yiqun T Chen and Daniela M Witten. Selective inference for k-means clustering. Journal of Machine Learning Research, 24 0 (152): 0 1--41, 2023
2023
-
[7]
Estimating the reproducibility of psychological science
Open Science Collaboration. Estimating the reproducibility of psychological science. Science, 349 0 (6251): 0 aac4716, 2015. doi:10.1126/science.aac4716. URL https://www.science.org/doi/abs/10.1126/science.aac4716
-
[8]
Generalized data thinning using sufficient statistics
Ameer Dharamshi, Anna Neufeld, Keshav Motwani, Lucy L Gao, Daniela Witten, and Jacob Bien. Generalized data thinning using sufficient statistics. Journal of the American Statistical Association, 0 (just-accepted): 0 1--26, 2024
work page 2024
Show all 43 references
-
[9]
On multiple decision methods for ranking population means
Vaclav Fabian. On multiple decision methods for ranking population means. The Annals of Mathematical Statistics, pages 248--254, 1962
1962
-
[10]
Selective sequential model selection
William Fithian, Jonathan Taylor, Robert Tibshirani, and Ryan Tibshirani. Selective sequential model selection. arXiv preprint arXiv:1512.02565, 2015
2015 arXiv
-
[11]
Optimal inference after model selection, 2017
William Fithian, Dennis Sun, and Jonathan Taylor. Optimal inference after model selection, 2017. URL https://arxiv.org/abs/1410.2597
2017 arXiv
-
[12]
Selective inference for hierarchical clustering
Lucy L Gao, Jacob Bien, and Daniela Witten. Selective inference for hierarchical clustering. Journal of the American Statistical Association, 119 0 (545): 0 332--342, 2024
2024
-
[13]
Fuzzy and randomized confidence intervals and p-values
Charles J Geyer and Glen D Meeden. Fuzzy and randomized confidence intervals and p-values. Statistical Science, pages 358--366, 2005
2005
-
[14]
Is the selected population the best? The Annals of Statistics, pages 456--461, 1987
Sam Gutmann and Zakhar Maymin. Is the selected population the best? The Annals of Statistics, pages 456--461, 1987
1987
-
[15]
Exact testing with random permutations
Jesse Hemerik and Jelle Goeman. Exact testing with random permutations. TEST, 27 0 (4): 0 811--825, 2018. doi:10.1007/s11749-017-0571-1. URL https://doi.org/10.1007/s11749-017-0571-1
2018 doi
-
[16]
Simultaneous confidence intervals for all distances from the" best"
Jason C Hsu. Simultaneous confidence intervals for all distances from the" best". The Annals of Statistics, pages 1026--1034, 1981
1981
-
[17]
Rank verification for exponential families
Kenneth Hung and William Fithian. Rank verification for exponential families . The Annals of Statistics, 47 0 (2): 0 758 -- 782, 2019. doi:10.1214/17-AOS1634. URL https://doi.org/10.1214/17-AOS1634
2019 doi
-
[18]
Statistical methods for replicability assessment
Kenneth Hung and William Fithian. Statistical methods for replicability assessment . The Annals of Applied Statistics, 14 0 (3): 0 1063 -- 1087, 2020. doi:10.1214/20-AOAS1336. URL https://doi.org/10.1214/20-AOAS1336
2020 doi
-
[19]
Post-selection inference for changepoint detection algorithms with application to copy number variation data
Sangwon Hyun, Kevin Z Lin, Max G'Sell, and Ryan J Tibshirani. Post-selection inference for changepoint detection algorithms with application to copy number variation data. Biometrics, 77 0 (3): 0 1037--1049, 2021
2021
-
[20]
Exact post model selection inference for marginal screening
Jason D Lee and Jonathan E Taylor. Exact post model selection inference for marginal screening. Advances in neural information processing systems, 27, 2014
2014
-
[21]
Exact post-selection inference, with application to the lasso
Jason D Lee, Dennis L Sun, Yuekai Sun, and Jonathan E Taylor. Exact post-selection inference, with application to the lasso. 2016
2016
-
[22]
Testing statistical hypotheses, volume 3
Erich Leo Lehmann, Joseph P Romano, and George Casella. Testing statistical hypotheses, volume 3. Springer, 1986
1986
-
[23]
AdaPT: An Interactive Procedure for Multiple Testing with Side Information
Lihua Lei and William Fithian. AdaPT: An Interactive Procedure for Multiple Testing with Side Information . Journal of the Royal Statistical Society Series B: Statistical Methodology, 80 0 (4): 0 649--679, 06 2018. ISSN 1369-7412. doi:10.1111/rssb.12274. URL https://doi.org/10...
2018 doi
-
[24]
Data fission: splitting a single data point
James Leiner, Boyan Duan, Larry Wasserman, and Aaditya Ramdas. Data fission: splitting a single data point. Journal of the American Statistical Association, pages 1--12, 2023
2023
-
[25]
Selective inference in regression models with groups of variables
Joshua R Loftus and Jonathan E Taylor. Selective inference in regression models with groups of variables. arXiv preprint arXiv:1511.01478, 2015
2015 arXiv
-
[26]
On closed testing procedures with special reference to ordered analysis of variance
Ruth Marcus, Peritz Eric, and K Ruben Gabriel. On closed testing procedures with special reference to ordered analysis of variance. Biometrika, 63 0 (3): 0 655--660, 1976
1976
-
[27]
Inference after black box selection
Jelena Markovic, Jonathan Taylor, and Jeremy Taylor. Inference after black box selection. arXiv preprint arXiv:1901.09973, 2019
1901 arXiv
-
[28]
Testing retrospective hypotheses
Zakhar Maymin and Sam Gutmann. Testing retrospective hypotheses. Canadian Journal of Statistics, 20 0 (3): 0 335--345, 1992
1992
-
[29]
Engine time to failure dataset, 2024
Alexander Molotaliev. Engine time to failure dataset, 2024. URL https://www.kaggle.com/datasets/m0ntecarl0/engine-time-to-failure
2024
-
[30]
Data thinning for convolution-closed distributions
Anna Neufeld, Ameer Dharamshi, Lucy L Gao, and Daniela Witten. Data thinning for convolution-closed distributions. Journal of Machine Learning Research, 25 0 (57): 0 1--35, 2024
2024
-
[31]
Exact selective inference with randomization
Snigdha Panigrahi, Kevin Fry, and Jonathan Taylor. Exact selective inference with randomization. Biometrika, page asae019, 2024
2024
-
[32]
Infer-and-widen versus split-and-condition: two tales of selective inference
Ronan Perry, Zichun Xu, Olivia McGough, and Daniela Witten. Infer-and-widen versus split-and-condition: two tales of selective inference. arXiv preprint arXiv:2408.06323, 2024
2024
-
[33]
A practical two-step method for testing moment inequalities
Joseph P Romano, Azeem M Shaikh, and Michael Wolf. A practical two-step method for testing moment inequalities. Econometrica, 82 0 (5): 0 1979--2002, 2014
1979
-
[34]
Nelson, and Joseph P
Uri Simonsohn, Leif D. Nelson, and Joseph P. Simmons. P-curve: A key to the file drawer. Cognitive Linguistics: Cognition, 2013. URL https://api.semanticscholar.org/CorpusID:8505270
2013
-
[35]
Post-selection inference for-penalized likelihood models
Jonathan Taylor and Robert Tibshirani. Post-selection inference for-penalized likelihood models. Canadian Journal of Statistics, 46 0 (1): 0 41--61, 2018
2018
-
[36]
Post-selection adaptive inference for least angle regression and the lasso
Jonathan Taylor, Richard Lockhart, Ryan J Tibshirani, and Robert Tibshirani. Post-selection adaptive inference for least angle regression and the lasso. arXiv preprint arXiv:1401.3889, 354, 2014
2014 arXiv
-
[37]
Taylor, Joshua R
Jonathan E. Taylor, Joshua R. Loftus, and Ryan J. Tibshirani. Inference in adaptive regression via the Kac--Rice formula . The Annals of Statistics, 44 0 (2): 0 743 -- 770, 2016. doi:10.1214/15-AOS1386. URL https://doi.org/10.1214/15-AOS1386
2016 doi
-
[38]
Selective inference with a randomized response
Xiaoying Tian and Jonathan Taylor. Selective inference with a randomized response. The Annals of Statistics, 46 0 (2): 0 679--710, 2018
2018
-
[39]
Selective inference with unknown variance via the square-root lasso
Xiaoying Tian, Joshua R Loftus, and Jonathan E Taylor. Selective inference with unknown variance via the square-root lasso. Biometrika, 105 0 (4): 0 755--768, 2018
2018
-
[40]
Exact post-selection inference for sequential regression procedures
Ryan J Tibshirani, Jonathan Taylor, Richard Lockhart, and Robert Tibshirani. Exact post-selection inference for sequential regression procedures. Journal of the American Statistical Association, 111 0 (514): 0 600--620, 2016
2016
-
[41]
Truncated product method for combining p-values
Dmitri V Zaykin, Lev A Zhivotovsky, Peter H Westfall, and Bruce S Weir. Truncated product method for combining p-values. Genetic Epidemiology: The Official Publication of the International Genetic Epidemiology Society, 22 0 (2): 0 170--185, 2002
2002
-
[42]
Tfisher: A powerful truncation and weighting procedure for combining p-values
Hong Zhang, Tiejun Tong, John Landers, and Zheyang Wu. Tfisher: A powerful truncation and weighting procedure for combining p-values. 2020
2020
-
[43]
Locally simultaneous inference
Tijana Zrnic and William Fithian. Locally simultaneous inference. The Annals of Statistics, 52 0 (3): 0 1227--1253, 2024
2024
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.