REVIEW 3 major objections 4 minor 60 references
This paper claims that a data-adaptive exponential randomization scheme for selecting top-k winners yields inference with exact conditional coverage in Gaussian models and asymptotically exact conditional coverage for asymptotically linear
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 15:05 UTC pith:25I4VVRD
load-bearing objection Elegant exact Gaussian pivot and regret-calibrated randomization, but the non-Gaussian asymptotics rest on an unproved plug-in step and the central conditional guarantee goes unchecked in simulations. the 3 major comments →
Flexible Inference for Winners with Conditional Validity
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim, stated as Theorem 3.5, is that for any winner j0 in the observed selected set E^o, the pivot Pivot_{mu_{j0}}^{E^o}(T_{j0}; T^perp_{j0}, sigma^2_{j0}) is exactly Uniform(0,1) conditional on {E = E^o} in the Gaussian model T ~ N(mu, Sigma). Corollary 3.1 turns this into exact 1-alpha conditional coverage for the interval obtained by inverting the pivot, along with exact p-values. Theorem 5.1 extends the claim to selection statistics with an asymptotic linear representation: the same pivot, computed with the true covariance blocks sigma^2_{j0} and Gamma_{j0}, converges in distribution to Uniform(0,1) conditional on the realized selection. The method therefore provides conditi
What carries the argument
The key object is the exponential mechanism over all k-subsets of [p], which selects the winner set E with probability proportional to exp{s_E(T)/(tau * sigma_s(T))}, where s_E is an additive score, sigma_s is the dispersion of scores across subsets, and tau is a temperature parameter. Scaling by sigma_s makes tau scale-free, and the paper derives a tuning-free choice tau = (log |E_k|)^{-1} q that bounds the expected standardized regret by q. The selection weight Lambda_{E^o}(u1, u2) equals the conditional selection probability P(E=E^o | T = r(u1,u2)), and because the mechanism is a softmax, this weight is available in closed form. The conditional density of the target statistic T_{j0} given
Load-bearing premise
The asymptotic validity of the pivot is proven only when the true covariance of the selection statistics is used; the paper's applications plug in estimated covariances, and no theorem shows the plug-in pivot retains its conditional uniform limit.
What would settle it
In a binomial A/B/n design with small n, apply the method with plug-in covariance and measure, over many replications, the empirical coverage of the 95% interval conditional on a fixed realized winner set E^o; if coverage does not converge to 0.95 as n grows, the nonparametric reach claimed in the paper is unsupported.
If this is right
- Intervals for each selected winner are conditionally valid at level 1-alpha, meaning they answer the question 'how large is this particular winner's true effect?' even though the winner was chosen from the same data.
- The method applies beyond Gaussian data to any setting where the selection statistics are asymptotically linear, including binomial A/B/n trials, Bradley–Terry–Davidson rankings, and feature-importance measures from black-box models.
- The temperature parameter is set automatically from a user-specified regret budget q, so the randomized rule's selection quality is provably close to the top-k rule.
- Conditional intervals adapt to the strength of the signal: they become shorter as the gap between the top-k and the rest grows, unlike fixed-width marginal intervals.
- The same pivot gives valid p-values and a bias-corrected point estimate for each selected winner.
Where Pith is reading between the lines
- If the plug-in covariance estimator preserves the asymptotic uniformity of the pivot—which the paper assumes in its applications but does not prove—the method would provide fully rigorous nonparametric conditional inference; currently that step is a gap between Theorem 5.1 and the implemented procedure.
- The closed-form selection weight suggests the same conditioning strategy could be applied to other randomized selection rules, such as additive noise or bootstrap perturbation, whenever the conditional selection probability is tractable.
- A natural extension is to construct simultaneous conditional intervals for the full vector of selected winners, or to test comparisons such as winner versus runner-up, by forming joint pivots from the shared conditioning set.
- The finite-sample behavior with estimated covariance is an empirical question; simulations in the paper show nominal coverage, but a formal higher-order analysis or a Berry–Esseen-type bound would clarify when the approximation is reliable.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a randomized selection rule based on a data-adaptive exponential mechanism for top-k winners, followed by conditional inference on the selected effects. In the exact Gaussian model, it defines a pivot that is the conditional CDF of the selected coordinate given the selection event and a nuisance-sufficient statistic, and proves exact conditional uniformity (Theorem 3.5). A tuning-free temperature is calibrated through a finite-sample regret bound (Corollary 2.1), and an O(pk) dynamic program is given for sampling and weight evaluation. For non-Gaussian applications, the paper states an asymptotic version (Theorem 5.1) for asymptotically linear selection statistics, and applies it to binomial A/B/n trials, Bradley–Terry–Davidson rankings, and nonparametric feature importance. Simulations compare the method to polyhedral inference, data splitting, and zoom correction on marginal coverage and interval length.
Significance. If the results hold as stated, the paper makes a substantive contribution: it provides exact conditional inference for winners without characterizing the selection event, a regret-based calibration of the randomization level, and efficient implementation. The Gaussian pivot argument is clean and self-contained, and the code is publicly available. The regret bound is simple and interpretable. However, the strongest advertised contribution—assumption-lean nonparametric conditional validity—rests on Theorem 5.1, and the theorem as stated does not cover the plug-in covariance estimators used in every non-Gaussian application and simulation. This is a load-bearing gap that must be addressed before the broad claims are supported.
major comments (3)
- [Section 5, Theorem 5.1] Theorem 5.1 is stated for the pivot with the true covariance blocks σ²_{jo} and Γ_{jo} held fixed; see Definition B.4 and eq. (28). The proof, including the standardization ζ_n in (22), Proposition B.1, and the Lindeberg bounds in Corollary B.1, treats eΣ as a fixed matrix. Section 5's introduction promises cases where Σ is consistently estimable, but no theorem or proof supplies the plug-in version. This matters because Sections 6 and C use estimated covariances in all non-Gaussian applications: the binomial design uses \hatΣ=Diag{\hatπ_j(1-\hatπ_j)}, and the BTD and feature-importance implementations use estimated Fisher information / covariance. Replacing Σ by \hatΣ changes Λ_{E^o} and T^⊥ through the reconstruction map r, so the conditional law can shift. A uniform-in-Σ argument or explicit rate conditions are required. Without this, the nonparametric conditional-coverage guarantee i
- [Section 7, Figures 3–5] The non-Gaussian simulations are the only empirical evidence for the asymptotic conditional guarantee, yet they report only marginal coverage. The method's advertised advantage over marginal baselines such as Zoom Correction is precisely conditional validity. The conditional-coverage demonstration in Figure 1(d) is for the exact Gaussian model, not for the asymptotically linear applications of Theorem 5.1. The paper should report conditional coverage for the binomial, BTD, and feature-importance designs—for example, averaged over the realized selected sets as implied by Corollary 3.1—to support the central claim that the nonparametric extension provides conditional validity.
- [Appendix B.5, Propositions B.10/B.11] The verification of Assumption 1(i) is incomplete in the BTD and feature-importance proofs. In Proposition B.10, the remainder R_n is bounded by δ_n‖Z_n‖ + (K/√n)‖U_n‖² with δ_n = o_p(1) random, and the proof asserts sup_n E[exp{4C_R‖R_n‖}] < ∞ from sub-Gaussianity of Z_n and U_n. This does not follow without a rate condition on δ_n or an additional argument bounding the exponential moment of the product. Proposition B.11 makes a similar leap, treating boundedness of exponential moments of the two displayed terms as sufficient for the remainder. These steps are needed for the ALR condition and should be made rigorous or replaced by explicit sufficient conditions.
minor comments (4)
- [Section 6.3] The sentence 'Here the term If no hold-out dataset is available...' is incomplete; it should introduce the cross-fitting alternative properly.
- [Figure captions] The captions of Figures 2–5 say 'selection equality' where 'selection quality' is meant.
- [Section 3.3, Definition 3.1] The notation Λ_{E^o}(u1,u2) is used before the standardized version F(v) is introduced in Appendix B; a short pointer in Section 3 would improve readability.
- [Section 5, Assumption 1] Assumption 1(i) defines C_R via c_f^{(1)} which is only fully defined in Proposition B.6 in Appendix B. This is a minor organizational issue but would benefit from a forward reference near Assumption 1.
Circularity Check
No circular reduction found: the pivot is a conditional CDF by construction and the main derivations are self-contained; minor self-citations are disclosed and non-load-bearing. The main caveat is an unproved plug-in covariance extension, which is a completeness gap, not circularity.
full rationale
The central construction (Definitions 3.1 and 3.2; Proposition 3.1; Theorem 3.5) defines the pivot as the conditional CDF of T_j0 under the exponential-mechanism selection law, and uniformity follows from the probability integral transform. This is the intended mechanism of a valid pivot, not a fitted prediction; no data-dependent parameter is tuned to force coverage. The regret bound (Propositions 2.1–2.2 and Corollary 2.1) is proved in-paper from the Gibbs variational principle (Lemma A.1, Csiszár 1975), so it does not depend on self-citation. The citations to Bakshi and Panigrahi (2025) and Wu et al. (2025) are explicitly framed as specific-model precursors and are not load-bearing. The one substantive gap is the plug-in covariance: Theorem 5.1 is stated with the true covariance blocks σ²_{j0} and Γ_{j0} fixed in the pivot, while Section C implements the Binomial design with "\Σ = Diag{\π_j(1−\π_j)}"; no theorem or proof covers this plug-in pivot. This is a missing-support/completeness issue for the nonparametric claims, not a circular reduction, so it is weighed in the verdict but does not raise the circularity score above the low range.
Axiom & Free-Parameter Ledger
free parameters (3)
- temperature parameter τ / regret budget q =
q=1 in simulations; τ=(log|E_k|)^{-1}
- stabilization constant ε in σ_s(t) =
unspecified; 'small constant'
- quadrature grid size G =
500; 2000 for Bradley-Terry-Davidson design
axioms (6)
- domain assumption Exact Gaussian model T ~ N_p(μ, Σ) with known Σ (Eq. 7)
- domain assumption Assumption 1: asymptotic linear representation with uniformly sub-Gaussian Y_{i,n} and exponentially integrable remainder R_n
- domain assumption Assumption 2: subset score functions s_E have globally bounded derivatives up to order 3
- domain assumption Additive score structure s_E(T)=Σ_{j∈E}s_j(T) and positive softmax probabilities
- domain assumption Application-specific regularity: binomial π_j∈(ε,1−ε); BTD strong convexity and positive match proportions; feature-importance boundedness plus o_P(n^{−1/4}) nuisance rates
- ad hoc to paper Plug-in covariance estimators are valid in the pivot, though Theorem 5.1 is proved with true Σ
read the original abstract
Researchers often select top-performing options or winners, based on a data-driven criterion, such as treatments, models, or model features and then report effect estimates for the selected winners. Naive post-selection estimates, however, are known to suffer from the winner's curse, producing systematically overoptimistic results. We introduce a flexible conditional inference method that corrects for this overoptimism through an adaptive exponential randomization scheme. Our method achieves selection quality that closely matches that of standard top-k selection, while also yielding shorter confidence intervals than existing approaches. Furthermore, our approach applies broadly to nonparametric settings with asymptotically linear selection statistics, covering wide-ranging applications such as inference for the efficacy of the most promising treatments in clinical trials, the abilities of top-ranked models on leaderboards, and the importance of the most predictive features in a model.
Figures
Reference graph
Works this paper leans on
-
[1]
The Econometrics Journal , volume =
Double/debiased machine learning for treatment and structural parameters , author =. The Econometrics Journal , volume =. 2018 , doi =
2018
-
[2]
Biostatistics , volume=
Bias-reduced estimators and confidence intervals for odds ratios in genome-wide association studies , author=. Biostatistics , volume=. 2008 , publisher=
2008
-
[3]
Journal of the American Statistical Association , volume=
Inference on the proportion of variance explained in principal component analysis , author=. Journal of the American Statistical Association , volume=. 2026 , publisher=
2026
-
[4]
Journal of the American Statistical Association , volume=
A general framework for inference on algorithm-agnostic variable importance , author=. Journal of the American Statistical Association , volume=. 2023 , publisher=
2023
-
[5]
Biometrics , volume=
Estimating heterogeneous treatment effects for general responses , author=. Biometrics , volume=. 2025 , publisher=
2025
-
[6]
Panigrahi, Snigdha and Fry, Kevin and Taylor, Jonathan , title =. Biometrika , volume =. 2024 , month =. doi:10.1093/biomet/asae019 , url =
-
[7]
Journal of machine learning research , volume=
Approximate post-selective inference for regression with the group lasso , author=. Journal of machine learning research , volume=
-
[8]
arXiv preprint arXiv:2411.15908 , year=
Selective Inference for Time-Varying Effect Moderation , author=. arXiv preprint arXiv:2411.15908 , year=
-
[9]
Journal of the American Statistical Association , volume=
Inference on selected subgroups in clinical trials , author=. Journal of the American Statistical Association , volume=. 2021 , publisher=
2021
-
[10]
arXiv preprint arXiv:2601.13514 , year=
Post-selection inference for penalized M-estimators via score thinning , author=. arXiv preprint arXiv:2601.13514 , year=
-
[11]
Biometrika , volume=
Splitting strategies for post-selection inference , author=. Biometrika , volume=. 2023 , publisher=
2023
-
[12]
Tibshirani and Alessandro Rinaldo and Rob Tibshirani and Larry Wasserman , title =
Ryan J. Tibshirani and Alessandro Rinaldo and Rob Tibshirani and Larry Wasserman , title =. The Annals of Statistics , number =. 2018 , doi =
2018
-
[13]
The Annals of Statistics , volume=
Carving model-free inference , author=. The Annals of Statistics , volume=. 2023 , publisher=
2023
-
[14]
Journal of the American Statistical Association , volume=
False discovery rate--adjusted multiple confidence intervals for selected parameters , author=. Journal of the American Statistical Association , volume=. 2005 , publisher=
2005
-
[15]
Lee, Jason D. and Sun, Dennis L. and Sun, Yuekai and Taylor, Jonathan E. , year=. Exact post-selection inference, with application to the lasso , volume=. The Annals of Statistics , publisher=. doi:10.1214/15-aos1371 , number=
-
[16]
Valid post-selection inference , volume=
Berk, Richard and Brown, Lawrence and Buja, Andreas and Zhang, Kai and Zhao, Linda , year=. Valid post-selection inference , volume=. The Annals of Statistics , publisher=. doi:10.1214/12-aos1077 , number=
-
[17]
SELECTIVE INFERENCE WITH A RANDOMIZED RESPONSE , urldate =
Xiaoying Tian and Jonathan Taylor , journal =. SELECTIVE INFERENCE WITH A RANDOMIZED RESPONSE , urldate =
-
[18]
2019 , eprint=
On the length of post-model-selection confidence intervals conditional on polyhedral constraints , author=. 2019 , eprint=
2019
-
[19]
arXiv preprint arXiv:1410.2597 , year=
Optimal inference after model selection , author=. arXiv preprint arXiv:1410.2597 , year=
-
[20]
Electronic Journal of Statistics , volume=
Confidence intervals for the means of the selected populations , author=. Electronic Journal of Statistics , volume=
-
[21]
arXiv preprint arXiv:1906.00505 , year=
Confidence intervals for selected parameters , author=. arXiv preprint arXiv:1906.00505 , year=
Pith/arXiv arXiv 1906
-
[22]
The Annals of Statistics , volume=
Locally simultaneous inference , author=. The Annals of Statistics , volume=. 2024 , publisher=
2024
-
[23]
The Annals of Statistics , volume=
A flexible defense against the winner’s curse , author=. The Annals of Statistics , volume=. 2025 , publisher=
2025
-
[24]
arXiv preprint arXiv:2604.09779 , year=
Inference conditional on selection: a review , author=. arXiv preprint arXiv:2604.09779 , year=
-
[25]
Biometrika , volume=
Hybrid confidence intervals for informative uniform asymptotic inference after model selection , author=. Biometrika , volume=. 2024 , publisher=
2024
-
[26]
Circular analysis in systems neuroscience: the dangers of double dipping , volume =
Kriegeskorte, Nikolaus and Simmons, W Kyle and Bellgowan, Patrick S F and Baker, Chris I , journal =. Circular analysis in systems neuroscience: the dangers of double dipping , volume =
-
[27]
Ball and Lindsay M
Tali M. Ball and Lindsay M. Squeglia and Susan F. Tapert and Martin P. Paulus , journal =. Double Dipping in Machine Learning: Problems and Solutions , volume =
-
[28]
2024 , eprint=
Combining T-learning and DR-learning: a framework for oracle-efficient estimation of causal contrasts , author=. 2024 , eprint=
2024
-
[29]
Rasines, D García and Young, G A , title = ". Biometrika , volume =. 2022 , month =. doi:10.1093/biomet/asac070 , url =
-
[30]
Journal of the American Statistical Association , volume=
Data fission: splitting a single data point , author=. Journal of the American Statistical Association , volume=. 2025 , publisher=
2025
-
[31]
Journal of the American Statistical Association , volume=
Generalized data thinning using sufficient statistics , author=. Journal of the American Statistical Association , volume=. 2025 , publisher=
2025
-
[32]
2023 , eprint=
Data fission: splitting a single data point , author=. 2023 , eprint=
2023
-
[33]
Jordan , journal =
Tijana Zrnic and Michael I. Jordan , journal =
-
[34]
Integrative methods for post-selection inference under convex constraints , volume =
Snigdha Panigrahi and Jonathan Taylor and Asaf Weinstein , journal =. Integrative methods for post-selection inference under convex constraints , volume =
-
[35]
Carving model-free inference , volume =
Snigdha Panigrahi , journal =. Carving model-free inference , volume =
-
[36]
Approximate Selective Inference via Maximum Likelihood , volume =
Snigdha Panigrahi and Jonathan Taylor , journal =. Approximate Selective Inference via Maximum Likelihood , volume =
-
[37]
The Annals of Statistics , keywords =
Richard Berk and Lawrence Brown and Andreas Buja and Kai Zhang and Linda Zhao , doi =. The Annals of Statistics , keywords =. 2013 , bdsk-url-1 =
2013
-
[38]
Terry , journal =
Ralph Allan Bradley and Milton E. Terry , journal =. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons , urldate =
-
[39]
Biometrics , pages=
On extending the Bradley-Terry model to incorporate within-pair order effects , author=. Biometrics , pages=. 1977 , publisher=
1977
-
[40]
arXiv preprint arXiv:2410.19212 , year=
Inference After Ranking with Applications to Economic Mobility , author=. arXiv preprint arXiv:2410.19212 , year=
-
[41]
2026 , eprint=
Inference After Ranking with Applications to Economic Mobility , author=. 2026 , eprint=
2026
-
[42]
The Quarterly Journal of Economics , volume=
Inference on winners , author=. The Quarterly Journal of Economics , volume=. 2024 , publisher=
2024
-
[43]
arXiv preprint arXiv:2512.06522 , year=
Hierarchical Clustering With Confidence , author=. arXiv preprint arXiv:2512.06522 , year=
-
[44]
arXiv preprint arXiv:2511.15068 , year=
Classification Trees with Valid Inference via the Exponential Mechanism , author=. arXiv preprint arXiv:2511.15068 , year=
-
[45]
arXiv preprint arXiv:2412.20535 , year=
Inference with randomized regression trees , author=. arXiv preprint arXiv:2412.20535 , year=
-
[46]
Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining , pages=
Winner's curse: Bias estimation for total effects of features in online controlled experiments , author=. Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining , pages=
-
[47]
PLoS genetics , volume=
Review and further developments in statistical corrections for Winner’s Curse in genetic association studies , author=. PLoS genetics , volume=. 2023 , publisher=
2023
-
[48]
Biostatistics , volume=
Selection-adjusted inference: an application to confidence intervals for cis-eQTL effect sizes , author=. Biostatistics , volume=. 2021 , publisher=
2021
-
[49]
Genetic epidemiology , volume=
Empirical Bayes correction for the Winner's Curse in genetic association studies , author=. Genetic epidemiology , volume=. 2013 , publisher=
2013
-
[50]
2023 , month =
Tao, Terence , title =. 2023 , month =
2023
-
[51]
Annals of Probability , year=
\ I\ -Divergence Geometry of Probability Distributions and Minimization Problems , author=. Annals of Probability , year=
-
[52]
A generalization of the Lindeberg principle , volume=
Chatterjee, Sourav , year=. A generalization of the Lindeberg principle , volume=. The Annals of Probability , publisher=. doi:10.1214/009117906000000575 , number=
-
[53]
Canadian Journal of Statistics , volume=
Post-selection point and interval estimation of signal sizes in Gaussian samples , author=. Canadian Journal of Statistics , volume=. 2017 , publisher=
2017
-
[54]
arXiv preprint arXiv:2509.13538 , year=
Selective and marginal selective inference for exceptional groups , author=. arXiv preprint arXiv:2509.13538 , year=
-
[55]
AEA Papers and Proceedings , volume=
Inference for losers , author=. AEA Papers and Proceedings , volume=. 2022 , organization=
2022
-
[56]
arXiv preprint arXiv:2408.02060 , year=
Winners with confidence: Discrete argmin inference with an application to model selection , author=. arXiv preprint arXiv:2408.02060 , year=
-
[57]
Statistica Sinica , pages=
Statistical applications of the Poisson-binomial and conditional Bernoulli distributions , author=. Statistica Sinica , pages=. 1997 , publisher=
1997
-
[58]
Information processing letters , volume=
Weighted random sampling with a reservoir , author=. Information processing letters , volume=. 2006 , publisher=
2006
-
[59]
International Conference on Machine Learning , pages=
Differentially private quantiles , author=. International Conference on Machine Learning , pages=. 2021 , organization=
2021
-
[60]
International Conference on Machine Learning , pages=
A Joint Exponential Mechanism For Differentially Private Top- k , author=. International Conference on Machine Learning , pages=. 2022 , organization=
2022
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.