Pith. sign in

REVIEW 2 major objections 5 minor 22 references

A Flexible Defense Against the Winner's Curse

T0 review · 2 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The zoom correction gives valid confidence intervals for the winner's mean under arbitrary dependence and without Gaussian assumptions.

desk verdict A genuinely new, mostly sound method for inference on the winner; its advertised flexibility is exactly as strong as the user-supplied tail bound, and the real-data intervals go beyond that guarantee. read the letter →

arxiv 2411.18569 v1 pith:PEHF54O6 submitted 2024-11-27 stat.ML math.STstat.MEstat.TH

Tijana Zrnic , William Fithian This is my paper · ORCID
classification stat.MLmath.STstat.MEstat.TH MSC 62F2562F07
keywords winner'scursezoomcorrectionpost-selectioninferencesimultaneousconfidenceintervalstop-kwinnersarbitrarydependenceselectionbias
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces the zoom correction, a method for building confidence intervals for the mean of the empirically best candidate, the winner, after the data have selected it. The method guarantees at least $1-\alpha$ coverage under arbitrary dependence between candidates and without assuming Gaussianity or independence; the only required input is a known tail bound on the estimation errors. The correction works by constructing a simultaneous confidence region for all candidates that spends its error budget only on those that could plausibly win, then projecting that region onto the selected coordinate. Because the projection is computed through a 'worst-case' mean vector, the interval can be evaluated with a one-dimensional search, and a step-down version gives explicit endpoint formulas. The same machinery extends to top-$k$ winners, the population-best value and identity, and near-winners.

What carries the argument

The load-bearing object is the zoom test's acceptance region. At a candidate mean $\theta$, the region is $\prod_{j=1}^m [\theta_j - \max\{r_\alpha, \Delta_j/2\}, \theta_j + \max\{r_\alpha, \Delta_j/2\}]$, where $\Delta_j = \max_\ell \theta_\ell - \theta_j$ is the suboptimality of coordinate $j$ and the active radius $r_\alpha$ is the smallest $r$ with $S(\max\{r, \Delta_j/2\}_{j=1}^m) \le \alpha$, so the region has coverage $1-\alpha$ under the tail bound $S$. Inverting this test produces the simultaneous confidence region, and Lemma 3.1 reduces projection onto the winner coordinate to a one-dimensional condition: for the worst-case vector $\theta^t$ with $\theta^t_{\hat\imath}=t$ and $\theta^t_j = \min\{2X_j/3 + t/3, t\}$ for $j\ne \hat\imath$, the point $t$ lies in the projected set exactly when $|X_{\hat\imath} - t| \le r_\alpha(\theta^t)$.

What would settle it

Run a large simulation (say 100,000 trials) with $m=1000$, $X_i \sim N(0,1)$ independent, all $\theta_i=0$, and $\alpha=0.1$; compute the Theorem 3.1 interval using the exact joint tail bound and check whether the empirical coverage of $\theta_{\hat\imath}$ is at least $0.9$ overall. A reproducible coverage deficit under the exact $S$ would refute the paper's central claim, while under-coverage only when $S$ is deliberately misspecified would confirm that the tail-bound input, not the method, is the fragile part.

Watch

Extended reading notes

Core claim

The central claim is Theorem 3.1: if $\hat\imath = \arg\max_i X_i$ is the empirical winner and $r_\alpha(\theta)$ is the smallest radius such that the tail bound $S$ is at most $\alpha$ around a candidate mean $\theta$, then the set $\hat C^\alpha_{\hat\imath}=\{t: |X_{\hat\imath}-t|\le r_\alpha(\theta^t)\}$, with $\theta^t$ the worst-case vector that makes $t$ as easy as possible to include, satisfies $P(\theta_{\hat\imath}\in[\min\hat C^\alpha_{\hat\imath},\max\hat C^\alpha_{\hat\imath}])\ge 1-\alpha$. The interval comes from inverting a pointwise hypothesis test, the zoom test, whose acceptance region is a rectangle centered at $\theta$ with half-width $\max\{r_\alpha,\Delta_j/2\}$ in coordinate $j$, where $\Delta_j$ is how far $\theta_j$ sits below the population maximum. This construction devotes the error budget to 'active' candidates, so the interval automatically widens when many candidates are close and narrows to essentially uncorrected inference when one candidate is clearly best.

Load-bearing premise

The guarantee assumes the user can supply a valid tail bound $S(v)$ with $P(\exists i : |\xi_i| > v_i) \le S(v)$, or valid marginal tail bounds for a union bound; if that input is too small or is calibrated on the same data, the coverage claim can fail.

Editorial extensions

If this is right

  • With only a known marginal tail bound and a union bound, the step-down zoom correction gives valid winner intervals even when the candidate estimates are dependent in unknown ways.
  • The interval width adapts to the data: it matches the fully simultaneous correction when candidates are tied, and approaches the uncorrected marginal interval when one candidate is far ahead.
  • The same inverted-region construction provides valid confidence sets for the top $k$ winners, for the population-best value $\theta^*$, and for the identity of the population winner.
  • In Gaussian settings, the paper's experiments show the method is competitive with hybrid inference and strictly tighter than locally simultaneous and fully simultaneous inference, with no covariance estimate required.
  • The variance-adaptive extension keeps the coverage guarantee when candidates have different known scales, by scaling each coordinate's radius by $\sigma_j$.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The worst-case-vector shortcut is a general template: any data-dependent query for which a dominating parameter vector exists can inherit a scalar projection check, so the construction may extend beyond argmax winners to other selection rules.
  • Since the proof only consumes the tail bound $S$ inside the acceptance region, users could plug in sharper dependence-aware bounds, such as those from martingale concentration, and obtain shorter intervals without changing the method.
  • If a user cannot supply a trustworthy prior tail bound and instead estimates $S$ from the same data, the coverage guarantee has no automatic rescue; the method's flexibility does not remove the need for an external or pre-specified error bound.
  • The paper treats the grid-search computation as practically negligible, but in the non-monotone equation for the lower endpoint $r_l$, a coarse grid could in principle miss the true boundary; an exact root-finding implementation would be a worthwhile robustness check.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper proposes the "zoom correction," a finite-sample confidence-interval method for the mean of the empirically selected best candidate. The idea is to invert a "zoom test" whose acceptance rectangle is deliberately widened around suboptimal coordinates, forming a simultaneous confidence region, and then to project this region onto the winning coordinate. Lemma 3.1 characterizes the projection exactly as {t : |X_hat - t| <= r_alpha(theta^t)}, which yields the main coverage theorem (Theorem 3.1). Under a union-bound tail bound, a step-down implementation is given in Theorem 4.1, with extensions to top-k winners, the population winner's value and identity, near-winners, and a variance-adaptive variant. The paper includes proofs, synthetic experiments, and applications to climate extremes, model leaderboards, and language devices.

Significance. If Theorem 3.1 stands, this is a valuable contribution: it provides valid inference on the empirical winner under arbitrary dependence and with no tuning parameters, requiring only a known tail bound on the errors. I checked the central proof steps (Lemmas 3.1, B.1-B.3, subsequent lemmas, and the step-down arguments) and found the mathematical core sound. The method strictly dominates Bonferroni and locally simultaneous inference in the examples, and the code is provided. The principal caveat is that the finite-sample guarantee is conditional on a correctly specified tail bound S; the real-data sections estimate this input, so the reported intervals do not automatically inherit the stated coverage. This caveat is manageable but needs to be made prominent and addressed.

major comments (2)
  1. [Sections 1.1 and 6.2-6.4] The validity statement in Section 1.1 is explicit that the method requires a valid tail bound S(v) with P(exists i: |xi_i| > v_i) <= S(v), and the coverage theorem is conditional on that object. In the real-data analyses, however, S is not known and is instead estimated: Section 6.2 uses a covariance matrix estimated from older years, Section 6.3 uses normal approximations for logistic regression, and Section 6.4 uses plug-in covariance estimates for logistic coefficients and Fisher z-transformed correlations. The resulting intervals therefore inherit an approximation error that Theorem 3.1 and Theorem 4.1 do not cover. This is the load-bearing input condition of the method, so the paper should either construct S in a way that supplies finite-sample guarantees (for example by using nonparametric tail bounds with a valid uncertainty correction), or explicitly reframe the empirical sections as approximate and study sensitivity of the endpoints to misspecification of S. Without this, the advertised 'nonparametric, arbitrary dependence' claim is stronger than what is delivered.
  2. [Section 3, paragraph after Theorem 3.1] The theorem defines tl and tu as the minimum and maximum of the exact set C_hat^alpha = {t : |X_hat - t| <= r_alpha(theta^t)}, but the implementation used in all experiments is a grid search over t in [X_hat - r_alpha(0), X_hat + r_alpha(0)]. The paper acknowledges this 'technically allows for some inaccuracy' and says it expects the inaccuracy to be negligible. As written, a finite grid can miss the true endpoints, so the plotted intervals are not the exact zoom-correction intervals and could undercover relative to 1 - alpha. Please provide an error-controlled implementation, such as conservative bounds on the discretization error or a root-finding based method, or at least report a numerical study of grid sensitivity to demonstrate that the practical coverage is not compromised.
minor comments (5)
  1. [Section 1.1 and Abstract] The abstract promises validity under 'arbitrary dependencies' without mentioning the known-tail-bound condition. Consider adding a qualifier in the abstract so that readers are immediately aware that the guarantee is conditional on a user-supplied S(v).
  2. [Lemma 4.1 proof] The proof invokes 'the continuity of the noise distribution' to assert the boundary condition tl = X_hat - r_alpha(theta^tl), but the paper assumes continuity of S, not of the error distribution. The step-down proof in Theorem 4.1 actually only needs inequalities Sl(rl) >= alpha and Su(ru) >= alpha; please rephrase to avoid an unstated distributional assumption.
  3. [Section 5.4] The phrase 'known estimate of the variance' is internally contradictory, and the text immediately says the procedure is valid for any choice of sigma_i. Please describe sigma_i as user-specified positive scale parameters (or known fixed constants), not as estimated variances, to avoid confusion.
  4. [Algorithm 2] The update alpha_{j+1} = alpha_j - S((bDelta_(j) + S^{-1}(alpha))/3) uses the fixed uncorrected quantile S^{-1}(alpha) rather than a quantity depending on the current step. The proof shows this is conservative, but a short intuitive explanation of why this particular update is chosen would improve readability.
  5. [Section 6.2] When saying that older data are used to estimate the covariance, please specify the estimator (e.g., sample covariance) and the number of years available, since the quality of the plug-in S is relevant to the first major comment.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the coverage guarantee is a direct consequence of inverting a test built on a user-supplied tail bound.

full rationale

The derivation chain is self-contained and non-circular. The zoom test acceptance region A_alpha(theta) is defined in Eq. (2) by choosing r_alpha as the smallest radius satisfying S(max{r, Delta/2}) <= alpha; because S is assumed to upper-bound P(exists i: |xi_i| > v_i), Proposition 2.1's statement that P(X in A_alpha(theta)) >= 1 - alpha is an immediate consequence of the construction, not an empirical prediction. Inverting this test gives a simultaneous confidence region, and projecting it along the winning coordinate gives the interval in Theorem 3.1; Lemma 3.1 supplies an exact worst-case characterization of the projection, so the theorem does not rely on any fitted constants or data-dependent tuning parameters. The step-down implementation in Theorem 4.1 is proven to return radii that upper-bound the exact endpoints, again using only the assumed marginal tail bound. The self-citations to Zrnic and Fithian's locally simultaneous inference and post-selection work are used as baselines or related-work comparisons, not as load-bearing justification for the zoom correction's validity, and no uniqueness theorem is imported from the authors' prior work. The main input assumption, a known valid tail bound S(v), is genuinely an assumption rather than an output of the method; in the real-data sections S is constructed from normal approximations and plug-in covariance estimates, so those reported intervals inherit approximation error and do not carry the finite-sample guarantee as stated. That is a correctness/assumption gap, but it is not circularity, because the theorem explicitly conditions on a valid S and the method does not redefine S in terms of the interval it is asked to produce.

Assumptions & free parameters 1 free parameters · 5 assumptions · 0 invented entities

The basic zoom correction has no free parameters; its validity rests on the supplied tail bound and standard probability. The variance-adaptive extension adds user-chosen scales σ_j that are not needed for the main theorem. No invented physical or mathematical entities are introduced.

free parameters (1)
  • per-candidate scale σ_j = user-specified or estimated from data
    Used only in the variance-adaptive extension (Section 5.4) to weight interval widths. The paper does not prove coverage if σ_j are estimated from the same data used for selection.
assumptions (5)
  • domain assumption A valid tail bound S(v) satisfying P(∃i:|ξ_i|>v_i)≤S(v) is known, continuous, and coordinate-wise decreasing.
    Main working assumption in Section 1.1; the coverage statement is conditional on this bound.
  • standard math Errors ξ=X−θ have a well-defined joint distribution and finite mean, so θ=E[X] is well defined.
    Needed to define the problem and the probability of the acceptance region.
  • domain assumption For the union-bound version, marginal tail bounds S_i with P(|ξ_i|>r)≤S_i(r) are valid for all i.
    Section 4 replaces S(v) by ΣS_i(v_i); validity relies only on these bounds.
  • domain assumption The noise distribution is continuous, used to assert boundary equalities for interval endpoints in Lemma 4.1.
    Invoked in the proof of Lemma 4.1; if not continuous, the exact characterization can fail, though the step-down bounds may remain conservative.
  • domain assumption For the variance-adaptive extension, σ_i are fixed positive constants (or estimated from independent data).
    Section 5.4; the proof treats σ as constants and does not cover data-dependent σ estimates.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Flexible Defense Against the Winner's Curse." pith.science (2026). https://pith.science/paper/PEHF54O6

@misc{pith2026241118569,
  author       = {Pith},
  title        = {Pith review of: A Flexible Defense Against the Winner's Curse},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PEHF54O6}},
  note         = {Machine review of arXiv:2411.18569}
}
read the original abstract

Across science and policy, decision-makers often need to draw conclusions about the best candidate among competing alternatives. For instance, researchers may seek to infer the effectiveness of the most successful treatment or determine which demographic group benefits most from a specific treatment. Similarly, in machine learning, practitioners are often interested in the population performance of the model that performs best empirically. However, cherry-picking the best candidate leads to the winner's curse: the observed performance for the winner is biased upwards, rendering conclusions based on standard measures of uncertainty invalid. We introduce the zoom correction, a novel approach for valid inference on the winner. Our method is flexible: it can be employed in both parametric and nonparametric settings, can handle arbitrary dependencies between candidates, and automatically adapts to the level of selection bias. The method easily extends to important related problems, such as inference on the top k winners, inference on the value and identity of the population winner, and inference on "near-winners."

Figures

Figures reproduced from arXiv: 2411.18569 by the authors.

Figure 1
Figure 1. Definition of the acceptance region Aα. One can think of the acceptance region as starting from a large radius r and shrinking it until the coverage equals 1−α. (a) We form boxes of radius ∆j 2 around each candidate θj . The radius of Aα around θj can never be smaller than that box. At the beginning, r is larger than all boxes. (b) Once r becomes equal to ∆j 2 for some j, that coordinate becomes inactive and its int… view at source ↗
Figure 2
Figure 2. Data X (brown crosses) and θ t (gray circles) for two candidate values of t. The intervals illustrate the acceptance region Aα(θ t ). (a) We consider a value t less than Xˆı and conclude that t ∈ Cbα ˆı because Xˆı is within the active radius rα(θ t ) around t. (b) We consider a value t greater than Xˆı and conclude that t ̸∈ Cbα ˆı because Xˆı is not within the active radius rα(θ t ) around t. Lemma 3.1. Fix t ∈ R … view at source ↗
Figure 3
Figure 3. Interval widths of the zoom correction compared to conditional [12], hybrid [2], simultaneous over the selected (SoS) [3], and locally simultaneous inference [21], in the synthetic data setting. The gray dashed line corresponds to the uncorrected interval width. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Zoom correction intervals compared to the intervals computed via conditional [12], hybrid [2], fully simultaneous, and locally simultaneous [21] inference, in extreme climate event analysis. inference on the warmest location, but in all other problems it is comparable …
Figure 5
Figure 5. Figure 5: Zoom correction intervals compared to the intervals computed via fully simultaneous and locally simultaneous [21] inference, in analyzing the performance of leaderboard winners. and Bradley-Terry scores of different models are dependent because they are collected throu…
Figure 6
Figure 6. Figure 6: Intervals computed via conditional [12], hybrid [2], simultaneous, and locally simultaneous [21] inference, and the zoom correction, in language device analysis. 7 Discussion We introduced the zoom correction: a novel, flexible approach for valid inference on the winne…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 15 canonical work pages

  1. [1]

    Inference for losers

    Isaiah Andrews, Dillon Bowen, Toru Kitagawa, and Adam McCloskey. Inference for losers. In AEA Papers and Proceedings, volume 112, pages 635–640. American Economic Association 2014 Broadway, Suite 305, Nashville, TN 37203, 2022

    work page 2014

  2. [2]

    Inference on winners.The Quarterly Journal of Economics, 139(1):305–358, 2024

    Isaiah Andrews, Toru Kitagawa, and Adam McCloskey. Inference on winners.The Quarterly Journal of Economics, 139(1):305–358, 2024

    2024

  3. [3]

    Confidence intervals for selected parame- ters

    Yoav Benjamini, Yotam Hechtlinger, and Philip B Stark. Confidence intervals for selected parame- ters. arXiv preprint arXiv:1906.00505, 2019

    arXiv 1906

  4. [4]

    Rank analysis of incomplete block designs: I

    Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons.Biometrika, 39(3/4):324–345, 1952

    1952

  5. [5]

    Chatbot arena: An open platform for evaluating LLMs by human preference

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating LLMs by human preference. InForty-first International Conference on Machine Learning, 2024

    work page 2024

  6. [6]

    A computational approach to politeness with application to social factors

    Cristian Danescu-Niculescu-Mizil, Moritz Sudhof, Dan Jurafsky, Jure Leskovec, and Christopher Potts. A computational approach to politeness with application to social factors. In51st Annual Meeting of the Association for Computational Linguistics, pages 250–259. ACL, 2013

    work page 2013

  7. [7]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–255. IEEE, 2009

    2009

  8. [8]

    Simultaneous Statistical Inference: With Applications in the Life Sciences

    Thorsten Dickhaus. Simultaneous Statistical Inference: With Applications in the Life Sciences. Springer Series in Statistics. Springer, 2014. doi: 10.1007/978-3-642-45183-2

    work page doi:10.1007/978-3-642-45183-2 2014

Show all 22 references
  1. [9]

    Optimal inference after model selection.arXiv preprint arXiv:1410.2597, 2014

    William Fithian, Dennis Sun, and Jonathan Taylor. Optimal inference after model selection.arXiv preprint arXiv:1410.2597, 2014

    2014 arXiv

  2. [10]

    Confidence intervals for the means of the selected populations

    Claudio Fuentes, George Casella, and Martin T Wells. Confidence intervals for the means of the selected populations. Electronic Journal of Statistics, 12:58–79, 2018

    2018

  3. [11]

    On the length of post-model-selection confidence intervals conditional on polyhedral constraints

    Danijel Kivaranovic and Hannes Leeb. On the length of post-model-selection confidence intervals conditional on polyhedral constraints. Journal of the American Statistical Association, 116(534): 845–857, 2021

    2021

  4. [12]

    Exact post-selection inference, with application to the lasso.The Annals of Statistics, 44(3):907–927, 2016

    Jason D Lee, Dennis L Sun, Yuekai Sun, and Jonathan E Taylor. Exact post-selection inference, with application to the lasso.The Annals of Statistics, 44(3):907–927, 2016

    2016

  5. [13]

    Data fission: splitting a single data point

    James Leiner, Boyan Duan, Larry Wasserman, and Aaditya Ramdas. Data fission: splitting a single data point. Journal of the American Statistical Association, pages 1–12, 2023

    2023

  6. [14]

    Miller Jr.Simultaneous Statistical Inference

    Rupert G. Miller Jr.Simultaneous Statistical Inference. Springer Series in Statistics. Springer-Verlag, New York, 2nd edition, 1981

    1981

  7. [15]

    Data thinning for convolution- closed distributions

    Anna Neufeld, Ameer Dharamshi, Lucy L Gao, and Daniela Witten. Data thinning for convolution- closed distributions. Journal of Machine Learning Research, 25(57):1–35, 2024

    2024

  8. [16]

    Splitting strategies for post-selection inference.Biometrika, 110(3):597–614, 2023

    Daniel G Rasines and G Alastair Young. Splitting strategies for post-selection inference.Biometrika, 110(3):597–614, 2023. 18

    2023

  9. [17]

    Weatherbench: a benchmark data set for data-driven weather forecasting.Journal of Advances in Modeling Earth Systems, 12(11):e2020MS002203, 2020

    Stephan Rasp, Peter D Dueben, Sebastian Scher, Jonathan A Weyn, Soukayna Mouatadid, and Nils Thuerey. Weatherbench: a benchmark data set for data-driven weather forecasting.Journal of Advances in Modeling Earth Systems, 12(11):e2020MS002203, 2020

    2020

  10. [18]

    Selective inference with a randomized response.The Annals of Statistics, 46(2):679–710, 2018

    Xiaoying Tian and Jonathan Taylor. Selective inference with a randomized response.The Annals of Statistics, 46(2):679–710, 2018

    2018

  11. [19]

    TorchVision: PyTorch’s computer vision library.https: //github.com/pytorch/vision, 2016

    TorchVision maintainers and contributors. TorchVision: PyTorch’s computer vision library.https: //github.com/pytorch/vision, 2016. GitHub repository

    2016

  12. [20]

    Confidence bounds based on the largest treatment mean

    JH Venter. Confidence bounds based on the largest treatment mean. South African Journal of Science, 84(5):340, 1988

    1988

  13. [21]

    Locally simultaneous inference.The Annals of Statistics, 52(3): 1227–1253, 2024

    Tijana Zrnic and William Fithian. Locally simultaneous inference.The Annals of Statistics, 52(3): 1227–1253, 2024

    2024

  14. [22]

    worst-case

    Tijana Zrnic and Michael I Jordan. Post-selection inference via algorithmic stability.The Annals of Statistics, 51(4):1666–1691, 2023. 19 A Proofs from Section 2 A.1 Proof of Proposition 2.2 Proposition 2.2. The active radiusrα is equal to the1 − α quantile of M (ξ, ∆). Proof....

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.