Pith. sign in

REVIEW 3 major objections 5 minor 15 references

Existing conformal selection methods break under noisy calibration labels; RCS replaces them with a covariate-adjusted weight estimate that provably controls FDR and recovers oracle power.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 03:56 UTC pith:GE4VU2HZ

load-bearing objection RCS is a genuinely new and mostly sound framework for conformalized selection under noisy calibration, but the 'general contamination' claim is narrower than advertised because the estimated localized weights can blow up. the 3 major comments →

arxiv 2607.22985 v1 pith:GE4VU2HZ submitted 2026-07-25 stat.ML cs.LGstat.ME

Robust Conformalized Selection with Noisy Responses

Chengyao Yu , Hongxin Wei , Bingyi Jing This is my paper
classification stat.ML cs.LGstat.ME
keywords Conformalized selectionFalse discovery rateLabel contaminationLocalized covariate shiftEmpirical BayesRandomized responseMultiple testingClass-conditional label noise
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Conformalized selection tells practitioners which candidates to trust, but its guarantees assume clean calibration responses. This paper shows that when calibration labels are contaminated, two existing conformal selection methods either lose control of the false discovery rate or become severely conservative, and it delivers a replacement: Robust Conformalized Selection (RCS). RCS reframes label noise as a localized covariate shift by conditioning on each predicted class, then estimates the false discovery proportion with class-specific weights. Under the paper's regularity conditions, RCS controls group-wise FDR asymptotically and reaches the best recovery power among all score-threshold rules. The upshot is that noisy labels from crowdsourcing, label differential privacy, or assay variability no longer need to be treated as clean.

Core claim

The paper's central claim is that a class-conditional weight w_k(x)=P(Y≠k|X=x)/P(˜Y≠k|X=x) makes contaminated calibration data as usable as clean data for selection. Weighting each calibration point by w_k(X_i) corrects the mismatch between clean and noisy labels and yields a covariate-adjusted empirical-Bayes estimate of the number of false selections; taking the largest threshold whose estimated FDP is at most α gives asymptotic group-wise FDR control, and the selected set's recovery power converges to the oracle optimum. The paper proves this for general label contamination, specializes it to class-conditional transition matrices and randomized response, shows that U*(X)=1−P(Y=\hat f(X)|X

What carries the argument

The load-bearing object is the localized weight w_k(x)=P(Y≠k|X=x)/P(˜Y≠k|X=x), the ratio of clean to noisy disagreement probabilities among points the classifier assigns to class k. Multiplying noisy calibration indicators by this weight converts the label-noise FDP into a covariate-shift FDP, and the group A_g estimate is a weighted count of calibration points with noisy label ≠k and low uncertainty, scaled by class-size ratio. Threshold t̂_A_g is the largest t with estimated FDP ≤ α_A_g; the theory shows this threshold tracks the oracle feasible frontier, delivering FDR control and asymptotic power optimality.

Load-bearing premise

The localized weights w_k(x)=P(Y≠k|X=x)/P(˜Y≠k|X=x) must be well-defined, uniformly bounded, and consistently estimable; for the class-conditional implementation this means the noise transition matrix and clean class probabilities must be known or reliably estimated.

What would settle it

Simulate K=10 classes with block-structured label noise (contamination only inside pairs of classes) and run RCS using a randomized-response transition matrix estimated from paired clean/noisy labels; then compute group-wise FDR from held-out true labels as K and noise rate grow. Theorem 14 predicts control when the estimated weights are L2-close; if FDR inflates here, it pins down when the consistency assumption is really needed.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Calibration data with contaminated responses can be used directly for trustworthy labeling; RCS keeps group-wise FDR at the target level as n,m grow.
  • The optimal score U*(X)=1−P(Y=\hat f(X)|X) shows that a well-calibrated probability estimate is the right confidence measure for maximum recovery power.
  • Under randomized response noise, RCS needs no weight estimation: a simple adjustment of the FDR threshold by ϵ(1−α−1/K) preserves validity.
  • Threshold-based regression selection (responses above a cutoff) inherits the same guarantees once binarized, giving a robust alternative to earlier conformal selection.
  • Estimating the contamination transition matrix from a small clean subset, or using a lower confidence bound, maintains FDR control in the paper's experiments.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • [Inference] The weight formulation separates the contamination model from the selection rule, so the same FDP estimate could be dropped into other false-discovery procedures whenever class-conditionally consistent weights are available.
  • [Inference] When the transition matrix is misspecified, the paper's formal guarantee (Theorem 14) weakens because the estimated weights are no longer L2-close; the experiments show only mild power loss, but extreme block-structured noise with a randomized-response working model is the regime to watch.
  • [Inference] The binary reduction for Task 2 suggests the approach extends beyond classification to any continuous screening problem where the response can be binarized at a threshold, provided one can estimate the localized weight.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies conformalized selection when the calibration responses are contaminated, covering both selection of correctly classified candidates (Task 1) and selection of candidates with responses above a threshold (Task 2). It first shows that existing methods (cfBH and PSP) can lose power or fail to control FDR under contamination. It then proposes Robust Conformalized Selection (RCS), which estimates the false-discovery proportion through class-localized weights w_k(x)=P(Y≠k|X=x)/P(tilde Y≠k|X=x), an estimate the authors interpret as a covariate-adjusted empirical-Bayes FDP. The main theorems assert asymptotic group-wise FDR control and recovery-power optimality when the weights are well-defined and uniformly bounded (Theorem 11), robustness when weights are estimated consistently in L2 (Theorem 14), and a simplified specialization for randomized response noise (Proposition 16) as well as a reduction of Task 2 to binary RCS (Corollary 17). The paper includes simulations on synthetic data and applications to CIFAR-10H and ACS income data, with code made available.

Significance. If the central claims hold, this is the first systematic treatment of conformalized selection under contaminated calibration data, a practically important gap relative to the extensive clean-calibration literature. The key identity underlying the FDP estimator is clean and is derived from a probability calculation rather than fitted to a target; the proofs in Appendix B are detailed and mostly standard. The asymptotic power-optimality argument for the oracle score U* is also a genuine contribution. The experiments cover several contamination models and real data, and the provision of reproduction code is a strength. However, the central theorem is conditional on the localized weights being well-defined and uniformly bounded, and the class-conditional plug-in implementation used in Algorithm 1 does not guarantee this condition. The robustness theorem covers only L2-consistent estimation of an already bounded oracle weight, and the claimed robustness to mis-specified transition matrices is not supported by a theorem. These gaps matter for the paper's headline claim of FDR control under general label contamination, although they do not invalidate the core derivation under the stated a

major comments (3)
  1. [Sec. 4.3, Eq. (11), Prop. 15, Algorithm 1, Thm. 11] This is a load-bearing gap because Algorithm 1 is the concrete recipe for the paper's main method.
  2. [Sec. 6.1.3, Thm. 14] This affects the generality of the main contribution, not just the presentation.
  3. [Sec. 5.2, Eq. (21), Cor. 17] This is the same class of issue as the first major comment, but for the second task.
minor comments (5)
  1. [Appendix B.4] In the proof of Theorem 4, the sentence 'by (B.3) and the dominated convergence theorem' appears to refer to equation (34) or (33); the reference is incorrect or the equation numbering is off.
  2. [Figure 1] The legend appears to list 'BH_clip (drift = 0.5)' twice and does not show the -0.5 drift case described in the text; please correct the legend or the description.
  3. [Appendix B.5 (proof of Thm. 11)] After defining t_delta, the text says 'Since eps > 0 is arbitrary' but the variable is delta; this is a typo.
  4. [Sec. 6.1.2 and 6.1.3] The text states that 'RCS maintains FDR control in all settings' and 'all RCS-based methods control FDR below 0.1'. These are simulation conclusions; I suggest saying 'in the experiments reported here' to avoid appearing to claim a general theorem.
  5. [Algorithm 1] The threshold in Eq. (13) is defined as a maximum over t in [0,1], while Algorithm 1 restricts to a finite grid. It would help to state that the max in (13) is attained or that the grid is used only for implementation.

Circularity Check

0 steps flagged

No significant circularity: the FDP estimator is derived from an exact probability identity, nuisance parameters are estimated on independent data, and the FDR-control theorems do not assume their own conclusions.

full rationale

The central claim (Theorem 11) is not circular. The localized weights w_k(x) in Eq. (11) are defined as a ratio of conditional probabilities, and the identity in Remark 9, E[w_k(X) I{hatY=k, tildeY≠k, U≤t}] = P(hatY=k, Y≠k, U≤t), follows exactly from the tower property and the definition of w_k; it is not a fitted quantity. The FDP estimator (12) is a plug-in importance-weighted average whose consistency is proved in Lemma 18 via Glivenko-Cantelli arguments under the stated boundedness of w. Theorem 14 requires only L2 consistency of independently estimated weights, and Theorem 11 explicitly conditions on w being well-defined and uniformly bounded; these are honest assumptions, not restatements of the target conclusion. Theorem 13's optimality of U* = 1 - P(Y=hatf(X)|X) is established by a Neyman-Pearson/relaxation argument over all measurable rules, not by defining the objective in terms of U*. The only self-citations (Yu et al. 2024 in Remark 9; Ming et al. 2025 in Appendix C.1.1) are non-load-bearing: the first is a contextual perspective citation, and the second is an optional random-weighting alternative while the actual experiments use bootstrap. The skeptic concern about misspecified T or unbounded weights is a limitation of the 'general contamination' scope, because no theorem is claimed for the mis-specified plug-in; the mis-specification results are empirical. Overall the derivation is self-contained with respect to its stated assumptions.

Axiom & Free-Parameter Ledger

4 free parameters · 7 axioms · 0 invented entities

The paper does not introduce new physical or ontological entities. Its contribution is conditional on nuisance estimation of the contamination mechanism: the localized weights and transition matrix are inferred from data, and the FDR guarantee depends on the consistency of those estimates.

free parameters (4)
  • Transition matrix T (or scalar noise rate ϵ) = Plug-in estimates from clean/noisy fitting datasets; bootstrap one-sided lower bounds in RCS (CI-lower T)
    Needed to construct localized weights via (17)/(18); mis-estimation directly changes the FDP estimate.
  • Localized weight functions w_k(x) = Estimated as \hw_k via plug-in \hμ and \hT, or score-localized binning for Task 2
    This infinite-dimensional nuisance is the covariate adjustment in Eq. 12; Theorem 14 requires L2-consistent estimation.
  • Uncertainty score U(x) = User-specified, e.g., U = 1 - max_k \hμ_k(x) or U = -\hμ(x)
    Choice affects power; Theorem 13 says U* = 1 - P(Y=\hf(X)|X) is optimal, but in practice it is approximated by the soft-classifier output.
  • Smoothing and binning constants (λ=1, B=3, B=200, β=0.05) = Implementation constants in Appendix C
    Used for estimating T and score-localized weights; no theoretical optimality is claimed.
axioms (7)
  • domain assumption Calibration and test triples (X_i, Y_i, \Y_i) are i.i.d.
    Section 2.1; basis of exchangeability and conformal validity.
  • domain assumption Predictor \f and score U are pre-trained and independent of calibration/test data.
    Remark 7; Theorems condition on the training stage.
  • domain assumption Localized weights are well-defined and uniformly bounded: 0 < w_k(x) < ∞.
    Eq. 11 and Theorem 11; fails if P(\Y\neq k|X) = 0 in a region.
  • domain assumption Class-conditional contamination \Y ⊥ X | Y with transition matrix T.
    Section 4.3, Eq. 16; used for plug-in weights and Proposition 15.
  • domain assumption Assumption 10: no atoms in U|Y,\hY; positive group masses; oracle feasible frontier is strictly approximable.
    Technical regularity for asymptotic threshold convergence and non-degenerate power.
  • domain assumption Estimated weights are L2-consistent: \|\hw - w\|_{L2(P_X)} = o_p(1).
    Theorem 14; not proven for the score-localized estimator in Appendix C.2.2.
  • domain assumption Under the randomized-response specialization, the noise rate ϵ is known or a valid lower bound is available.
    Section 5.1 and Section 6.1.2 bounded-contamination experiments.

pith-pipeline@v1.3.0-alltime-deepseek · 43524 in / 13445 out tokens · 135226 ms · 2026-08-01T03:56:19.286740+00:00 · methodology

0 comments
read the original abstract

Conformalized selection has been widely applied to select high-quality candidates from large datasets with rigorous uncertainty quantification, such as reliable labeling, drug discovery, and the alignment of large language models. Nevertheless, existing methods assume clean responses on calibration data, an assumption that rarely holds in practice. In this paper, we formulate the above tasks as selecting candidates with true predicted labels or with responses exceeding certain values. We demonstrate that existing conformal selection methods fail to control the false discovery rate (FDR) or suffer from severe power loss under contaminated calibration data. To that end, we propose Robust Conformalized Selection (RCS), a unified framework for selective classification with valid FDR control under general label contamination. The key insight of RCS lies in a novel statistical reduction: by separately conditioning on different classes, we translate the intractable label noise into a localized covariate shift problem, which then enables a covariate-adjusted empirical-Bayes-type estimate of the number of false selections. Statistical properties such as the asymptotic FDR control, power optimality, and robustness of RCS are established. We further develop an instantiation of RCS under randomized response model, and also apply RCS to the task of selecting candidates with large response values. Extensive experiments on both simulated and real-world datasets demonstrate the effectiveness of RCS.

Figures

Figures reproduced from arXiv: 2607.22985 by Bingyi Jing, Chengyao Yu, Hongxin Wei.

Figure 1
Figure 1. Figure 1: Empirical FDR and power of BH clip under clean and contaminated calibration responses as the target FDR level varies. The dashed diagonal in the FDR panel represents the nominal target level. Points and error bars denote the mean and one standard deviation over 500 independent repetitions. sample sizes are set as ntr = n = m = 100K. The probabilities P(Y = k), k ∈ [K] are given by (Z1, . . . , ZK)/ P k∈[K]… view at source ↗
Figure 2
Figure 2. Figure 2: Empirical FDR and power of PSP under clean calibration labels and randomized￾response contamination as the target FDR level varies. All results are averaged over 500 independent repetitions. 6 [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Results under a known randomized response model in overall classification. [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Results under a known asymmetric label noise in overall classification. [PITH_FULL_IMAGE:figures/full_fig_p018_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Results under a known asymmetric label noise in class-wise classification. [PITH_FULL_IMAGE:figures/full_fig_p018_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Performance of methods under a randomized response model ( [PITH_FULL_IMAGE:figures/full_fig_p019_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Robustness to estimated randomized response models. [PITH_FULL_IMAGE:figures/full_fig_p020_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Robustness to structural mis-specification of the contamination model. [PITH_FULL_IMAGE:figures/full_fig_p021_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Results for Task 2 under two constant threshold-status contamination models. [PITH_FULL_IMAGE:figures/full_fig_p022_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Results for Task 2 under additive Gaussian noise. The left and right panels [PITH_FULL_IMAGE:figures/full_fig_p023_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Performance on CIFAR-10H image data with imperfect labels under different [PITH_FULL_IMAGE:figures/full_fig_p024_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Performance on ACS income data with label differential privacy under different [PITH_FULL_IMAGE:figures/full_fig_p024_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Performance of methods under asymmetric label noises ( [PITH_FULL_IMAGE:figures/full_fig_p053_13.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 6 linked inside Pith

  1. [5]

    ACS: An interactive framework for conformal selection.arXiv preprint arXiv:2507.15825,

    Yu Gui, Ying Jin, Yash Nair, and Zhimei Ren. ACS: An interactive framework for conformal selection.arXiv preprint arXiv:2507.15825,

    Pith/arXiv arXiv

  2. [8]

    Feedback- enhanced online multiple testing with applications to conformal selection.arXiv preprint arXiv:2509.03297,

    Lin Lu, Yuyang Huo, Haojie Ren, Zhaojun Wang, and Changliang Zou. Feedback- enhanced online multiple testing with applications to conformal selection.arXiv preprint arXiv:2509.03297,

    Pith/arXiv arXiv

  3. [9]

    Yash Nair, Ying Jin, James Yang, and Emmanuel J Cand` es

    Accepted. Yash Nair, Ying Jin, James Yang, and Emmanuel J Cand` es. Diversifying conformal selec- tions.arXiv preprint arXiv:2506.16229,

    arXiv

  4. [11]

    Published online

    doi: 10.1080/01621459.2026.2634937. Published online. Jack W Scannell, James Bosley, John A Hickman, Gerard R Dawson, Hubert Truebel, Guilherme S Ferreira, Duncan Richards, and J Mark Treherne. Predictive validity in drug discovery: what it is, why it matters and how to improve it.Nature Reviews Drug Discovery, 21(12):915–931,

    arXiv 2026

  5. [12]

    Cheap and fast–but is it good? Evaluating non-expert annotations for natural language tasks

    57 YU, Wei and Jing Rion Snow, Brendan O’Connor, Dan Jurafsky, and Andrew Y Ng. Cheap and fast–but is it good? Evaluating non-expert annotations for natural language tasks. InProceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 254–263,

    2008

  6. [1995]

    Probably approximately correct labels.arXiv preprint arXiv:2506.10908,

    Emmanuel J Cand` es, Andrew Ilyas, and Tijana Zrnic. Probably approximately correct labels.arXiv preprint arXiv:2506.10908,

    arXiv

  7. [2008]

    Training convolutional networks with noisy labels.arXiv preprint arXiv:1406.2080,

    Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, and Rob Fergus. Training convolutional networks with noisy labels.arXiv preprint arXiv:1406.2080,

    Pith/arXiv arXiv 2080

  8. [2012]

    Selection from hierarchical data with conformal e-values

    Yonghoon Lee and Zhimei Ren. Selection from hierarchical data with conformal e-values. arXiv preprint arXiv:2501.02514,

    Pith/arXiv arXiv

  9. [2013]

    A conformal prediction score that is robust to label noise.arXiv preprint arXiv:2405.02648,

    Coby Penso and Jacob Goldberger. A conformal prediction score that is robust to label noise.arXiv preprint arXiv:2405.02648,

    Pith/arXiv arXiv

  10. [2014]

    A unified framework for large-scale inference of classification: Error rate control and optimality.arXiv preprint arXiv:2504.07321,

    Yinrui Sun and Yin Xia. A unified framework for large-scale inference of classification: Error rate control and optimality.arXiv preprint arXiv:2504.07321,

    arXiv

  11. [2020]

    A generalized e- value feature detection method with FDR control at multiple resolutions.arXiv preprint arXiv:2409.17039,

    Chengyao Yu, Ruixing Ming, Min Xiao, Zhanfeng Wang, and Bingyi Jing. A generalized e- value feature detection method with FDR control at multiple resolutions.arXiv preprint arXiv:2409.17039,

    arXiv

  12. [2023]

    Theoretical founda- tions of conformal prediction.arXiv preprint arXiv:2411.11824,

    Anastasios N Angelopoulos, Rina Foygel Barber, and Stephen Bates. Theoretical founda- tions of conformal prediction.arXiv preprint arXiv:2411.11824,

    Pith/arXiv arXiv

  13. [2024]

    Conformal selective prediction with general risk control.arXiv preprint arXiv:2603.24704,

    Tian Bai and Ying Jin. Conformal selective prediction with general risk control.arXiv preprint arXiv:2603.24704,

    arXiv

  14. [2025]

    Controlling the false discovery rate via knockoffs.The Annals of Statistics, 43(5):2055–2085,

    Rina Foygel Barber and Emmanuel J Cand` es. Controlling the false discovery rate via knockoffs.The Annals of Statistics, 43(5):2055–2085,

    2055

  15. [2026]

    Davood Karimi, Haoran Dou, Simon K Warfield, and Ali Gholipour

    doi: 10.1093/biomet/ asaf066. Davood Karimi, Haoran Dou, Simon K Warfield, and Ali Gholipour. Deep learning with noisy labels: Exploring techniques and remedies in medical image analysis.Medical Image Analysis, 65:101759,

    doi:10.1093/biomet/