{"id":"e242762e-dedf-47b5-a0fa-408cc694e053","arxiv_id":"2608.12100","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The thesis proposes noise-aware temperature scaling, conformal prediction, label-private conformal prediction, and target-domain calibration that reconstruct clean confidence from noisy or privatized data.","lead":"A machine learning thesis develops calibration and conformal prediction methods that stay reliable under noisy labels, privacy constraints, and unsupervised domain shift. It offers finite-sample coverage guarantees for many-class tasks and tests the methods on medical and natural image datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"NACP's k-independent coverage guarantee is unsupported: Lemma 4.3.4 applies DKW to a quantity that is not an empirical CDF of i.i.d. observations.","rationale":"The reader's weakest-assumption pick, real annotator noise, is a genuine limitation and Table 4.6 supports it. However, the thesis's headline theoretical result is explicitly conditional on uniform noise with known epsilon, so a violation of that condition limits the method's scope rather than falsifying the theorem. The more load-bearing issue is internal: the proof of Lemma 4.3.4 applies DKW to \\hat{F}^r, which is not an empirical CDF of n i.i.d. scalar observations, and the claimed k-independence of Delta rests on exactly that step. If this cannot be repaired, the central guarantee is unproven even in the idealized setting the theorem assumes. This is not a disagreement with the reader's verdict; I agree that the contribution is substantial and likely patchable, and that the manuscript should either supply a valid empirical-process bound for \\hat{F}^r or narrow the stated guarantee. The proposed simulation test isolates the issue cleanly: it directly checks whether the claimed 1-delta uniform deviation bound holds, and whether the true deviation has a k-dependence that the paper's Delta omits.","tokens_in":58906,"tokens_out":9587,"duration_ms":90563,"concrete_test":"Simulate Lemma 4.3.4 under ideal uniform noise with known epsilon. Use a trained classifier with k classes, e.g. ImageNet ResNet-18 with k=1000, calibration size n=5000, epsilon=0.2, delta=0.001. Repeatedly draw calibration sets; compute the exact F^c(q) from a large clean labeled set and \\hat{F}^c(q) via the NACP estimator, and record M = sup_q |\\hat{F}^c(q) - F^c(q)|. Compare the empirical (1-delta) quantile of M to Delta = sqrt(log(4/delta)/(2 n h^2)). If the quantile exceeds Delta materially, Lemma 4.3.4's DKW step is invalid and Theorem 4.3.5 is not proven by the paper's argument. Repeat with k=10 and k=100 to isolate whether the true uniform deviation grows with k.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Lemma 4.3.4 is the linchpin of Theorem 4.3.5, the paper's headline claim that the finite-sample correction Delta is independent of the number of classes under uniform noise. The proof applies the DKW inequality separately to \\hat{F}^n and \\hat{F}^r and union-bounds the two events. \\hat{F}^n is a genuine empirical CDF of noisy conformity scores, so DKW applies to it. But \\hat{F}^r(q) = (1/n) \\sum_i |C_q(x_i)|/k = (1/(nk)) \\sum_{i,j} 1{S(x_i,j) \\le q} is not an empirical CDF of n i.i.d. scalar observations: for a fixed x_i, the k terms share the same input and can have different marginal distributions, and across i the groups are independent but the within-group entries are dependent. It is an average of conditional CDFs, not an empirical CDF. The uniform-in-q deviation of such an average can carry a k-dependent constant (e.g. via empirical-process or VC-type bounds), so the claimed Delta = sqrt(log(4/delta)/(2 n h^2)), which has no k dependence, does not follow from the stated argument. Thus the central coverage guarantee is not established as written, even in the idealized uniform-noise setting with known epsilon. This is an internal-consistency concern, distinct from the real-noise limitation: fixing the real-noise assumption would not repair the proof gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This PhD-thesis manuscript develops methods for confidence calibration and conformal prediction under label noise, under local differential privacy, and under unsupervised domain shift. The main technical contributions are: a noise-robust temperature scaling method for noisy validation labels (Chapter 3); two noisy-label conformal prediction methods, NRSCP and NACP, with NACP intended to provide a finite-sample coverage guarantee whose correction term is independent of the number of classes under uniform label noise (Chapter 4); two locally differentially private conformal prediction frameworks, LDP-CP-L and LDP-CP-S (Chapter 5); and a target-domain calibration method, UTDC, that rescales source-domain binwise accuracy by an estimated accuracy ratio (Chapter 6). The experiments are extensive and cover medical and natural image datasets, multiple architectures, several noise models, and both synthetic and real annotation noise.","tokens_in":59184,"tokens_out":9643,"duration_ms":94519,"significance":"If the central NACP guarantee were valid, the claim that the finite-sample correction is independent of the number of classes for uniform noise would be a substantial advance over the prior noisy-label conformal prediction bounds of ACNL and CRCP, whose correction terms grow with k. The empirical work is careful in many respects: results are averaged over 1000 splits for the conformal experiments, several model architectures are compared, and code links are provided. The value of NACP as an efficient practical method and the strength of the UTDC domain-adaptation results are not in question. However, the theoretical guarantee that is the paper's headline claim is supported by an invalid application of the DKW inequality, so the significance of the theoretical contribution is currently conditional.","major_comments":[{"comment":"The DKW inequality is applied to \\hat F^r(q) = (1/n) \\sum_i |C_q(x_i)|/k, but this quantity is not an empirical CDF of n i.i.d. scalar observations. For a fixed x_i, the k terms S(x_i,j) share the same input and are dependent, and the function q \\mapsto |C_q(x_i)|/k is a conditional CDF rather than an indicator of one observed scalar. A uniform-in-q deviation bound for such an average of random functions does not follow from DKW as stated; a correct argument would need empirical-process or VC-type machinery and would in general introduce a k-dependent constant. Therefore the k-independent form of \\Delta in Theorem 4.3.5, which is the paper's main claim for many-class problems, is not established.","section":"§4.3.2, Lemma 4.3.4"},{"comment":"There is a mismatch between the theorem and the implemented algorithm. Theorem 4.3.5 proves coverage at least 1-\\alpha only when q is chosen so that \\hat F^c(q) = 1 - \\alpha + \\Delta, but Algorithm 4, line 4, selects q satisfying \\hat F^c(q) = 1 - \\alpha, and the guarantee printed in Algorithm 4, line 6, states coverage at least 1 - \\alpha - \\Delta. The claim that the unadjusted procedure reaches nominal coverage in the experiments is an empirical observation and does not replace the missing proof. The same issue appears in Algorithm 7 and Theorem 5.4.1 in Chapter 5.","section":"§4.3.2, Algorithm 4, and Theorem 4.3.5"},{"comment":"The proof of the LDP-CP-L coverage guarantee is not supplied. The text says that substituting \\beta and n into Theorem 4.3.5 yields the bound, but Theorem 4.3.5 requires the condition \\hat F^c(q) = 1 - \\alpha + \\Delta and gives a two-sided coverage statement 1 - \\alpha \\le p(y \\in C_q(x)) \\le 1 - \\alpha + 2\\Delta; neither the condition nor the claimed one-sided 1 - \\alpha - \\Delta form is verified for the binary-search output of Algorithm 7. Additionally, because the raw features x_i are sent unperturbed to the aggregator, k-RR on labels provides label-LDP, not \\epsilon-LDP for the full data pair (x,y); the theorem statement should be aligned with the label-DP setting that Section 5.2 actually describes.","section":"§5.4, Theorem 5.4.1"},{"comment":"The proof of Theorem 5.4.2 is too terse to establish the stated conformal coverage guarantee. The theorem gives a sample complexity for the private binary-search quantile estimator, but the proof does not show how an error in the estimated (1-\\alpha)-quantile of conformity scores translates into the stated bound on p(y \\in C_{\\hat q}(x)). The argument also implicitly assumes that the scores are bounded in a known interval [Q_min, Q_max] and that the binary-search interval satisfies the \\tau-condition, but these assumptions are not stated in the theorem. A complete derivation connecting the quantile estimator's error to conformal coverage is needed.","section":"§5.3, §5.4, Theorem 5.4.2"}],"minor_comments":[{"comment":"Equation (2.10) as written is not a stochastic matrix: for the uniform noise model that flips to one of the remaining k-1 classes, the off-diagonal entry should be \\epsilon/(k-1), not \\epsilon/k, otherwise the row sum is 1 - \\epsilon/k rather than 1. The text says the two uniform-noise conventions are used interchangeably, but the displayed matrix should be corrected or the normalization should be explained.","section":"§2.4, Eq. (2.10)"},{"comment":"The proof refers to \"Lemma 3.1\" when it means Lemma 4.3.1; the cross-reference should be fixed.","section":"§4.3.1, proof of Lemma 4.3.1"},{"comment":"On CIFAR-10N-aggregate, NACP with rand-APS reports 88.49% coverage, below the 90% target. The authors do acknowledge that real annotator noise is not uniform, but this violation of the key assumption should be stated directly in the text near the table, since it shows that the coverage guarantee does not transfer to the real-noise experiments.","section":"§4.5, Table 4.6"},{"comment":"The assumption that the source-to-target accuracy ratio is constant across confidence bins is validated only qualitatively and on a single task in Figure 6.4. Since this ratio rescaling is the core of UTDC, it would be helpful to report a quantitative measure of the binwise ratio variability across the different benchmark tasks.","section":"§6.2, Eq. (6.1)"}],"recommendation":"major_revision","confidential_remarks":"The main risk is not the empirical results but the validity of the headline k-independent coverage bound. The DKW argument in Lemma 4.3.4 is formally incorrect for the quantity \\hat F^r, and the theorem/algorithm mismatch in Algorithms 4 and 7 means that even a corrected lemma would not immediately imply the stated procedure's guarantee. If the authors can supply a correct finite-sample proof, or must weaken the claim to a k-dependent correction, the paper needs a substantial revision of its central theoretical contribution before it can be accepted. The UTDC and NTS chapters appear sounder and empirically strong; the revision should clearly separate those contributions from the part of Chapter 4 whose theory is currently unsupported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. This is a PhD thesis that packages several published papers, and it deserves a serious referee, but not acceptance as-is. The genuinely new material is real: NTS reconstructs clean binwise accuracy from noisy validation labels and works well on medical imaging; NACP's uniform-noise correction term is the first I know of whose finite-sample bound does not grow with k; LDP-CP-L/S adapt the noisy-channel idea to local differential privacy; UTDC calibrates on the target domain without labels by rescaling source binwise accuracy with an estimated accuracy ratio. Code links are included for the main methods. The empirical comparisons are extensive, and the gains on ImageNet, CIFAR-100, and TinyImageNet over ACNL and CRCP are striking.\n\nThe soft spot is the proof of the headline claim. Lemma 4.3.4 applies DKW to \\hat F^r(q), which is an average over i of |C_q(x_i)|/k—not an empirical CDF of i.i.d. observations. Within a fixed x_i the k score terms are dependent and can have different marginals; DKW simply does not apply. Consequently Theorem 4.3.5, the k-independent coverage guarantee for NACP, is not established even in the idealized uniform-noise setting with known epsilon. The gap is repairable in principle, but a valid empirical-process or VC-type bound would likely reintroduce a k dependence, so the paper's key selling point may survive only in weakened form. Theorem 5.4.1 inherits this problem, so the LDP-CP-L coverage guarantee has the same issue.\n\nTwo smaller issues. The privacy statement for LDP-CP-L is overstated: the algorithm emits raw features x, so it is label-DP, not epsilon-LDP for the full data point; the prose later admits this, but the theorem label does not. And on CIFAR-10N, where the noise is not uniform, NACP sometimes undercovers (88.5% vs. the 90% target), which is expected given the assumption but should be presented as a limitation rather than a success.\n\nWho is this for? Researchers working on conformal prediction under label noise or calibration in medical/high-stakes settings will get value from the empirical chapters and the NTS/UTDC parts. The NACP guarantee needs a fix before it should be cited as a theorem. I'd send it to a serious referee. My verdict would be major revision, conditional on fixing Lemma 4.3.4.","headline":"Genuinely useful empirical chapters and a promising idea, but the k-independent coverage guarantee rests on a DKW step that does not apply; deserves a serious referee and major revision.","tokens_in":59703,"tokens_out":2843,"would_cite":true,"duration_ms":28470,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Noisy labels need not inflate conformal prediction sets","keywords":["confidence calibration","conformal prediction","label noise","temperature scaling","coverage guarantee","local differential privacy","unsupervised domain adaptation","prediction sets"],"falsifier":"Run NACP on a synthetic validation set with 1000 classes where the noise probability is twice as high for low-confidence inputs; if empirical coverage falls below $1-\\alpha-\\Delta$, or if the prediction set size grows with the number of classes under this feature-dependent noise, then the uniform x-independent noise model is the failing premise.","tokens_in":58665,"feed_emoji":"🎯","tokens_out":3355,"duration_ms":35479,"temperature":0.7,"pith_summary":"The thesis claims that confidence calibration and conformal prediction can be made reliable when validation labels are noisy, by modeling the noise and reconstructing the clean-label behavior. Its central method, NACP, estimates clean coverage from noisy coverage and average set size, then chooses a threshold whose clean coverage equals the target. The key payoff is a finite-sample coverage guarantee whose correction term does not grow with the number of classes, so tasks with hundreds or thousands of classes can keep small prediction sets. The same noise-modeling idea is applied to temperature scaling for medical imaging, to locally differentially private calibration, and to unsupervised domain adaptation.","feed_headline":"Noisy labels need not inflate prediction sets","feed_subtitle":"A noise-aware threshold recovers clean-label coverage even with many classes, keeping 90% sets small.","key_machinery":"The central identity is the noisy-coverage decomposition $F^n(q) = (1-\\epsilon)F^c(q) + \\epsilon F^r(q)$, where a validation label is either the true label or a uniformly random label. NACP estimates $F^n$ from noisy validation scores, estimates $F^r$ from the average prediction-set size $|C_q(x)|/k$, inverts the identity, and searches for the threshold satisfying the target clean coverage. The finite-sample guarantee is carried by a DKW-type uniform concentration bound on the estimated clean CDF, producing the correction $\\Delta = \\sqrt{\\log(4/\\delta)/(2nh^2)}$ with $h=(1-\\epsilon)/(1+\\epsilon)$, which is independent of the number of classes.","core_discovery":"On its own terms, the paper establishes that under uniform, input-independent label noise with rate $\\epsilon$, the clean conformal coverage $F^c(q)$ can be reconstructed from the noisy coverage $F^n(q)$ and the random-label coverage $F^r(q)$ through $\\hat{F}^c(q) = (\\hat{F}^n(q) - \\epsilon \\hat{F}^r(q))/(1-\\epsilon)$. Choosing the threshold where $\\hat{F}^c(q) = 1-\\alpha$ yields prediction sets whose size tracks the clean-label oracle, with a finite-sample coverage guarantee $p(y \\in C_q(x)) \\ge 1-\\alpha-\\Delta$ in which $\\Delta$ depends on validation set size and noise rate but not on the number of classes. The thesis further argues that ordinary temperature scaling fails badly on noisy validation labels, that a confusion-matrix correction recovers clean calibration for general noise matrices, and that the same reconstruction identity turns local differential privacy mechanisms into a known noise channel for conformal prediction.","pith_inferences":["Because the class-independent correction holds only for uniform noise, a natural next test is replacing the scalar $\\epsilon$ with per-input noise estimates to handle instance-dependent label corruption; the thesis does not prove such an extension.","The reconstruction identity also suggests a unified view of privacy and noise: any mechanism that acts as a known label-noise channel, beyond k-ary randomized response, could be plugged into the same threshold search with an appropriate correction term.","A testable extension is applying NACP in online or streaming conformal settings where noisy calibration labels arrive incrementally and the coverage estimate must be updated without re-searching the whole grid."],"forward_implications":["For validation sets corrupted by uniform label noise, practitioners can set conformal thresholds using noisy labels and still obtain prediction sets close to clean-label size.","In many-class tasks with 100 to 1000 classes, methods whose correction terms scale with the number of classes become useless, while NACP's class-independent correction keeps them viable.","Temperature scaling applied to noisy validation sets can match clean-label calibration on medical imaging benchmarks instead of overheating the model.","Local differential privacy can be added to conformal prediction by treating the privacy mechanism as a known noise channel, with coverage adjusted by a finite-sample term.","Calibrating on unlabeled target data by rescaling source accuracy substantially improves calibration under domain shift compared with importance-weighted source calibration."],"supporting_citations":[{"why":"Provides the conformal prediction framework and the quantile-threshold coverage theorem that NACP builds on.","marker":"[93]"},{"why":"Supplies the APS conformal score used in the experiments and the randomized-score coverage guarantee.","marker":"[82]"},{"why":"Defines the Noisy-CP baseline that ignores label noise, which NACP is compared against and outperforms in set size.","marker":"[21]"},{"why":"Provides a prior noise-aware conformal method whose class-dependent correction term becomes ineffective for many classes.","marker":"[86]"},{"why":"Provides another contamination-robust conformal baseline whose correction term scales with the number of classes.","marker":"[13]"},{"why":"Supplies the Dvoretzky-Kiefer-Wolfowitz inequality used in the finite-sample coverage guarantee for NACP.","marker":"[60]"},{"why":"Provides the noise-robust training method and noise matrix estimation used in the end-to-end calibration experiments.","marker":"[52]"}],"fun_headline_variants":["Noise-aware calibration: keep prediction sets tight","A formula to strip label noise from calibration","Recover clean coverage without clean validation data","Clean confidence estimates even with noisy labels","Private conformal prediction that stays calibrated"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The coverage guarantee assumes the label noise is uniform and independent of the input, with the noise rate known or well estimated; real annotator noise can violate this, and the paper's own CIFAR-10N experiment shows coverage falling below the 90 percent target on the aggregate set.","fun_headline_variants_meta":{"raw":{"variants":["Noise-aware calibration: keep prediction sets tight","A formula to strip label noise from calibration","Recover clean coverage without clean validation data","Clean confidence estimates even with noisy labels","Private conformal prediction that stays calibrated"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000382,"raw_usage":{"total_tokens":2054,"prompt_tokens":1005,"completion_tokens":1049,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":984}},"tokens_in":621,"tokens_out":1049,"duration_ms":10596,"temperature":1.0,"reasoning_tokens":984,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:16:40.752219+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run NACP on a synthetic validation set with 1000 classes where the noise probability is twice as high for low-confidence inputs; if empirical coverage falls below $1-\\alpha-\\Delta$, or if the prediction set size grows with the number of classes under this feature-dependent noise, then the uniform x-independent noise model is the failing premise.","supporting_citations":[{"cited_title":"Classification with valid and adaptive coverage","cited_arxiv_id":null,"evidence_quote":"Supplies the APS conformal score used in the experiments and the randomized-score coverage guarantee."},{"cited_title":"Provably end-to-end label-noise learning without anchor points","cited_arxiv_id":null,"evidence_quote":"Provides the noise-robust training method and noise matrix estimation used in the end-to-end calibration experiments."}],"review_version":1}