{"id":"98bd6e91-e670-469a-9c4d-f09c173f2bbf","arxiv_id":"2411.17113","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A distributionally robust training objective with likelihood-ratio pseudo-labels improves classification accuracy under noisy crowd labels.","lead":"This paper trains classifiers when crowd labels are unreliable by treating the estimated probability of each true label as uncertain and protecting against the worst case within a small robustness radius. It derives fast closed-form updates and a likelihood-ratio rule that assigns pseudo-labels only to confident examples, reporting strong accuracy gains on noisy-label benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central robustness guarantee in Eq. (1) is vacuous unless the estimated reference P̂_{y|x,ey} lies within the ε-Wasserstein ball of the true posterior; the paper does not verify this, and the warm-up small-loss heuristic can fail under high label noise.","rationale":"The paper has genuine strengths: the duality derivation in Proposition 2.1 is detailed and appears sound, Theorem 3.3 gives a closed-form robust risk, and the experiments span many baselines and real datasets with code released. The central claim, however, is that CDRO with an ε-Wasserstein ball provides robustness to misspecification of the true-label posterior. That claim rests on the reference being within some ε of the truth; otherwise the minimax formulation optimizes for the wrong center and the 'worst-case' bound is beside the point. The reader's weakest assumption correctly identifies this. My additional reading found a secondary theory-practice gap: the pseudo-label threshold in Remark 3.1 is derived for concave T, while the experiments use convex cross-entropy, so the hard 0/1 pseudo-labeling is not the optimal action from Theorem 3.1 for the actual loss. This is worth noting, but it is not the primary load-bearing issue: even if the threshold were corrected, the coverage of the true posterior by the ambiguity set would still be unverified. I therefore agree with the reader's CONDITIONAL verdict: the paper should either verify the reference coverage empirically or qualify the robustness claim. No change to the verdict is needed from this pass.","tokens_in":51965,"tokens_out":6599,"duration_ms":60059,"concrete_test":"On CIFAR-10 with synthetic IDN-HIGH (≈50% noise, Algorithm 2 of [29] with known instance-dependent transition), compute the true posterior P*(y|x,ey) from the known noise model and class prior for every training instance. After the 30-epoch warm-up, compute the estimated reference P̂(y|x,ey) exactly as in Algorithm 1 (frequency-counting τ̂ from D*_0, ψ from the warm-up model). Estimate the average conditional total-variation (or W_1 with 0-1 cost) distance between P* and P̂ over the training set. If this average distance exceeds the smallest ε for which the method's accuracy is reported (e.g., ε=0.001–0.06 for CIFAR-10), then for those data points the true posterior is outside the ambiguity set and the guarantee in Eq. (1) does not apply.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation (1) minimizes the worst-case risk over Q ∈ Γ_ε(P_{y|x,ey}), so the resulting bound only controls deviations from the reference P. In Section 3.3, P is built as P̂_j(x,ey) ∝ ψ^{(ι)}_j(x) · τ̂_j(ey), with τ̂_j(ey) estimated by frequency counting on the m smallest-loss samples D*_0 of a warm-up model trained on noisy labels. If the warm-up model has memorized noisy labels—which the paper's own training curves (Figures 6-7) show occurring at high noise—then D*_0 contains mislabeled examples, τ̂ is biased, and P̂ can be far from the true posterior P*. The theoretical results (Theorem 2.2, Corollary 2.3) only relate the empirical robust risk to its population counterpart for the same fixed center P; they say nothing about center misspecification. The limitations section explicitly states that transition matrix estimation is not the focus, yet the DRO guarantee is meaningful only if P* ∈ Γ_ε(P̂) for the ε values used ((0,1/K), e.g., 0.001–0.06 for CIFAR-10). No experiment measures the distance between P̂ and P*, or checks the coverage rate. Without that check, the claim that the method is 'robust under misspecified true-label posteriors' is not supported: if the true posterior is outside the ball, the worst-case risk in (1) is not an upper bound for the actual risk, and the algorithm reduces to a heuristic pseudo-labeling method with no distributional robustness guarantee.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a conditional distributionally robust optimization (CDRO) framework for learning from noisy crowdsourced labels. The true-label posterior P_{y|x,ey} is treated as an uncertain distribution inside a Wasserstein ambiguity ball centered at an estimated reference distribution. The authors derive a dual formulation, an analytical per-instance solution, a closed-form expression for the empirical robust risk, and an adaptive update for the Lagrange multiplier. These ingredients lead to Algorithm 1 (AdaptCDRP), which uses likelihood-ratio-based robust pseudo-labels and a pseudo-empirical reference distribution. Experiments on CIFAR-10, CIFAR-100, CIFAR-10N, CIFAR-100N, LabelMe, and Animal-10N report consistent improvements over a broad set of baselines.","tokens_in":52286,"tokens_out":9100,"duration_ms":85826,"significance":"If the proposed framework is fully validated, it would be a useful contribution to noisy-label learning: it gives a clean DRO treatment of posterior misspecification, provides closed-form dual quantities that are computationally attractive, and demonstrates strong empirical gains, especially at high noise. The paper ships detailed appendix proofs, code, and extensive experiments including real human annotations and ablation studies, which are strengths. However, the central theoretical guarantee is conditional on the estimated reference distribution being close to the true posterior, and the paper does not verify this condition or quantify the effect of reference misspecification. The link between the concave-case pseudo-label theory and the convex cross-entropy implementation is also not fully established. These issues do not invalidate the empirical findings, but they do limit the force of the paper's robustness claims.","major_comments":[{"comment":"The DRO guarantee in Eq. (1) controls only deviations of Q from the reference P̂_{y|x,ey}. Theorem 2.2 and Corollary 2.3 quantify the difference between empirical and population robust risk for a fixed center, but they say nothing about what happens when the true posterior P* lies outside the ε-Wasserstein ball. In Section 3.3, P̂ is built from warm-up model predictions and frequency-counted transition estimates on the smallest-loss subset; Figures 6–7 show that the warm-up model can overfit strongly at high noise, so D*_0 may contain mislabeled examples, and τ̂_j(ey) may be biased. For the small ε values used in the experiments (e.g., 0.001–0.06 for CIFAR-10), the paper provides no evidence that P* ∈ Γ_ε(P̂). Without such a coverage check, the worst-case risk in Eq. (1) is not an upper bound for the actual risk, and the theoretical claim of robustness under misspecified true-label posteriors is not supported. I recommend adding controlled experiments that measure or estimate the distance between P̂ and the true posterior, or a sensitivity analysis showing how accuracy degrades as the center is perturbed.","section":"§2.1, Eq. (1); §3.3"},{"comment":"The hard-threshold likelihood-ratio form in Remark 3.1 is derived specifically for a concave function T. For a convex T, Theorem 3.1 instead gives intermediate optimal values t*_0 and t*_1 in the middle probability regime, not a hard 0/1 pseudo-label. The experiments use the cross-entropy loss, i.e., T(t) = -log t, which is convex (Remark 3.5), yet Algorithm 1 line 6 and the surrounding text apply the hard threshold C from Remark 3.1 without stating which case of Theorem 3.1 is being used or deriving a convex analogue. This leaves a gap between the analytic per-instance solution and the deployed pseudo-labeling rule. If the hard threshold is intended as a heuristic rather than a direct consequence of Theorem 3.1, the paper should say so explicitly and justify it empirically.","section":"§3.1, Theorem 3.1 and Remark 3.1; §3.3, Algorithm 1; §4"},{"comment":"The adaptive update γ_t = γ0,t - (1/λ){ε^p - E_{P*_{t,\\iota}} c^p(y',Y)} is the unconstrained minimizer of a quadratic surrogate, not the optimal Lagrange multiplier of the empirical robust risk in Eq. (5). The paper states that this is a 'one-step update method' and 'a principled balance', but no convergence or optimality guarantee is provided for this update, and it is not shown that repeated updates approach the γ*_ψ characterized in Theorem 3.3. Since the robust loss and the resulting classifier update depend on γ, this heuristic is load-bearing. Please either analyze the surrogate update (e.g., under what conditions it preserves the minimization direction of Theorem 3.3) or clearly label it as an approximation and report sensitivity to λ.","section":"§3.3, Algorithm 1 line 13"}],"minor_comments":[{"comment":"The first displayed inequality in Corollary 2.3 reads 'Rϵ(ψ̂_ϵ,n; P) ≤ Rϵ(ψ̂_ϵ,n; P)', with the same quantity on both sides; this is likely a typo and should be corrected (for example, to relate bRϵ and Rϵ).","section":"Corollary 2.3"},{"comment":"The paper says the small-loss subset size m is '1 minus the estimated noise rate', but it does not specify how the noise rate is estimated before the transition-matrix estimation. Please provide the estimation procedure or the chosen m/n values used in the experiments.","section":"§3.3"},{"comment":"The hyperparameters C (pseudo-label threshold), λ (Lagrange multiplier step), and κ (label metric scale) are not reported in the main text or in Appendix B.1. Since these directly control pseudo-label selection and the γ update, please report their values and, if possible, their sensitivity.","section":"§3.3, Algorithm 1; §4"},{"comment":"In the opening of Appendix A.8, the notation 'For i ∈ [K] and j ∈ [K]' should presumably be 'i ∈ [n] and j ∈ [K]', since the data index ranges over the n samples.","section":"Appendix A.8"},{"comment":"There are small typos in the appendix, including 'Algorihtm 2' and 'Lines 11-12 f Algorithm 1'; these should be cleaned up before publication.","section":"Appendix B.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a machine learning journal and the empirical work is substantial. My main concern is not with the LP duality or the closed-form derivations per se, but with the gap between the DRO guarantee (which is conditional on the reference distribution) and the actual construction of that reference from a warm-up model under high noise. This is fixable with additional analysis or experiments, so I do not recommend rejection, but the robustness claim should be either verified or carefully qualified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, it is a serious piece of work: the conditional Wasserstein DRO formulation for noisy-label posterior misspecification is new, the per-sample analytic solutions are derived carefully, and the empirical gains are real—CIFAR-10 IDN-HIGH at 86.62 versus 78.67 for the best baseline is a substantial jump. Second, the central robustness claim is conditional in a way that the paper does not fully check. The worst-case risk in Eq. (1) only bounds deviations from the estimated reference P̂; if the true posterior P* is not inside the ε-ball, the bound says nothing about actual risk. The paper never measures the distance between P̂ and P*, and the small-loss warm-up heuristic that builds P̂ can be contaminated under high noise—their own Figures 6 and 7 show the warm-up model overfitting. That is the load-bearing soft spot, and the limitations section quietly acknowledges it by saying transition estimation is not the focus.\n\nWhat the paper does well: the LP duality is standard but cleanly applied, the closed-form robust risk in Theorem 3.3 is a nice contribution, and the pseudo-label likelihood-ratio rule is a genuine twist. The experiments are broad, include real crowdsourced datasets, report standard errors, and the authors released code. The appendix proofs are thorough and mostly check out on a first pass.\n\nWhere I would push back: the pseudo-label threshold in Remark 3.1 is derived for a concave T, but the experiments use convex cross-entropy; Remark 3.5 clips the input but does not close that gap. The adaptive γ update is a heuristic quadratic surrogate, not a direct consequence of Theorem 3.3. And the hyperparameters C and λ are not reported in the main text, which is an easy fix but still a strike against reproducibility. These are moderate issues, not fatal ones. The paper is honest about limitations, and the central idea is likely defensible if the reference coverage is tested or the claims are softened.\n\nWho is this for: anyone working on label noise or crowdsourcing. It introduces a framework that will likely be built on even if the current version has loose ends. It deserves a serious referee, not a desk reject. For a journal, I would recommend major revision: add an experiment measuring or bounding the distance between P̂ and P*, report all hyperparameters, and either reconcile the concave/convex mismatch or clearly relegate the threshold rule to a heuristic.","headline":"Genuinely new CDRO formulation for noisy-label posteriors with strong empirical gains, but the robustness guarantee rests on an unverified reference posterior and the theory-practice link has gaps.","tokens_in":52821,"tokens_out":2786,"would_cite":true,"duration_ms":29398,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that noisy-label learning can be formulated as conditional distributionally robust optimization, solved in closed form per data point, and that the resulting pseudo-labeling algorithm outperforms competing methods on…","keywords":["noisy labels","crowdsourcing","conditional distributionally robust optimization","Wasserstein ambiguity set","pseudo-labeling","true label posterior","robust classification","label noise"],"falsifier":"On a dataset whose clean labels are known, deliberately corrupt the noise-transition estimates (for instance by warming up on heavily memorized noisy labels), then measure the Wasserstein distance between the true posterior and the estimated reference; if accuracy falls below a non-robust baseline whenever that distance exceeds $\\epsilon$, the claim that the ambiguity ball covers realistic misspecification is refuted.","tokens_in":51716,"feed_emoji":"🏷️","tokens_out":9872,"duration_ms":77592,"temperature":0.7,"pith_summary":"The paper sets out to show that learning from crowdsourced noisy labels can be made robust without needing an accurate estimate of the true label posterior. It reformulates the problem as conditional distributionally robust optimization: minimize the worst-case expected loss over a Wasserstein ball of radius $\\epsilon$ around the estimated posterior $P_{y|x,\\tilde y}$ for each instance and annotation vector. The paper derives the dual of this problem, solves the per-data-point robust risk in closed form, and uses that solution as a likelihood-ratio pseudo-labeling rule with an adaptively chosen Lagrange multiplier. If the central claim is correct, the method hedges against misspecification of the posterior rather than requiring the estimate to be right, and the reported results show consistent accuracy gains over competing approaches on synthetic and real crowdsourced benchmarks.","feed_headline":"A closed-form robust risk wins on noisy crowdsourced labels","feed_subtitle":"It hedges against posterior misspecification and beats 14 competing methods on synthetic and real datasets.","key_machinery":"The object that carries the argument is the conditional ambiguity set $\\Gamma_\\epsilon(P_{y|x,\\tilde y})$, a Wasserstein ball of radius $\\epsilon$ around the estimated true-label posterior for each $(x,\\tilde y)$. Its dual (Proposition 2.1) turns the worst-case expectation into an infimum over a scalar Lagrange multiplier $\\gamma$, which is what makes the robust risk solvable. Theorem 3.3 then identifies the optimal $\\gamma$ with the sorted gap $\\alpha_{(s^\\star)}/\\kappa^p$, so the algorithm can update its robustness level in a single step instead of solving a nested optimization at every epoch.","core_discovery":"The central discovery is that the conditional distributionally robust risk has a tractable dual whose per-point solution is a likelihood-ratio test. In the binary case with a concave loss transform $T$, the optimal output for an instance is class $0$ or $1$ when the corresponding posterior probability exceeds $\\varrho(\\epsilon)+\\varpi_1$, and the abstaining uniform value $1/2$ otherwise; a convex $T$ introduces an additional soft region solved by a one-dimensional equation. For multi-class problems, the optimal action puts mass $1/k_0$ on the $k_0$ largest posterior probabilities and zero elsewhere. Because the optimal Lagrange multiplier is available in closed form as $\\gamma^\\star_\\psi=\\alpha_{(s^\\star)}/\\kappa^p$, the empirical robust risk reduces to the nominal loss plus a penalty that prevents overconfident predictions. The paper builds the AdaptCDRP algorithm on these facts, using confidently pseudo-labeled points to form a pseudo-empirical distribution as the reference for CDRO.","pith_inferences":["Editorial inference: the threshold $C_1$ in the pseudo-label rule is an explicit precision-recall dial; tuning it per class or per annotator could trade coverage against pseudo-label accuracy without changing the CDRO derivation.","Editorial inference: because the closed-form robust risk adds a penalty on overconfident points, the method is likely compatible with logit-normalization or confidence-penalty tricks, which may further stabilize very high noise rates.","Editorial inference: the algorithm's reference distribution is modular, so substituting a better transition estimator should improve accuracy; the paper reports partial evidence for this, and a systematic comparison across estimators would be a direct test."],"forward_implications":["If the central claim holds, practitioners can train on noisy crowdsourced labels without first obtaining high-quality posterior estimates, because the CDRO objective hedges against errors in those estimates.","The likelihood-ratio pseudo-label rule only trains on instances where the top posterior dominates the second by a threshold, so the method should remain stable in high-noise regimes where argmax pseudo-labeling degrades.","The closed-form optimal Lagrange multiplier provides a one-step, principled update for the robustness level, avoiding expensive bi-level optimization during training.","The generalization bounds imply the empirical robust risk tracks its population counterpart at rate $O(n^{-1/2})$, so the reported gains are not explained by overfitting the training set."],"supporting_citations":[{"why":"Supplies the conditional distributionally robust optimization framework that the paper adopts.","marker":"[16]"},{"why":"Provides the Wasserstein distance definition and the ambiguity-set construction used in the formulation.","marker":"[17]"},{"why":"Supplies the strong-duality result for Wasserstein distributionally robust optimization that underpins Proposition 2.1.","marker":"[18]"},{"why":"The baseline method whose posterior-weighted loss the paper's analytical solution encompasses as a special case, and a key comparison in the experiments.","marker":"[9]"},{"why":"Provides the instance-dependent noise transition and posterior update used to build the reference distribution, and sources several baseline results in the tables.","marker":"[10]"},{"why":"Supplies the algorithm used to generate synthetic instance-dependent noisy labels for the CIFAR experiments.","marker":"[29]"},{"why":"Provides the real-world human-annotated noisy label datasets (CIFAR-10N and CIFAR-100N) used in evaluation.","marker":"[22]"}],"fun_headline_variants":["Closed-form CDRO tames noisy crowdsourced labels","Robust pseudo-labels via a likelihood-ratio test","A closed-form worst-case risk for noisy labels","CDRO offers a tractable hedge against label noise"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that for every instance the true conditional distribution of the label lies inside the epsilon-Wasserstein ball around the estimated posterior; if the warm-up phase has memorized the noisy labels, the transition estimates are biased and may put the true posterior outside any reasonable ball.","fun_headline_variants_meta":{"raw":{"variants":["Closed-form CDRO tames noisy crowdsourced labels","Robust pseudo-labels via a likelihood-ratio test","A closed-form worst-case risk for noisy labels","CDRO offers a tractable hedge against label noise"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000304,"raw_usage":{"total_tokens":1769,"prompt_tokens":990,"completion_tokens":779,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":716}},"tokens_in":606,"tokens_out":779,"duration_ms":8015,"temperature":1.0,"reasoning_tokens":716,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:30:21.822833+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a dataset whose clean labels are known, deliberately corrupt the noise-transition estimates (for instance by warming up on heavily memorized noisy labels), then measure the Wasserstein distance between the true posterior and the estimated reference; if accuracy falls below a non-robust baseline whenever that distance exceeds $\\epsilon$, the claim that the ambiguity ball covers realistic misspecification is refuted.","supporting_citations":[{"cited_title":"Conditional distributionally robust functionals","cited_arxiv_id":null,"evidence_quote":"Supplies the conditional distributionally robust optimization framework that the paper adopts."},{"cited_title":"Quantifying distributional model risk via optimal transport","cited_arxiv_id":null,"evidence_quote":"Provides the Wasserstein distance definition and the ambiguity-set construction used in the formulation."},{"cited_title":"Distributionally robust stochastic optimization with Wasserstein distance","cited_arxiv_id":null,"evidence_quote":"Supplies the strong-duality result for Wasserstein distributionally robust optimization that underpins Proposition 2.1."},{"cited_title":"Label correction of crowdsourced noisy annotations with an instance-dependent noise transition model","cited_arxiv_id":null,"evidence_quote":"Provides the instance-dependent noise transition and posterior update used to build the reference distribution, and sources several baseline results in the tables."},{"cited_title":"Part-dependent label noise: Towards instance-dependent label noise","cited_arxiv_id":null,"evidence_quote":"Supplies the algorithm used to generate synthetic instance-dependent noisy labels for the CIFAR experiments."}],"review_version":1}