{"id":"bb858260-ca26-4ac2-81db-731e818c6663","arxiv_id":"2501.13389","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"AEON jointly estimates closed-set and open-set label noise rates via learnable parameters and uses them to adaptively weight samples in one-stage training, achieving state-of-the-art accuracy on several noisy-label benchmarks.","lead":"AEON is a new training method that estimates both in-distribution and out-of-distribution label noise rates while learning image classifiers, and it reports higher accuracy than prior methods on noisy-label benchmarks. It also introduces a new benchmark that simulates realistic instance-dependent ID and OOD noise.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Learned scalar 'noise rates' are not identifiable as measurements; a clean-label control is needed before the estimation claim can stand.","rationale":"I read the paper in good faith: AEON's stated contribution is a one-stage method that simultaneously estimates and adapts to instance-dependent ID and OOD label noise, with a new benchmark and strong empirical results. The central claim, however, has a load-bearing identifiability weakness. The noise rates in Eqs. (1) and (5) are single scalar outputs of sigmoids, and their only learning signal is the total loss in Eq. (9). The thresholds in Eqs. (4) and (8) use these scalars to define which samples are treated as clean or noisy, creating a circular dependence: the same objective that trains the classifier also determines what the 'noise rate estimate' means. Without an independent measurement of the noise fraction, there is no reason to expect γ to converge to the true rates, and the clean-label control would reveal this directly. The reader's weakest assumption focused on the Gaussian inverse-CDF thresholds; I agree that this is a real calibration risk, but it is downstream of the deeper issue that the learned scalars are not identifiable as rates. I also note the paper's own limitation statement that temperature parameters require manual tuning, plus the statement in Section 5.2.3 that hyperparameters were selected on a held-out portion of the training set, which strains the 'no clean validation data' claim. These are addressable concerns. The empirical accuracy results, the benchmark, and the ablation studies are valuable and suggest the method works as a robust learning procedure even if the estimation vocabulary is overstated. For that reason I do not move the verdict from the reader's CONDITIONAL: the paper should be accepted only if the authors either provide clean-label and controlled-rate experiments showing γ tracks true rates, or reframe the central claim as adaptive loss weighting rather than noise rate estimation. The existing reproducibility weaknesses (no released code or benchmark) further support requiring those experiments before final acceptance.","tokens_in":18885,"tokens_out":7053,"duration_ms":64455,"concrete_test":"Run AEON under the paper's exact protocol on a clean version of ciFAIR-100/CIFAR-100 with no injected noise (rood = 0, rid = 0) and record the final (ηid, ηood). If either estimate is not essentially 0 (e.g., > 0.05), the learned scalars do not measure noise rates. As a second arm, use synthetic data with known rates, e.g., (0.2, 0), (0, 0.2), and (0.2, 0.4), plot estimated versus injected rates across seeds and initial γ values, and check whether different γ initializations converge to the same final rates at similar training loss. If the mapping is flat or non-monotone, or if final estimates depend strongly on initialization, γ acts as a free loss weight rather than an identifiable rate estimator.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that AEON jointly estimates true instance-dependent ID/OOD noise rates, but in Section 3.2 the quantities called rates are two learnable scalars, ηood = σ(γood; Tood) and ηid = σ(γid; Tid). They are updated only through the gradient of Eq. (9), whose objective is classification, energy, and contrastive loss, with no term that rewards agreement with any known noise rate. The thresholds τood and τid in Eqs. (4) and (8) then partition the batch according to the estimated η values, but nothing ties those η values to the actual fraction of corrupted samples. The optimizer can therefore choose γ values that reduce training loss while deviating arbitrarily from the true rates; even on a perfectly clean dataset, treating some samples as noisy might lower the loss, so nonzero final η estimates would not indicate real noise. The estimates near injected rates in Tables 1 and 2 are not evidence of measurement unless a control demonstrates that γ tracks true rates. This identifiability problem is more fundamental than the Gaussian quantile concern and would persist even if the batch scores were exactly Gaussian.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AEON, a one-stage method for learning image classifiers under mixed in-distribution (ID) and out-of-distribution (OOD) instance-dependent label noise. AEON introduces two learnable scalars (gamma_id, gamma_ood) whose sigmoid transforms are called estimated noise rates (eta_hat_id, eta_hat_ood). These rates set adaptive thresholds for energy scores and losses via the inverse CDF of a Gaussian with empirical mean/variance, producing soft sample weights. The training objective combines supervised loss on likely-clean ID samples, unsupervised loss on likely-noisy ID samples, an energy-based OOD loss, and contrastive losses. The paper also proposes a new benchmark (ciFAIR-100 with Places-IDN OOD noise and part-dependent ID noise) and reports state-of-the-art accuracy on this benchmark, on CIFAR-100 with synthetic mixed noise, and on real-world datasets (Clothing1M, mini-WebVision, WebFG-496) with about 1.2x training overhead over PLS.","tokens_in":19233,"tokens_out":3565,"duration_ms":42528,"significance":"If the noise-rate-estimation claim were validated, AEON would be a meaningful step: it tackles ID and OOD noise jointly in one stage, requires no clean validation set, and is computationally efficient. The new benchmark and the consistent accuracy gains are useful empirical contributions. However, the central claim of 'estimating instance-dependent ID and OOD noise rates' is not supported as stated: the reported eta_hat values are fitted parameters that are not shown to be identifiable from the true noise rates. The paper also provides no proof or empirical demonstration that the learned gamma converges to the true rates, and the self-referential construction of thresholds from the same fitted rates weakens the interpretation. The accuracy results are creditable, but the estimation claim needs to be either rigorously validated (e.g., with a clean-label control) or substantially softened.","major_comments":[{"comment":"The estimated noise rates eta_hat_ood and eta_hat_id are learnable scalars that are optimized only through the training objective in Eq. (9), which contains no term that rewards agreement with the true noise rates. The adaptive thresholds in Eqs. (4) and (8) are then derived from these fitted values, so the entire procedure is self-referential: the partition of a batch into clean/noisy is defined by the estimated rates, and the loss then depends on that partition. There is therefore no mechanism that ties the learned gamma to the actual fraction of corrupted samples; on a perfectly clean dataset, nonzero eta_hat could reduce the loss by downweighting hard examples. The near-matching estimates in Tables 1 and 2 are not evidence of measurement without a control experiment. Please add an experiment on clean-label data (e.g., CIFAR-100 with no injected noise) showing that eta_hat converges to approximately 0, or add an identifiable objective (e.g., a clean validation set or a statistical test on the Gaussian mixture) that forces gamma to track the true rates. Without such a test, the paper should not claim to estimate noise rates.","section":"Section 3.2, Eqs. (1), (5), (9)"},{"comment":"The thresholds tau_ood and tau_id are set to the inverse cumulative distribution function of a Gaussian with the empirical mean and variance of the batch energy scores or loss values. This assumes that the batch energy and loss distributions are approximately Gaussian, but no empirical evidence or theoretical justification is provided. Under instance-dependent noise, loss distributions are often skewed or multimodal (e.g., a mixture of clean and noisy samples), and energy scores for OOD samples may not be Gaussian in practice. If the distributions deviate from Gaussian, the quantile will not correspond to the top eta_hat fraction of noisy samples, causing miscalibrated sample weights and degraded accuracy. Please provide empirical validation of the Gaussian assumption (e.g., histograms or QQ-plots at several training epochs) or replace the inverse-CDF with a nonparametric quantile estimator that does not rely on distributional shape.","section":"Section 3.2, Eqs. (4) and (8)"},{"comment":"The sentence \"These hyperparameters are selected based on performance on a held-out portion of the training set\" conflicts with the paper's claim of not requiring clean validation data. A held-out portion of the noisy training set is not a clean set, and selecting hyperparameters on it may still leak noisy-label information into model selection. More importantly, the reported noise rate estimates in Tables 1 and 2 are compared with the injected rates, but the hyperparameters (T, beta, margins, etc.) were presumably tuned on the same benchmark, so the claimed agreement may reflect tuning rather than estimation. Please clarify how the held-out selection was performed, whether the same protocol was applied to all baselines, and how this affects the interpretation of the reported eta_hat values.","section":"Section 5.2.3"},{"comment":"The paper uses Expected Calibration Error (ECE) to claim that the dual-stream detection mechanism and noise rate estimation are reliable (\"ECE evaluates the reliability of our dual-stream detection mechanism in Eqs. (3) and (7) and noise rate estimation in Eqs. (1) and (5)\"). ECE measures the calibration of the classifier's predicted probabilities on the test set; it does not measure the accuracy of eta_hat as an estimate of the true noise rate. The reported ECE improvements therefore do not validate the noise-rate-estimation claim. Please either remove this interpretation or add a direct evaluation of eta_hat against the injected rates over training (e.g., a plot analogous to Fig. 2 for multiple random seeds and noise configurations, with standard deviations).","section":"Section 5.3, Tables 1 and 2"}],"minor_comments":[{"comment":"The notation \"Phi^{-1}(1 - eta_hat; mu, sigma^2)\" is nonstandard: the inverse CDF of a Gaussian is typically parameterized by mean and standard deviation, not variance. Please clarify or correct the notation.","section":"Section 3.2, Eqs. (4) and (8)"},{"comment":"The sentence \"Following our algorithm to implement the benchmark explained in Section 4\" is circular in self-reference; it should refer to Section 4's construction method as \"the algorithm in Section 4\".","section":"Section 5.1.1"},{"comment":"The margin hyperparameter in Eq. (13) is written as \"mood\", while the text later refers to \"m_out\" and \"m_in\". Please unify the notation for clarity.","section":"Section 3.3, Eq. (13)"},{"comment":"The final loss in Algorithm 1 adds Lcont,sup_i and Lcont,uns_i separately, but Eq. (14) already defines Lcont_i as their sum and Eq. (9) includes Lcont_i. Please double-check that the loss is not double-counted.","section":"Algorithm 1, line 27"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the identifiability of the learned noise rates. The authors should be asked to provide a clean-label control or otherwise demonstrate that the gamma parameters track actual noise rates, and to soften the estimation claim if such a demonstration is not possible. The Gaussian-threshold assumption is also load-bearing and needs empirical justification. The paper is otherwise well-written and the empirical accuracy results are valuable; the benchmark contribution alone may justify publication if the estimation claim is reframed as 'adaptive sample weighting' rather than 'noise rate estimation'."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — AEON is worth a serious referee, but the central estimation claim should not be taken at face value. What is genuinely new: a one-stage method that tries to handle closed-set and open-set noise with a single network, plus a new benchmark that replaces ciFAIR images with semantically similar Places365 images, which is more realistic than random OOD substitution. The experiments are extensive and the accuracy gains on their benchmark are large (about 9 points over PLS at 40/40 noise), with consistent wins on Clothing1M and WebFG. If the accuracy results replicate, that is a useful contribution on its own.\n\nThe soft spot — and it is load-bearing for the phrase \"noise rate estimation\" — is that the quantities called rates are just sigmoid transforms of two learned scalars, trained by a classification/energy/contrastive objective. Nothing in the loss rewards agreement with actual noise rates. The estimated values near the injected rates in Tables 1 and 2 are therefore not evidence of measurement; on a fully clean dataset the optimizer could still prefer nonzero rates if that reduces training loss. A clean-label control (train the same pipeline on a noise-free dataset and show the rates collapse toward zero) is the minimum needed before the estimation claim stands. Without it, the claims should be phrased as \"learned mixing weights,\" not estimated rates.\n\nTwo smaller issues. First, the thresholds in Eqs. 4 and 8 are inverse-Gaussian quantiles, but there is no evidence energy scores or losses are approximately Gaussian. This is a calibration risk; the paper's sensitivity analysis covers temperatures but not distributional fit. Second, Section 5.2.3 says hyperparameters are selected on a held-out portion of the training set, which undercuts the \"no clean validation data\" claim. For Clothing1M they say the validation set was not considered; those two statements sit in tension.\n\nAlso worth noting: Table 1 baselines are reproduced locally, Table 2 are taken from other papers, so the two tables are not perfectly comparable. The paper acknowledges MDM beats AEON in one row of Table 2, then uses that to argue the new benchmark is more reliable; that reasoning is weak when differences are under a point. A concrete comparison of baselines run under identical conditions would be stronger.\n\nThe paper ships no code or benchmark data, which matters for a benchmark paper. That said, the method is coherent, the ablation is informative, and the limitations are at least acknowledged (manual temperature tuning, no theory). I would send this to review, with the clear ask that the estimation claim be reframed or supported by a clean-label control. If I were in the noisy-label area, I would read it but would not cite the noise-rate results until that control exists.","headline":"A useful empirical method and benchmark, but the 'noise rate estimation' claim is fitted scalars until a clean-label control shows the rates track actual corruption.","tokens_in":19620,"tokens_out":2767,"would_cite":false,"duration_ms":704181,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AEON claims to be the first method that jointly estimates instance-dependent in-distribution and out-of-distribution label noise rates during training, without clean validation data, and it reports accuracy gains over existing methods on…","keywords":["noisy labels","instance-dependent label noise","out-of-distribution detection","energy-based scoring","noise rate estimation","semi-supervised learning","contrastive learning","ciFAIR-100 benchmark"],"falsifier":"Record the empirical distributions of energy scores and losses on a corrupted batch with known ID and OOD noise rates and compare the Gaussian quantile $\\Phi^{-1}(1-\\hat\\eta;\\mu,\\sigma^2)$ with the empirical quantile at that rate; if they disagree by more than a small fraction of samples on the datasets AEON evaluates, the threshold mechanism is miscalibrated. The direct experiment is to replace Eqs. (4) and (8) with empirical batch quantiles and check whether accuracy changes.","tokens_in":18687,"feed_emoji":"🏷️","tokens_out":10640,"duration_ms":83680,"temperature":0.7,"pith_summary":"The paper sets out to show that jointly estimating the two kinds of label corruption seen in real image datasets—closed-set noise, where the true class is inside the training label set, and open-set noise, where it is outside—makes robust training work better than simply detecting and re-weighting noisy samples. Its method, AEON, learns two scalar noise-rate parameters during training and converts them, through Gaussian-quantile thresholds, into soft per-sample weights that route each sample among supervised, unsupervised, and contrastive losses. The authors claim this is the first method to estimate instance-dependent in-distribution and out-of-distribution noise rates together without a clean validation set, and they report accuracy gains on CIFAR-100, ciFAIR-100, Clothing1M, mini-WebVision, and WebFG-496 at about 1.2 times the runtime of the most efficient comparison method. They also construct a benchmark that injects instance-dependent noise of both types into ciFAIR-100, and report that method rankings on this benchmark track real-world dataset rankings more closely than older synthetic benchmarks do. The paper acknowledges that its temperature parameters still require manual tuning.","feed_headline":"AEON estimates both ID and OOD label noise in one pass","feed_subtitle":"A dual-stream soft mask and adaptive quantile thresholds keep training robust to closed- and open-set corruption.","key_machinery":"The load-bearing object is a dual-stream soft mask with Gaussian-quantile adaptive thresholds. The OOD stream uses an energy score $E_i = -T_E \\log \\sum_{k=1}^C e^{f_\\theta^{(k)}(\\hat{x}_i)/T_E}$; the ID stream uses the per-sample classification loss $L_i^s = -\\hat{y}_i^\\top \\log(\\mathrm{softmax}(f_\\theta(\\hat{x}_i)))$. The thresholds of Eqs. (4) and (8), $\\tau = \\Phi^{-1}(1-\\hat\\eta;\\mu,\\sigma^2)$, convert the current estimated rates into cut positions under the assumption that each score distribution is Gaussian, and sigmoid functions with widths $\\beta_{\\mathrm{ood}}$ and $\\beta_{\\mathrm{id}}$ produce the weights $w_i^{\\mathrm{ood}}$ and $w_i^{\\mathrm{id}}$. These weights are the only connection between the estimated rates and the loss, so the method's behavior hinges on them.","core_discovery":"AEON's central claim is that explicit, jointly estimated noise rates—not just per-sample detection—are what make learning robust to mixed closed- and open-set label noise. The method carries two learnable scalars $\\gamma_{\\mathrm{id}}$ and $\\gamma_{\\mathrm{ood}}$, mapped through sigmoids to the rates $\\hat{\\eta}_{\\mathrm{id}}$ and $\\hat{\\eta}_{\\mathrm{ood}}$. Each batch it computes energy scores $E_i$ and supervised losses $L_i^s$, fits their empirical mean and variance, and sets adaptive thresholds $\\tau_{\\mathrm{ood}} = \\Phi^{-1}(1-\\hat{\\eta}_{\\mathrm{ood}}; \\mu_E, \\sigma_E^2)$ and $\\tau_{\\mathrm{id}} = \\Phi^{-1}(1-\\hat{\\eta}_{\\mathrm{id}}; \\mu_L, \\sigma_L^2)$. Sigmoid weights built from these thresholds route each sample among a supervised loss, a pseudo-label unsupervised loss, and a contrastive loss, so the estimated rates directly control the training objective. The paper reports that this one-stage estimator reaches the best accuracies it compares against on CIFAR-100, ciFAIR-100, Clothing1M, mini-WebVision, and WebFG-496, with estimated rates near the injected or previously reported rates.","pith_inferences":["Untested by the paper: replacing the inverse-CDF thresholds in Eqs. (4) and (8) with empirical batch quantiles would separate the benefit of explicit rate estimation from the cost of the Gaussian assumption.","Because the rates are learned end-to-end, AEON could in principle track noise rates that drift during training, such as as the model memorizes; the paper reports only converged values, so this remains an open extension.","The paper's own limitation note says temperature parameters are manually tuned; automating them is future work, so transferring AEON to new datasets may require per-dataset hyperparameter adjustment.","The similarity between the new benchmark's rankings and real-world rankings suggests a possible low-cost screening protocol for noisy-label methods, though the paper does not formalize one."],"forward_implications":["Robust training can be done in one stage: mixing supervised, pseudo-label, and contrastive objectives with learned weights removes the need for a separate clean-versus-noisy assignment stage.","The reported gains grow with noise severity: up to roughly 9 accuracy points on the proposed benchmark at 40% closed-set plus 40% open-set noise, and larger gains at high noise rates on synthetic CIFAR-100.","The roughly 1.2x runtime overhead over PLS makes joint noise-rate estimation cheap enough for large-scale datasets such as the 1M-image Clothing1M set.","Estimated noise rates are informative: they stay near injected rates in synthetic settings and near previously measured rates on real-world data, which supports using them to audit a dataset's corruption level.","The new benchmark's rankings match real-world web-noise rankings, giving the community a cheaper testbed for screening noisy-label methods."],"supporting_citations":[{"why":"PLS is the efficiency and weighting baseline that AEON compares against, claiming 1.2x overhead and better accuracy.","marker":"[1]"},{"why":"The authors' prior graphical-model method estimates only ID noise; AEON positions itself as its joint ID/OOD extension.","marker":"[20]"},{"why":"DivideMix supplies the semi-supervised pseudo-label machinery that AEON adapts for ID noisy samples.","marker":"[28]"},{"why":"Part-dependent label noise provides the instance-dependent closed-set corruption used in the benchmark and the evaluation setup.","marker":"[49]"},{"why":"Places365 is the source of semantically similar open-set images used to create instance-dependent OOD noise.","marker":"[57]"},{"why":"ciFAIR-100 is the base dataset the paper's new ID+OOD benchmark is built on.","marker":"[8]"},{"why":"A contrastive embedding method for in- and out-of-distribution noise that AEON reports against in the comparisons.","marker":"[2]"},{"why":"Clothing1M is the large-scale real-world dataset where AEON reports its highest accuracy and real-world noise-rate estimates.","marker":"[50]"}],"fun_headline_variants":["One-stage AEON estimates both ID and OOD noise rates","Jointly estimate closed- and open-set noise rates in one pass","Adaptive thresholds from estimated noise rates improve robust learning","Instance-dependent noise rates learned in one pass for robust training","AEON: one-stage estimation of mixed label noise for robust learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The clean/noisy cutoffs are computed by treating each batch's energy scores and loss values as approximately Gaussian, so if those score distributions are not Gaussian the thresholds will cut at the wrong quantiles and the sample weights will be miscalibrated.","fun_headline_variants_meta":{"raw":{"variants":["One-stage AEON estimates both ID and OOD noise rates","Jointly estimate closed- and open-set noise rates in one pass","Adaptive thresholds from estimated noise rates improve robust learning","Instance-dependent noise rates learned in one pass for robust training","AEON: one-stage estimation of mixed label noise for robust learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000243,"raw_usage":{"total_tokens":1570,"prompt_tokens":1027,"completion_tokens":543,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":643,"completion_tokens_details":{"reasoning_tokens":458}},"tokens_in":643,"tokens_out":543,"duration_ms":5693,"temperature":1.0,"reasoning_tokens":458,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:59:24.373258+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Record the empirical distributions of energy scores and losses on a corrupted batch with known ID and OOD noise rates and compare the Gaussian quantile $\\Phi^{-1}(1-\\hat\\eta;\\mu,\\sigma^2)$ with the empirical quantile at that rate; if they disagree by more than a small fraction of samples on the datasets AEON evaluates, the threshold mechanism is miscalibrated. The direct experiment is to replace Eqs. (4) and (8) with empirical batch quantiles and check whether accuracy changes.","supporting_citations":[{"cited_title":"Is your noise correction noisy? pls: Robust- ness to label noise with two stage detection","cited_arxiv_id":null,"evidence_quote":"PLS is the efficiency and weighting baseline that AEON compares against, claiming 1.2x overhead and better accuracy."},{"cited_title":"Instance-dependent noisy-label learning with graphical model based noise-rate estimation, 2024","cited_arxiv_id":null,"evidence_quote":"The authors' prior graphical-model method estimates only ID noise; AEON positions itself as its joint ID/OOD extension."},{"cited_title":"DivideMix: Learning with noisy labels as semi-supervised learning","cited_arxiv_id":null,"evidence_quote":"DivideMix supplies the semi-supervised pseudo-label machinery that AEON adapts for ID noisy samples."},{"cited_title":"Part-dependent label noise: Towards instance-dependent label noise","cited_arxiv_id":null,"evidence_quote":"Part-dependent label noise provides the instance-dependent closed-set corruption used in the benchmark and the evaluation setup."},{"cited_title":"Places: A 10 million image database for scene recogni- tion","cited_arxiv_id":null,"evidence_quote":"Places365 is the source of semantically similar open-set images used to create instance-dependent OOD noise."},{"cited_title":"Do we train on test data? purging cifar of near-duplicates","cited_arxiv_id":null,"evidence_quote":"ciFAIR-100 is the base dataset the paper's new ID+OOD benchmark is built on."},{"cited_title":"Embedding con- trastive unsupervised features to cluster in- and out-of-distribution noise in corrupted 15 image datasets","cited_arxiv_id":null,"evidence_quote":"A contrastive embedding method for in- and out-of-distribution noise that AEON reports against in the comparisons."},{"cited_title":"Learning from massive noisy labeled data for image classification","cited_arxiv_id":null,"evidence_quote":"Clothing1M is the large-scale real-world dataset where AEON reports its highest accuracy and real-world noise-rate estimates."}],"review_version":1}