{"id":"9996672d-ea4f-469a-9703-66484c3fe5aa","arxiv_id":"2505.15195","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"For binary classification with noisy labels, the paper derives the Bayes-optimal function for combining a model's current predictions with the given labels during retraining, and shows a fitted version improves linear probing at high label noise.","lead":"This paper finds the optimal way to merge an AI model's own predictions with noisy training labels when retraining it for binary classification. In theory and on two image datasets, the resulting retraining rule beats earlier heuristics, especially when labels are very noisy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.1's state-evolution recursion (7) is internally inconsistent: with g_t = y_hat it fails the exact identity theta_{t+1}=theta_t, and the same factor-alpha error propagates into Eq. (11) but not into (12)/(14).","rationale":"The reader identified the validity of AMP state evolution as the weakest assumption. My check sharpens this: the issue is not only that the proof of Theorem 3.1 is omitted, but that the state-evolution recursion as displayed is internally inconsistent with the exact AMP dynamics for a simple admissible aggregator. This is a concrete, checkable algebraic error rather than a mere gap in exposition. The rest of the paper, including Eqs. (12), (14), (15), and the synthetic experiments, appears to use the corrected factor 1/sqrt(alpha), so the mathematical core may be salvageable after a revision. However, because the central optimality result rests on the state-evolution theorem, the manuscript in its current form is not reliable as written. The verdict remains CONDITIONAL: the authors should correct the factor in Eq. (7) and re-verify the subsequent derivations, and the simple g_t = y_hat test provides a minimal verification of the corrected recursion.","tokens_in":1324,"tokens_out":1105,"duration_ms":311235,"concrete_test":"Run the GMM AMP updates (4)-(5) with g_t(y, y_hat) = y_hat for all t, e.g., n = 10^4, d = 8*10^3, gamma = 1.5, p = 0.3. Since C_t = 0, the iterates satisfy theta_{t+1}=theta_t, so the empirical coefficient d^{-1} sum_j M_j theta_{t,j} / (d^{-1} sum_j M_j^2) must be approximately gamma(1-2p)/sqrt(alpha) for every t. Compare this measured coefficient with the two candidate recursions: Eq. (7) predicts gamma sqrt(alpha)(1-2p) for t >= 2, while the corrected m_{t+1} = (gamma/sqrt(alpha)) E[Y g_t] predicts the observed constant value. The measured value will settle which factor is correct.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The state-evolution recursion stated in Theorem 3.1, Eq. (7), is not consistent with the AMP update rule and is contradicted by the paper's own later equations. Take the admissible Lipschitz aggregator g_t(y, y_hat) = y_hat for all t. Then C_t=0 and Eqs. (4)-(5) give theta_{t+1}=theta_t and y_{t+1}=y_t, so the state-evolution mean must remain m_1 = gamma(1-2p)/sqrt(alpha). Substituting g_t = y_hat into Eq. (7) at t=1, however, gives m_2 = gamma sqrt(alpha)(1-2p), which differs from m_1 by a factor alpha. The same factor error appears in Eq. (11), where m_{t+1} = gamma sqrt(alpha) E[(g_t^*)^2] and sigma_{t+1}^2 = E[(g_t^*)^2] imply eta_{t+1}^2 = gamma^2 alpha E[(g_t^*)^2], yet the next line and Eq. (14) assert gamma^2/alpha E[(g_t^*)^2]. The correct relation, consistent with Eq. (12), is m_{t+1} = (gamma/sqrt(alpha)) E[Y g_t], and this corrected form is apparently what the optimality derivation and simulations use. As printed, the foundational state-evolution theorem is not self-consistent; the optimal-aggregator claim depends on a recursion that the paper has mis-stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies iterative retraining of a binary classifier when the observed labels are noisy. It proposes an AMP-style iterative update in which, at each round, the current soft predictions and the noisy labels are combined by an aggregator function, and it derives the Bayes-optimal aggregator for two data models: a Gaussian mixture model (GMM) and a generalized linear model (GLM). For the GMM it gives a state-evolution recursion, a closed-form optimal aggregator, and a monotonicity result showing when retraining helps or hurts; for the GLM it gives an analogous state-evolution characterization and an optimal aggregator expressed through an integral. The paper also proposes a practical variant, BayesMix RT, obtained by fitting a bimodal Gaussian to the logits of a pretrained model, and reports experiments on two image classification tasks in the high-label-noise regime.","tokens_in":29190,"tokens_out":12586,"duration_ms":109374,"significance":"If the state-evolution theorems are correct, the paper provides a principled and fairly complete answer to a question that previous retraining literature addressed only heuristically: how to optimally combine a model's own predictions with noisy labels during retraining. The derivation of the optimal aggregator as a Bayes posterior ratio is clean, checkable, and is a genuine conceptual contribution. The paper also ships synthetic simulations that verify the corrected state evolution, and the proposed BayesMix RT method is a natural, falsifiable practical extension. These strengths make the paper potentially valuable for the AMP and noisy-label retraining communities, provided the recursion inconsistencies described below are resolved.","major_comments":[{"comment":"The state-evolution recursion is stated with an incorrect factor. Taking the admissible aggregator g_t(y, yhat) = yhat gives theta_{t+1} = theta_t and y_{t+1} = y_t, so the state evolution must satisfy m_{t+1} = m_1 = gamma (1-2p)/sqrt(alpha). Substituting g_t = yhat into Eq. (7) at t=1 gives m_2 = gamma sqrt(alpha) (1-2p), which differs by a factor alpha. The correct update is m_{t+1} = (gamma/sqrt(alpha)) E[Y g_t(barm_t Y + barsigma_t G, Yhat)], which is the form implicitly used by Eq. (10), Eq. (12), and Eq. (14). Eq. (11) repeats the gamma sqrt(alpha) error and contradicts Eq. (12), because Eq. (12) implies m_t = (gamma/sqrt(alpha)) sigma_t^2. As printed, Theorem 3.1 is false, and the optimal-aggregator claim in Theorem 3.2 depends on this recursion.","section":"Section 3.2, Eq. (7); Section 3.3, Eqs. (10)-(12) and (14)"},{"comment":"The state-evolution theorems are not proved. Appendix A states that the proof 'follows by adapting techniques from standard AMP analysis ... and so omitted here', and Theorem 4.2 is delegated to the GAMP results of [36]. Because the updates (4)-(5) and (16)-(17) include memory corrections and a data-dependent initialization g_0(., yhat) = yhat, this is not a routine invocation unless a precise external theorem covers exactly this recursion and initialization. This is load-bearing: Theorem 3.2, Theorem 4.3, and the test-error formulas (9) and (23) all rest on these state-evolution results. Please provide a complete proof or a precise statement of the external theorem being applied, and re-check the initialization argument after the recursion factors are corrected.","section":"Appendix A (proof of Theorem 3.1) and Appendix G (Theorem 4.2)"},{"comment":"The GLM state evolution contains an analogous factor-alpha inconsistency. Theorem 4.2 as printed states the limiting empirical distribution of beta_t with sigma_t sqrt(alpha) multiplying the Gaussian variable, while the derivation in Appendix G computes E[(mu_t betabar + sigma_t/sqrt(alpha) G)^2] and uses the sigma_t/sqrt(alpha) form to obtain the expression eta_t gamma / sqrt(eta_t^2 gamma^2 + 1/alpha) in Eq. (23). These two displays differ by a factor alpha. As typeset, the GLM state-evolution theorem is internally inconsistent; please correct either the statement of Theorem 4.2 or the appendix computation and verify the initialization mu_1 = 2 E[Z hhat_p(Z)]/(alpha gamma^2), sigma_1 = sqrt(alpha) against the corrected recursion.","section":"Section 4, Theorem 4.2 vs. Appendix G"}],"minor_comments":[{"comment":"In the experiments, each method uses a different retraining learning rate (eta_1 = 5e-4, 1e-4, or 5e-6 across methods). Since the learning rate can strongly affect linear probing results, please include a matched-learning-rate comparison or an ablation over learning rates so the reported gains of BayesMix RT are not attributable to this difference.","section":"Section 5 and Appendix K"},{"comment":"The text says that 'if the initial model is already good, retraining can actually hurt its performance', but Proposition 3.3(i) only proves that the sequence is non-decreasing when eta_1 <= eta_*. It does not prove monotone decrease for eta_1 above the smallest fixed point. Please add a proof or qualify the claim.","section":"Section 3.3, discussion after Eq. (15) and Proposition 3.3"},{"comment":"The learning-rate symbols eta_0 and eta_1 in Appendix K conflict with the state-evolution variable eta_t used throughout the paper; consider renaming the learning rates to avoid confusion.","section":"Appendix K"}],"recommendation":"major_revision","confidential_remarks":"The factor-alpha inconsistencies in Eqs. (7), (11), and Theorem 4.2 look like typographical errors rather than conceptual flaws, because the surrounding equations (10), (12), (14), and (23) use the corrected form and the simulations in Appendix D match the corrected state evolution. However, since these equations are the foundational theorems on which the optimality claims rest, the manuscript cannot be accepted in its current form. The revision should correct the recursions, supply or precisely cite the missing state-evolution proofs, and rerun the consistency checks. The paper is otherwise a good fit for the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper asks the right question—how to optimally combine noisy labels with model predictions when retraining—and gives a principled AMP-based answer for binary GMM and GLM settings. The optimal aggregator is the posterior-mean (2q−1) type function, which is a clean application of the standard Cauchy-Schwarz optimality argument, and the paper correctly identifies that. The state-evolution analysis is appropriate for this problem, and the synthetic experiments show the SE curve matching. That part is worth taking seriously.\n\nBut there is a load-bearing typo, or worse, in the central recursion. In Eq. (7), the mean update is stated as m_{t+1} = γ√α E[g_t(...) Y]. That cannot be right: taking g_t = y_hat (which should reproduce the initial estimator) gives m_2 = γ√α(1−2p) = α m_1, while the AMP update (4)-(5) with g_t = y_hat gives θ_{t+1}=θ_t and so m should stay m_1 = γ(1−2p)/√α. The same factor appears in Eq. (11), and it is contradicted by the paper’s own Eq. (10), Eq. (12), and Eq. (14), which all use γ/√α. So the intended recursion is clear, but Theorem 3.1, the theorem that everything rests on, is stated with the wrong formula and its proof is omitted, referring to standard AMP techniques. The reader cannot verify the key step from the paper itself.\n\nThe GLM section relies on GAMP results, which is standard practice and acceptable, but here too the proof is mostly a reference. The experiments have a learning-rate confound: Appendix K shows BayesMix RT uses η1=5e−6 while full RT uses 5e−4 and consensus uses 1e−4 on Food-101. That is a 100-fold difference, and it is hard to attribute the 18-point improvement to the aggregator alone. On the other hand, the paper acknowledges the practical variant is heuristic.\n\nThese are real soft spots, but the core idea—optimal aggregation via AMP, with the posterior mean as the aggregator—is sound and the synthetic validation supports it. The factor error is likely a typo, since the rest of the paper uses the correct version, but a theorem statement with a false formula cannot be left as is. This paper deserves a serious referee, but only after the authors fix the recursion, supply the missing proof as an appendix, and control the learning rates.\n\nWho for: people working on label noise, self-training, and AMP theory will want to see this. I'd cite the conceptual result, but not the current arXiv version. Recommendation: send to an editor for peer review, with a note to the authors to correct Eq. (7) and (11) before resubmission.","headline":"A clean optimal-aggregator derivation with a real factor-α error in the stated state evolution; worth refereeing after a careful revision.","tokens_in":29677,"tokens_out":5897,"would_cite":true,"duration_ms":47090,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For binary classification with noisy labels, the paper derives the Bayes-optimal rule for combining a model's predictions with the given labels, and shows it minimizes prediction error across retraining rounds.","keywords":["retraining","label noise","approximate message passing","state evolution","Bayes-optimal aggregator","Gaussian mixture model","generalized linear model","self-distillation"],"falsifier":"Run the AMP updates (4)-(5) on finite-$n$ GMM data with the claimed optimal aggregator and compare the realized test error to the predicted $\\Phi(-\\gamma\\eta_t/\\sqrt{\\eta_t^2+1})$ for a sequence of growing $n$ and fixed $t$: a persistent gap, or a failure of the empirical distribution of entries of $\\theta_t$ to match a Gaussian with variance $m_t^2+\\sigma_t^2$, would show the state-evolution premise fails. A second check is to search numerically over aggregators for one that beats $g_t^*$ in the $m_{t+1}/\\sigma_{t+1}$ ratio at any round, which would contradict Theorem 3.2 directly.","tokens_in":28611,"feed_emoji":"🔁","tokens_out":6914,"duration_ms":58210,"temperature":0.7,"pith_summary":"The paper asks how a binary classifier should retrain when the only labels available are noisy: should it trust its own predictions, the given labels, or some mix? Using approximate message passing (AMP) to model iterative retraining, it derives the Bayes-optimal aggregator that combines the current soft predictions with the noisy labels, for both Gaussian mixture and generalized linear ground-truth models. The aggregator maximizes the signal-to-noise ratio of the next model at every round, and in the GMM case the monotone state-evolution map makes this choice optimal for all future rounds. The paper also gives a practical version for linear probing with cross-entropy loss and reports that it outperforms full and consensus-based retraining under severe label noise.","feed_headline":"Bayes-optimal retraining beats noisy-label heuristics","feed_subtitle":"A closed-form aggregator from approximate message passing cuts test error over repeated rounds, especially under high label noise.","key_machinery":"The machinery is the AMP update (4)-(5), whose Onsager correction terms debias the iterates so their empirical distributions become asymptotically Gaussian, and the companion state-evolution recursion (7) that tracks the means and variances $(m_t, \\sigma_t)$. The decisive object is the posterior probability $q_t = P(Y=1 \\mid \\bar{m}_t Y + \\bar{\\sigma}_t G, \\hat{Y})$; a Cauchy-Schwarz argument shows that any aggregator's signal-to-noise ratio $m_{t+1}/\\sigma_{t+1}$ is bounded by $E[(2q_t-1)^2]$, and this bound is attained exactly by the Bayes-optimal rule $g_t^* = 2q_t-1$. The formulas (13) and (25) are the explicit Bayes-rule evaluations of that posterior mean in the two data models, and the recursions (14) and (26) reduce the whole retraining trajectory to the evolution of the single ratio $\\eta_t = m_t/\\sigma_t$.","core_discovery":"The central claim is that the optimal aggregator at round $t$ is the posterior-mean rule $g_t^*(y,\\hat{y}) = 2/(1+(p/(1-p))^{\\hat{y}} \\exp(-2\\gamma^2 y/(\\alpha(\\eta_t^2+1))) \\pi_-/\\pi_+) - 1$ for the GMM, with an analogous integral-based formula for GLMs (Theorems 3.2 and 4.3). This choice is optimal in the sense of maximizing the ratio $m_{t+1}/\\sigma_{t+1}$ that controls the next model's test error; because the state-evolution map is non-decreasing in the GMM case, the aggregator also minimizes the classification error at every subsequent round. The paper further shows that retraining with this rule can be characterized by a one-dimensional recursion $\\eta_{t+1}^2 = F(\\eta_t^2)$, so the effect of many rounds is deterministic and quantifiable.","pith_inferences":["The optimal aggregator has a readable shrinkage form: it down-weights the noisy label by a factor depending on the model's soft prediction, effectively treating the model output as an extra noisy observation of the true label; the same principle could be ported to nonlinear models by estimating the needed moments from logits.","Because AMP is conjectured optimal among polynomial-time algorithms for many high-dimensional estimation problems, the paper's framework suggests the Bayes-optimal retraining rule may also be computationally optimal for these data models, a claim the paper does not itself make.","A direct testable extension is to estimate the label-flip probability $p$ online and re-fit the bimodal GMM each round; the state-evolution map predicts the gap between BayesMix RT and consensus-based retraining should widen as $p$ approaches $1/2$ and shrink as $p$ decreases, which Table 5 qualitatively supports."],"forward_implications":["With the optimal aggregator, each retraining round is guaranteed (in the asymptotic limit) not to decrease the signal-to-noise ratio when starting below the fixed point, and the one-dimensional map $F$ predicts exactly how test error improves over rounds.","If the initial model is poor ($\\eta_1$ below the fixed point), retraining strictly reduces test error; if the initial model is already very good, Proposition 3.3 shows retraining can actually hurt, matching the paper's observation that the fixed point is a separator.","In the GMM setting, the Bayes-optimal choice at a single round also minimizes test error at all future rounds because $F$ is non-decreasing, so there is no trade-off between short-term and long-term gains.","The practical BayesMix RT rule, which fits a bimodal GMM to logits and applies a similarly derived aggregator, improves on full retraining and consensus-based retraining by a large margin on Food-101 Pho-vs-Ramen at label-flip probability $p=0.45$ after 10 rounds.","Synthetic experiments in the paper show the AMP theory with the optimal aggregator tracks simulation closely, and the optimal map dominates the full-retraining and consensus maps at every $\\eta$ in the examples."],"supporting_citations":[{"why":"Supplies the AMP-style iterative update and Onsager correction that the paper adapts into (4)-(5).","marker":"[13]"},{"why":"Provides the generalized AMP theory that Theorem 4.2's GLM state evolution is built on.","marker":"[36]"},{"why":"Establishes state evolution for general AMP algorithms, the backbone of the GMM result in Theorem 3.1.","marker":"[18]"},{"why":"The unifying AMP tutorial whose asymptotic Gaussianity machinery the paper invokes for its recursions.","marker":"[14]"},{"why":"Supplies AMP results for low-rank and structured matrices used in the proof trick and in the proposed multi-class extension.","marker":"[30]"},{"why":"Defines full retraining and consensus-based retraining, the baselines the optimal aggregator is compared against.","marker":"[9]"},{"why":"Contributes the Stein's lemma simplification that the GLM state-evolution derivation follows.","marker":"[27]"}],"fun_headline_variants":["Bayes-optimal aggregator wins for self-boost retraining","AMP reveals optimal retraining rule for noisy labels","Optimal retraining beats heuristics in high label noise","How to best combine model predictions and noisy labels","Posterior-mean aggregator from AMP cuts test error"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that AMP state evolution correctly describes the empirical distribution of the estimates for the paper's recursions with data-dependent initialization, a result quoted from standard AMP/GAMP theory (with the GMM proof omitted) rather than proved here.","fun_headline_variants_meta":{"raw":{"variants":["Bayes-optimal aggregator wins for self-boost retraining","AMP reveals optimal retraining rule for noisy labels","Optimal retraining beats heuristics in high label noise","How to best combine model predictions and noisy labels","Posterior-mean aggregator from AMP cuts test error"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000181,"raw_usage":{"total_tokens":1301,"prompt_tokens":934,"completion_tokens":367,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":288}},"tokens_in":550,"tokens_out":367,"duration_ms":4142,"temperature":1.0,"reasoning_tokens":288,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:22:27.247974+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the AMP updates (4)-(5) on finite-$n$ GMM data with the claimed optimal aggregator and compare the realized test error to the predicted $\\Phi(-\\gamma\\eta_t/\\sqrt{\\eta_t^2+1})$ for a sequence of growing $n$ and fixed $t$: a persistent gap, or a failure of the empirical distribution of entries of $\\theta_t$ to match a Gaussian with variance $m_t^2+\\sigma_t^2$, would show the state-evolution premise fails. A second check is to search numerically over aggregators for one that beats $g_t^*$ in the $m_{t+1}/\\sigma_{t+1}$ ratio at any round, which would contradict Theorem 3.2 directly.","supporting_citations":[{"cited_title":"L., Maleki, A., and Montanari, A.Message-passing algorithms for compressed sensing.Proceedings of the National Academy of Sciences 106, 45 (2009), 18914–18919","cited_arxiv_id":null,"evidence_quote":"Supplies the AMP-style iterative update and Onsager correction that the paper adapts into (4)-(5)."},{"cited_title":"In2011 IEEE International Symposium on Information Theory Proceedings(2011), IEEE, pp","cited_arxiv_id":null,"evidence_quote":"Provides the generalized AMP theory that Theorem 4.2's GLM state evolution is built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes state evolution for general AMP algorithms, the backbone of the GMM result in Theorem 3.1."},{"cited_title":"Y., Venkataramanan, R., Rush, C., Samworth, R","cited_arxiv_id":null,"evidence_quote":"The unifying AMP tutorial whose asymptotic Gaussianity machinery the paper invokes for its recursions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies AMP results for low-rank and structured matrices used in the proof trick and in the proposed multi-class extension."},{"cited_title":"InInternational Conference on Artificial Intelligence and Statistics(2021), PMLR, pp","cited_arxiv_id":null,"evidence_quote":"Contributes the Stein's lemma simplification that the GLM state-evolution derivation follows."}],"review_version":1}