{"id":"bcd294c5-472c-400c-9d4d-161a8149c7ab","arxiv_id":"2603.01221","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Effective AI debate requires high initial disagreement kept under control, and uncertainty-guided multi-agent reinforcement learning improves both individual accuracy and long-horizon debate stability.","lead":"This paper studies why multi-agent debate among AI models sometimes improves math answers and sometimes fails, by splitting uncertainty into disagreement between models and internal randomness of each model. It then trains the models with reinforcement learning that rewards informative, low-noise debating, and reports accuracy gains over standard baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"UMAD's uncertainty guidance optimizes a different quantity (Eq. 4 token-level NLL) than the Sys-AU it claims to control (§3.2); without a correlation check, the mechanism and gains are unsubstantiated.","rationale":"The reader's weakest_assumption already flagged token-level NLL as a questionable proxy, and I agree that is the soft spot. However, I want to sharpen it: the problem is not only that the proxy is imperfect, but that the proxy used in training (Eq. 4) is a different level of quantity from the Sys-AU defined and measured in Section 3.2. Sys-AU is an answer-distribution entropy over K=16 rollouts; Û_i is a per-response token sequence negative log-likelihood. The paper provides no correlation or calibration evidence that these track each other, and §8 explicitly concedes they can diverge. This is load-bearing because the UMAD algorithm's only uncertainty-specific mechanism is the Eq. (4) advantage shaping; if Û_i is not a faithful surrogate for Sys-AU, then the observed accuracy gains may come from a generic entropy/confidence regularization rather than from the theorized epistemic-gain/aleatoric-cost trade-off. I do not think this forces rejection: the empirical results may still be real, and the proxy could turn out to correlate well. But the current manuscript does not demonstrate that, so the appropriate verdict remains CONDITIONAL. My concern is slightly narrower than the reader's: I do not object to K=16 rollouts as an estimator per se; the more acute issue is the mismatch between what is measured in the analysis and what is optimized in training. Hence agreement_with_reader is 'partial' rather than 'agree'. The proposed test—a correlation/disagreement diagnostic on existing rollouts—would settle the concern cheaply and without re-training, and a variant replacing Eq. (4)'s weight with answer-level entropy could further confirm whether the mechanism is responsible for UMAD's gains.","tokens_in":26288,"tokens_out":5207,"duration_ms":52594,"concrete_test":"On the existing heterogeneous MATH500 evaluation rollouts, compute for each response both (a) Û_i as defined in Eq. (4) and (b) the answer-level entropy contribution to Sys-AU from the K=16 rollouts (or a semantic-entropy cluster version). Report the Spearman rank correlation between (a) and (b), plus the fraction of responses where the Eq. (4) weight would move the advantage in the opposite direction from the answer-level entropy weight. If the rank correlation is below ~0.7 or the disagreement fraction is substantial (>10%), Eq. (4) is not a faithful proxy for the aleatoric cost in §3.2, and UMAD's claimed mechanism is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical narrative is that effective debate keeps Sys-AU low (§3.2) and that UMAD's uncertainty-guided reward implements this trade-off (§5.2, Fig. 3). But the quantity analyzed and the quantity optimized are not the same. In §3.2 and Appendix C.5, Sys-AU is defined as 1/N Σ H(p_i,t), the mean entropy of answer-level predictive distributions estimated from K=16 independent rollouts per agent per turn. In §4.2, Eq. (4) shapes GRPO advantages with Û_i = −(1/|y_i,t|) Σ_j log π_θ(y_i,t,j | ...), a token-level mean negative log-likelihood of a single generated response. These are different objects: token-level NLL conflates response length, tokenization, phrasing, and decoding temperature, and it is not answer-level entropy. A long correct derivation can have high NLL while a short wrong `\\boxed{42}` can have low NLL. The paper itself concedes in §8 that \"token-level NLL are not always perfectly correlated with semantic uncertainty, especially for poorly calibrated models.\" Moreover, training uses G=5 rollouts at temperature 0.8 (Table 2), while Sys-AU is measured with K=16 rollouts at temperature 1.0 (Appendix C.5); no evidence is provided that the proxy transfers across these regimes. Because Eq. (4) is the only uncertainty-specific mechanism distinguishing UMAD from standard IPPO, the reported 5.1% improvement over IPPO could reflect confidence/entropy regularization rather than reduction of the aleatoric cost identified in the decomposition. Figure 3's \"aleatoric uncertainty declines\" could simply be token-NLL shrinkage. This mismatch is load-bearing for the claim that UMAD operationalizes the Bayesian uncertainty decomposition.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies multi-agent debate (MAD) for math reasoning through an uncertainty-decomposition lens. It models the system’s predictive distribution as a mixture of agents’ answer-level distributions and uses the textbook identity H(mixture) = JSD + mean entropy to define system epistemic uncertainty (Sys-EU) and system aleatoric uncertainty (Sys-AU). It claims that, empirically, successful debates reduce Sys-EU while keeping Sys-AU low, and that heterogeneous debates start with larger Sys-EU and therefore have greater epistemic potential. On this basis it introduces UMAD, a multi-agent GRPO variant whose advantages are weighted by token-level negative log-likelihood and whose reward is augmented by an epistemic intrinsic reward that credits an agent for improving peers’ correctness. Experiments compare UMAD with Zero-Shot MAD and IPPO on homogeneous/heterogeneous Qwen pairs across MATH, GSM8K, AMC, and AIME.","tokens_in":26754,"tokens_out":8708,"duration_ms":83011,"significance":"If the empirical claims held, this would be a useful step toward making MAD a stable, learnable mechanism rather than a fragile inference-time heuristic. Proposition 3.2 is correct, though elementary; the more interesting contributions are the UMAD training objective and the attempted empirical link between uncertainty dynamics and debate success. The paper is transparent about prompts, compute, and limitations, and the IPPO ablation is a reasonable control. However, the current manuscript does not establish that the quantity optimized (token-level NLL) is the quantity measured and theorized (answer-level Sys-AU), and the statistical evidence is too weak to support the headline claims. The theoretical Theorem 3.3 is circular as stated and should not be presented as a theorem.","major_comments":[{"comment":"The 'heterogeneous novelty condition' is essentially the claim being proved. The theorem states Gepi(m_hetero) ≥ Gepi(m_homo) for Gepi(m):=I(φ;m|x,c), but the proof only shows, under the stated condition, that I(φ;m_homo,m_hetero|x,c) ≥ I(φ;m_homo,m'_homo|x,c). Because the condition is conditional on m_homo, it does not imply the claimed ordering of unconditional mutual informations. If 'epistemic gain' is instead read as the incremental information after m_homo, then the conclusion is just the assumption restated. I recommend removing the theorem label and presenting the condition as a testable empirical hypothesis.","section":"§3.3, Theorem 3.3"},{"comment":"The training signal Ĥ_i in Eq. (4) is the mean token-level negative log-likelihood of a single generated response, whereas Sys-AU in Eq. (3) is the entropy of an answer-level distribution estimated from K=16 rollouts at temperature 1.0. The paper itself concedes in §8 that 'token-level NLL are not always perfectly correlated with semantic uncertainty.' Training uses G=5 rollouts at temperature 0.8, while the decomposition is computed with K=16 at temperature 1.0. No evidence is provided that the ranking of Ĥ_i correlates with Sys-AU or that the proxy transfers across temperature/group-size regimes. Because Eq. (4) is the only uncertainty-specific mechanism distinguishing UMAD from IPPO, the reported gains could be due to confidence/entropy regularization rather than to controlling the aleatoric cost identified in §3.2. Please add a calibration/correlation analysis and an ablation with a","section":"§4.2, Eq. (4) vs. §3.2, Eq. (3)"},{"comment":"Results are reported without error bars, multiple seeds, or significance tests. Several datasets are very small: AMC23 has 40 problems and AIME24/25 have 30 each. For example, the heterogeneous A1 AIME24 T=5 difference of 10.0 vs 3.3 is 3 vs 1 correct answers, and many AIME rows are 0.0 for A0 across all methods. The claims of 'consistent improvements' and 'largest gains at T=5' are not supportable without an estimate of variability. I recommend reporting mean ± std over at least three seeds, or bootstrapped confidence intervals for the headline T=5 numbers.","section":"§5.2, Table 1"},{"comment":"The success/neutral/fail trajectories are selected post hoc by final accuracy outcome. The observation that 'successful debates can keep Sys-AU low or decreasing' is therefore a selected pattern; it is not established on the full dataset. The figure shows only representative trajectories, with no aggregate statistics such as mean Sys-AU over successful versus failing trajectories, or a correlation between ΔSys-AU and final accuracy. Please report the full distribution or a quantitative regression over all trajectories.","section":"§3.2, Figure 6"},{"comment":"The K sampling protocol is contradictory. Appendix B.1 states that K=16 rollouts are generated at t=0 and K=1 for all t>0, while Algorithm 1 and Appendix C.5 state K=16 at every turn. Since Sys-AU in Eq. (3) is the mean entropy of per-agent answer distributions at each turn, K=1 after the first turn would make Sys-AU uncomputable. The manuscript must clarify which protocol is used for the decomposition results and which is used for training; if B.1 describes training rollouts and C.5 describes evaluation, this should be stated explicitly.","section":"Appendix B.1 vs. Algorithm 1 / Appendix C.5"}],"minor_comments":[{"comment":"Typos: 'develope' should be 'develop'; 'manifesting as a tend for consensus' should be 'tendency'; 'Corret to Both Wrong' should be 'Correct to Both Wrong'.","section":"§1, §3.3, Appendix E.1"},{"comment":"The caption is inconsistent: panel labels repeat 'Success Homogeneous MAD' (f appearing twice), and the figure appears to duplicate Figure 2. Please fix the labels and unify the figure numbering.","section":"Figure 6 caption"},{"comment":"The text references Figure 6 in §3.2 and Figure 2 earlier; it should be clear which figure supports which claim. Consider consolidating or renumbering.","section":"§3.2 vs. Figure 2/6"},{"comment":"The caption says 'The pseudo-code is demonstrated in Table 1'; it should refer to Algorithm 1.","section":"Algorithm 1 caption"},{"comment":"Eq. (5) defines r_eu_{i,t} = η/(N−1) Σ_{j≠i} ΔR_peer, but Algorithm 2 line 22 sets R_total = R_corr + η·ΔR_peer. Clarify whether the average over peers is included in η or is part of the implementation.","section":"§4.2, Eq. (5) and Algorithm 2"},{"comment":"The compute statement says 'rollout group size of G=16' but Table 2 lists Group Size (G)=5; it also mentions models 3B to 7B, while the experiments use 3B and 4B models. Please align these numbers.","section":"Appendix G vs. Table 2"},{"comment":"The law-of-total-entropy decomposition uses a latent variable φ_i, but no posterior over φ_i is ever computed. This is fine as conceptual framing, but the paper should state explicitly that φ_i is a formal device rather than an implemented Bayesian quantity.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"The UMAD training recipe is concrete, reproducible, and the IPPO control is informative, so I would not reject the paper outright. However, the theoretical wrapper (Theorem 3.3) is circular, and the central mechanism claim is undercut by the mismatch between the optimized proxy (token NLL) and the measured quantity (answer-level Sys-AU). The empirical case also needs error bars or significance testing, especially on the small AMC/AIME sets. These issues are fixable, but they are load-bearing for the paper's stated contributions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful thing here is the empirical finding. Training agents with an uncertainty-weighted advantage plus a peer-improvement reward consistently lifts the weaker agent in heterogeneous debate while preserving the stronger one, and the gains grow with debate rounds. Table 1 shows a steady, if modest, improvement over IPPO and zero-shot baselines, and the ablation that isolates the uncertainty guidance (5.1% over IPPO) is the right experiment. I believe that result. The observation that successful debates keep system-level aleatoric uncertainty low, while failures show it exploding, is also a credible pattern worth studying. Credit where due: the paper is explicit about its limitations in Section 8, which is more than many papers do.\n\nNow the soft spots. Proposition 3.2 is just the standard JSD-plus-mean-entropy identity, not a new theory. Theorem 3.3 assumes its own conclusion: the heterogeneous novelty condition is exactly the inequality being proved, so it doesn't explain anything. More importantly, there's a real mismatch the authors wave at but don't resolve. In Section 3.2 they measure Sys-AU as answer-level entropy from K=16 rollouts at temperature 1.0, but the training objective in Eq. 4 shapes the GRPO advantage with token-level mean negative log-likelihood of a single response at temperature 0.8. Those are different quantities. Token NLL conflates length, phrasing, and decoding noise; it is not answer-level entropy. The paper concedes in Section 8 that NLL may not correlate with semantic uncertainty. Without a correlation check showing that the proxy tracks the measured quantity, the mechanism story in Figure 3—\"aleatoric uncertainty declines\"—could just be token-NLL shrinkage. That is a fair reading of the stress-test note, and I think it lands.\n\nMinor but real: no error bars or significance tests, small AIME subsets, and the success/neutral/fail trajectories in Figure 6 are post-hoc selected. These are weaknesses, not fatal ones. The central empirical claim is independent of the theory wrappers and stands on its own plausibility.\n\nWho is this for? People working on multi-agent LLM reasoning or RLVR will get something out of the UMAD algorithm and the ablation. A serious referee should see it, but the authors should be pushed to either reframe the paper as purely empirical or do the missing correlation analysis. My recommendation: send to review, but require code, data, and a direct comparison of the two uncertainty measures before accepting.","headline":"Read this for the UMAD training results, not the theory: the empirical win is real but the theoretical wrapping is circular and the optimized uncertainty proxy doesn't match the analyzed one.","tokens_in":27250,"tokens_out":1488,"would_cite":true,"duration_ms":17876,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Effective multi-agent debate depends on keeping the gain from disagreement ahead of the cost of decoding noise; the authors train agents to do exactly that.","keywords":["multi-agent debate","uncertainty decomposition","epistemic uncertainty","aleatoric uncertainty","reinforcement learning","GRPO","mathematical reasoning","large language models"],"falsifier":"Re-run the same debate pairs using semantic clustering—grouping responses by meaning before computing entropy—instead of token-level NLL. If the success/failure separation by Sys-AU disappears or reverses, the claimed trade-off is an artifact of the uncertainty proxy; if the separation survives, the mechanism is robust. Alternatively, find a debate where Sys-AU stays low but Sys-EU never falls and accuracy still improves; that would break the necessity of the epistemic-gain condition.","tokens_in":26203,"feed_emoji":"🤖","tokens_out":6650,"duration_ms":63801,"temperature":0.7,"pith_summary":"Multi-agent debate between language models can sharpen reasoning or degrade it, and this paper proposes that the difference is a matter of uncertainty. It decomposes a debate system's total predictive uncertainty into epistemic uncertainty (how much the agents' answer distributions disagree) and aleatoric uncertainty (how noisy each agent's own decoding is). Across homogeneous and heterogeneous model pairs solving math problems, successful debates keep the second term low or falling while the first falls; failures show aleatoric noise exploding and swamping the benefit of consensus. The authors therefore design UMAD, an uncertainty-guided multi-agent reinforcement learning algorithm that penalizes internal noise and rewards evidence that actually helps peers, and they report it lifts the weaker agent's accuracy while preserving the stronger agent's. A sympathetic reader comes away with a quantitative explanation for the paradox that debate accuracy can rise while token entropy rises.","feed_headline":"Uncertainty-guided training lifts AI debate accuracy by up to 14 points","feed_subtitle":"A Bayesian split of disagreement versus noise explains debate success—and a reinforcement learner uses it to keep gains.","key_machinery":"The carrying identity is the system-level decomposition TU = Sys-EU + Sys-AU, obtained by applying generalized Jensen–Shannon divergence to the mixture of agents' answer-level empirical distributions; Sys-EU measures inter-agent disagreement, Sys-AU measures intra-agent decoding noise. On the training side, two mechanisms do the work: an aleatoric-uncertainty-aware advantage that weights the GRPO advantage by an exponential of the standardized token-level mean negative log-likelihood, and an epistemic-influence intrinsic reward that credits an agent for the average correctness improvement its reference solution produces in peers at the next turn. The decomposition turns the measured trade-of","core_discovery":"The central claim is that debate effectiveness is governed by a trade-off between epistemic gain and aleatoric cost. Total predictive uncertainty decomposes exactly into Sys-EU, the Jensen–Shannon divergence among agents' answer distributions, and Sys-AU, the mean per-agent entropy. Empirically, Sys-EU falls monotonically in every debate—debate is consensus-seeking regardless of correctness—so what separates success from failure is whether Sys-AU stays low or explodes. Heterogeneous pairs start with higher Sys-EU and thus larger potential gain, but benefit only when aleatoric cost is controlled; a theorem shows heterogeneous evidence yields strictly larger epistemic gain under a novelty cond","pith_inferences":["The decomposition suggests a practical early-stopping rule: if Sys-AU starts growing after Sys-EU has flattened, additional rounds are more likely to hurt than help—an intervention the paper does not itself propose.","The log-odds update lemma is task-agnostic: any setting where answers reduce to a correctness event should exhibit the same evidence-versus-noise structure, so the framework likely transfers beyond math to factual or code reasoning.","Because the epistemic-influence reward depends on a verifiable correctness signal, applying UMAD to open-ended domains would require a judge or learned proxy and could otherwise reward persuasive but wrong arguments—a risk the paper's impact statement acknowledges.","Replacing token-level NLL with semantic clustering would test whether Sys-AU measures genuine decoding noise or merely surface wording variance; the paper itself flags NLL as imperfectly correlated with semantic uncertainty."],"forward_implications":["If the trade-off is right, tracking Sys-EU and Sys-AU across rounds yields a direct, quantitative diagnostic of whether a debate is on track or about to collapse.","Heterogeneous pairs are not automatically better: their larger epistemic potential pays off only when aleatoric noise is controlled, so model pairing should balance diversity with stability.","UMAD trained on two debate rounds generalizes to five, so long-context stability can be obtained without expensive long-horizon training.","Uncertainty-guided training improves individual post-debate accuracy even without majority voting, implying the model internalizes the debate benefit instead of relying on aggregation."],"fun_headline_variants":["Debate success depends on epistemic gain minus aleatoric cost","Why AI debate works: high gain, low noise","Uncertainty-guided training improves AI debate by 14 points","Bayesian decomposition predicts debate productivity","Control aleatoric cost to unlock debate's potential"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that K=16 independent answer rollouts faithfully estimate an agent's predictive distribution and that token-level mean negative log-likelihood is an adequate proxy for aleatoric uncertainty; the paper itself concedes in Section 8 that token-level NLL is not always perfectly correlated with semantic uncertainty, especially for poorly calibrated models.","fun_headline_variants_meta":{"raw":{"variants":["Debate success depends on epistemic gain minus aleatoric cost","Why AI debate works: high gain, low noise","Uncertainty-guided training improves AI debate by 14 points","Bayesian decomposition predicts debate productivity","Control aleatoric cost to unlock debate's potential"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000979,"raw_usage":{"total_tokens":3964,"prompt_tokens":682,"completion_tokens":3282,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":426,"completion_tokens_details":{"reasoning_tokens":3220}},"tokens_in":426,"tokens_out":3282,"duration_ms":20604,"temperature":1.0,"reasoning_tokens":3220,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T19:39:55.923599+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same debate pairs using semantic clustering—grouping responses by meaning before computing entropy—instead of token-level NLL. If the success/failure separation by Sys-AU disappears or reverses, the claimed trade-off is an artifact of the uncertainty proxy; if the separation survives, the mechanism is robust. Alternatively, find a debate where Sys-AU stays low but Sys-EU never falls and accuracy still improves; that would break the necessity of the epistemic-gain condition.","supporting_citations":[],"review_version":1}