{"id":"a4116b59-db79-44a3-9477-68d18f1b6cab","arxiv_id":"2608.08764","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A fully unsupervised self-distillation method uses answer-level consensus and minority-trajectory contrast to improve LLM math reasoning without labels or verifiers.","lead":"CoDA is a training loop that lets a language model improve its math reasoning from unlabeled answers, using repeated-answer consensus as a teacher signal and rare answers as a penalty signal. The paper reports gains on competition benchmarks without gold solutions, though the supporting numbers are internally inconsistent and lack error bars.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 5.3 shows consensus-only training collapses; Eq. (9) suppresses all minority trajectories regardless of correctness, so the disagreement branch is not shown to stabilize false consensus, and Tables 1/3 disagree on the 1.7B consensus-only baseline.","rationale":"CoDA is a genuinely interesting attempt to replace external verification with internal agreement structure, and the paper deserves credit for diagnosing false consensus in Section 5.3 and for including an oracle-filtered ablation even as a diagnostic. My concern is not that the idea is impossible; it is that the central claim, that minority-trajectory contrast stabilizes training against erroneous consensus, is not supported by the reported evidence. The oracle-filtered ablation shows that removing incorrect modal prompts makes training stable, while CODA-Full is only described as intermediate in Figure 4, with no quantitative final accuracy on the false-consensus subset. More importantly, Eq. (9) is not a neutral gentle penalty that keeps minority tokens at their reference probability; it is a monotone loss that drives their log-ratios toward negative infinity. In a false-consensus regime, the correct trajectory is a minority trajectory, so this branch works against the very information the method needs. Whether the net effect is positive depends on the relative frequency of true versus false consensus and on lambda, but lambda is not reported. The cross-table discrepancy for the Qwen3-1.7B Consensus-Only baseline (34.37 in Table 1 versus 35.28 in Table 3) further weakens confidence in the small reported gains. These are empirical questions, not logical contradictions, so the reader's CONDITIONAL verdict remains appropriate; the authors should provide the oracle-split and lambda-sensitivity analysis to settle the concern.","tokens_in":11164,"tokens_out":7748,"duration_ms":86808,"concrete_test":"On Qwen3-1.7B with the main setup, run CODA-Full and Consensus Only for 5 seeds; after training, use gold labels only as a test-time diagnostic to split the evaluation prompts by whether the s=10 modal answer is correct. Compare accuracy on the correct-consensus and false-consensus subsets, and sweep lambda in {0, 0.01, 0.1, 1} while reporting per-seed variability. If CODA-Full's advantage is concentrated in correct-consensus prompts and it leaves false-consensus accuracy at or below Consensus Only, then minority calibration does not achieve the claimed false-consensus stabilization. Reconcile the 1.7B Consensus-Only discrepancy between Table 1 and Table 3 in the same rerun.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that answer-level consensus from s=10 self-rollouts is a reliable privileged signal, and that the disagreement branch repairs the cases where it is not. The paper's own Section 5.3 shows the first half is not secure: unfiltered Consensus Only improves early and then collapses, oracle-filtered consensus is stable, and CODA-Full is described only as intermediate in Figure 4. The disagreement branch cannot distinguish a true minority (wrong) from a correct minority under false consensus, because both are treated as y^- in Eq. (9). Moreover, Eq. (9), softplus(beta * log(pi_theta/pi_0)) - log 2, is minimized by driving every minority-token log-ratio toward -infinity, not by holding it at its initial reference level; the reference anchor only appears in the denominator. Thus on false-consensus prompts the method actively suppresses the correct unique-answer trajectory while the positive branch amplifies the wrong modal one. The paper provides no oracle-split analysis showing the net effect on false-consensus prompts, and the Qwen3-1.7B Consensus-Only average differs between Table 1 (34.37) and Table 3 (35.28), so the reported 0.8-1.7 point gain over Consensus Only is not clearly established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CoDA (Consensus and Disagreement Alignment), an unsupervised on-policy self-distillation method for improving reasoning in language models. The method samples s rollouts for each unlabeled prompt, groups them by normalized final answer, and uses a representative modal trajectory as privileged context for a frozen self-teacher, whose token-level distributions are distilled onto a fresh student rollout. A second branch penalizes minority trajectories with a reference-anchored, KTO-style softplus objective. Experiments on Qwen3-1.7B and Qwen3-4B over AIME 2024/2025/2026, HMMT 2025, and AMO-Bench report gains over self-generated baselines, with the full method approaching or exceeding supervised baselines on aggregate accuracy. The paper also studies trajectory selection, consensus sample size, and the effect of oracle filtering as a diagnostic.","tokens_in":11438,"tokens_out":5441,"duration_ms":62215,"significance":"If the central claim is correct, the paper would be a meaningful contribution: it would show that a reasoning model can improve on competition-level mathematics using only its own unlabeled rollouts and without gold solutions, verifiers, or reward models. The on-policy formulation is clearly specified, the comparison set includes both supervised and self-generated baselines, and the oracle-filtered diagnostic in Section 5.3 is a useful way to probe the false-consensus failure mode. However, the evidentiary basis is fragile: the internal tables disagree on the same baseline, the headline gains are within what could be a few questions on small benchmarks, no uncertainty quantification is provided, and one table reports a result the text says is still in progress. The method's central theoretical claim about the anchoring behavior of the disagreement objective is also not supported by the mathematics of Eq. (9). For these reasons the contribution is not yet established to the standard expected for this venue.","major_comments":[{"comment":"The reference-anchored disagreement objective does not behave as claimed. softplus(βℓ_t^-) − log 2 is strictly decreasing in ℓ_t^- and approaches −log 2 as ℓ_t^- → −∞, so minimizing L_dis drives every minority-token log-ratio toward −∞, i.e., it drives πθ(y_t^- | x, y_<t^-) toward zero. The frozen π0 in the denominator only changes the gradient scale; it does not create a lower bound or an anchor at the reference policy's probability. The text's assertion in §3.4 that the objective 'merely ensures that observed disagreement modes are not amplified beyond their initial probability' is therefore not supported by the equation. If the intended behavior is a one-sided penalty that is inactive at or below the reference level, the objective must be reformulated (e.g., with a hinge or an explicit reference point), or the claim must be revised.","section":"§3.4, Eq. (9)"},{"comment":"The reported Consensus-Only baseline is internally inconsistent: for Qwen3-1.7B, Table 1 gives an average of 34.37% while Table 3 gives 35.28% for the same configuration. This 0.91-point discrepancy is larger than the 0.84-point gain of CODA-Full over Consensus-Only reported in Table 3, and comparable to the 1.75-point gain reported in §5.1. In addition, Figure 4 shows that CODA-Full still declines late in training, merely more gradually than unfiltered Consensus Only, and the oracle-filtered variant in Table 3 is often comparable or better on average. Without error bars, multiple seeds, or significance tests, the central claims that disagreement calibration 'stabilizes training against erroneous consensus' and that CODA 'significantly improves' are not established.","section":"§5.1 and §5.3, Tables 1 and 3"},{"comment":"No error bars, number of seeds, or significance tests are reported anywhere in the paper. The benchmarks are small (AIME has 30 problems per year), so an average difference of 0.8–1.7 points can be driven by one or two correct answers across multiple benchmarks. Additionally, the joint objective in Eq. (10) depends on the weight λ, but λ is never reported in Section 4.3 or elsewhere, even though β, τ, s, and K are given. This makes the main results non-reproducible and prevents the reader from assessing the relative strength of the two branches.","section":"§4.3 and Tables 1–4"},{"comment":"Table 4 reports a final average of 35.41% for s = 20, while the text immediately below the table states that 'the s = 20 final evaluation is still in progress and is not estimated from training metrics.' These statements are contradictory: either the 35.41% is a completed final evaluation (and the caveat should be removed), or it is not a final evaluation (and the table is misleading). The section's conclusion about 'consensus saturation' and diminishing returns therefore rests on an incomplete or undefined result and must be clarified with a completed evaluation.","section":"§5.4, Table 4"},{"comment":"The paper never directly tests the effect of the disagreement branch on false-consensus prompts. When the modal answer is wrong, the correct trajectory is a minority trajectory and receives the same penalty in Eq. (9) as any incorrect minority trajectory. The authors argue the penalty is 'gentle' and serves as a regularizer, but no oracle-split analysis is provided to show the net effect on the subset of prompts where consensus is false. Consequently, the central mechanism claimed to break the harmful feedback loop is not empirically isolated; the aggregate comparison in Figure 4 conflates true-consensus and false-consensus prompts.","section":"§5.3, Eq. (9) and Figure 4"}],"minor_comments":[{"comment":"The divergence orientation is inconsistent: Eq. (7) with α = 0 defines forward KL Dα(q_t^+, p_θ,t), i.e., KL(q_t^+ || p_θ,t), while Figure 2 panel (B) labels the alignment loss as KL(p_θ,t || q_t^+). Please make the direction consistent in the figure and the text.","section":"Figure 2 and Eq. (7)"},{"comment":"The paper cites Wang et al. (2023) for self-consistency and for SFT-Self, but the reference list entry is the Self-Instruct paper. The self-consistency reference appears to be missing, and the Self-Instruct citation is likely incorrect for both usages.","section":"References, §1 and §4.2"},{"comment":"The implementation section does not report batch size, number of training steps, or wall-clock time. For an on-policy method whose cost is dominated by s = 10 rollouts per prompt, this information is important for reproducibility and for interpreting the accuracy–efficiency trade-off in Section 5.4.","section":"Section 4.3"},{"comment":"The framework name is spelled inconsistently as 'CODA' in the abstract and most of the text but 'CoDA' in the title, Figure 2, and some headings. Please standardize the capitalization.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and could be a good fit for this venue if the empirical evidence is made reliable. The internal inconsistencies (Table 1 vs Table 3, the 'in progress' row in Table 4) and the mismatch between Eq. (9) and its claimed behavior are the main blockers; these require re-running or re-analyzing experiments and a careful rewrite of Section 3.4. I would not recommend acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is worth taking seriously: use answer-level consensus to build privileged context for on-policy self-distillation, and use the minority trajectories as a regularizer. That combination is genuinely new, and the framing—no gold answers, no verifiers, no reward models—is a real distinction from prior work. The paper is clearly written, the experimental setup is mostly sound, and the false-consensus failure mode it identifies is real; its own Figure 4 illustrates that hazard nicely. Comparing against TTRL and an oracle-filtered SFT baseline is informative.\n\nThe soft spots are not minor. First, the central baseline is internally inconsistent: Table 1 reports Qwen3-1.7B consensus-only as 34.37, while Table 3 reports the same setting as 35.28. That is exactly the baseline against which the 0.8–1.7 point gains are claimed, and there are no seeds, error bars, or significance tests anywhere. Second, the disagreement branch does not do what the paper says. Eq. (9) is minimized by driving every minority log-ratio toward negative infinity, not by holding it at the reference level; the reference anchor only fixes the zero point. So on false-consensus prompts, the method actively suppresses the correct unique-answer trajectory while the positive branch amplifies the wrong modal one. The paper never analyzes this case, and its claim that the objective 'gently penalizes' or prevents amplification is not supported by the math. Third, Figure 4 shows CODA-Full is intermediate between consensus-only and oracle-filtered consensus: it slows the collapse, it does not prevent it. That undercuts the 'stabilizing training against erroneous consensus' headline.\n\nThese are fixable in principle: rerun with seeds, report lambda and all hyperparameters, reconcile the tables, and do an oracle-split analysis on false-consensus prompts to show the net effect. But until that is done, the evidence is fragile and the mechanism is imprecise.\n\nThis paper is for people working on self-improvement and on-policy distillation. It deserves a serious referee—the idea is good and the problem is important—but the referee should demand those fixes before acceptance. I would not cite it in its current form.","headline":"Promising label-free self-distillation idea that is not yet supported by its own numbers or its own loss definition.","tokens_in":12017,"tokens_out":2069,"would_cite":false,"duration_ms":23855,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that a language model can improve its mathematical reasoning using only the agreement-and-disagreement structure of its own unlabeled rollouts, with no gold solutions, verifiers, or reward models.","keywords":["self-distillation","on-policy learning","self-consistency","mathematical reasoning","false consensus","KTO-style calibration","unsupervised training","large language models"],"falsifier":"Run CoDA and the consensus-only ablation on a question set engineered so that the base model's most frequent answer is wrong on most prompts (for example, a systematic digit-reversal bias that makes ten samples converge on an incorrect result). If the full method's accuracy declines at the same rate as the consensus-only variant, the disagreement branch is not actually protecting against false consensus and the central claim fails.","tokens_in":10935,"feed_emoji":"🧮","tokens_out":14322,"duration_ms":140757,"temperature":0.7,"pith_summary":"The paper sets out to establish that a language model can improve its own mathematical reasoning using no gold solutions, verifiers, or reward models, only the latent agreement structure of its own unlabeled rollouts. The proposed method, CoDA (Consensus and Disagreement Alignment), samples several on-policy reasoning attempts per question, groups them by final answer, and treats the most frequent answer group as privileged context for a frozen copy of the model, which then guides a fresh student rollout token by token. Trajectories outside the modal group are not ignored: they are gently penalized against the original policy, because agreement can be wrong and consensus-only training can amplify correlated errors into a false consensus. On five competition-level math benchmarks with 1.7B and 4B models, CoDA is reported to outperform self-generated baselines, including an oracle-filtered self-imitation reference, and to approach or beat a supervised on-policy distillation baseline. A sympathetic reader would care because the result would mean unlabeled question banks are enough to post-train reasoning models.","feed_headline":"No gold answers needed: self-consensus trains better math reasoning","feed_subtitle":"Agreement becomes the teacher, disagreement the brake, and no label, verifier, or reward model is used.","key_machinery":"The load-bearing machinery is the answer-level consensus partition of the evidence set. CoDA tallies empirical support $c(z)=s\\hat{p}_\\theta(z|x)$ by grouping $s$ unlabeled rollouts according to a normalized final answer, with invalid or missing answers mapped to $\\bot$ and excluded from voting, and keeps a prompt only if the modal group has at least two members. From that modal pool a representative trajectory $y^+$ is selected (shortest, random, or longest) and packaged as privileged context $\\phi(x,y^+)$ for the frozen teacher. The positive branch then aligns a fresh student rollout $r$ to the teacher's distributions via $L_{\\mathrm{con}}(x)=\\frac{1}{|M_r|}\\sum_{t\\in M_r}\\min(D_\\alpha(q_t^+,p_{\\theta,t}),\\tau)$, a clipped generalized divergence over response tokens. The negative branch computes the student-to-reference log-ratio $\\ell_t^-(\\theta)=\\log\\frac{\\pi_\\theta(y_t^-|x,y_{<t}^-)}{\\pi_0(y_t^-|x,y_{<t}^-)}$ and minimizes $\\mathrm{softplus}(\\beta\\ell_t^-(\\theta))-\\log 2$ for selected minority tokens, so unstable alternatives are kept near their initial likelihood rather than promoted. The frozen reference $\\pi_0$ anchors both branches and is what makes the whole update unsupervised.","core_discovery":"On the paper's own terms, the central discovery is that answer-level consensus inside a model's own rollouts is a sufficient source of privileged information for self-teaching, provided disagreement is used as a regularizer. CoDA keeps a prompt only when at least two of the $s$ sampled trajectories share the same normalized final answer; a representative trajectory from that modal pool is revealed only to the frozen initial policy, and the student is trained on fresh rollouts by minimizing a clipped divergence to the teacher's next-token distributions. Minority trajectories, defined by answers outside the modal pool, receive a reference-anchored KTO-style penalty that reduces their relative likelihood under the student without declaring them semantically false. The paper reports that this dual-signal loop improves average accuracy over the base model on competition-level benchmarks, surpasses the fully unsupervised TTRL baseline and an oracle-filtered self-imitation baseline, approaches or exceeds supervised OPSD, and converts the late-training collapse seen in the consensus-only ablation into a gradual decline. The conclusion is that the uncertainty structure of unlabeled rollouts is a usable foundation for self-improvement.","pith_inferences":["Because the teacher and the reference are frozen copies of the initial policy, the method's ceiling is set by the initial policy's answer distribution; periodically refreshing the reference could escape that ceiling but would re-open the amplification loop the frozen anchor closes.","The answer-grouping step requires a canonical final-answer extractor, so the method transfers most directly to problems with machine-checkable outputs; a semantic-equivalence judge would be a natural extension to open-ended writing or dialogue reasoning.","The comparison with oracle filtering suggests disagreement calibration is doing something beyond deleting bad prompts, possibly variance reduction on teacher targets or entropy control; an ablation replacing the minority loss with plain token-entropy or KL regularization on evidence trajectories would separate those mechanisms."],"forward_implications":["Unlabeled problem banks become usable training data for reasoning improvement, removing the need for expert-written solutions or verifier labels.","Positive-only consensus distillation is unsafe: the paper's own training curves show it improves early then suffers late-stage collapse, so any self-improvement loop needs a disagreement-based regularizer.","The disagreement branch can recover much of the stability of oracle filtering without oracle access, since CoDA-Full stays well above the consensus-only trajectory as training progresses.","Using more consensus samples ($s=20$ versus $s=10$) stabilizes optimization and lowers the alignment loss, but final accuracy gains saturate because extra samples cannot fix a systematically wrong modal answer.","When selecting what to reveal to the teacher, a concise consensus trajectory is better than a verbose one, though this advantage shrinks as the model's own outputs get shorter."],"supporting_citations":[{"why":"Cited as the origin of self-consistency, the premise that answer agreement across sampled reasoning paths reveals a stable policy mode.","marker":"Wang et al. 2023"},{"why":"Supplies the unpaired KTO objective that CoDA adapts into its reference-anchored minority-trajectory penalty.","marker":"Ethayarajh et al. 2024"},{"why":"Provides the reference-anchored preference idea behind the KTO-style calibration point used in the negative branch.","marker":"Ouyang et al. 2022"},{"why":"Defines on-policy distillation from self-generated mistakes, the training paradigm CoDA extends.","marker":"Agarwal et al. 2024"},{"why":"Introduces privileged self-distillation (OPSD), the supervised baseline whose gold-solution requirement CoDA removes.","marker":"Zhao et al. 2026"},{"why":"Supplies the OpenThoughts problem prompts used as the unlabeled training questions.","marker":"Guha et al. 2025"},{"why":"Provides the Qwen3 instruction-tuned base models on which CoDA is trained and evaluated.","marker":"Yang et al. 2025"},{"why":"The closest multi-rollout distillation method; CoDA's novelty is removing its ground-truth correctness requirement.","marker":"Yu et al. 2026"},{"why":"Gives TTRL, the fully unsupervised baseline CoDA outperforms, plus the 'lucky hit' view of negative evidence.","marker":"Zuo et al. 2026"}],"fun_headline_variants":["Consensus from rollouts teaches; disagreement regularizes","No labels, no verifier: consensus teaches, disagreement corrects","Answer consensus as teacher, minority rollouts as brake","Self-consensus trains math reasoning without any labels","Teach with consensus, brake with disagreement: label-free self-distillation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that on the training questions, the most common final answer among ten sampled attempts is, on net, a more reliable guide than the student's own unaided predictions, and that the gentle penalty on minority answers is strong enough to compensate when that most common answer is wrong.","fun_headline_variants_meta":{"raw":{"variants":["Consensus from rollouts teaches; disagreement regularizes","No labels, no verifier: consensus teaches, disagreement corrects","Answer consensus as teacher, minority rollouts as brake","Self-consensus trains math reasoning without any labels","Teach with consensus, brake with disagreement: label-free self-distillation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001059,"raw_usage":{"total_tokens":4472,"prompt_tokens":1001,"completion_tokens":3471,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":617,"completion_tokens_details":{"reasoning_tokens":3389}},"tokens_in":617,"tokens_out":3471,"duration_ms":26853,"temperature":1.0,"reasoning_tokens":3389,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:24:51.111739+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CoDA and the consensus-only ablation on a question set engineered so that the base model's most frequent answer is wrong on most prompts (for example, a systematic digit-reversal bias that makes ten samples converge on an incorrect result). If the full method's accuracy declines at the same rate as the consensus-only variant, the disagreement branch is not actually protecting against false consensus and the central claim fails.","supporting_citations":[],"review_version":1}