REVIEW 5 major objections 4 minor 22 references
Learning from Consensus and Disagreement: Unsupervised On-Policy Self-Distillation with Minority-Trajectory Contrast
T0 review · 5 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper argues that a language model can improve its mathematical reasoning using only the agreement-and-disagreement structure of its own unlabeled rollouts, with no gold solutions, verifiers, or reward models.
desk verdict Promising label-free self-distillation idea that is not yet supported by its own numbers or its own loss definition. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is the answer-level consensus partition of the evidence set. CoDA tallies empirical support $c(z)=s\hat{p}_\theta(z|x)$ by grouping $s$ unlabeled rollouts according to a normalized final answer, with invalid or missing answers mapped to $\bot$ and excluded from voting, and keeps a prompt only if the modal group has at least two members. From that modal pool a representative trajectory $y^+$ is selected (shortest, random, or longest) and packaged as privileged context $\phi(x,y^+)$ for the frozen teacher. The positive branch then aligns a fresh student rollout $r$ to the teacher's distributions via $L_{\mathrm{con}}(x)=\frac{1}{|M_r|}\sum_{t\in M_r}\min(D_\alpha(q_t^+,p_{\theta,t}),\tau)$, a clipped generalized divergence over response tokens. The negative branch computes the student-to-reference log-ratio $\ell_t^-(\theta)=\log\frac{\pi_\theta(y_t^-|x,y_{<t}^-)}{\pi_0(y_t^-|x,y_{<t}^-)}$ and minimizes $\mathrm{softplus}(\beta\ell_t^-(\theta))-\log 2$ for selected minority tokens, so unstable alternatives are kept near their initial likelihood rather than promoted. The frozen reference $\pi_0$ anchors both branches and is what makes the whole update unsupervised.
What would settle it
Run CoDA and the consensus-only ablation on a question set engineered so that the base model's most frequent answer is wrong on most prompts (for example, a systematic digit-reversal bias that makes ten samples converge on an incorrect result). If the full method's accuracy declines at the same rate as the consensus-only variant, the disagreement branch is not actually protecting against false consensus and the central claim fails.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that answer-level consensus inside a model's own rollouts is a sufficient source of privileged information for self-teaching, provided disagreement is used as a regularizer. CoDA keeps a prompt only when at least two of the $s$ sampled trajectories share the same normalized final answer; a representative trajectory from that modal pool is revealed only to the frozen initial policy, and the student is trained on fresh rollouts by minimizing a clipped divergence to the teacher's next-token distributions. Minority trajectories, defined by answers outside the modal pool, receive a reference-anchored KTO-style penalty that reduces their relative likelihood under the student without declaring them semantically false. The paper reports that this dual-signal loop improves average accuracy over the base model on competition-level benchmarks, surpasses the fully unsupervised TTRL baseline and an oracle-filtered self-imitation baseline, approaches or exceeds supervised OPSD, and converts the late-training collapse seen in the consensus-only ablation into a gradual decline. The conclusion is that the uncertainty structure of unlabeled rollouts is a usable foundation for self-improvement.
Load-bearing premise
The method assumes that on the training questions, the most common final answer among ten sampled attempts is, on net, a more reliable guide than the student's own unaided predictions, and that the gentle penalty on minority answers is strong enough to compensate when that most common answer is wrong.
Editorial extensions
If this is right
- Unlabeled problem banks become usable training data for reasoning improvement, removing the need for expert-written solutions or verifier labels.
- Positive-only consensus distillation is unsafe: the paper's own training curves show it improves early then suffers late-stage collapse, so any self-improvement loop needs a disagreement-based regularizer.
- The disagreement branch can recover much of the stability of oracle filtering without oracle access, since CoDA-Full stays well above the consensus-only trajectory as training progresses.
- Using more consensus samples ($s=20$ versus $s=10$) stabilizes optimization and lowers the alignment loss, but final accuracy gains saturate because extra samples cannot fix a systematically wrong modal answer.
- When selecting what to reveal to the teacher, a concise consensus trajectory is better than a verbose one, though this advantage shrinks as the model's own outputs get shorter.
Reading between the lines
- Because the teacher and the reference are frozen copies of the initial policy, the method's ceiling is set by the initial policy's answer distribution; periodically refreshing the reference could escape that ceiling but would re-open the amplification loop the frozen anchor closes.
- The answer-grouping step requires a canonical final-answer extractor, so the method transfers most directly to problems with machine-checkable outputs; a semantic-equivalence judge would be a natural extension to open-ended writing or dialogue reasoning.
- The comparison with oracle filtering suggests disagreement calibration is doing something beyond deleting bad prompts, possibly variance reduction on teacher targets or entropy control; an ablation replacing the minority loss with plain token-entropy or KL regularization on evidence trajectories would separate those mechanisms.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CoDA (Consensus and Disagreement Alignment), an unsupervised on-policy self-distillation method for improving reasoning in language models. The method samples s rollouts for each unlabeled prompt, groups them by normalized final answer, and uses a representative modal trajectory as privileged context for a frozen self-teacher, whose token-level distributions are distilled onto a fresh student rollout. A second branch penalizes minority trajectories with a reference-anchored, KTO-style softplus objective. Experiments on Qwen3-1.7B and Qwen3-4B over AIME 2024/2025/2026, HMMT 2025, and AMO-Bench report gains over self-generated baselines, with the full method approaching or exceeding supervised baselines on aggregate accuracy. The paper also studies trajectory selection, consensus sample size, and the effect of oracle filtering as a diagnostic.
Significance. If the central claim is correct, the paper would be a meaningful contribution: it would show that a reasoning model can improve on competition-level mathematics using only its own unlabeled rollouts and without gold solutions, verifiers, or reward models. The on-policy formulation is clearly specified, the comparison set includes both supervised and self-generated baselines, and the oracle-filtered diagnostic in Section 5.3 is a useful way to probe the false-consensus failure mode. However, the evidentiary basis is fragile: the internal tables disagree on the same baseline, the headline gains are within what could be a few questions on small benchmarks, no uncertainty quantification is provided, and one table reports a result the text says is still in progress. The method's central theoretical claim about the anchoring behavior of the disagreement objective is also not supported by the mathematics of Eq. (9). For these reasons the contribution is not yet established to the standard expected for this venue.
major comments (5)
- [§3.4, Eq. (9)] The reference-anchored disagreement objective does not behave as claimed. softplus(βℓ_t^-) − log 2 is strictly decreasing in ℓ_t^- and approaches −log 2 as ℓ_t^- → −∞, so minimizing L_dis drives every minority-token log-ratio toward −∞, i.e., it drives πθ(y_t^- | x, y_<t^-) toward zero. The frozen π0 in the denominator only changes the gradient scale; it does not create a lower bound or an anchor at the reference policy's probability. The text's assertion in §3.4 that the objective 'merely ensures that observed disagreement modes are not amplified beyond their initial probability' is therefore not supported by the equation. If the intended behavior is a one-sided penalty that is inactive at or below the reference level, the objective must be reformulated (e.g., with a hinge or an explicit reference point), or the claim must be revised.
- [§5.1 and §5.3, Tables 1 and 3] The reported Consensus-Only baseline is internally inconsistent: for Qwen3-1.7B, Table 1 gives an average of 34.37% while Table 3 gives 35.28% for the same configuration. This 0.91-point discrepancy is larger than the 0.84-point gain of CODA-Full over Consensus-Only reported in Table 3, and comparable to the 1.75-point gain reported in §5.1. In addition, Figure 4 shows that CODA-Full still declines late in training, merely more gradually than unfiltered Consensus Only, and the oracle-filtered variant in Table 3 is often comparable or better on average. Without error bars, multiple seeds, or significance tests, the central claims that disagreement calibration 'stabilizes training against erroneous consensus' and that CODA 'significantly improves' are not established.
- [§4.3 and Tables 1–4] No error bars, number of seeds, or significance tests are reported anywhere in the paper. The benchmarks are small (AIME has 30 problems per year), so an average difference of 0.8–1.7 points can be driven by one or two correct answers across multiple benchmarks. Additionally, the joint objective in Eq. (10) depends on the weight λ, but λ is never reported in Section 4.3 or elsewhere, even though β, τ, s, and K are given. This makes the main results non-reproducible and prevents the reader from assessing the relative strength of the two branches.
- [§5.4, Table 4] Table 4 reports a final average of 35.41% for s = 20, while the text immediately below the table states that 'the s = 20 final evaluation is still in progress and is not estimated from training metrics.' These statements are contradictory: either the 35.41% is a completed final evaluation (and the caveat should be removed), or it is not a final evaluation (and the table is misleading). The section's conclusion about 'consensus saturation' and diminishing returns therefore rests on an incomplete or undefined result and must be clarified with a completed evaluation.
- [§5.3, Eq. (9) and Figure 4] The paper never directly tests the effect of the disagreement branch on false-consensus prompts. When the modal answer is wrong, the correct trajectory is a minority trajectory and receives the same penalty in Eq. (9) as any incorrect minority trajectory. The authors argue the penalty is 'gentle' and serves as a regularizer, but no oracle-split analysis is provided to show the net effect on the subset of prompts where consensus is false. Consequently, the central mechanism claimed to break the harmful feedback loop is not empirically isolated; the aggregate comparison in Figure 4 conflates true-consensus and false-consensus prompts.
minor comments (4)
- [Figure 2 and Eq. (7)] The divergence orientation is inconsistent: Eq. (7) with α = 0 defines forward KL Dα(q_t^+, p_θ,t), i.e., KL(q_t^+ || p_θ,t), while Figure 2 panel (B) labels the alignment loss as KL(p_θ,t || q_t^+). Please make the direction consistent in the figure and the text.
- [References, §1 and §4.2] The paper cites Wang et al. (2023) for self-consistency and for SFT-Self, but the reference list entry is the Self-Instruct paper. The self-consistency reference appears to be missing, and the Self-Instruct citation is likely incorrect for both usages.
- [Section 4.3] The implementation section does not report batch size, number of training steps, or wall-clock time. For an on-policy method whose cost is dominated by s = 10 rollouts per prompt, this information is important for reproducibility and for interpreting the accuracy–efficiency trade-off in Section 5.4.
- [Throughout] The framework name is spelled inconsistently as 'CODA' in the abstract and most of the text but 'CoDA' in the title, Figure 2, and some headings. Please standardize the capitalization.
Circularity Check
No circularity: CoDA's self-generated consensus and disagreement signals are algorithmic objectives evaluated on external benchmarks, not predictions reduced to their own inputs by construction.
full rationale
CoDA is an empirical method paper. The claimed derivation is algorithmic rather than a formal derivation: the positive branch (Eq. 7) aligns the student to a frozen teacher conditioned on a consensus trajectory, and the negative branch (Eq. 9) penalizes minority trajectories relative to a reference policy. These are training objectives, not predictions, and the paper's central claims are evaluated on held-out competition benchmarks (AIME 2024/2025/2026, HMMT 2025, AMO-Bench). No equation reduces to its own input, and no fitted parameter is renamed as a prediction. The self-citations (Guo et al. 2026a,b) are references to prior work on error accumulation and preference optimization; they are not load-bearing for CoDA's design and do not invoke an external uniqueness theorem. Section 5.3 explicitly tests the central assumption that consensus can be wrong, using oracle-filtered diagnostics, so the claim is externally falsifiable rather than assumed by construction. The paper is self-contained against external benchmarks, and any concerns about selector choice on the same benchmarks or the Table 1 vs. Table 3 Consensus Only inconsistency are correctness/overfitting issues, not circularity. Section 5.4 also notes that the s=20 final evaluation is 'still in progress,' which is a missing result but does not affect the circularity analysis.
Assumptions & free parameters
free parameters (6)
- divergence clipping threshold tau =
0.05
- KTO calibration strength beta =
0.1
- disagreement loss weight lambda
- consensus sample size s =
10
- minority trajectories sampled K =
1
- representative consensus selector g =
shortest for 1.7B, random for 4B
assumptions (3)
- domain assumption Answer-level consensus among s self-rollouts is a reliable proxy for correct reasoning on the training prompts.
- domain assumption Minority trajectories are predominantly undesirable relative to the initial policy.
- domain assumption The frozen initial policy pi0 provides a stable reference such that KTO-style calibration does not distort the student's general abilities.
Cite this review
Pith. "Pith review of Learning from Consensus and Disagreement: Unsupervised On-Policy Self-Distillation with Minority-Trajectory Contrast." pith.science (2026). https://pith.science/paper/Y2E6XHWK
@misc{pith2026260808764,
author = {Pith},
title = {Pith review of: Learning from Consensus and Disagreement: Unsupervised On-Policy Self-Distillation with Minority-Trajectory Contrast},
year = {2026},
howpublished = {\url{https://pith.science/paper/Y2E6XHWK}},
note = {Machine review of arXiv:2608.08764}
}
read the original abstract
On-policy self-distillation improves language-model reasoning by querying a teacher on states actually visited by the student. Recent methods create a powerful information asymmetry by exposing the teacher to privileged context, yet they fundamentally rely on external supervision---such as gold solutions or verifiers---to construct this advantage. We introduce CoDA (Consensus and Disagreement Alignment), a fully unsupervised framework that creates reliable privileged information entirely from the latent uncertainty structure of a model's own unlabeled rollouts. CoDA extracts two complementary signals. In the positive branch, answer-level consensus identifies a stable reasoning mode, which conditions a frozen self-teacher to provide dense distributional guidance on fresh student trajectories. However, because agreement does not guarantee correctness, positive-only distillation risks amplifying correlated errors into a false consensus. To break this harmful feedback loop, CoDA incorporates a negative branch that exploits disagreement: minority trajectories are treated as unstable alternatives and gently penalized via a reference-anchored, KTO-style calibration objective. This unpaired binary feedback provides robust regularization without requiring the strong assumption that the consensus is the absolute ground truth. Empirical evaluations on competition-level mathematical benchmarks demonstrate that CoDA significantly improves reasoning, outperforming self-generated baselines and effectively stabilizing training against erroneous consensus.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[2]
Fu, Y .; Huang, H.; Jiang, K.; Liu, J.; Jiang, Z.; Zhu, Y .; and Zhao, D
Kto: Model alignment as prospect theoretic optimization.arXiv preprint arXiv:2402.01306. Fu, Y .; Huang, H.; Jiang, K.; Liu, J.; Jiang, Z.; Zhu, Y .; and Zhao, D
-
[6]
Jia, N.; Yang, H.; Ma, X.; Lian, J.; Zhang, S.; Zhang, W.; Zeng, K.; Cai, X.; and Sun, Z
Reinforcement Learning via Self-Distillation.arXiv preprint arXiv:2601.20802. Jia, N.; Yang, H.; Ma, X.; Lian, J.; Zhang, S.; Zhang, W.; Zeng, K.; Cai, X.; and Sun, Z
-
[7]
Jin, Y .; Wang, Y .; Fu, L.; Xiao, Y .; Luo, Y .; Liu, H.; Prakash, B
Asymmetric on-policy distillation: Bridging exploitation and imitation at the token level.arXiv preprint arXiv:2605.06387. Jin, Y .; Wang, Y .; Fu, L.; Xiao, Y .; Luo, Y .; Liu, H.; Prakash, B. A.; Hester, J.; Wang, J.; and Kumar, S
-
[8]
Ko, J.; Kim, S.; Chen, T.; and Yun, S.-Y
UniSD: To- wards a Unified Self-Distillation Framework for Large Lan- guage Models.arXiv preprint arXiv:2605.06597. Ko, J.; Kim, S.; Chen, T.; and Yun, S.-Y
-
[9]
arXiv preprint arXiv:2402.03898
Distillm: Towards streamlined distillation for large language models. arXiv preprint arXiv:2402.03898. Ouyang, L.; Wu, J.; Jiang, X.; Almeida, D.; Wainwright, C. L.; Mishkin, P.; Zhang, C.; Agarwal, S.; Slama, K.; Ray, A.; et al
-
[11]
Press, O.; Zhang, M.; Min, S.; Schmidt, L.; Smith, N
Privileged Informa- tion Distillation for Language Models.arXiv preprint arXiv:2602.04942. Press, O.; Zhang, M.; Min, S.; Schmidt, L.; Smith, N. A.; and Lewis, M
-
[13]
Ross, S.; Gordon, G.; and Bagnell, D
Direct preference optimization: Your language model is secretly a reward model.arXiv preprint arXiv:2305.18290. Ross, S.; Gordon, G.; and Bagnell, D
-
[14]
Shen, G.; Cheng, X.; Zhao, C.; Huang, L.; Li, J.; Zhao, D.; and Yu, X
DeepSeek- Math: Pushing the Limits of Mathematical Reasoning in Open Language Models.arXiv preprint arXiv:2402.03300. Shen, G.; Cheng, X.; Zhao, C.; Huang, L.; Li, J.; Zhao, D.; and Yu, X. 2026a. Anti-Self-Distillation for Reason- ing RL via Pointwise Mutual Information.arXiv preprint arXiv:2605.11609. Shen, G.; Huang, L.; Cheng, X.; Zhao, C.; Li, J.; Zha...
Show all 22 references
-
[15]
arXiv preprint arXiv:2602.20574
GATES: Self- distillation under privileged context with consensus gating. arXiv preprint arXiv:2602.20574. Wang, Y .; Kordi, Y .; Mishra, S.; Liu, A.; Smith, N. A.; Khashabi, D.; and Hajishirzi, H
-
[16]
Xu, Y .; Sang, H.; Zhou, Z.; He, R.; and Wang, Z
Chain-of- thought prompting elicits reasoning in large language mod- els.Advances in neural information processing systems, 35: 24824–24837. Xu, Y .; Sang, H.; Zhou, Z.; He, R.; and Wang, Z. 2026a. PACED: Distillation and On-Policy Self-Distillation at the Frontier of Student ...
-
[17]
Yang, C.; Qin, C.; Si, Q.; Chen, M.; Gu, N.; Yao, D.; Lin, Z.; Wang, W.; Wang, J.; and Duan, N
Qwen3 technical report.arXiv preprint arXiv:2505.09388. Yang, C.; Qin, C.; Si, Q.; Chen, M.; Gu, N.; Yao, D.; Lin, Z.; Wang, W.; Wang, J.; and Duan, N
-
[18]
Ye, T.; Dong, L.; Wu, X.; Huang, S.; and Wei, F
Self-Distilled RLVR.arXiv preprint arXiv:2604.03128. Ye, T.; Dong, L.; Wu, X.; Huang, S.; and Wei, F
-
[19]
Yu, W.; Li, X.; Zhao, Y .; Liu, X.; Zhang, R.; Wang, H.; Luo, Y .; Wu, C
On-Policy Context Distillation for Language Models.arXiv preprint arXiv:2602.12275. Yu, W.; Li, X.; Zhao, Y .; Liu, X.; Zhang, R.; Wang, H.; Luo, Y .; Wu, C. H.; Mittal, G.; Fredrikson, M.; and Hu, Y
-
[20]
Zhao, S.; Xie, Z.; Liu, M.; Huang, J.; Pang, G.; Chen, F.; and Grover, A
Multi-Rollout On-Policy Distillation via Peer Successes and Failures.arXiv preprint arXiv:2605.12652. Zhao, S.; Xie, Z.; Liu, M.; Huang, J.; Pang, G.; Chen, F.; and Grover, A
-
[21]
Zheng, B.; Ma, X.; Liang, Y .; Ruan, J.; Fu, X.; Lin, K.; Zhu, B.; Zeng, K.; and Cai, X
Self-Distilled Reasoner: On-Policy Self-Distillation for Large Language Models.arXiv preprint arXiv:2601.18734. Zheng, B.; Ma, X.; Liang, Y .; Ruan, J.; Fu, X.; Lin, K.; Zhu, B.; Zeng, K.; and Cai, X
-
[22]
Zuo, Y .; Zhang, K.; Sheng, L.; Qu, S.; Cui, G.; Zhu, X.; Li, H.; Long, X.; Hua, E.; Qi, B.; et al
Scope: Signal-calibrated on-policy distillation enhancement with dual-path adaptive weighting.arXiv preprint arXiv:2604.10688. Zuo, Y .; Zhang, K.; Sheng, L.; Qu, S.; Cui, G.; Zhu, X.; Li, H.; Long, X.; Hua, E.; Qi, B.; et al
-
[2015]
Distill- ing the knowledge in a neural network.arXiv preprint arXiv:1503.02531. Hu, E. J.; Shen, Y .; Wallis, P.; Allen-Zhu, Z.; Li, Y .; Wang, S.; Wang, L.; Chen, W.; et al
-
[2022]
Penaloza, E.; Vattikonda, D.; Gontier, N.; Lacoste, A.; Charlin, L.; and Caccia, M
Training language models to fol- low instructions with human feedback.arXiv preprint arXiv:2203.02155. Penaloza, E.; Vattikonda, D.; Gontier, N.; Lacoste, A.; Charlin, L.; and Caccia, M
-
[2023]
InFindings of the Association for Computational Linguistics: EMNLP 2023, 5687–5711
Measuring and narrowing the com- positionality gap in language models. InFindings of the Association for Computational Linguistics: EMNLP 2023, 5687–5711. Rafailov, R.; Sharma, A.; Mitchell, E.; Ermon, S.; Manning, C. D.; and Finn, C
2023
-
[2024]
InInternational Conference on Learn- ing Representations, volume 2024, 21246–21263
On- policy distillation of language models: Learning from self- generated mistakes. InInternational Conference on Learn- ing Representations, volume 2024, 21246–21263. Ethayarajh, K.; Xu, W.; Muennighoff, N.; Jurafsky, D.; and Kiela, D
2024
-
[2025]
Guo, J.; Sun, H.; Zhang, W.; Fan, X.; and Zhang, Y
OpenThoughts: Data Recipes for Reasoning Models.arXiv preprint arXiv:2506.04178. Guo, J.; Sun, H.; Zhang, W.; Fan, X.; and Zhang, Y . 2026a. Mitigating error accumulation in knowledge editing for multi-hop question answering. InProceedings of the AAAI Conference on Artificial ...
2026 arXiv
-
[2026]
Gu, Y .; Dong, L.; Wei, F.; and Huang, M
Revisiting on-policy distillation: Em- pirical failure modes and simple fixes.arXiv preprint arXiv:2603.25562. Gu, Y .; Dong, L.; Wei, F.; and Huang, M
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.