Pith. sign in

REVIEW 3 major objections 4 minor 16 references

This paper proposes CANON, showing that a language model's own majority-vote consensus can act as token-level privileged context for label-free self-distillation, improving reasoning accuracy without any gold labels.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 04:35 UTC pith:LOAZ7VWQ

load-bearing objection A careful, well-posed label-free distillation method with genuinely strong transductive results; the abstract overclaims one inductive majority-vote result, but the core is worth engaging seriously. the 3 major comments →

arxiv 2607.13643 v1 pith:LOAZ7VWQ submitted 2026-07-15 cs.LG cs.AIcs.CL

Consensus as Privileged Context for Label-Free Self-Distillation

John Gkountouras , Josip Juki\'c , Ivan Titov This is my paper
classification cs.LG cs.AIcs.CL
keywords self-distillationconsensuslabel-free trainingtoken-level supervisionmajority votingtest-time trainingreasoning LLMsJensen-Shannon divergence
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to establish that the agreement among a language model's own sampled solutions—the consensus behind majority voting—can play the role that gold solutions usually play in self-distillation. Rather than filtering or rewarding whole trajectories, CANON conditions a frozen copy of the model on the most confident consensus-reaching solution and trains the live model to match that teacher's next-token distributions on its own rollouts. The authors argue that this dense, token-level, label-free supervision improves pass@1 by up to 12 points, approaches a gold-conditioned oracle teacher, transfers to held-out benchmarks, and goes beyond mere sharpening: the model solves prompts it never solved and its majority vote becomes more accurate. If correct, it gives a cheap, label-free post-training recipe that captures most of the benefit of ensemble voting in a single pass.

Core claim

CANON turns the model's own majority-vote consensus into dense token-level training signal. Specifically, it samples N solutions, extracts the majority answer, picks the highest mean-token-log-probability rollout among those reaching that answer, and uses a frozen snapshot of the model conditioned on that solution as a teacher. The student minimizes a full-vocabulary Jensen-Shannon divergence to the teacher over all of its own rollouts, including those that disagree with the consensus. The paper reports that this recovers most of the advantage of a gold-conditioned teacher and outperforms label-free reinforcement learning at a fraction of the compute.

What carries the argument

The consensus-anchored frozen-snapshot teacher: a rollout that reaches the majority answer and has the highest mean token log-probability, inserted into the teacher's context via a fixed template; the frozen teacher scores every token of every student rollout, yielding full next-token distributions. The Jensen-Shannon divergence objective (bounded, symmetric) and one epoch of parameter-efficient training over a single generation pass complete the mechanism. This machinery converts a scalar vote into dense positional supervision while preventing teacher-student co-adaptation.

Load-bearing premise

The load-bearing premise is that the most confident majority-answer rollout is a good enough line of reasoning that forcing the student to imitate its next-token distributions improves the student on all rollouts; the paper's own analysis shows this fails when the model is confidently wrong (a measurable but rare case).

What would settle it

A decisive test: apply CANON to any model-benchmark pair where the base model's majority-vote accuracy is not above its single-sample accuracy (no consensus headroom). The claim predicts near-zero or negative avg@32 change; if strong positive gains appear, the proposed mechanism—distilling the gap between expressed and consensus accuracy—is not what is operating.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, majority-vote ensembling can be amortized into a single greedy pass, making test-time accuracy nearly free at inference.
  • Label-free post-training can match gold-conditioned distillation and gold-reward training on held-out benchmarks, suggesting consensus can substitute for labels in math and science reasoning.
  • Dense token-level supervision carries most of the signal: replacing it with sequence-level fine-tuning on the same consensus solution loses most of the gain.
  • The method's applicability is predictable from two measurable quantities—the gap between consensus and single-sample accuracy and the reliability of the majority—giving a practical selection rule.
  • Iterating self-distillation beyond one round yields little and can reverse gains, so the one-round recipe is near-optimal.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to replace exact-match voting with medoid or minimum-Bayes-risk selection, which the paper notes as future work; if that works, the same anchoring mechanism could apply to free-form generation.
  • The failure-mode analysis implies that confidently-wrong consensus is the main risk; combining CANON with an abstention or uncertainty gate might turn a rare -3.0-point harm into a safeguard.
  • Because one generation pass supplies both consensus and training substrate, CANON could plausibly be embedded in batch serving pipelines as a continuous self-improvement loop, though the paper does not test this.
  • The per-prompt difficulty analysis suggests the method should transfer to other verifiable-output domains—code, formal proofs, multiple-choice science—whenever the base model's consensus is informative but not saturated.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes CANON, a label-free self-distillation method that converts the majority-answer consensus of multiple sampled rollouts into dense, token-level supervision. For each unlabeled prompt, CANON samples N solutions, extracts the majority answer, and conditions a frozen snapshot of the model on a consensus solution; the resulting teacher-forced next-token distributions supervise the student on its own rollouts via a Jensen-Shannon divergence. Training lasts one epoch and requires a single generation pass. Experiments on competition mathematics and graduate-level science benchmarks compare CANON against label-free baselines (LMSI, ScPO, TTRL, EMPO, SCRL, TTRL-Guard), label-full baselines (RFT, GRPO, gold-conditioned oracle distillation), and evaluate both a transductive regime (training on the test prompts) and an inductive regime (training on a disjoint pool and evaluating on held-out benchmarks). The paper also reports ablations isolating the conditioning content and design choices, plus an analysis of when consensus supervision helps or hurts.

Significance. If the main claims hold, this is a significant contribution: it shows that the model's own majority consensus can serve as a privileged context for dense self-distillation, extracting more from an unlabeled signal than scalar-reward RL and matching much of the value of gold-conditioned teachers at a fraction of the compute. The evaluation is unusually careful: a single frozen evaluation harness, hyperparameters selected once on disjoint development slices, paired bootstrap confidence intervals for the central avg@32 comparisons, contamination audits with positive controls, faithful baseline implementations, and an ablation ladder that cleanly separates the contribution of the reasoning content from format and answer leakage. The live-teacher collapse documented in Appendix F is an important practical caution. The main caveat is scope: several consensus-level and coverage claims are supported only by transductive evidence or by very small prompt counts, and the sole inductive majority-vote result points in the opposite direction.

major comments (3)
  1. [Abstract; §5.1, Table 4; §4.4, Table 3] The unqualified sentence 'its majority vote itself becomes more accurate' is supported only by transductive evidence. Every maj@32 gain in Table 4 is transductive, while the only inductive pooled experiment—the design that actually separates adaptation from genuine improvement—reports maj@32 falling from 56.7 to 53.3 for CANON-pool on held-out AIME 2024 (Table 3). With n=30 the estimate is noisy, but it is currently the only inductive consensus-level evidence and it points opposite to Table 4. Consequently, the claim that gains are 'not pure distribution sharpening' is not established outside the transductive setting. Please qualify the abstract/conclusion claim, or add a larger inductive experiment (or an aggregated transition analysis over held-out benchmarks) that supports majority-vote improvement under held-out evaluation.
  2. [§5.1, Table 4, Figure 4; Appendix J] The coverage claims rest on pass@32 and maj@32 deltas on 30-prompt benchmarks (AIME24: +10.0 pass@32, +3.3 maj@32; AIME25: +16.7 pass@32, +3.3 maj@32). At n=30 one prompt moves maj@32 by 3.3 points, so these point estimates are not self-evaluating. Appendix E reports paired bootstrap intervals only for avg@32, not for pass@32 or maj@32, and the per-prompt transition counts in Appendix J are small (e.g., 4 coverage gains on AMC, 4 on AIME24). Since RQ4—whether gains go beyond sharpening—depends on these exact statistics, please report confidence intervals for pass@32/maj@32 or aggregate the transition analysis across benchmarks with a formal test. This is fixable, but as written the 'beyond sharpening' conclusion rests on a small number of prompts.
  3. [Appendix B (oracle construction); §4.2] The text says the oracle differs from CANON 'only in which solution anchors the teacher,' but for GPQA the deep-resampled oracle provides gold conditioning for only 179 of 198 prompts and the remaining prompts fall back to consensus conditioning. The appendix asserts that restricting to covered prompts changes no conclusion, but the restricted comparison is not shown. Since GPQA is the cell where the oracle retains its largest advantage (Table 6 / Appendix E), please report the per-coverage analysis or otherwise substantiate the 'only difference' statement.
minor comments (4)
  1. [§4.2] The sentence reporting AIME 2024 oracle numbers (48.0 vs 44.6) and GPQA numbers (62.5 vs 54.6) cites '(Table 6)', but the table shown at that location is the conditioning ladder and contains no AIME column. The numbers appear to belong to Appendix E; please fix the cross-reference.
  2. [Figure 4] The hatched bar for the pass@32 change is not labeled in the legend. Add a legend entry or a direct label so the reader can distinguish the two statistics.
  3. [§5.2, Table 5] The +8.0 point gain in the 0–0.2 vote-share band, the largest band by prompt count (n=413), is in some tension with the prose characterization that 'the consensus is often too weak to anchor reliably.' One sentence reconciling the positive aggregate gain with the weak-consensus characterization would help.
  4. [§7.1, Limitations] The limitation statement about iterating the procedure ('at most one additional productive round') is supported by Appendix F, but the appendix result is not cross-referenced here. Adding the cross-reference would help the reader verify the claim.

Circularity Check

0 steps flagged

No significant circularity: CANON is self-training by design, but all load-bearing claims are checked against external labels and controlled ablations.

full rationale

CANON's teacher is literally the model itself: a frozen snapshot conditioned on a consensus rollout selected from the model's own samples, and the student is trained on its own rollouts. That is self-referential in the machine-learning sense, but it is not circular in the derivation sense: the paper does not define a predicted quantity in terms of the fitted quantity, and no reported result is equal to the training objective by construction. The central claims (pass@1, maj@32, held-out transfer, and the 'more than sharpening' analysis) are evaluated with external gold labels used only for scoring, and the oracle arm provides an independent gold-conditioned reference. The conditioning ladder (Table 6) explicitly shows that content-free placeholders, bare consensus answers, and another prompt's consensus solution do not reproduce the gains, so the result is not an artifact of the training procedure or of any self-citation. The paper contains no load-bearing self-citations: the privileged-context and self-distillation methods it builds on (Zhao et al., 2026a; Hübotter et al., 2026; Stein et al., 2026) are from other groups. The one legitimate caveat is evidential, not circular: in the pooled inductive setting, Table 3 reports CANON-pool maj@32 53.3 versus base 56.7, which the paper itself flags ('one prompt moves maj@32 by 3.3 points'). That is a measurement/evidence problem for the unqualified 'majority vote itself becomes more accurate' claim, not a reduction of the output to the input. Consequently, there are no circular steps to report; score 0.

Axiom & Free-Parameter Ledger

6 free parameters · 6 axioms · 0 invented entities

The method's contribution is empirical, not derivational. It rests on a handful of recipe constants and the domain assumption that majority agreement is a trustworthy teacher signal. No parameters are fitted to the evaluation benchmarks; hyperparameters were selected on disjoint development slices. The free-parameter count is moderate but explicitly disclosed and ablated in the appendices. No new formal or physical entities are introduced.

free parameters (6)
  • rollout count N = 32
    Consensus and distillation substrate size; recipe constant. Ablations show monotonic gains to 32 and plateau at 64, so 32 is a chosen operating point, not derived.
  • learning rate = 1e-5 for Qwen3-4B / Qwen3.5-4B; 3e-5 SmolLM3; 5e-6 LFM2.5
    Selected once per model on a 64-prompt OmniMath development slice; frozen for all reported benchmarks.
  • LoRA rank / alpha = rank 64, alpha 128
    Adapter capacity chosen by hand; no sweep reported for rank.
  • supervision divergence = JSD
    Selected on dev slice; Appendix F reports forward KL gives 78.2 vs JSD 76.5 on AMC, so the choice is not uniquely best.
  • training epochs / rounds = 1
    Single epoch; second round +0.9, third reverses; a recipe decision.
  • consensus solution selector = argmax mean token log-prob among majority rollouts
    Ad hoc selection rule (Eq. 4); no theoretical justification that the most confident rollout is the best teacher.
axioms (6)
  • domain assumption Majority-vote answers are a reliable label-free proxy for correctness.
    Foundation of the supervision signal; cited to self-consistency (Wang et al. 2022); §5.2 shows it fails when consensus is confidently wrong.
  • domain assumption Teacher-forced distributions conditioned on the selected consensus solution are useful targets for all rollouts, including minority rollouts.
    Assumed in Eq. (2); ablation 'masking minority-answer rollouts costs 3.5 points' supports it empirically but it is not proven.
  • domain assumption A single epoch with a frozen snapshot keeps rollouts sufficiently on-policy.
    Appendix F: stale rollouts lose 1–5 points, so freshness is load-bearing.
  • domain assumption LoRA fine-tuning with the specified rank and LR preserves and improves base capabilities rather than corrupting them.
    Empirical premise; no formal guarantee.
  • domain assumption The teacher prompt template adds no information beyond the inserted solution.
    Appendix C; Table 6 ladder with a content-free placeholder controls for this.
  • standard math Jensen-Shannon divergence gradient formula (Eq. 6) and probability calculus.
    Used to derive the objective; the formula is correct under a fixed teacher.

pith-pipeline@v1.3.0-alltime-deepseek · 23561 in / 16481 out tokens · 170063 ms · 2026-08-02T04:35:15.371373+00:00 · methodology

0 comments
read the original abstract

Sampling multiple solutions and returning the majority answer is among the most reliable ways to improve the reasoning accuracy of large language models without labels, and a growing family of methods converts this consensus signal into training supervision. However, existing approaches use consensus only in restricted forms: as a filter that selects solutions for fine-tuning, as a preference between answers, or as a scalar reward for reinforcement learning, discarding most of the information that the agreeing solutions contain. We present CANON (Consensus-ANchored self-distillatiON), a label-free training method that turns consensus into dense, token-level supervision. For each unlabeled prompt, CANON samples multiple solutions, extracts the majority answer, and conditions a frozen snapshot of the model on a solution that reaches it; this consensus-anchored teacher then supervises the model on its own rollouts at every token. Experiments on mathematical and scientific reasoning benchmarks show that CANON improves pass@1 by up to 12 points, outperforming label-free reinforcement learning by 6 points at a seventh of its compute and approaching a teacher conditioned on gold solutions; trained on pooled unlabeled data, it transfers to held-out benchmarks, matching training methods that use gold labels. Analysis suggests that the improvements are not pure distribution sharpening: after training, the model solves problems it previously never solved in 32 attempts, and its majority vote itself becomes more accurate.

Figures

Figures reproduced from arXiv: 2607.13643 by Ivan Titov, John Gkountouras, Josip Juki\'c.

Figure 1
Figure 1. Figure 1: CANON training framework. Given an unlabeled prompt, the model samples N solutions and extracts the majority answer. A frozen snapshot of the model, conditioned on a consensus solution, serves as a dense per-token teacher; the student is trained on its own rollouts to follow the lead line set by their consensus, without seeing it. Gradients never flow into the teacher. A single generation pass provides bot… view at source ↗
Figure 2
Figure 2. Figure 2: Transductive AMC, main comparisons (Qwen3-4B-Instruct-2507). (a) Accuracy against measured wall-clock on identical hardware, including generation. CANON reaches within half a point of the gold-conditioned oracle in 1.2 GPU-hours; gold-reward GRPO needs roughly 11 hours to match it and 19 to surpass it, and TTRL plateaus below CANON at every budget. Filled markers and solid lines denote label-free methods; … view at source ↗
Figure 3
Figure 3. Figure 3: Held-out AIME 2024 during pooled label-free training. The RL-based methods climb slowly toward, and do not reach, the level CANON attains in a single epoch (dashed guide); TTRL re￾gresses past its peak checkpoint. Filled markers and solid lines denote label-free methods; open markers and dashed lines denote methods that use gold labels. one epoch. That a label-free method matches gold-supervised training o… view at source ↗
Figure 4
Figure 4. Figure 4: Where the gains land. Change in avg@32 by base-model difficulty band (AMC, transductive). Gains concentrate on the hardest prompts, including a +33.3-point pass@32 gain on never-solved prompts (hatched). never solved n=12 0-25% n=10 25-50% n=7 50-75% n=7 75-100% n=9 solved n=38 0 10 20 30 40 50 60 change in avg@32 (points) +51.9 +31.7 +13.4 +4.2 -0.7 +2.3 +33.3 pass@32 Further controls are provided in Appe… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

16 extracted references · 1 canonical work pages

  1. [1]

    On-policy distillation of language models: Learning from self- generated mistakes

    Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self- generated mistakes. InInternational Conference on Learning Representations, volume 2024, pages 21246–21263,

    2024

  2. [4]

    Prompts that enter the hardest bands partly through an unlucky draw would then show spurious gains under any re-evaluation, and the never-solved band can only move up

    A regression-to-the-mean control for the difficulty analysis.The difficulty bands of Figure 4 are assigned from the base model’s observed accuracy over 32 samples, and the same finite-sample estimate ordinarily serves as the baseline when computing gains. Prompts that enter the hardest bands partly through an unlucky draw would then show spurious gains un...

    2025

  3. [5]

    against the snapshot run. Snapshot Live teacher training loss≈0.150.00 generation entropy≈0.280.04 gated prompts (frac.) 1.0 0.0 responses at length cap 0.11 to 0.34 1.00 20 Consensus width and gating.The remaining two choices are matters of simplicity rather than necessity (values in Table 7). Using three consensus solutions instead of one, implemented a...

    2024

  4. [6]

    When the majority votes wrong, the intervention timing for test-time reinforcement learning hides in the extinction window.arXiv preprint arXiv:2605.19444,

    Hongxiang Lin, Zhirui Kuai, Erpeng Xue, and Lei Wang. When the majority votes wrong, the intervention timing for test-time reinforcement learning hides in the extinction window.arXiv preprint arXiv:2605.19444,

    Pith/arXiv arXiv

  5. [7]

    net/forum?id=tqTNOpkP5j

    URLhttps://openreview. net/forum?id=tqTNOpkP5j. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y . K. Li, Y . Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.CoRR, abs/2402.03300,

    Pith/arXiv arXiv

  6. [8]

    Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu

    URLhttps://doi.org/10.48550/ arXiv.2402.03300. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework.arXiv preprint arXiv: 2409.19256,

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.03300

  7. [9]

    Mingyang Song and Mao Zheng

    URL https://openreview.net/forum?id=am22IukDiKf. Mingyang Song and Mao Zheng. A survey of on-policy distillation for large language models.arXiv preprint arXiv:2604.00626,

    Pith/arXiv arXiv

  8. [10]

    Gates: Self-distillation under privileged context with consensus gating.arXiv preprint arXiv:2602.20574,

    Alex Stein, Furong Huang, and Tom Goldstein. Gates: Self-distillation under privileged context with consensus gating.arXiv preprint arXiv:2602.20574,

    arXiv

  9. [12]

    ISBN 979-8-89176-390-6

    Association for Computational Linguistics. ISBN 979-8-89176-390-6. doi: 10.18653/v1/2026.acl-long.1337. URLhttps://aclanthology. org/2026.acl-long.1337/. Zheng Yuan, Hongyi Yuan, Chengpeng Li, Guanting Dong, Keming Lu, Chuanqi Tan, Chang Zhou, and Jingren Zhou. Scaling relationship on learning mathematical reasoning with large language models,

    work page doi:10.18653/v1/2026.acl-long.1337 2026

  10. [13]

    on four-GPU nodes; wall-clock figures are reported for a single reference node with four RTX 6000 Ada GPUs (48 GB). Each model’s generation mode is fixed across train- ing and evaluation (Table 8), multiple-choice training on GPQA uses a 2,048-token prompt cap, and answers are extracted from\boxed{}with math-verify equivalence checking for mathematics and...

    2023

  11. [14]

    answer: X

    C PROMPTTEMPLATES ANDANSWEREXTRACTION Every rollout, in training and evaluation, is generated from a single-turn chat prompt with no sys- tem message: the user message is the problem followed by the standard suffix used by TTRL, wrapped with the tokenizer’s chat template (generation prompt appended; thinking disabled for hybrid-thinking models). The stude...

    2023

  12. [2020]

    Self-consistency improves chain of thought reasoning in language models

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171,

    Pith/arXiv arXiv

  13. [2023]

    doi: 10.18653/v1/2023.emnlp-main.67

    Association for Computational Linguis- tics. doi: 10.18653/v1/2023.emnlp-main.67. URLhttps://aclanthology.org/2023. emnlp-main.67/. Jonas Hübotter, Frederike Lübeck, Lejs Behric, Anton Baumann, Marco Bagatella, Daniel Marta, Ido Hakimi, Idan Shenfeld, Thomas Kleine Buening, Carlos Guestrin, et al. Reinforcement learn- ing via self-distillation.arXiv prepr...

    Pith/arXiv arXiv 2023

  14. [2024]

    The surprising effectiveness of test-time training for few-shot learning.arXiv preprint arXiv:2411.07279,

    Ekin Akyürek, Mehul Damani, Adam Zweiger, Linlu Qiu, Han Guo, Jyothish Pari, Yoon Kim, and Jacob Andreas. The surprising effectiveness of test-time training for few-shot learning.arXiv preprint arXiv:2411.07279,

    Pith/arXiv arXiv

  15. [2025]

    Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han

    URL https://openreview.net/forum?id=WJaUkwci9o. Jiaxin Huang, Shixiang Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve. In Houda Bouamor, Juan Pino, and Kalika Bali, editors,Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, pages 1051–1068, Singapore, December

    2023

  16. [2026]

    Let’s verify step by step

    Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. InInternational Conference on Learning Representations, volume 2024, pages 39578–39601,

    2024