{"id":"1a3225c2-e44c-4598-a7fd-e3aad5d36739","arxiv_id":"2608.06296","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A model self-improves on math reasoning by majority-voting its own rollouts, using the winning answer as a pseudo-solution, and distilling itself on the disagreeing rollouts, without any labeled data.","lead":"This paper introduces U-OPSD, a training method that lets a language model improve itself on math problems using only its own sampled answers, with no ground-truth labels or external teacher. In tests, it matches or beats supervised methods like OPSD and GRPO on several competition-math benchmarks, while using only unlabeled problems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline margins over OPSD may be inflated by selecting the best of six test-set checkpoints without seed replication; a fixed-protocol rerun is needed before the claim that consensus replaces gold supervision is accepted.","rationale":"The reader identifies pseudo-label accuracy as the weakest assumption, and that is a genuine limitation for generalization. However, the paper acknowledges it in Section 5, and the in-domain probe of 86.7% agreement suggests the mechanism is active on the training distribution. The more immediate threat to the specific headline claim is that the reported U-OPSD-over-OPSD margins are selected maxima over test checkpoints with no error bars. Because both methods are evaluated on the same test benchmarks and the best checkpoint is chosen per method, any difference in checkpoint-to-checkpoint variance or peak timing transfers directly into the reported margin. Figure 3 showing U-OPSD consistently above OPSD on AIME24 and AIME25 in non-thinking mode is evidence against a pure artifact, but the five-benchmark average and the 8B results could still shift under a fixed protocol. A rerun with seeds and validation-based selection is the minimal experiment that would settle whether the 3.2 and 2.3 point margins are genuine. If they survive, the central claim is supported; if not, the claim should be weakened to U-OPSD matching OPSD without labels. A pseudo-label SFT baseline would be a useful secondary mechanistic check, but the evaluation protocol is the primary load-bearing concern. I therefore keep the reader's conditional verdict unchanged, with the evaluation protocol as the key condition.","tokens_in":19964,"tokens_out":9596,"duration_ms":94357,"concrete_test":"Run U-OPSD and OPSD on Qwen3-4B and Qwen3-8B non-thinking mode with three independent seeds each, and report (i) the step-150 checkpoint average, (ii) the best checkpoint selected on a held-out validation split rather than on the five test benchmarks, and (iii) the per-seed margin with a confidence interval for U-OPSD minus OPSD on the five-benchmark average. If the validation-selected or step-150 margin is not consistently positive across seeds, the headline outperformance claim should be downgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that U-OPSD beats supervised OPSD by 3.2 and 2.3 points in non-thinking mode (Table 1). The protocol in Section 4.1 evaluates checkpoints every 25 steps up to 150 and reports the best score for each method on the same five test benchmarks. Taking a maximum over six correlated checkpoint evaluations is an upward-biased estimator of expected performance, and the bias is method-specific: a method whose accuracy keeps rising until step 150, as U-OPSD's AIME curves in Figure 3 suggest, receives a larger boost from best-checkpoint selection than one that peaks earlier. Section 5 explicitly states that seed-replicated error bars for every training setup are pending. Without seeds, the 3.2 and 2.3 point margins could be within run-to-run noise. The ablations in Figure 4 use the same best-checkpoint protocol, so the hyperparameter conclusions inherit the same issue. Thus the load-bearing premise that pseudo-solutions can replace gold solutions in the strong sense of outperforming OPSD is not yet securely established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes U-OPSD, an unsupervised variant of on-policy self-distillation that removes all external supervision from the teacher context. For each unlabeled prompt, the method draws G rollouts from a stop-gradient copy of the policy, majority-votes the parsed answers to form a pseudo-answer, selects the longest agreeing rollout as the pseudo-solution, and then minimizes a token-level forward-KL divergence between the teacher distribution conditioned on the pseudo-solution and the student distribution along the disagreeing rollouts. Experiments on five mathematical reasoning benchmarks with Qwen3-4B and Qwen3-8B in both thinking and non-thinking modes, plus two instruct models, report consistent gains over the base models and average gains over supervised OPSD of 3.2% and 2.3% in non-thinking mode, while roughly tying OPSD in thinking mode. The paper also reports ablations over the confidence threshold, rollout count, teacher update strategy, teacher reference choice, distillation target, and divergence computation strategy. Section 5 explicitly acknowledges the regime dependence of the gains, the dependence on base-model competence, and that seed-replicated error bars are pending.","tokens_in":20155,"tokens_out":5069,"duration_ms":46512,"significance":"If the empirical claims hold, U-OPSD is a meaningful conceptual advance: it shows that the privileged context in on-policy self-distillation need not be a ground-truth solution and that the model's own consensus can serve as the teacher reference. The method is simple, the objective in Eq. (5) is clear, and the comparisons against supervised OPSD, GRPO, SFT, and label-free RL baselines address the right counterfactuals. The paper also has real strengths in presentation: it states its limitations candidly, reports the pseudo-label agreement rate, and will release code. The self-referential training signal is not circular, because the central claim is evaluated on external benchmarks with known answers. However, the headline margins over OPSD rest on an evaluation protocol that selects the best checkpoint without seed replication, and the compute comparison between U-OPSD and OPSD is not fully specified; these issues must be resolved before the central claim is secure.","major_comments":[{"comment":"The main comparison uses an upward-biased checkpoint-selection protocol without error bars. Section 4.1 states that checkpoints are evaluated every 25 steps up to 150 and the best score is reported for U-OPSD and OPSD, while Section 5 explicitly states that seed-replicated error bars are pending. Taking the maximum over six correlated checkpoint evaluations inflates expected performance, and the inflation is method-specific when learning curves have different shapes; Figure 3 suggests U-OPSD is still climbing on AIME24 in non-thinking mode while other methods plateau. The headline claims of 3.2% and 2.3% average gains over OPSD in Table 1, as well as the ablation conclusions in Figure 4, cannot currently be distinguished from run-to-run noise. Please provide at least three seeds for the main U-OPSD and OPSD comparisons and report a preset checkpoint or a last-k average as the primary protocol, with best-checkpoint results retained only as a secondary analysis.","section":"Section 4.1 / Tables 1-2"},{"comment":"The mechanism's ceiling is measured on only 64 training prompts. The pseudo-label quality probe reports 86.7% agreement with gold answers and 13.3% wrong pseudo-labels; this is the load-bearing quantity for the claim that consensus can replace gold supervision, yet no error bar or breakdown is given, and the manuscript does not report how pseudo-label accuracy evolves during training or across the five evaluation distributions. Section 5 appropriately acknowledges the dependence on base-model competence, but the main tables would be materially stronger if pseudo-label accuracy were reported per benchmark and per checkpoint. Without this, the reader cannot tell whether the reported gains are driven by the intended self-correction mechanism or by a few unrepresentative high-consistency prompts.","section":"Section 4.3 / Section 5"},{"comment":"The cost-matched comparison between U-OPSD and OPSD is not fully specified. The text says U-OPSD uses G=8 rollouts per prompt with a maximum completion length of 4,096 tokens, while supervised OPSD draws a single rollout from each of 32 prompts per optimizer step with a maximum length of 1,024 tokens, but it does not state how many prompts U-OPSD uses per optimizer step. If U-OPSD is allowed more prompts, more tokens, or more total sampling per step, part of its advantage could be attributable to compute rather than to consensus-based conditioning. Please report prompts per step, total sampled tokens per step, and wall-clock time for both methods, and verify the claim that Tables 1 and 2 are cost-matched.","section":"Section 4.1 / Implementation details"}],"minor_comments":[{"comment":"The sentence 'we keep the teacher in thinking mode and distill its behavior into a student in non-thinking mode, while evaluating the resulting model in thinking mode' appears contradictory; please clarify whether the student is trained in thinking mode, non-thinking mode, or both, and how the evaluation mode is chosen.","section":"Section 4.1 / Thinking mode"},{"comment":"The algorithm and the objective do not specify how the agreeing reference y+ and the subset B^-_x of disagreeing rollouts are selected; the experiments later refer to 'longest-1' and Figure 5 explores longest/random/shortest. Please state the default selection rule directly in Algorithm 1 and in the text around Eq. (5).","section":"Algorithm 1 / Eq. (5)"},{"comment":"The divergence family D_beta is introduced as a generalized Jensen-Shannon divergence whose beta -> 0 limit is forward KL, but Table 5 labels beta=0.5 as JSD and beta=1 as reverse KL without explaining the mapping; please add a sentence defining the beta convention used in the table.","section":"Table 5 / Section 3.1"},{"comment":"The horizontal axis in Figure 3 runs from 25 to 100, while the text says checkpoints are evaluated every 25 steps up to 150; please make the axis range and the number of plotted checkpoints consistent with the protocol.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be of interest to the post-training and self-improvement community, and the conceptual claim is attractive. The main risk is not the method's soundness but whether the reported margins survive a fixed-protocol evaluation with seed replication; the authors already acknowledge this in Section 5, so I would frame the revision as a verification task rather than a redesign. I would also ask the editor to ensure that the OPSD baseline is evaluated under exactly the same checkpoint rule as U-OPSD, and that the compute comparison is made explicit."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea is new and simple: let the model vote on its own rollouts, use the longest agreeing rollout as the teacher's privileged context, and distill only on the disagreeing rollouts. That removes the gold solution from OPSD, and it works on competition math. In non-thinking mode, average gains over base are 8.5 and 10.7 points at 4B/8B, and it edges supervised OPSD by about 3 points. The gains are consistent across five benchmarks, and Figure 3 shows U-OPSD above the pack at every checkpoint on AIME24 and AIME25, which is a stronger claim than a single best-checkpoint number.\n\nWhat's good: the method is clearly presented, the ablations are thoughtful (teacher reference vs. distillation target, divergence type, token budget), and the limitations section is honest about the regime-dependence and the base-model competence ceiling. They even state that the default tau=0.5 is not the best in their sweep and that G=12 beats G=8. That is a sign of careful reporting.\n\nThe main weakness is the evaluation protocol. They report the best of six test-set checkpoints for U-OPSD and OPSD, and peak performance for GRPO. That is an upward-biased estimate, and the bias can differ across methods. The paper states seed-replicated error bars are pending. So the 3.2 and 2.3 point margins over OPSD could move. That said, the training curves mitigate this on AIME24 and AIME25, so I do not think the main conclusion collapses; it just needs a fixed protocol and seeds before we trust the exact margins.\n\nSecond soft spot: there is no pseudo-label SFT baseline. SFT on the majority-vote pseudo-solutions would isolate how much of the gain comes from dense distillation on disagreeing rollouts versus merely imitating the pseudo-label. The label-only ablation in Figure 5 partially addresses this by showing that conditioning on just the answer fails, but that is not the same as SFT on the full pseudo-solution.\n\nThird: the hyperparameter sweeps use the same best-checkpoint protocol, so the conclusions about tau and G inherit the same uncertainty. The fact that the default tau is not optimal is awkward but not fatal.\n\nOverall: a solid paper with a real idea and honest reporting. The central claim—consensus can serve as a teacher reference in on-policy self-distillation—is plausible and supported by consistent gains, but the evaluation protocol needs hardening. It deserves peer review, with a request for seed-replicated error bars, a fixed checkpoint protocol, and a pseudo-label SFT baseline. I would cite it if I worked on self-distillation.","headline":"U-OPSD is a genuinely unsupervised distillation recipe with a real idea, but the headline margins over supervised OPSD rest on best-checkpoint selection and no seeds; the claim is plausible, not yet secure.","tokens_in":20718,"tokens_out":2031,"would_cite":true,"duration_ms":17380,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On-policy self-distillation can work with no ground-truth labels: the model's own majority-vote consensus serves as the teacher, and the label-free variant beats supervised OPSD by up to 3.2% on five math benchmarks.","keywords":["on-policy self-distillation","unsupervised post-training","self-consistency","majority vote","knowledge distillation","mathematical reasoning","large language models","label-free training"],"falsifier":"Measure the gold-agreement of majority-vote pseudo-labels on a labeled holdout from the training distribution; if agreement falls well below the 86.7% reported on the 64-prompt probe, final accuracy should stop improving and eventually fall below the base model, which would falsify the claim that consensus alone can carry the distillation.","tokens_in":19752,"feed_emoji":"🧮","tokens_out":9474,"duration_ms":75211,"temperature":0.7,"pith_summary":"This paper claims that on-policy self-distillation of a large language model does not need any ground-truth solution, external feedback, or a stronger teacher. The proposed method, U-OPSD, samples several rollouts per problem, takes the majority-voted final answer as a pseudo-solution when a confidence threshold is met, and then distills the model's own solution-conditioned next-token distribution into the student along the rollouts that disagreed with the vote. On five competition-math benchmarks this label-free procedure improved Qwen3-4B and Qwen3-8B by 8.5% and 10.7% over the base model in non-thinking mode, and outperformed the supervised OPSD baseline by 3.2% and 2.3%. The resulting picture is that the binding constraint for this kind of self-improvement is not access to gold solutions but the machinery for surfacing and correcting the model's own inconsistencies.","feed_headline":"A model's own majority vote can replace gold answers in self-distillation","feed_subtitle":"U-OPSD lifts Qwen3 math scores by up to 10.7% with no labels, matching or beating supervised OPSD and GRPO.","key_machinery":"The load-bearing mechanism is consensus as an endogenous privileged context. U-OPSD samples $G$ rollouts from a stop-gradient copy of the policy, parses their final answers, and lets a majority vote fix a pseudo-answer; when the winning fraction crosses $\\tau = 1/2$ and at least one rollout disagrees, the longest agreeing rollout $y^+$ replaces the ground-truth solution as the teacher reference. The loss is a per-token divergence $D_\\beta(\\bar{\\pi}(\\cdot | x, y^+, y^-_{<n}) \\| \\pi_\\theta(\\cdot | x, y^-_{<n}))$ summed over the prefixes of the disagreeing rollouts, where $D_\\beta$ is the generalized Jensen-Shannon divergence whose $\\beta \\to 0$ limit is the forward KL divergence. This turns self-consistency into dense corrective supervision at precisely the positions where the model's own samples contradict its consensus.","core_discovery":"The central claim is that the ground-truth solution in on-policy self-distillation can be replaced by the model's own consensus, and that this replacement is not a degradation but often an improvement. Concretely, U-OPSD draws $G$ independent rollouts from a stop-gradient copy of the policy, extracts and canonicalizes their final answers, and takes the plurality answer $\\tilde{a}(x)$ as the pseudo-answer. If the winning fraction passes threshold $\\tau = 1/2$ and at least one rollout disagrees, the longest agreeing rollout $y^+$ is used as the teacher reference in place of $y^\\star$, and the model is trained to match the teacher's next-token distribution, conditioned on $(x, y^+)$, along prefixes of the disagreeing rollouts. The same checkpoint plays teacher and student, so no external signal enters; the paper reports that this unsupervised variant matches or surpasses supervised OPSD, SFT, and GRPO across five math benchmarks in both thinking and non-thinking modes.","pith_inferences":["As an editorial extension, the vote margin could be converted from a binary threshold into a per-prompt confidence weight, so that low-margin consensus contributes less gradient; this is directly testable with the existing code.","Soft consensus, for example embedding similarity instead of exact-match answers, would extend the same machinery to open-ended generation, where canonicalizable final answers are unavailable.","The reported 86.7% pseudo-label gold-agreement suggests a practical diagnostic: check pseudo-label accuracy on a small labeled holdout before training, since the method's ceiling is bounded by that accuracy."],"forward_implications":["Post-training no longer requires labeled solutions: any unlabeled problem set with extractable answers can supply both the teacher reference and the student trajectories.","Consensus-derived pseudo-solutions are more effective as a conditioning context for token-level distillation than as a scalar reward: label-free RL baselines under the same budget gained at most 1.5%, while U-OPSD gained 8.5-10.7% in non-thinking mode.","Improvement concentrates on the competence frontier: prompts with reliable votes but residual disagreement receive the correction, so the curriculum is set by the model's own vote statistics.","In strong thinking-mode models the gains shrink to a tie with supervised OPSD, indicating ceiling effects rather than a failure of the unsupervised signal."],"supporting_citations":[{"why":"Supplies the OPSD recipe that U-OPSD modifies and the supervised baseline it is compared against.","marker":"[Zhao et al., 2026a]"},{"why":"Supplies the majority-vote self-consistency mechanism used to form pseudo-answers.","marker":"[Wang et al., 2023]"},{"why":"Supplies GRPO, the group-relative sparse-reward baseline paired with gold answers.","marker":"[Shao et al., 2024]"},{"why":"Supplies the on-policy distillation objective with full-vocabulary logit divergence that U-OPSD adopts.","marker":"[Agarwal et al., 2024]"},{"why":"Supplies the on-policy distillation formulation and the rationale for dense token-level supervision.","marker":"[Gu et al., 2024]"},{"why":"Supplies the Qwen3 base models used in all experiments.","marker":"[Yang et al., 2025]"},{"why":"Supplies the OpenThoughts problem set whose unlabeled statements U-OPSD trains on.","marker":"[Guha et al., 2025]"},{"why":"Supplies the MATH500 benchmark used in evaluation.","marker":"[Hendrycks et al., 2021, Lightman et al., 2024]"}],"fun_headline_variants":["Majority vote replaces gold answers in self-distillation","Self-distillation without labels: majority vote suffices","Model consensus matches supervised self-distillation on math","Unsupervised self-distillation: internal consistency is enough"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a majority vote over the model's own rollouts produces a pseudo-solution that is correct often enough (the paper's 64-prompt probe found 86.7% agreement with gold answers) to serve as a teacher reference, and that this accuracy holds on the training distribution.","fun_headline_variants_meta":{"raw":{"variants":["Majority vote replaces gold answers in self-distillation","Self-distillation without labels: majority vote suffices","Model consensus matches supervised self-distillation on math","Unsupervised self-distillation: internal consistency is enough"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000302,"raw_usage":{"total_tokens":1831,"prompt_tokens":1125,"completion_tokens":706,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":741,"completion_tokens_details":{"reasoning_tokens":643}},"tokens_in":741,"tokens_out":706,"duration_ms":6451,"temperature":1.0,"reasoning_tokens":643,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:12:55.129308+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the gold-agreement of majority-vote pseudo-labels on a labeled holdout from the training distribution; if agreement falls well below the 86.7% reported on the 64-prompt probe, final accuracy should stop improving and eventually fall below the base model, which would falsify the claim that consensus alone can carry the distillation.","supporting_citations":[],"review_version":2}