{"id":"80c8a139-110f-4dc8-ab6b-483f0d4438a2","arxiv_id":"2506.06395","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"RLSC, a method that uses a language model's self-confidence as reward, is shown to improve math benchmark accuracy, but the results are undermined by training on the AIME test set and the method reduces to known self-distillation.","lead":"This paper proposes RLSC, a reinforcement learning method that uses a language model's own output probability as the reward signal, requiring no labels or external reward models. The authors report large accuracy gains on math benchmarks from only 10 or 20 training steps, but the evaluation appears to train on the same AIME exam used for testing.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The loss in Eqs. (5)-(7) does not maximize the claimed self-confidence objective: it is self-distillation to p_old, its gradient vanishes at initialization, and the reported gains cannot be attributed to RLSC as derived.","rationale":"The paper's central claim is that RLSC improves reasoning by using the model's own confidence as a reward, formalized as maximizing F(p_θ)=Σ p_θ^2. The derivation in Eqs. (3)-(5) is internally inconsistent: the gradient formula ignores the dependence of the expectation on p_θ and substitutes a frozen p_old, and the resulting loss L1 is standard self-distillation. At the starting point p_θ=p_old, the gradient vanishes, so no learning occurs without the α term; with α, the update is a uniformizing regularizer, not a confidence sharpener. This is not a matter of disagreement with community consensus; it is a mathematical error that invalidates the claimed mechanism. The reader's AIME24 train/eval overlap concern is real and would invalidate the flagship benchmark, but it is secondary because even clean generalization results on MATH500 etc. would not establish the paper's mechanism if the loss does not implement it. I agree with the reader's REJECT verdict, though on a different and more fundamental ground. I do not question the integrity of the authors; the concern is that the described algorithm cannot do what is claimed. No machine-checked proofs or reproducible code are provided to rescue the derivation. A toy categorical simulation would settle the matter in minutes without training a 7B model.","tokens_in":9797,"tokens_out":5401,"duration_ms":56304,"concrete_test":"Analytical check on a toy categorical distribution: set p_old=(0.6,0.3,0.1) and initialize p_θ=p_old. With α=0, compute the L1 gradient at this point; it is exactly 0, so no update occurs. With α=0.1, one L2 step moves p_θ toward uniform, decreasing Σp^2 from 0.46 toward 0.38, i.e., reducing confidence. Independently re-derive Eq. (4) from Eq. (3) using the chain rule: ∇F=2E_{y~p_θ}[p_θ(y)∇log p_θ(y)], which differs from the paper's Eq. (4) by a factor of 2 and by placing p_θ inside the expectation. If the authors' code instead computes the loss on current-model log-probabilities without detaching p_θ, that is a different algorithm and must be stated; otherwise, the claimed objective is provably not optimized.","verdict_should_be":"REJECT","load_bearing_attack":"Section 2.1 defines F(p_θ)=Σ_y p_θ(y)^2 (Eq. 3) and claims maximizing it 'sharpens' the distribution. The gradient in Eq. (4) is incorrect: ∇F=2Σ_y p_θ(y)∇p_θ(y)=2E_{y~p_θ}[p_θ(y)∇log p_θ(y)], not E_{y~p_old}[p_old(y)∇log p_θ(y)]. With p_old frozen, the proposed L1=-Σ_y p_old(y)log p_θ(y) (Eq. 5) has gradient -Σ_y p_old(y)∇log p_θ(y); at initialization p_θ=p_old this equals -Σ_y∇p_θ(y)=0. Thus no parameter update occurs without the α term. With α>0, the L2 gradient at p_θ=p_old is -αΣ_y(1/p_old(y))∇p_θ(y), which is the gradient of a uniformizing entropy regularizer, not of mode sharpening. Minimizing L1 is equivalent to minimizing KL(p_old||p_θ), whose minimizer is p_θ=p_old. Therefore the procedure cannot increase Σ_y p_θ(y)^2 beyond the teacher's value; 'mode sharpening' is not implemented. Since this mechanism is the central contribution, the large benchmark gains, even if numerically reproducible, must stem from another component (e.g., format adaptation, evaluation mismatch, or the AIME24 train/eval overlap flagged by the reader), not from self-confidence RL. This is an internal inconsistency in the derivation, independent of benchmark leakage.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Reinforcement Learning via Self-Confidence (RLSC), a fine-tuning method that uses the model's own output probability distribution as a reward signal, with the stated goal of 'sharpening' the distribution toward its mode and thereby eliminating the need for labels, preference models, or reward engineering. The authors derive a self-confidence objective F(p_theta) = sum_y p_theta(y|x)^2, propose losses L1 and L2 based on a frozen copy of the model, and apply the method to Qwen2.5-Math-7B using only 16 samples per question and 10 or 20 training steps on an AIME2024 dataset. They report large accuracy gains on AIME24, MATH500, Minerva Math, Olympiadbench, and AMC23, and claim that the model learns concise reasoning without prompting. The paper also includes qualitative examples and a short ablation deferral.","tokens_in":10125,"tokens_out":7660,"duration_ms":72195,"significance":"If the method worked as described, it would be a noteworthy contribution: zero-label, low-compute post-training for reasoning models, with a simple differentiable objective replacing handcrafted rewards or majority-vote pseudo-labeling. The paper is transparent about its loss and training setup, and re-evaluating baselines with a common public script is good practice. However, the central gradient derivation is mathematically incorrect, so the implemented loss does not maximize the stated self-confidence objective; the reported gains therefore cannot be attributed to the proposed mechanism. Additionally, the AIME24 training/evaluation overlap and an unreliable answer label in a qualitative example further undermine the empirical claims. The idea of deriving self-training from ensemble agreement is appealing, but as presented the central claim is not supported and would require a corrected derivation and substantially clarified evaluation before a fresh assessment.","major_comments":[{"comment":"Equation (4) is not the gradient of F(p_theta) = sum_y p_theta(y)^2. The correct gradient is nabla F = 2 sum_y p_theta(y) nabla p_theta(y) = 2 E_{y~p_theta}[p_theta(y) nabla log p_theta(y)], whereas Eq. (4) uses E_{y~p_old}[p_old(y) nabla log p_theta(y)] with p_old frozen. Consequently, L1 in Eq. (5) is the cross-entropy from p_old to p_theta; its minimizer is p_theta = p_old, and at initialization its gradient is exactly zero. Adding the alpha term in L2 introduces a maximum-entropy regularizer -alpha sum_y log p_theta, which pushes the distribution toward uniform rather than toward the mode. None of these losses implements the 'mode sharpening' objective that the paper's central claim rests on, so the reported gains cannot be attributed to RLSC as derived.","section":"§2.2, Eqs. (4)–(7)"},{"comment":"The training set is described in §2.3 as 'the AIME2024 dataset' and the evaluation benchmark in §3.1 is 'AIME24 [14]', with no train/test split stated; reference [14] is the same NuminaMath corpus in both places. If the 16 samples per question are generated from the same problems that appear in the AIME24 evaluation, then the reported +13.4% improvement on AIME24 reflects memorization or overlap rather than generalization. The paper must state explicitly whether the AIME24 evaluation problems are disjoint from the training questions and, if not, report results on a held-out split.","section":"§2.3 and §3.1"},{"comment":"The qualitative example in Appendix Case 1 labels the RLSC output 'Final Answer (Correct): 2', but for a = 2 the piecewise function does not intersect y = 2017 twice (the second-branch intersection x = 1006.5 lies outside x < a), so this answer appears incorrect. This indicates that the evaluation pipeline does not reliably verify answers, which calls into question the accuracy numbers in Table 2 and the 'emergent behavior' claims in §3.2.","section":"§3.3 / Appendix Case 1"},{"comment":"The text says the original Qwen model 'often fails to function properly' under direct evaluation. If the baseline's outputs are not in the expected format and are counted as wrong by the shared evaluation script, the large deltas may be largely due to format adaptation rather than improved reasoning. The paper should provide a breakdown of parse failures versus mathematical errors for the baseline and the tuned model to support the attribution of the improvements to RLSC.","section":"§3.1, Table 2"}],"minor_comments":[{"comment":"The author affiliation 'Skotech' should be 'Skoltech'.","section":"Title page"},{"comment":"'massage templates' should be 'message templates'.","section":"§1, Contribution 3"},{"comment":"The text contains a typo, 'absense' should be 'absence'.","section":"§2.3"},{"comment":"The delta entries for GSM8K and MMLU are written as '2.0' and '5.3' without the '+' sign used elsewhere in the row; please make the formatting consistent.","section":"Table 2"},{"comment":"Table 1 is difficult to read: the 'Loss function' and 'Functional' columns appear garbled (e.g., 'RLHF lossp old logpE pθ [pθ]'); please reformat the table so each row clearly associates a loss with its optimized functional.","section":"Table 1"},{"comment":"The pseudocode contains formatting issues, including broken line breaks inside variable names and missing indentation around 'loss.backward()' and 'optimizer.step()', which make the algorithm harder to follow.","section":"Algorithm 1"},{"comment":"References [14] and [15] are identical; the paper uses [14] for both the training corpus and the AIME24 evaluation set, which should be clarified in the text.","section":"References"}],"recommendation":"reject","confidential_remarks":"I agree with the rejection recommendation. The gradient derivation error in Eq. (4) is load-bearing: the implemented loss does not maximize the stated self-confidence objective, so the paper's central mechanism is not what is being tested. The AIME24 train/eval overlap and the mislabeled qualitative example are additional red flags that would need to be resolved. If the authors correct the derivation and re-run the experiments with a disjoint evaluation split, the underlying idea might merit a fresh review, but the current manuscript does not support its claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the headline results are not supported by the paper's own math, and the AIME evaluation looks like it trains and tests on the same problems. I would not trust the numbers.\n\nWhat's genuinely nice: the paper is clearly written, the intuition that majority voting sharpens the model's distribution is reasonable, and the experiments are impressively cheap (16 samples, 10 steps). A label-free, reward-free post-training recipe that actually worked would be useful.\n\nBut the derivation doesn't hold. Equation (4) miscomputes the gradient of F(p_θ)=Σ p_θ(y)^2. The correct gradient is E_{y~p_θ}[p_θ(y)∇log p_θ(y)], not E_{y~p_old}[p_old(y)∇log p_θ(y)]. The loss L1 = −Σ p_old log p_θ is exactly self-distillation; at initialization its gradient is zero, so without the α term there is no update. Minimizing L1 drives p_θ toward p_old, not toward a sharper distribution. The α variant adds a uniform floor, which is a different objective. The central mechanism is thus internally inconsistent with the equations.\n\nThe experimental problems match. The paper trains on the AIME2024 dataset and evaluates on AIME24 without stating a split; the training set likely contains the same problems, which makes the +13.4% AIME gain uninterpretable. The qualitative Case 1 in the appendix shows an answer labeled correct that is not correct for the stated problem, which raises real doubts about the evaluation pipeline.\n\nThese are load-bearing flaws. The core claim — that RLSC improves reasoning by maximizing self-confidence — is not established by either the math or the experiments. I don't see a quick fix; the method would need to be rederived and the evaluation rerun with a clean split.\n\nWho would benefit from reading this? Someone studying self-distillation in LLMs might find it a curious data point, but it is not a reliable source. I'd desk reject it, not send it to referees. If you want a cautionary example for a reading group on derivation errors, it works, but I wouldn't cite it.","headline":"The headline gains are not supported by the paper's own math: the loss is self-distillation, not self-confidence RL, and the AIME evaluation likely leaks the training set.","tokens_in":10717,"tokens_out":5491,"would_cite":false,"duration_ms":52434,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Using a language model's own confidence as the reward, RLSC lifts math accuracy by 9–22 points with no labels and only 10–20 training steps.","keywords":["reinforcement learning","self-confidence","zero-label learning","mode sharpening","language model post-training","mathematical reasoning","few-shot RL"],"falsifier":"Train RLSC on the AIME2024 set and evaluate only on held-out AIME problems that are provably absent from the training questions; if the +13.4% gain evaporates, the result is memorization rather than mode sharpening. A second decisive test is to apply RLSC to a model whose confidence is adversarially miscalibrated — for example, one that is overconfident on plausible-looking distractors — and check whether accuracy rises along with confidence.","tokens_in":9585,"feed_emoji":"🧮","tokens_out":11741,"duration_ms":99653,"temperature":0.7,"pith_summary":"RLSC is the claim that post-training for reasoning can be driven entirely by the model's own self-confidence: the reward is the probability the model assigns to its own sampled answers, so no labels, preference models, or handcrafted reward functions are needed. The paper derives this objective from Test-Time RL's majority voting, showing that voting is equivalent to sharpening the output distribution toward its mode, and turns that into a differentiable loss. With just 16 generations per question and 10–20 optimizer steps on Qwen2.5-Math-7B, the method reports accuracy gains of roughly 10–22 points across five competition-math benchmarks. If the claims hold, post-training becomes dramatically cheaper and simpler, and a pretrained model's internal signal may be enough to improve itself.","feed_headline":"Self-confidence as reward lifts math accuracy by up to 22%","feed_subtitle":"No labels, no preference models, no reward design: 16 samples per question and a few training steps beat five math benchmarks.","key_machinery":"The central object is the self-confidence functional $F(p_\\theta) = \\sum_y p_\\theta(y|x)^2$, the probability that two i.i.d. samples from the model's output distribution coincide. The paper calls the operation that maximizes it 'mode sharpening': it concentrates probability mass on the most likely response, which is the same effect majority voting achieves, but expressed as a differentiable objective. The load-bearing identity is the log-trick gradient $\\nabla_\\theta F = \\mathbb{E}_{y \\sim p_{\\mathrm{old}}}[p_{\\mathrm{old}}(y|x)\\nabla_\\theta \\log p_\\theta(y|x)]$, which converts the functional into a weighted maximum-likelihood loss $L_1 = -\\sum_y p_{\\mathrm{old}}(y|x)\\log p_\\theta(y|x)$; the smoothed variant $L_2 = -\\sum_y(p_{\\mathrm{old}}(y|x)+\\alpha)\\log p_\\theta(y|x)$ adds a constant to stabilize optimization. Because the weighting probabilities come from a frozen copy of the model, the reward is entirely self-generated, and gradients flow only into the updated model's log-likelihoods.","core_discovery":"The paper's central claim is that maximizing the self-confidence functional $F(p_\\theta) = \\mathbb{E}_{y \\sim p_\\theta(y|x)}[p_\\theta(y|x)] = \\sum_y p_\\theta(y|x)^2$ is a sufficient training objective for improving reasoning accuracy in a strong pretrained model. This functional is the collision probability of two independent samples from the model's output distribution, and the paper argues that maximizing it is exactly what majority-voting pseudo-labeling does implicitly: it sharpens the distribution toward its mode. The gradient identity $\\nabla_\\theta F = \\mathbb{E}_{y \\sim p_{\\mathrm{old}}}[p_{\\mathrm{old}}(y|x)\\nabla_\\theta \\log p_\\theta(y|x)]$, obtained with a frozen copy $p_{\\mathrm{old}}$, yields the reward-weighted loss $-\\sum_y p_{\\mathrm{old}}(y|x)\\log p_\\theta(y|x)$, optionally smoothed by a constant $\\alpha$. Trained for 10–20 steps on the AIME2024 set with 16 samples per question, the Qwen2.5-Math-7B model gains +13.4 points on AIME24, +21.2 on MATH500, +21.7 on Minerva Math, +20.8 on Olympiadbench, and +9.7 on AMC23; the paper reports the same pattern, more weakly, at the 1.5B scale and attributes the effect to mode sharpening producing more concise and direct reasoning.","pith_inferences":["Editorial: if RLSC transfers beyond mathematics, it suggests a general post-training recipe — maximize the collision probability of the model's own output distribution — which would turn self-consistency from a decoding-time heuristic into a training signal for code, factoid QA, and other tasks where the mode is often correct.","Editorial: the method's validity depends on the pretrained model's confidence being correlated with correctness. A direct way to test this is to apply RLSC to a deliberately overconfident or adversarially miscalibrated model; if accuracy fails to rise along with confidence, the reported gains may be a calibration artifact rather than genuine reasoning improvement.","Editorial: because the paper trains on the AIME2024 dataset and evaluates on AIME24 without stating a train/test split, the headline +13.4% number should be read cautiously until the overlap is checked; a cleaner design trains on one competition set and evaluates on another, disjoint one.","Editorial: the self-confidence functional is estimated from only 16 Monte-Carlo samples per question, so the gradient is noisy; the smoothing constant $\\alpha$ may be stabilizing the optimization more than the paper's preliminary ablation suggests, and a full sweep over sample counts and $\\alpha$ would clarify how much of the robustness is built-in versus tuned."],"forward_implications":["Post-training for reasoning no longer needs human labels, preference models, verifiers, or reward engineering; the frozen model's own output probabilities are the reward signal.","Very small compute budgets suffice: 16 samples per question and 10–20 optimizer steps on a single training set transfer to gains on AIME24, MATH500, Minerva Math, Olympiadbench, and AMC23, with smaller gains on GSM8K and MMLU-Stem.","RLSC removes the preprocessing burden of TTRL, which needed 64 samples per question and a clean separation of answer from reasoning trace for majority-vote pseudo-labels; RLSC needs only the model's token log-probabilities.","Fine-tuned models become more concise: the paper observes the RLSC model answering correctly with shorter, more direct reasoning, without any 'think step by step' prompting, though it does not formally quantify the length reduction."],"supporting_citations":[{"why":"Supplies the Test-Time RL majority-voting baseline that RLSC's self-confidence objective is derived from and compared against.","marker":"[24]"},{"why":"Technical report for the Qwen2.5-Math base model used in all experiments.","marker":"[22]"},{"why":"NuminaMath dataset, the source of the AIME2024 training questions used for the 10–20 fine-tuning steps.","marker":"[14]"},{"why":"Provides the public evaluation script used to re-evaluate baseline and RLSC checkpoints under identical settings.","marker":"[6]"},{"why":"The training-verifiers paper referenced alongside [6] as the common evaluation protocol for math benchmarks.","marker":"[5]"},{"why":"The MATH dataset publication from which the MATH500 evaluation set is drawn.","marker":"[11]"},{"why":"Olympiadbench, one of the central evaluation benchmarks where RLSC reports a +20.8 gain.","marker":"[9]"},{"why":"Minerva Math, the benchmark where RLSC reports its largest gain at +21.7 points.","marker":"[12]"},{"why":"DeepSeek-R1, the reinforcement-learning-for-reasoning line that motivates the RL framing and supplies the GPQA-Diamond benchmark.","marker":"[7]"}],"fun_headline_variants":["Self-confidence reward makes math models 22% sharper","Few-shot RL with self-confidence beats five math benchmarks","Train math models with self-confidence, no labels needed","Model's own confidence as reward: +22% on math tests","RLSC: self-confidence reward, few samples, big math gains"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire method rests on the assumption that a pretrained model's self-assigned probability on a sampled answer is a trustworthy reward signal — that sharpening the model's confidence moves it toward correct answers rather than merely toward overconfident ones.","fun_headline_variants_meta":{"raw":{"variants":["Self-confidence reward makes math models 22% sharper","Few-shot RL with self-confidence beats five math benchmarks","Train math models with self-confidence, no labels needed","Model's own confidence as reward: +22% on math tests","RLSC: self-confidence reward, few samples, big math gains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000299,"raw_usage":{"total_tokens":1769,"prompt_tokens":1026,"completion_tokens":743,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":642,"completion_tokens_details":{"reasoning_tokens":661}},"tokens_in":642,"tokens_out":743,"duration_ms":8652,"temperature":1.0,"reasoning_tokens":661,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:17:08.560260+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train RLSC on the AIME2024 set and evaluate only on held-out AIME problems that are provably absent from the training questions; if the +13.4% gain evaporates, the result is memorization rather than mode sharpening. A second decisive test is to apply RLSC to a model whose confidence is adversarially miscalibrated — for example, one that is overconfident on plausible-looking distractors — and check whether accuracy rises along with confidence.","supporting_citations":[{"cited_title":"Training verifiers to solve math word problems, 2021","cited_arxiv_id":null,"evidence_quote":"The training-verifiers paper referenced alongside [6] as the common evaluation protocol for math benchmarks."},{"cited_title":"Evalchemy: Automatic evals for llms, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the public evaluation script used to re-evaluate baseline and RLSC checkpoints under identical settings."}],"review_version":1}