{"id":"d85cab08-a0a5-478c-8501-9572c06529c4","arxiv_id":"2506.08745","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CoVo trains LLMs with a self-generated reward based on the consistency and volatility of intermediate reasoning states, matching supervised RL performance without external labels.","lead":"This paper proposes CoVo, a training method that lets a language model score its own reasoning by checking whether its intermediate steps consistently point to the same final answer. If it works, AI models could improve their reasoning without needing human-labeled or externally verified answers.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core assumption that consistency/volatility separates correct from incorrect trajectories is validated only at initialization; the paper never shows this separation survives training under the changing policy, so the self-reward may decouple from correctness.","rationale":"The reader's weakest-assumption analysis and my stress-test converge on the same load-bearing point: the entire self-rewarding mechanism depends on consistency/volatility remaining predictive of correctness as the policy changes. The paper convincingly demonstrates the base-model separation (Table 2, Figs. 2 and 4), and the proposed vector aggregation is a reasonable way to combine the two features, but the reward is computed from the training policy's own likelihoods at each step. Since the policy is the object being optimized, there is no guarantee the separation survives; a policy that becomes uniformly more confident could make wrong trajectories look consistent and low-volatility. Figure 5 gestures at 'reward accuracy' during training, but the definition is absent and no checkpoint-level feature statistics are shown, so it does not close the gap. The theoretical section does not help: Proposition 3 assumes the reward ranks correct answers highest and rarely misclassifies incorrect ones, which is precisely the property needing empirical verification. I do not see a fatal flaw in the experimental setup, and the method is plausible, but the central claim is conditional on a measurable persistence property. Since the reader already issued CONDITIONAL for essentially this reason, the verdict should remain unchanged: accept only if the persistence check is added and passes. I credit the paper for the base-model analysis and for reporting feature distributions across four domains, but those do not cover the training-time regime.","tokens_in":27738,"tokens_out":6940,"duration_ms":86371,"concrete_test":"Freeze CoVo training checkpoints at t=0, 100, 200, 300, 400, and 500 for Qwen2.5-3B-Instruct (or Llama3.2-3B-Instruct). On a held-out labeled set (e.g., 200 MATH-500 prompts plus 200 MMLU-Pro prompts), sample 16 trajectories per prompt with each frozen policy, compute the CoVo intrinsic reward r_int from Eq. (9) per answer group, and measure (i) the AUC of r_int for distinguishing correct from incorrect groups and (ii) the d-prime separation of the consistency/volatility features between correct and incorrect trajectories. If AUC or d-prime decays materially (e.g., AUC falls below about 0.6) while the RL reward still rises, the reward has decoupled from correctness and the central claim fails. Report the identical checkpoint curves for the majority-voting reward for comparison.","verdict_should_be":"UNCHANGED","load_bearing_attack":"CoVo's reward is computed from Eq. (1), where distances use the current policy pi_theta, and consistency and volatility are derived from those distances. The empirical support for the reward-correctness link (Table 2, Fig. 2, Fig. 4) is established on the base model, with ground truth used only to stratify trajectories. During training, pi_theta changes, and the reward is recomputed from the updated policy's own likelihoods. Nothing in the paper demonstrates that the separation between correct and incorrect trajectories in consistency/volatility space persists under optimization. If the policy becomes globally more confident, wrong trajectories can also acquire high consistency and low volatility relative to alternatives, causing r_int to rank answer groups incorrectly even while the RL objective rises. Figure 5 is invoked as evidence against reward hacking, but it reports an unspecified 'reward accuracy,' does not define its computation, shows no checkpoint-wise feature distributions, and covers a single model; it cannot establish persistence. Moreover, the formal convergence result (Prop. 3) simply assumes as premises that the CoVo reward ranks y_gamma highest and has low misclassification probability; it does not prove these properties hold for the trained policy. Thus the central claim is conditional on an unverified persistence assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CoVo, a self-rewarding reinforcement learning method for LLM reasoning. It defines two trajectory-level features—consistency (the fraction of intermediate states whose likelihood is closest to the trajectory's own final answer) and volatility (the normalized position of the last state that deviates from the own answer)—computed from a distance matrix based on the policy's token log-likelihoods. Trajectories are grouped by final answer, and an intrinsic reward is computed via linear or vector aggregation of these features; a curiosity bonus based on average token log-probability and a KL penalty is added to encourage exploration. The policy is optimized with Reinforce++. Experiments on three model families (Llama3.2-3B, Qwen2.5-3B, Qwen2.5-7B) across math, commonsense, and science benchmarks report that CoVo matches or exceeds rule-based supervised RL baselines, with additional diversity analyses and ablations. Theoretical sections provide propositions on model collapse of majority voting, a variational interpretation, and a convergence bound.","tokens_in":28004,"tokens_out":7362,"duration_ms":75401,"significance":"If the empirical claims hold, CoVo offers a meaningful step toward label-free RL for reasoning: it uses intermediate reasoning structure rather than only final-answer aggregation, potentially reducing some forms of reward hacking, and it is demonstrated on multiple model sizes with released code. The paper also includes careful ablations of reward components and an empirical analysis of trajectory patterns. However, the current evidence is not yet conclusive: the main comparisons lack statistical uncertainty, the training data selection excludes a potentially substantial subset of prompts, the curiosity reward's KL penalty is ill-specified in pseudocode, and the persistence of the consistency-volatility separation under the evolving policy is not established. These gaps are fixable but require additional empirical analysis and clarification.","major_comments":[{"comment":"Table 1 reports single runs without error bars or multiple seeds. Many differences against supervised RL baselines are within 0.2–0.4 percentage points (e.g., Llama3.2-3B MATH-500: CoVo 51.2 vs GRPO 51.8; Qwen2.5-3B GSM8K: CoVo 88.7 vs RLOO 89.1; Qwen2.5-7B MATH-500: CoVo 78.4 vs Reinforce++ 78.2). The headline claim 'comparable to or even surpassing supervised RL' is therefore not statistically supported. Please report standard deviations or confidence intervals across multiple seeds for CoVo and baselines, or at least demonstrate that the ordering is consistent across seeds.","section":"Table 1 and Section 4.2"},{"comment":"The reward computation and training deliberately exclude prompts for which all sampled answers are identical ('we focus on prompts where πθ generates multiple final answers'). This post-hoc filtering depends on the policy's own sampling and biases the learning signal toward prompts with answer diversity. The paper does not quantify the fraction of discarded prompts or compare against baselines trained on the same filtered subset. Without this control, one cannot attribute the performance gains to the consistency/volatility reward rather than to the selection of easier or more diverse prompts. Please report the number of prompts retained per dataset and include an ablation that applies the same filtering to supervised RL baselines.","section":"Section 2 and Section 3.1"},{"comment":"The curiosity reward p_KL is not well-defined as written. In Eq. (10), p_KL = ln[KL(P_{i+1}, U) + 1], where P_{i+1} is described as the token probability distribution of state s_{i+1}; however, Algorithm 2 accumulates P as a list of log-probabilities (line 8) and then computes KL(P, U) at line 11. A KL divergence requires two probability distributions, not a list of scalar log-probabilities. Please specify the exact construction of P_{i+1} and correct the pseudocode so the reward is reproducible.","section":"Section 3.3.2, Eq. (10), Algorithm 2"},{"comment":"The core assumption that correct trajectories have higher consistency and lower volatility than incorrect ones is validated only on the base model (Table 2, Fig. 2, Fig. 4). During RL, the reward is recomputed from the current policy's likelihoods (Eq. (1)), so the separation must be re-verified under distribution shift. Figure 5 reports 'reward accuracy' but never defines how it is computed, shows no checkpoint-wise feature distributions, and covers only one model. Similarly, Proposition 3 assumes that the CoVo reward ranks y_γ highest and has low misclassification probability, but does not prove these properties hold for the trained policy. Please provide an analysis of consistency/volatility stratifications at multiple training checkpoints, and either prove or empirically demonstrate that the reward-correctness association persists under the evolving policy.","section":"Section 4.3, Fig. 5, Proposition 3"}],"minor_comments":[{"comment":"The table header contains a typo: 'V olatility' should be 'Volatility'. Additionally, some standard deviations are very large (e.g., 0.867±0.785 for GPQA), which suggests heavy-tailed distributions; consider reporting medians or interquartile ranges for clarity.","section":"Table 2"},{"comment":"The legend contains a typo: 'Groud Truth' should be 'Ground Truth'. Please also clarify what the three reward curves in the left panel represent and how 'Reward Accuracy' is computed, as this is central to the reward-hacking claim.","section":"Figure 5"},{"comment":"The volatility formula uses a max over a set containing a boolean expression (|D[i,0] ≠ min_k D[i,k]|), which is confusing. It would be clearer to define Vol(τ) = (1/T) · max{ i : D[i,0] ≠ min_{0≤k<K} D[i,k] }, with the convention that the maximum is 0 if the set is empty.","section":"Section 3.2, Eq. (5)"},{"comment":"The monotonicity proof for rV_int with respect to volatility is heuristic: the step 'When Δv causes Vi to deviate from the current group, it tends to be sizable. Therefore (Vi − Vj + Δv/2) > 0, which leads to ΔrV_int(Vi)<0' is not a formal argument. Since the paper states that a formal proof is provided, please tighten this derivation or explicitly state the additional conditions needed for the sign to hold.","section":"Appendix B.3.1"},{"comment":"The notation 'r(s,x,y) ∈ [0,1] ∝ log πθ(y|x⊕s)' mixes proportionality with a bounded range; log-probabilities are non-positive, so the stated range is incompatible with a positive proportionality constant. Please clarify the exact relationship, e.g., by defining r as a normalized or shifted version of log πθ.","section":"Proposition 2, Eq. (12)"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal. The main technical risk is the unverified persistence assumption: the reward-correctness link is shown only at initialization, and the convergence theorem assumes rather than proves the required reward properties. I would not reject on this basis, but the revision must include checkpoint-wise empirical evidence. Please also verify that the comparisons with EMPO and TTRL are fair with respect to data filtering and compute, since the manuscript excludes single-answer prompts without noting whether baselines are subject to the same filtering."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the CoVo paper. The new thing is real: using consistency and volatility of intermediate reasoning states as an intrinsic reward, with a vector aggregation and a curiosity bonus, is a genuine departure from TTRL and EMPO, which only look at final answers. The base-model analysis in Table 2 and Figure 2 does show that correct and incorrect trajectories separate in this feature space, and the ablation (Table 3) shows both reward components contribute. The vector aggregation robustness proof in Appendix B.3 is legitimate, and the code is available.\n\nThe problems are exactly where the reader put them. First, no error bars or multiple seeds, and the margins over baselines are often within noise (e.g., +0.4 on MATH-500 for Qwen2.5-3B). Second, the reward computation excludes prompts where all sampled answers are identical, a selection bias that removes exactly the cases where the intrinsic reward has no contrast to work on. Third, the curiosity reward in Eq. (10) defines a KL divergence between a list of logprobs and a uniform distribution without specifying the distributional form; as written it is not well-defined. Fourth, and most importantly, the paper validates the consistency/volatility signal only on the base model. The reward is recomputed from the training policy's own likelihoods during RL, and nothing in the paper shows that the separation between correct and incorrect trajectories persists under optimization. Figure 5's 'reward accuracy' is never defined, shows no checkpoint-wise distributions, and covers a single model. Proposition 3 does not fix this: it simply assumes as premises that the reward ranks y_gamma highest and has low misclassification probability. Those are exactly the properties that need to be established for the trained policy.\n\nThat said, the central idea is worth taking seriously. The paper is not a toy; it reports consistent gains across three backbones and seven benchmarks, and the base-model pattern is robust across domains. I'd send it to a serious referee, but with a request for major revision: add seeds, define and empirically verify the persistent separation, fix the KL term, and either prove Prop 3 under conditions that hold during training or reframe it as a heuristic motivation. This paper is for researchers working on self-rewarding RL and label-free reasoning; they will want to read it, but they should not take the headline claim at face value.","headline":"CoVo's consistency/volatility observation is real on the base model, but the paper never shows the signal survives training, so the central claim remains conditional.","tokens_in":28526,"tokens_out":1996,"would_cite":false,"duration_ms":24005,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a language model can reinforce its own reasoning with no ground-truth labels or external reward models, using a reward built from how consistently its intermediate steps converge on the final answer, and reports…","keywords":["self-rewarding reinforcement learning","LLM reasoning","intrinsic reward","consistency and volatility","curiosity bonus","trajectory analysis","label-free RL"],"falsifier":"Measure the separation between correct and incorrect trajectories in consistency and volatility (the statistics of Table 2) using the training policy's likelihoods at several checkpoints during a CoVo run; a significant narrowing or inversion of that gap while reported accuracy still rises would show the reward has decoupled from correctness. A complementary test: run CoVo on a prompt set where the model produces confidently wrong, homogeneous solutions, and check whether the intrinsic reward increases while accuracy falls — that pattern would be direct evidence of the reward-hacking the paper claims to avoid.","tokens_in":27571,"feed_emoji":"🧠","tokens_out":11579,"duration_ms":113602,"temperature":0.7,"pith_summary":"This paper sets out to show that the reasoning ability of a language model can be improved by its own internal signals alone, with no answer labels, no verifiers, and no trained reward models. Its central observation is that when a model samples many solutions to the same prompt, correct solutions differ measurably from wrong ones in trajectory shape: the intermediate steps of correct answers are consistently closest to their own final answer and stop wandering toward other candidates early, while wrong answers fluctuate and converge late. From this the paper builds CoVo, a reward that scores each sampled trajectory by that consistency and volatility, aggregates the scores robustly across solutions sharing the same answer, adds a curiosity bonus to keep exploration alive, and optimizes with an off-the-shelf policy-gradient algorithm. On math, commonsense, and science benchmarks across three models, CoVo matches or beats supervised RL that uses ground-truth answers, which matters because it points to a scalable path for improving reasoning from unlabeled prompts.","feed_headline":"Consistent reasoning paths teach LLMs without any labels","feed_subtitle":"Unlabeled prompts alone can drive reasoning RL to match or beat supervised training on math and science benchmarks.","key_machinery":"The load-bearing object is the likelihood distance matrix $D\\in\\mathbb{R}^{T\\times K}$ between the $T$ intermediate reasoning states of one sampled trajectory and the $K$ distinct final answers produced by all sampled trajectories for the same prompt, where each entry is $d(s_i,y_k) = -\\frac{1}{|y_k|}\\sum_j \\log\\pi_\\theta(y_k[j]\\mid s_i, y_k[:j])$ — the per-token negative log-likelihood the policy assigns to candidate answer $y_k$ given prefix $s_i$. Two scalar features are read off this matrix: consistency $\\mathrm{Con}(\\tau)$, the fraction of intermediate states for which the trajectory's own final answer is the nearest candidate, and volatility $\\mathrm{Vol}(\\tau)$, the normalized index of the last state that is nearest to a different candidate. These two features are the reward's entire content: each trajectory becomes a vector $\\mathrm{Con}(\\tau_i)[\\cos(\\mathrm{Vol}(\\tau_i)),\\sin(\\mathrm{Vol}(\\tau_i))]$, vectors are summed within groups that share a final answer, and the intrinsic reward is the group-mean magnitude of the combined vector — a polar-coordinate aggregation that the paper proves is monotone in both features and more robust to outliers than the linear difference $\\mathrm{Con}-\\mathrm{Vol}$. A curiosity reward, the average token log-probability along a transition penalized by $\\ln[\\mathrm{KL}(P_{i+1},U)+1]$ against a uniform distribution, is added to keep the sampled solution space diverse. The combined reward feeds a Reinforce++ update with a KL reference term, and the paper's variational bound (Proposition 2) is what connects this reward to a principled objective over latent reasoning paths.","core_discovery":"The discovery the paper defends is that intermediate reasoning states carry a reliable, self-supervised signal of answer correctness, and that this signal is strong enough to replace external supervision in RL training for reasoning. Concretely, the paper defines a distance matrix $D\\in\\mathbb{R}^{T\\times K}$ where each entry is the per-token negative log-likelihood from an intermediate state $s_i$ to a candidate final answer $y_k$, computed with the policy's own probabilities; from this matrix, a trajectory is summarized by its consistency $\\mathrm{Con}(\\tau)$ (the fraction of states nearest to the trajectory's own answer) and volatility $\\mathrm{Vol}(\\tau)$ (the normalized position of the last state that deviates toward another answer). Correct and incorrect trajectories separate cleanly in these two features across math, commonsense, and science prompts. CoVo converts the separation into a reward: trajectories are grouped by final answer, each trajectory is embedded as a vector $\\mathrm{Con}(\\tau_i)[\\cos(\\mathrm{Vol}(\\tau_i)),\\sin(\\mathrm{Vol}(\\tau_i))]$, the group vectors are summed, and the magnitude of the sum is the intrinsic reward, with a proof that this geometric aggregation keeps the desired monotonicity while damping outlier influence. A curiosity bonus based on token log-probabilities with a KL-from-uniform penalty is added to counter diversity collapse. The paper further claims a theoretical grounding: optimizing this reward is equivalent to variational inference over latent reasoning trajectories, and it proves that majority-voting rewards push the policy toward its current mode even when that mode is wrong, while the consistency-based reward does not. The empirical claim closes the argument: on MATH-500, GSM8K, AMC-23, Olympiad Bench, MMLU-Pro, GPQA, and CommonsenseQA, CoVo reaches accuracy comparable to or higher than supervised RL baselines on Llama3.2-3B, Qwen2.5-3B, and Qwen2.5-7B.","pith_inferences":["A stress test the paper does not run: recompute the consistency–volatility gap between correct and incorrect trajectories using the training policy's own likelihoods at later checkpoints; if the gap narrows as the policy becomes globally more confident, the reward's grounding in correctness weakens and the method would need periodic revalidation.","Because the reward needs several distinct final answers per prompt to be informative, CoVo's signal thins out on problems with large, flat answer spaces or small sampling budgets; testing it on code-generation or long-form proof tasks with sparse answer agreement would reveal the boundary of the mechanism.","The paper explicitly leaves visual-language reasoning untested; a natural extension is whether consistency of reasoning states still separates correct from incorrect answers when states mix language and perception, or whether perceptual grounding breaks the likelihood-based distance.","The vector aggregation is essentially a smoothed measure of how many steps commit to the final answer; a cheaper implementation that counts committed steps directly would test whether the geometric form earns its extra complexity."],"forward_implications":["Reasoning RL no longer needs answer keys: any prompt set that yields multiple distinct sampled solutions becomes usable training data, since the reward comes from the model's own trajectory geometry.","The majority-voting failure mode is addressed head-on: the paper proves vote-counting rewards collapse the policy onto the current sample mode even when that mode is wrong, and reports that CoVo's reward accuracy stays stable over 500 training steps where majority voting drifts.","Training diversity is preserved: CoVo keeps sampled reasoning paths more diverse than rule-based GRPO on semantic-similarity, entropy, and lexical-overlap metrics, suggesting the curiosity bonus offsets the homogenization RL usually causes.","Math-only unlabeled training transfers: improvements appear on commonsense and science benchmarks the model never saw in training, indicating the learned behavior is general reasoning rather than answer-format memorization.","The approach is backbone-agnostic and scales in parameters: consistent gains appear on 3B and 7B models with different base checkpoints, which supports the paper's claim of a scalable unsupervised pathway."],"supporting_citations":[{"why":"supplies the definition of distance from an intermediate reasoning state to a final answer, which is the foundation of the distance matrix.","marker":"[65]"},{"why":"inspires the intra-group relative reward design by grouping sampled trajectories, which CoVo adapts to consistency and volatility features.","marker":"[37]"},{"why":"provides the Reinforce++ policy-gradient backbone with clipped normalized advantages used to optimize the self-reward.","marker":"[13]"},{"why":"is the source of the unlabeled math-reasoning instruction prompts used for training without labels.","marker":"[15]"},{"why":"is the majority-voting self-rewarding baseline the paper contrasts with, including the reward-hacking failure mode Proposition 1 formalizes.","marker":"[68]"},{"why":"is the semantic-clustering self-rewarding baseline whose answer-level reward CoVo's trajectory-level reward is designed to beat.","marker":"[61]"},{"why":"supplies the optimization-dynamics theory on which the convergence bound for CoVo's reward is built.","marker":"[32]"}],"fun_headline_variants":["Consistency and volatility: self-rewarding RL for LLM reasoning","CoVo uses trajectory consistency as intrinsic reward","Free reward from reasoning paths, no labels needed","Self-rewarding reasoning RL matches supervised baselines","CoVo: consistency-based self-reward rivals supervised training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method rests on the premise that correct answers reliably look more consistent and less volatile than wrong answers in the model's own probabilities, and that this gap survives while the policy is being updated; the paper verifies the gap on the starting model but not on the training policy, so if the model becomes uniformly more confident, wrong answers could come to look \"consistent\" too and the reward would decouple from correctness.","fun_headline_variants_meta":{"raw":{"variants":["Consistency and volatility: self-rewarding RL for LLM reasoning","CoVo uses trajectory consistency as intrinsic reward","Free reward from reasoning paths, no labels needed","Self-rewarding reasoning RL matches supervised baselines","CoVo: consistency-based self-reward rivals supervised training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000496,"raw_usage":{"total_tokens":2531,"prompt_tokens":1143,"completion_tokens":1388,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":759,"completion_tokens_details":{"reasoning_tokens":1310}},"tokens_in":759,"tokens_out":1388,"duration_ms":14473,"temperature":1.0,"reasoning_tokens":1310,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:03:04.740895+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the separation between correct and incorrect trajectories in consistency and volatility (the statistics of Table 2) using the training policy's likelihoods at several checkpoints during a CoVo run; a significant narrowing or inversion of that gap while reported accuracy still rises would show the reward has decoupled from correctness. A complementary test: run CoVo on a prompt set where the model produces confidently wrong, homogeneous solutions, and check whether the intrinsic reward increases while accuracy falls — that pattern would be direct evidence of the reward-hacking the paper claims to avoid.","supporting_citations":[],"review_version":1}