{"id":"74d6b81d-9312-49eb-b6fa-3f72eadc29ee","arxiv_id":"2506.23235","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The probability an LLM assigns to a response is treated as a reward, formally justified via an equivalence between next-token prediction and offline inverse RL, and then used to fine-tune the model itself.","lead":"Any LLM trained to predict the next token already contains a built-in reward signal: the model's own output probabilities can be used to score responses, with no extra training. The authors prove a formal connection to inverse reinforcement learning and report experiments where this internal score matches or beats dedicated reward models.","discovery_kind":"unification","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'any LLM' scope claim depends on Eq. (13), which assumes the training data are expert demonstrations for the target reward; otherwise the endogenous reward is only α log bπ plus a potential and Theorem 2 does not apply.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing step: the expert policy πE is assumed to be the entropy-regularised optimal policy for the true reward (Eq. 13). My stress-test reaches the same point: Proposition 1 is a tautology in the sense that the offline IRL objective and the MLE objective are the same up to a constant, so the 'equivalence' does not, by itself, make the recovered reward a good generalist reward. The theorems that give the paper its normative content both require Eq. (13). For a truly general pretraining corpus, there is no single coherent reward r* that the data-generating process optimises, so the condition is not satisfied. This is why I consider the concern load-bearing: it directly undermines the abstract's 'any LLM' claim, not merely a numerical detail. I did not choose the exact-RL caveat as the single primary issue because the paper explicitly acknowledges 'we ignore any optimisation error', making that limitation visible; the expert-optimality assumption is more fundamental because it is silently inherited from MaxEnt IRL and is not flagged as a limitation. The proposed concrete test targets the scope claim directly: if a base pretrained model's EndoRM performs poorly on RM-Bench, then the empirical success in Table 1 comes from instruction-tuned data that approximate expert demonstrations, not from next-token prediction in general. Since the reader's verdict was already CONDITIONAL with a request to temper the overclaims, my analysis does not change the verdict; it adds a sharper reason for why the 'any LLM' phrasing should be corrected.","tokens_in":23226,"tokens_out":7513,"duration_ms":88718,"concrete_test":"Run the paper's EndoRM protocol from Appendix C on RM-Bench using a base pretrained checkpoint (e.g., Qwen2.5-7B without instruction tuning or SFT) and compare against Table 1. If accuracy on the base model is near chance or systematically favours high-log-probability text regardless of response quality, the 'any LLM' claim is falsified and Eq. (13)'s expert-optimality assumption is load-bearing. A companion control on a model trained on shuffled or randomised text would isolate whether the reward's usefulness comes from the data being expert demonstrations rather than from next-token prediction per se.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Proposition 1 is algebraically correct but nearly content-free: maximising Eq. (6) over Q is equivalent to maximising α times the next-token log-likelihood, so any softmax-parameterised MLE policy's logits solve the 'IRL' objective. The substantive claim is that this reward is the true reward and that RL on it is provably superior. Both Theorem 1 and Theorem 2 rely on Eq. (13), which assumes the data-generating expert πE is the entropy-regularised optimal policy for the unknown true reward r*. Under that assumption r* is, up to a potential, α log πE, and the endogenous reward is, up to a potential, α log bπ; the theorems measure only how well bπ's log-probabilities approximate πE's. For the paper's headline claim about 'any LLM trained via standard next-token prediction', the training distribution is web-scale text, which is not generated by an expert optimising a coherent reward r* (or at least no argument is given that it is). Without Eq. (13), Theorem 2's O(H) bound does not follow, and the endogenous reward is just an LLM's own log-likelihood, with no proved superiority over the base policy. The theory therefore supports only the weaker, known statement 'if you have expert demonstrations for the target reward, MLE log-probability is a reward', not 'a powerful generalist reward model is found inside any LLM'. The empirical section also deviates from the theory (Appendix C uses discounted and floored rewards, γ<1, β>0), so the experiments cannot independently rescue the broad scope claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper claims that any LLM trained by standard next-token prediction already contains a 'generalist reward model': its logits are shown to solve an offline inverse reinforcement learning (IRL) objective (Proposition 1), and the resulting 'endogenous reward' is used for reinforcement learning fine-tuning, with a claimed O(H) sub-optimality bound versus O(H^2) for behavior cloning (Theorem 2). The authors also provide experiments on RM-Bench, Multifaceted-Bench, DSP, and MATH-lighteval, reporting that the training-free endogenous reward outperforms LLM-as-a-judge baselines and is competitive with explicitly trained reward models, and that RL fine-tuning with it improves math reasoning. The theoretical core consists of Proposition 1, Theorem 1 (a total-variation bound on preference distributions), and Theorem 2 (a policy sub-optimality comparison), all built on the assumption that the expert policy is entropy-regularized optimal for the true reward.","tokens_in":23583,"tokens_out":4308,"duration_ms":48083,"significance":"If the central claims held, the paper would be significant: it would provide a theoretical justification for replacing the reward-modeling stage of RLHF with a training-free elicitation of log-probabilities from a base model, and it would prove a concrete advantage of RL over behavior cloning in the LLM setting. The paper also makes several useful secondary observations: Eq. (12) explicitly connects the outcome reward to α log π(τ|s1) plus a prompt-dependent potential; the reward-shaping perspective in §3.3 is correct; and the framing of existing generative verifiers as special cases of log-probability rewards is a useful unification. The proofs are transparent and the derivations in Appendix A are internally consistent under the stated idealizations. However, the significance is substantially reduced by the gap between the 'any LLM' scope claim and the actual assumptions: Eq. (13) assumes the training data are expert demonstrations for a coherent target reward, so the theorems apply to MLE policies trained on expert data, not to arbitrary web-scale pretrained models.","major_comments":[{"comment":"The 'any LLM' scope claim is contradicted by the assumption underlying both theorems. Eq. (13) asserts that the expert policy πE is the entropy-regularized optimal policy for the true reward r⋆; this is an assumption about the data-generating process, not a property of all next-token-prediction training. For web-scale pretraining data, no reward r⋆ is specified and no argument is given that the data are expert demonstrations for a coherent objective. Consequently, Proposition 1 shows only that the logits of any MLE policy solve the offline IRL objective for the data distribution actually used in training; Theorem 1 and Theorem 2 then measure how close bπ's log-probabilities are to πE's, not the discovery of a generalist reward in an arbitrary LLM. The paper should explicitly re-scope its claims to 'models trained on expert demonstrations for the target task' or provide a separate argument for the pretraining case.","section":"§4.1, Eq. (13)"},{"comment":"The O(H) bound in Theorem 2 assumes exact RL: the theorem statement defines πRL = argmaxπ Vπ_br and the text says 'we ignore any optimization error in solving the optimal policy.' The experiments, however, use practical RL algorithms with KL penalties, sampling, and finite rollouts, so the stated bound does not apply to the reported RLFT results. To support the empirical claim of self-improvement, the paper needs either (i) a bound that includes optimization error, or (ii) an explicit argument that the experimental setup approximately satisfies the exact-RL idealization, ideally with a measurement of the optimization gap.","section":"§4.2, Theorem 2"},{"comment":"The empirical reward computation deviates from the theoretical outcome reward. Eq. (12) is derived by telescoping br(sh, ah) = α log bπ(ah|sh) + V(s_h) − V(s_{h+1}), which requires the undiscounted, unfloored sum. Appendix C instead uses max(γ^{h−1}, β) with γ = 0.95 or 0.93 and β = 0 or 0.03. This breaks the telescoping, so the reward used in the RM-Bench, Multifaceted-Bench, and RL experiments is not the reward analyzed in Theorem 1 and Theorem 2. Either the experiments should use the exact theoretical reward from Eq. (12), or the theory should be extended to discounted and floored rewards, including the corresponding value-function and error-bound analysis.","section":"Appendix C, Eq. (10)-(12)"},{"comment":"Proposition 1 is presented as the 'theoretical cornerstone' and as evidence that a reward model is 'latently present' in any LLM, but the argument is definitional: Eq. (8) is the next-token prediction objective from Eq. (1) rewritten with a softmax parameterization of the policy, so any softmax-parameterized MLE policy's logits solve the offline IRL objective by construction. The paper should state this plainly and build its case on the conditions under which the MLE policy is a good reward, rather than presenting the algebraic equivalence itself as the discovery. As written, the proposition gives no information about whether the endogenous reward is the true reward for any downstream task.","section":"§3.2, Eq. (8) and Proposition 1"},{"comment":"The abstract and introduction claim 'the first theoretical proof of the effectiveness of reinforcement learning for LLMs.' This is not supported by the manuscript: the quadratic-to-linear improvement in horizon dependence is a known result in imitation learning (e.g., Ross and Bagnell, 2010; Xu et al., 2021; Rajaraman et al., 2020), and Theorem 2 is a re-derivation of that style of bound in a token-level MDP under an additional entropy-regularization assumption. The claim should be softened and the prior bounds cited and differentiated, or the specific novelty (the reward-extraction mechanism) should be identified independently of the H-dependence comparison.","section":"Introduction and §4.2"}],"minor_comments":[{"comment":"There are several typos: 'This paper discover' should be 'This paper discovers' (§1); 'closed to' should be 'close to' (after Theorem 1); 'quadractic' should be 'quadratic' (§4.2); 'Multifacted-Bench' and 'Multifaceted-Bench' are used inconsistently; and the reference to 'prepring' should be 'preprint'.","section":"Throughout"},{"comment":"The temperature settings are inconsistent: §5.1 says 'The temperatures for training and test are set to 1.0 and 0.6, respectively,' while Appendix C says 'All sampling temperatures are set to 1.0.' Please clarify which temperature is used for evaluation.","section":"§5.1 and Appendix C"},{"comment":"The RL method is described as using the approach of Shao et al. [2024], but the cited reference is DeepSeekMath, which does not introduce GRPO in the form used here; please cite the actual algorithm paper or describe the RL procedure precisely, including the KL coefficient and the exact objective being optimized.","section":"§5.4"},{"comment":"The proof of Theorem 1 contains a duplicated inequality line: the two identical lines bounding |log πE(τ) − log bπ(τ)| + |log πE(τ') − log bπ(τ')| appear consecutively; the duplicate should be removed.","section":"Appendix B.2"},{"comment":"The discussion of extending the method to image, video, and audio generation claims that any autoregressive model can be prompted to evaluate its own output, but it does not address the absence of a well-defined 'token-level reward' or the fact that the theoretical results in §4 assume a token-level MDP with the same vocabulary for actions and states. Adding a caveat or a short argument for the multimodal extension would improve rigor.","section":"§7.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is built on an interesting and correct algebraic observation, but the presentation overstates both its novelty and its scope. The authors should be encouraged to rework the claims around the conditional statement 'if the base model's training distribution is expert demonstrations for a coherent reward, then its log-probability is an IRL-derived reward and RL on it can improve the horizon-dependence of the error bound.' The empirical mismatch with Appendix C should be resolved before publication, and the 'first theoretical proof' claim should be removed or carefully qualified. I believe these changes are achievable within a major revision; the underlying algebra and the experimental ideas are worth preserving."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe core observation is a clean identity: for any policy that maximizes next-token likelihood on a dataset, the logits solve the offline IRL objective, and the extracted reward equals α times the model's own log-probability up to a potential. That is correct and worth stating clearly. The paper also shows existing generative reward models (GenRM, verifiers) are special cases, and it correctly notes that iterating this self-improvement converges immediately. Those parts are solid.\n\nThe trouble starts with the interpretation. The O(H) versus O(H^2) bound in Theorem 2 relies on Eq. (13), which assumes the expert is the entropy-regularized optimal policy for the true reward. Under that assumption the true reward is itself α log πE plus a potential, so the whole analysis measures how well the base model's log-probabilities approximate the expert's. For a pretrained model on web-scale text, that assumption is not argued. The 'any LLM' claim is thus not supported; the theory supports only the weaker, known statement: if you have expert demonstrations, MLE log-probability is a reward. The abstract's 'first theoretical proof' is also an overclaim given existing imitation-learning bounds from Ross and Bagnell, Xu et al., and others.\n\nEmpirically, the paper is upfront that results are initial (footnote in Section 5), but the experiments remain single-run point estimates with no variance. Appendix C's discounted, floored reward breaks the exact telescoping in Eq. (12), so the implementation doesn't test the theory as stated. The RM-Bench average (70.2) is a hair above Skywork (70.1), not a clear 'surpasses trained RMs' result. The RL gains are modest and unseeded. Section 7.5 flags self-reinforcing biases, which is honest, but it doesn't temper the title's promise of a 'generalist' reward inside any LLM.\n\nWorth a serious referee? Yes. The algebraic core is correct, the unifying view of generative verifiers is useful, and the assumptions are stated well enough that a careful reader can see where they break. But the paper needs revision: scope claims must be narrowed to the expert-demonstration setting, the empirical section needs seeds and variance, and the abstract should not sell the result as a proof that RL works for all LLMs. I'd send it for review with the expectation of major revision.","headline":"A clean algebraic identity yields standard imitation-learning bounds, but the 'any LLM' scope claim outruns the evidence.","tokens_in":24189,"tokens_out":6091,"would_cite":false,"duration_ms":61861,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Any language model trained by ordinary next-token prediction already contains a generalist reward model: its logits provably solve an offline inverse reinforcement learning objective, so a training-free reward can be elicited and used for…","keywords":["endogenous reward","reward model","inverse reinforcement learning","next-token prediction","LLM alignment","RLHF","soft Q-learning","logits"],"falsifier":"Construct a synthetic token-level MDP with a known true reward, compute the entropy-regularized optimal expert policy, sample demonstrations, train a tabular next-token policy by maximum likelihood, extract the endogenous reward, and re-train exactly to optimality; if the extracted reward is not the true reward up to the stated potential term, or if the re-trained policy's advantage over the base policy does not follow the predicted linear-versus-quadratic behavior in the horizon $H$, the paper's central identification fails.","tokens_in":23000,"feed_emoji":"🎯","tokens_out":11816,"duration_ms":116283,"temperature":0.7,"pith_summary":"The paper sets out to show that the reward-model stage of reinforcement learning from human feedback (RLHF) is unnecessary when the base model was trained with ordinary next-token prediction. It proves that a next-token prediction model's logits are a valid solution to an offline inverse reinforcement learning objective, so a reward signal can be read out through the inverse soft Bellman operator without any extra training. It further proves that reinforcement learning on this elicited reward has an error bound linear in the response length, $O(H)$, whereas the base behavior-cloned policy has a quadratic bound, $O(H^2)$. Experiments indicate that the elicited 'endogenous reward' beats LLM-as-a-judge baselines on preference benchmarks, matches or exceeds explicitly trained reward models, and improves math reasoning when used for RL fine-tuning.","feed_headline":"Any next-token-trained LLM contains a reward model in its logits","feed_subtitle":"The paper proves logits solve an offline inverse-RL objective, so the expensive reward-model training stage can be skipped.","key_machinery":"The carrying mechanism is the inverse soft Bellman operator of inverse soft Q-learning, $r(s_h,a_h)=Q(s_h,a_h)-\\alpha\\log\\sum_{a_{h+1}}\\exp(Q(s_{h+1},a_{h+1})/\\alpha)$, applied to the language model's logits treated as an estimated Q-function. The key identity is that the next-token prediction maximum-likelihood objective and the offline IRL objective are the same optimization problem, so the softmax of the logits is simultaneously the policy and the Q-function. Substituting it into the operator turns each token's log-probability into a dense token-level reward whose value-function terms are a potential-based reward shaping, which preserves optimal policies; at the trajectory level the reward collapses to the log-likelihood of the whole response plus a prompt-only term.","core_discovery":"On the paper's own terms, the discovery is that generation and evaluation are two uses of the same trained object. A next-token prediction model $\\hat{\\pi}$ with logits $\\hat{f}$, written $\\hat{\\pi}(\\cdot|s_h)=\\mathrm{softmax}(\\hat{f}(s_h,\\cdot);\\alpha)$, is the maximum-likelihood solution of the ordinary next-token objective; rewriting the offline inverse soft Q-learning objective shows it is the same maximization, so the logits are a Q-function for the data distribution. Feeding those logits into the inverse soft Bellman operator gives the endogenous reward $\\hat{r}(s_h,a_h)=\\alpha\\log\\hat{\\pi}(a_h|s_h)+V_{\\hat{Q}}(s_h)-V_{\\hat{Q}}(s_{h+1})$, a potential-shaped version of the token log-probability reward. Over a full response this telescopes to $\\alpha\\log\\hat{\\pi}(\\tau|s_1)$ plus a prompt-only constant, so the model's own probability of a response is its reward. The paper claims this is not a heuristic but the very reward an offline IRL algorithm would learn, and that RL on it corrects the compounding errors of imitation learning, giving the first theoretical proof it knows of that RL fine-tuning improves on the base model.","pith_inferences":["An implication the paper does not draw out: because the reward is elicited rather than learned, its practical ceiling is exactly the quality of the base model's probability estimates; on preferences where those probabilities encode stylistic bias rather than correctness, RL on the endogenous reward could amplify the bias.","The linear-in-$H$ advantage is proven for exact RL; under the approximate RL actually used in training, the margin over behavior cloning should narrow, and an ablation varying optimization steps or KL penalty would map where the theoretical benefit survives.","The same identity suggests a cheap transfer test: any autoregressive multi-modal generator can be probed with this operator, and the method should transfer wherever token-level likelihood is well calibrated, and fail where it is not."],"forward_implications":["The separate reward-model training stage of RLHF can be replaced by reading log-probabilities out of the base model, eliminating the need to collect preference labels or train a scorer.","Because the reward is a shaped version of the base model's own log-probabilities, RL fine-tuning on it provably reduces the compounding-error penalty from $O(H^2)$ to $O(H)$; the math-reasoning results are the empirical form of that prediction.","The endogenous reward is promptable: changing the system instruction changes the evaluation criterion of the same frozen model, which yields domain-specific judges without retraining.","Existing generative reward models that score by the probability of a 'Yes' token or by pairwise 'Yes/No' comparisons are special cases of this formulation and inherit its theoretical grounding.","Any autoregressive model, including image, video, or audio generation models, can in principle be aligned the same way, which extends RL-based alignment to modalities where human preference data is impractical to collect."],"supporting_citations":[{"why":"Supplies the offline inverse soft-Q-learning objective and the inverse soft Bellman operator that Proposition 1 identifies with next-token prediction logits.","marker":"[Garg et al., 2021]"},{"why":"Provides the maximum-entropy IRL formulation and the assumption that the expert policy is entropy-regularized optimal, which anchors Theorem 1 and Theorem 2.","marker":"[Ziebart et al., 2008]"},{"why":"Gives the reward-shaping theorem used to show that the value-function terms in the endogenous reward do not change optimal policies.","marker":"[Ng et al., 1999]"},{"why":"Provides the regularized-MDP theory, including the softmax form of the optimal policy and the Bellman fixed point, used throughout the proofs.","marker":"[Geist et al., 2019]"},{"why":"Supplies the quadratic compounding-error bound for imitation learning that Theorem 2 contrasts with the linear bound for RL on the endogenous reward.","marker":"[Ross and Bagnell, 2010]"},{"why":"Gives the imitation error-bound analysis used in the proof of the behavior-cloning half of Theorem 2, and the infinite-horizon extension noted there.","marker":"[Xu et al., 2021]"},{"why":"One of the imitation-learning error bounds the proof of the base policy's quadratic suboptimality builds on.","marker":"[Xu et al., 2020]"},{"why":"The generative-verifier formulation that Section 3.3 shows is a special case of the endogenous reward, and one of the baselines in the RM-Bench comparison.","marker":"[Zhang et al., 2024]"},{"why":"The pairwise and pointwise generative reward models used as training-free baselines and shown to be special cases of the endogenous-reward framework.","marker":"[Mahan et al., 2024]"}],"fun_headline_variants":["LLMs hide a reward model inside their logits","Next-token logits are a proven reward model","Skip reward training: LLM logits are a reward model","The generalist reward model was inside the LLM all along","Your LLM already knows how to score: logits are rewards"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the demonstration data came from a policy that is itself the entropy-regularized optimal policy for the true reward at the fixed temperature coefficient $\\alpha$, and that the follow-up reinforcement learning is solved exactly rather than approximately.","fun_headline_variants_meta":{"raw":{"variants":["LLMs hide a reward model inside their logits","Next-token logits are a proven reward model","Skip reward training: LLM logits are a reward model","The generalist reward model was inside the LLM all along","Your LLM already knows how to score: logits are rewards"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000684,"raw_usage":{"total_tokens":3153,"prompt_tokens":1047,"completion_tokens":2106,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":663,"completion_tokens_details":{"reasoning_tokens":2023}},"tokens_in":663,"tokens_out":2106,"duration_ms":16153,"temperature":1.0,"reasoning_tokens":2023,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:49:11.427934+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a synthetic token-level MDP with a known true reward, compute the entropy-regularized optimal expert policy, sample demonstrations, train a tabular next-token policy by maximum likelihood, extract the endogenous reward, and re-train exactly to optimality; if the extracted reward is not the true reward up to the stated potential term, or if the re-trained policy's advantage over the base policy does not follow the predicted linear-versus-quadratic behavior in the horizon $H$, the paper's central identification fails.","supporting_citations":[],"review_version":1}