{"id":"42273e74-c386-434b-99a4-1d566c9387e1","arxiv_id":"2506.02553","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Under the assumption that the response reward equals the discounted sum of token rewards, response-level rewards suffice for unbiased token-level policy gradients in LLMs.","lead":"This paper proves a mathematical identity: when a response-level reward is the discounted sum of per-token rewards, policy-gradient training can use only the response-level reward and still estimate the per-token gradient without bias. The authors use this to argue that PPO, GRPO, ReMax, and RLOO already capture token-level reward information, and they propose a new variant, TRePO.","discovery_kind":"unification","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The transfer from Theorem 1 to PPO/GRPO is unjustified: GRPO's group-normalized baseline and PPO's learned critic are not state-dependent baselines, so the claimed unbiased token-level gradients for these algorithms are not established.","rationale":"The reader's weakest-assumption flag (additive decomposition of RM) is not the actual soft spot: any response-level reward can be represented by a terminal-only token reward, so the additive premise in Theorem 1 is not restrictive, and the roll-out estimator of Eq. (21) is unbiased for the REINFORCE objective E[RM] without needing to know per-token rewards. The real load-bearing gap is the paper's extension of the theorem to the concrete algorithms it claims to justify. GRPO's group normalization and PPO's learned critic introduce biases that Theorem 1 does not cover, yet the abstract and Corollary 1 assert that these methods 'inherently possess' token-level modeling capability. This is an internal-support problem: the theorem establishes unbiasedness for a roll-out estimator, but the surveyed algorithms implement different estimators. The paper may be correctible by (i) restricting the unbiasedness claim to REINFORCE-style estimators with Monte Carlo or leave-one-out baselines, (ii) proving the PPO case only under an exact value function or explicit roll-out value estimator, and (iii) treating GRPO's STD normalization as a biased but often effective heuristic. These are addressable, so conditional acceptance remains the right verdict; the central roll-out theorem itself appears sound.","tokens_in":17990,"tokens_out":31299,"duration_ms":309991,"concrete_test":"Build a two-token, binary-action MDP with a response-level reward RM(a1,a2). Enumerate all group outcomes for K=2 and compute the exact expected gradient of GRPO's advantage estimator from Eq. (34), including the STD normalization, and compare it with the exact policy gradient ∇E[RM]. If the two differ for any policy weights, Corollary 1's unbiasedness claim fails. Repeat with PPO by using a deliberately misspecified linear critic in GAE and comparing the expected advantage gradient with the true gradient; any difference demonstrates that the actor-critic extension in 'Trajectory Policy Gradient Theorem 1' requires an exact or roll-out-based value estimate, which PPO does not provide.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Theorem 1 (Eq. 21) is unbiased for the roll-out estimator in which, for each token t, one draws a full response from the current policy conditioned on the prefix W_{0,t} and uses RM of that continuation. The proof and Corollary 1 then identify this with the core formulas of PPO, GRPO, ReMax, and RLOO. That identification is where the central claim overreaches. For actor-critic methods, the paper's 'Trajectory Policy Gradient Theorem 1' proof equates Q−V with roll-out estimates; this holds only if V is the exact value function or is itself replaced by an independent Monte Carlo return. Standard PPO uses a learned critic inside GAE, and an imperfect critic makes the advantage estimator biased. For GRPO, Eq. (34) subtracts the mean of the group that contains the current response and divides by the group standard deviation; the mean is not a state-dependent baseline independent of the current trajectory, and the STD normalization is nonlinear. The exact expectation of the GRPO gradient is therefore not equal to the policy gradient of E[RM], even in the infinite-sample limit of Theorem 1. Corollary 1 reduces all these methods to the formula E_W Σ_t RM(W) ∇log π(w_t), which is REINFORCE, not GRPO's normalized group estimator. Thus the central claim that these practical algorithms 'inherently possess the capacity' for unbiased token-level modeling is unsupported by the theorem as stated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies reinforcement learning for large language models under the Zero-Reward Assumption, where only the final token of a response receives a reward. It proposes a Trajectory Policy Gradient Theorem claiming that, if the response-level reward RM(W) is exactly the discounted sum of per-token rewards, the policy gradient for the true token-level objective can be unbiasedly estimated using only RM(W). The paper then argues that PPO, GRPO, ReMax, and RLOO therefore possess an inherent token-level modeling capability, proposes a new algorithm called TRePO, and surveys related RL and RL-free methods. No experimental evaluation is reported.","tokens_in":18313,"tokens_out":14190,"duration_ms":124038,"significance":"The paper contains a correctly derived standard policy-gradient identity for LLM token MDPs (Lemma 1) and a correct baseline-invariance result (Lemma 2), and it is transparent about the steps of its derivations. However, the main theorem is essentially the REINFORCE identity for the expected value of a full-trajectory reward: for an arbitrary response-level reward RM(W), Eq. (21) is the policy gradient of E[RM(W)], and it coincides with the true token-level objective only under the additive discounted decomposition assumed in Theorem 1. The significance of the result is therefore much smaller than the title suggests unless that decomposition is a guaranteed property of practical reward models. The transfer to GRPO and PPO is not proven, and the Theorem 2 baseline derivation contains a mathematical error. These issues undermine the paper's central claims as stated, although the core identity could be rehabilitated with a more careful formulation. TRePO is presented without experiments, so its practical promise remains untested.","major_comments":[{"comment":"Theorem 1 assumes RM(W)=Σ_{t=1}^T γ^{t-1} r(W_{0,t-1},w_t) for the true token rewards, but the Abstract concludes that the policy gradient can be unbiasedly estimated 'using only a response-level reward model' without stating this decomposition as a caveat. A learned or arbitrary response-level reward model is not guaranteed to satisfy this additive discounted form; if it does not, Eq. (21) is the policy gradient of E[RM(W)] rather than the gradient of the true token-level objective. Please state the decomposition as an explicit standing assumption in the abstract and in the statement of Theorem 1, or revise the claims to match the theorem that actually follows.","section":"§3.1 (Theorem 1) and Abstract"},{"comment":"The transfer of Theorem 1 to GRPO and PPO is not established. Corollary 1 replaces GRPO's actual estimator, Eq. (34), by the unnormalized REINFORCE form Eq. (27); the group mean and standard deviation in Eq. (34) are not a state-dependent baseline, and the nonlinear standardization of a group containing the current response is not shown to preserve unbiasedness relative to Eq. (21). For PPO, the proof after Eq. (29) equates Q−V with rollout estimates, but PPO's GAE uses a learned critic V_φ; an inaccurate critic introduces bias that Theorem 1 does not cover. Provide explicit unbiasedness proofs for the actual GRPO and PPO estimators, or restrict the conclusion to REINFORCE-style estimators without group normalization.","section":"§3.1, Corollary 1 and Trajectory Policy Gradient Theorem proof"},{"comment":"The derivation of the near-optimal baseline is mathematically incorrect. The proof obtains b=Cov(X1X2,X2)/Var(X2), then says one can 'assume X1 and X2 are dependent' and concludes b=EX1; the conclusion b=EX1 requires X1 and X2 to be independent. For dependent X1 and X2, Cov(X1X2,X2) is not EX1 Var(X2). Equation (28) remains an unbiased estimator because b=E_{W^{(t)}}RM(W^{(t)}) depends only on the prefix W_{0,t-1}, but the variance-reduction claim that motivates TRePO is unsupported. Correct this step, or state only that b is an admissible state-dependent baseline.","section":"§3.1, Theorem 2 proof"},{"comment":"The paper claims the theorem is independent of γ, but RM is defined in terms of γ, and changing γ changes the objective being optimized. Remark 3's derivation leads to Eq. (26) with a 1/γ^{t-1} factor and calls this 'undesired,' which appears to contradict the claim of independence. Clarify whether Lemma 1 and Theorem 1 use the same γ and state precisely what 'independent of γ' means.","section":"§3.1, Remark 3"},{"comment":"TRePO is not specified in enough detail for a reader to implement it. Line 15 of Algorithm 1 refers to 'the average clipped surrogate gradient loss in Eqn. (30)', but Eq. (30) is a formula for merging response-level and token-level reward models, not a loss function; no clipped surrogate loss for TRePO is defined anywhere. Please supply the actual TRePO objective and update rule, and reconcile the pseudocode with Theorem 2.","section":"§3.2, Algorithm 1"}],"minor_comments":[{"comment":"The notation for the continuation trajectory is inconsistent: Eq. (24) uses W^{(t)} with prefix W_{0,t-1}, while Eq. (21) and the final formula use W^{(t+1)}; to estimate V(W_{0,t}), the continuation must share the prefix W_{0,t}, so the indexing should be unified.","section":"§3.1, Eqs. (24)-(25)"},{"comment":"G_t(W) is defined for token t, while Eq. (24) uses G(W^{(t)}_{t,M}) without a definition; align the notation.","section":"Table 1 and Eq. (24)"},{"comment":"The symbol 'STD' is not defined; specify whether it is the population or sample standard deviation of the group rewards.","section":"§4.1, Eq. (34)"},{"comment":"The RLOO baseline is written as (1/(K-1))Σ_{k=1}^K r_k, which should sum over o≠k; correct the index.","section":"§4.1, Eq. (36)"},{"comment":"R3HF is cited as [15], but the reference list identifies [17] as R3HF; check the citation keys.","section":"§4.2, R3HF citation"},{"comment":"The definition R(τ)=Σ_{t=0}^{T-1} γ^{T-1} r_t appears to use the wrong discounting convention; this is likely a typo, but it affects the notation used in the proofs.","section":"§2.1, definition of R(τ)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads as an early preprint: it contains multiple typographical errors, an incorrect reference to Eq. (30) as a loss, and the 'dependent'/'independent' slip in Theorem 2. More substantively, the central theorem is close to a known REINFORCE identity, and the paper overstates its applicability to PPO and GRPO. I would not support acceptance in the present form; a focused revision that states the decomposition assumption explicitly, removes or proves the PPO/GRPO claims, and corrects Theorem 2 could make the paper publishable as a theoretical note."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. Theorem 1 is correct but much more modest than the abstract advertises. Under the explicit assumption that the response-level reward is exactly the discounted sum of per-token rewards, it is a clean statement of a standard REINFORCE identity: replacing each token's Q-value with a full-return rollout adds only state-dependent baseline terms that vanish in expectation. Lemma 1 and Lemma 2 are derived correctly, and the authors deserve credit for stating that assumption in Section 3.1 and for Remark 8, which honestly notes that VinePPO derives the same algorithm as TRePO. As a unifying survey of zero-reward and partial-reward RL methods for LLMs, the paper has value.\n\nThe soft spots are real and they are concentrated in the claims built on top of Theorem 1. Corollary 1 says PPO, GRPO, ReMax, and RLOO all have token-level modeling capability. For ReMax and RLOO this is essentially REINFORCE with a state-dependent baseline, so it mostly works, up to the same additive-decomposition caveat. For GRPO, the group mean and standard deviation are not state-dependent baselines; the expectation of the normalized group advantage is not the policy gradient even in the infinite-sample limit. For PPO, the theorem assumes an exact value function or independent Monte Carlo returns, while standard PPO uses a learned critic inside GAE, so the conclusion does not follow for the practical algorithm. The stress-test note lands on both of these points.\n\nTheorem 2 has a concrete derivation error. It writes the optimal baseline as b = Cov(X1X2, X2)/Var(X2), then effectively assumes X1 and X2 are independent to get b = E[X1]. They generally are not independent, so this is not the optimum. That is fixable, but as written the variance-reduction claim does not hold.\n\nTRePO is presented as a practical method but has no experiments, and the limitations section admits this. Since the authors also acknowledge it is the same algorithm as VinePPO, the novelty is thin. The citation pattern is honest, and I do not see self-citation chains or missing references that would change the assessment.\n\nThe paper is for readers who want a clean derivation of why response-level rewards can estimate token-level gradients under the decomposition assumption, plus a useful map of existing RL and RL-free methods. It deserves a serious referee, but the revision must rewrite the abstract to state the decomposition assumption, fix Theorem 2, restrict the Corollary 1 claims to the methods for which they actually hold, and either add experiments or drop TRePO's empirical promise.","headline":"A correct but standard REINFORCE baseline identity is dressed up as a general theorem, and the claims about PPO/GRPO extend beyond what the proof supports.","tokens_in":18861,"tokens_out":3106,"would_cite":false,"duration_ms":30625,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A theorem shows that response-level rewards suffice to estimate the true token-level policy gradient in LLM reinforcement learning, even when the Zero-Reward Assumption fails.","keywords":["response-level reward","zero-reward assumption","token-level credit assignment","policy gradient theorem","reinforcement learning for LLMs","PPO","GRPO","TRePO"],"falsifier":"Construct a response-level reward function that cannot be written as $RM(W)=\\sum_{t=1}^T \\gamma^{t-1} r(W_{0,t-1}, w_t)$ for any token rewards $r$ (for example, a reward that depends on token-pair interactions or on the exact order of two distant tokens), then compute the gradient estimate of Theorem 1 on a small MDP and compare it with the true policy gradient computed from known token rewards; a nonzero difference would show the unbiasedness claim fails for non-additive reward functions.","tokens_in":17754,"feed_emoji":"🎯","tokens_out":2746,"duration_ms":27321,"temperature":0.7,"pith_summary":"The paper claims that in LLM fine-tuning, the policy gradient computed from the true but unknown token-level rewards can be estimated unbiasedly using only a response-level reward. This holds for REINFORCE-style and actor-critic algorithms regardless of whether the Zero-Reward Assumption is true. The result gives a common theoretical foundation to PPO, GRPO, ReMax, and RLOO, showing they already have token-level modeling capability. It also justifies focusing engineering effort on improving the response-level reward model rather than on token-level supervision.","feed_headline":"Response-level rewards suffice for unbiased token-level gradients","feed_subtitle":"A theorem shows PPO, GRPO, ReMax and RLOO can model token-level rewards without token-level supervision.","key_machinery":"The load-bearing identity is the additive decomposition $RM(W)=\\sum_{t=1}^T \\gamma^{t-1} r(W_{0,t-1}, w_t)$ combined with Lemma 2, the state-dependent baseline trick. Because the policy gradient at time $t$ is multiplied by $\\nabla \\log \\pi_\\theta(w_t | W_{0,t-1})$, any function depending only on $W_{0,t-1}$ has zero expectation and can be added freely. Setting that function to the accumulated known prefix reward converts the unknown future reward into a sampled total response reward, producing an unbiased estimator.","core_discovery":"The central claim is Theorem 1: when the response-level reward is defined as $RM(W)=\\sum_{t=1}^T \\gamma^{t-1} r(W_{0,t-1}, w_t)$ for real token rewards $r$ and any discount factor $\\gamma$, then the policy gradient $\\nabla J(\\theta)$ can be written as an expectation over response-level rewards of full trajectories sampled from the current policy. The proof uses Lemma 2, which adds a state-dependent constant $c(W_{0,t-1})$ to the per-token term without changing the expectation, and then chooses the constant to be the sum of rewards of previously generated tokens. This causes the unknown token rewards after time $t$ to be replaced by the observable response-level reward of a fresh trajectory that shares the prefix. The paper concludes that REINFORCE-based methods (GRPO, ReMax, RLOO) and actor-critic methods (PPO) estimate this quantity unbiasedly, and that their differences reduce to baseline selection and approximation error.","pith_inferences":["The theorem's guarantee depends on the response-level reward being exactly a discounted sum of per-token rewards; a learned reward model is not guaranteed to have this decomposition, so the unbiasedness proof may not transfer to arbitrary trained reward functions.","If the additive decomposition fails, the prefix-reward cancellation in Lemma 2 breaks, and the extra trajectories sampled at each time step would no longer correct the gradient; measuring this gap could be done by comparing the gradient from Theorem 1 against a ground-truth token-reward gradient on a controlled task.","The same cancellation technique may extend to other sequential decision problems beyond language generation, wherever a terminal reward can be expressed as a discounted additive function of per-step rewards, potentially simplifying credit assignment in multi-turn agent training.","A practical test of the theory: on a reasoning dataset where intermediate-step correctness is independently known, compare the policy-gradient estimate from response-level rewards with the estimate computed from explicit token rewards; if they differ substantially, the response-level reward model in use is not additive."],"forward_implications":["GRPO, ReMax, RLOO, and PPO have unbiased token-level gradient estimates in expectation even when intermediate tokens carry nonzero task reward.","The remaining differences among these algorithms are baseline choice and approximation error, so the theorem gives a principled way to compare them rather than relying on empirical folklore.","Developers can treat the RL training algorithm as a black box and concentrate on improving the response-level reward model, since the algorithm already models token-level signals.","The paper's proposed TRePO algorithm directly implements Theorem 2 and removes the critic network, matching GRPO in memory use while aiming at a closer approximation of the exact policy gradient.","The analysis offers an explanation for why PPO often outperforms DPO: PPO uses token-level information more efficiently through its baseline, even when both are trained with response-level rewards."],"supporting_citations":[{"why":"Supplies the classical policy gradient theorem that Lemma 1 adapts to the LLM setting where states are token prefixes and sampling comes directly from the policy.","marker":"[19]"},{"why":"Defines PPO, the actor-critic algorithm that the paper analyzes as a zero-reward approach whose gradient is shown to be unbiased.","marker":"[7]"},{"why":"Defines GRPO, a REINFORCE-family algorithm whose response-level advantage estimator is shown to be a realization of Theorem 1.","marker":"[12]"},{"why":"Defines ReMax, the greedy-baseline REINFORCE method covered by Corollary 1.","marker":"[11]"},{"why":"Defines RLOO, the leave-one-out REINFORCE method covered by Corollary 1.","marker":"[13]"},{"why":"Provides the GAE advantage estimation that the paper contrasts with the baseline choices of GRPO, ReMax, and RLOO.","marker":"[21]"}],"fun_headline_variants":["Zero-reward assumption not needed for unbiased token gradients","Response-level rewards suffice for policy gradients in LLMs","Theorem: token rewards estimated by response-level models","PPO, GRPO, ReMax, RLOO: token-level rewards via response rewards","Unbiased token gradients from response-level rewards alone"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The theorem holds only when the response-level reward is exactly the discounted sum of per-token rewards for some real token reward function; a learned response-level reward model is not guaranteed to have this additive decomposition, and if it does not, the prefix-reward cancellation in the proof no longer applies.","fun_headline_variants_meta":{"raw":{"variants":["Zero-reward assumption not needed for unbiased token gradients","Response-level rewards suffice for policy gradients in LLMs","Theorem: token rewards estimated by response-level models","PPO, GRPO, ReMax, RLOO: token-level rewards via response rewards","Unbiased token gradients from response-level rewards alone"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00055,"raw_usage":{"total_tokens":2675,"prompt_tokens":1043,"completion_tokens":1632,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":659,"completion_tokens_details":{"reasoning_tokens":1549}},"tokens_in":659,"tokens_out":1632,"duration_ms":11845,"temperature":1.0,"reasoning_tokens":1549,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:22:20.213637+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a response-level reward function that cannot be written as $RM(W)=\\sum_{t=1}^T \\gamma^{t-1} r(W_{0,t-1}, w_t)$ for any token rewards $r$ (for example, a reward that depends on token-pair interactions or on the exact order of two distant tokens), then compute the gradient estimate of Theorem 1 on a small MDP and compare it with the true policy gradient computed from known token rewards; a nonzero difference would show the unbiasedness claim fails for non-additive reward functions.","supporting_citations":[{"cited_title":"Reinforcement learning: An introduction","cited_arxiv_id":null,"evidence_quote":"Supplies the classical policy gradient theorem that Lemma 1 adapts to the LLM setting where states are token prefixes and sampling comes directly from the policy."},{"cited_title":"High-dimensional continu- ous control using generalized advantage estimation, 2018","cited_arxiv_id":null,"evidence_quote":"Provides the GAE advantage estimation that the paper contrasts with the baseline choices of GRPO, ReMax, and RLOO."}],"review_version":1}