{"id":"f89a7a15-246f-4743-8796-4e45b8544e7e","arxiv_id":"2411.08302","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"RED assigns token-level rewards by taking the difference of a sequence reward model's scores on adjacent prefixes, improving RLHF training without additional reward-model training.","lead":"This paper presents RED, a way to turn a single overall score for an AI's whole answer into separate small scores for each word. The method uses only the existing reward model and adds almost no extra training, and the authors show better answers on several language tasks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"RED's central mechanism assumes the reward model's value head gives meaningful scores on incomplete prefixes, but Eq. 6 is only validated by a coarse 20-annotator study; if those prefix scores are uncalibrated, the claimed per-token credit is a potential-shaping term with no demonstrated benefit.","rationale":"The reader identified the out-of-distribution prefix-score assumption as the weakest point, and I agree that this is the most load-bearing empirical risk. The theoretical claim of unchanged optimal policy is actually robust: because Eq. 6 is a state-dependent potential difference, the policy-gradient direction and optimal policy are preserved even for arbitrary prefix potentials, so the concern is not that RED breaks RLHF theory. Rather, the concern is that the method's advantage over sparse-reward RLHF depends on prefix scores carrying real credit information, and that premise is only weakly tested. I also flag that the SafeRLHF results in Tables 3 and 4 undermine the 'consistent superiority' claim, which strengthens the need for a direct prefix-validity check plus multi-seed evaluation. The reader's CONDITIONAL verdict already captures these addressable risks, so I do not propose changing the verdict; I would only sharpen the requested validation.","tokens_in":30111,"tokens_out":8956,"duration_ms":112257,"concrete_test":"Use a fixed reward model and a held-out set of 100 completions. Compute Eq. 6 token rewards for each response, then obtain independent token-level importance judgments by having human annotators mark each token as essential, neutral, or harmful to the response's quality. Report the Spearman rank correlation and per-sequence sign agreement between Eq. 6 rewards and these judgments. If the correlation is near zero or negative on a substantial fraction of sequences, the premise that prefix scores reflect token credit fails; if it is consistently positive, the OOD concern is substantially mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central mechanism is Eq. 6: r_t^RM = R_phi(x, y_<=t) - R_phi(x, y_<=t-1). The reward model is trained by Eq. 2 on complete responses only, so its value head is never trained to output calibrated scores for prefixes. For a causal transformer, applying a final-position-trained head to every hidden state yields deterministic but potentially arbitrary numbers. The telescoping-sum property preserves the total return, so the optimal-policy argument in Section 3.3 is not threatened; what is threatened is the claimed benefit. The method is only better than sparse RLHF if these prefix scores encode genuine token-level credit. The paper's only direct support is Appendix C.2: 20 annotators, 10 questions each, 97% 'reasonable' -- a coarse check of whether highlighted tokens look important, not a measurement of prefix-score calibration or correlation with actual token influence. Moreover, Tables 3 and 4 show RED is not consistently superior: on SafeRLHF/LLaMA3, PPO-R.S-RED scores 14.242 vs. 14.870 for PPO-R.S (win rate 47.66% vs. 58.08%), and RLOO-R.S-RED raises the cost score from 0.852 to 2.049. Since the headline claim is that token-level rewards from holistic feedback improve RLHF, the out-of-distribution prefix-score assumption is the load-bearing, under-validated step.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RED (REward reDistribution), a method that converts the sparse, sequence-level reward produced by an off-the-shelf reward model into token-level rewards by taking the temporal difference of the reward model's scores on partial sequences (Eq. 6). The authors argue that the sum of these redistributed rewards equals the original sequence reward up to a prompt-dependent constant, so the optimal policy is preserved via potential-based reward shaping or return-equivalent Sequence-MDP arguments. They integrate RED into PPO and RLOO and report experiments on question answering (Nectar), summarization (TL;DR), and safety/helpfulness (SafeRLHF), comparing against sparse-reward baselines, attention-based credit assignment (ABC), reward shaping, and Lagrangian methods. The central theoretical claim is that token-level credit can be obtained at negligible cost and without retraining the reward model; the central empirical claim is that RED consistently improves the underlying RLHF methods.","tokens_in":30356,"tokens_out":4506,"duration_ms":53708,"significance":"If the claims hold, RED is an attractive and simple contribution to RLHF: it requires no additional training, no reward-model modification, and no human token-level labels, and it is grounded in a clean potential-shaping argument. The derivation is transparent and the connection to existing sequence-level RLHF is clearly drawn. The paper also gives credit to the literature on return-equivalent SDPs and potential-based shaping, and it explicitly attempts to evaluate the quality of the redistributed rewards (Appendix C.2). However, the empirical evidence as presented does not yet support the strong claim of consistent superiority: several table entries contradict the headline, no uncertainty quantification is reported, the reward-model evaluation is partly circular, and the core assumption that reward models produce meaningful scores on incomplete prefixes is only weakly validated. The manuscript is likely to be of interest to the RLHF community, but the empirical claims need substantial revision and additional evidence before publication.","major_comments":[{"comment":"The method's central mechanism assumes that R_phi(x, y_<=t), the reward model's output when its value head is applied to an incomplete prefix, is a meaningful, calibrated estimate of partial-sequence quality. However, the reward model is trained by Eq. (2) exclusively on complete responses, so nothing guarantees that these prefix scores are not arbitrary or noisy. The only direct evidence provided is Appendix C.2, a coarse human study in which 20 annotators rated 10 questions each and 97% of token-wise reward sequences were deemed 'reasonable'; this checks whether highlighted tokens look important, not whether the prefix scores are calibrated or correlate with actual token influence. Since the claimed benefit of RED over sparse RLHF depends entirely on this assumption, the manuscript needs direct validation: e.g., comparison of prefix-score calibration against held-out partial-response judgments, correlation of RED token rewards with token-level human importance ratings, or ablations with deliberately corrupted prefix scores. As written, the OOD prefix-score concern is load-bearing and under-validated.","section":"Section 3.2, Eq. (6), and Figure 2"},{"comment":"The paper's abstract and Section 4 claim that RED 'consistently improves' the baseline methods, but Table 4 contradicts this claim. On SafeRLHF with LLaMA3, PPO-R.S-RED achieves a reward score of 14.242 and a win rate of 47.66%, both worse than PPO-R.S's 14.870 and 58.08%; on the same setup, RLOO-R.S-RED raises the cost score from 0.852 to 2.049 and lowers the safe rate from 44.19% to 36.99%. These are not small or ignorable discrepancies, and they undermine the central empirical claim. The authors should either present a more nuanced claim, analyze why RED fails in these configurations, or provide additional evidence that the overall pattern is nevertheless positive. The current presentation is not proportionate to the data.","section":"Section 4.4, Tables 3 and 4"},{"comment":"The reward-model evaluation is partially circular. The primary evaluation metric is the average reward score of an off-the-shelf reward model, and this same reward model is used to generate the dense training signal in Eq. (6) and Eq. (7). It is therefore expected that RED-trained policies obtain higher scores from that reward model, even if the underlying human-aligned quality is unchanged or worse. The paper does include GPT-4 and human evaluations, which is commendable, but those are not reported for every configuration and in some cases (e.g., Table 2's TL;DR results) the GPT-4 advantage is much smaller than the reward-score advantage. The authors should report the full set of GPT-4/human evaluations for all main configurations and, ideally, show the correlation between reward-model scores and independent human judgments to establish that the reward-score gains are not merely an artifact of training toward the same model.","section":"Section 4.1 and Tables 1, 2, 4"},{"comment":"No error bars, standard deviations, or significance tests are reported anywhere in the empirical sections. All tables present single point estimates, despite the stochastic nature of RL training and the small evaluation sets (e.g., GPT-4 win rates on what appear to be 200-example sets). Given the variability visible in the training curves in Figure 5 and the mixed results in Table 4, the current evidence is insufficient to support the conclusion that RED is consistently superior. The authors should report means and standard deviations over multiple seeds and, where feasible, statistical significance tests or at least bootstrap confidence intervals for the main win-rate comparisons.","section":"Section 4, Tables 1-4 and Appendix C.2"}],"minor_comments":[{"comment":"The word 'reward redistribition' is misspelled in the abstract and in the first occurrence in Section 1; it should be 'reward redistribution'.","section":"Abstract and Section 1"},{"comment":"The notation R_phi(x, y_<=-1) is introduced in the display equation after Eq. (6), but it is used before it is defined. Define R_phi(x, y_<=-1) := R_phi(x, ∅) immediately before Eq. (6).","section":"Section 3.2"},{"comment":"The numbers in the right panel of Figure 1 are difficult to read: '0.400.10' and similar strings appear to be concatenated values. The figure should be reformatted so that each token's reward is clearly separated and legible.","section":"Figure 1"},{"comment":"The entry '48..0%' in the RLOO-RED vs. RLOO TL;DR row appears to be a typo; it should likely be '48.0%'.","section":"Table 12"},{"comment":"The formula for the reward-shaping baseline, r_t^agg = 1/2 * (r_tilde_t + alpha * c_tilde_t) with alpha = -1, is stated with a sign convention that is easy to misread. Since the cost model 'assesses how harmful each generation is', the text should clarify whether higher cost scores are worse and how the alpha = -1 combination maps to the reward used in training.","section":"Section 4.4"},{"comment":"The GPT-4 evaluation prompt for SafeRLHF explicitly instructs the evaluator to prioritize harmlessness, which explains why the GPT-4 results diverge from the reward-model results. This is a relevant methodological point and should be mentioned in the main text when the mismatch is discussed, rather than only in the appendix.","section":"Appendix B.5 and Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The theoretical core of the paper is sound and the method is simple enough to be widely adopted if the empirical claims can be substantiated. The main barrier is the gap between the strong claim of consistent superiority and the actual table entries, combined with the lack of uncertainty quantification and the partly circular reward evaluation. I would encourage the editors to ask for a revised version that addresses the OOD prefix-score validation, reanalyzes the negative results in Table 4, and reports variance across seeds. Also note that no code or data release is mentioned; given the simplicity of the method, a public implementation would materially help reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know about this one: it applies potential-based reward shaping to RLHF by using the reward model's own value head on intermediate prefixes. The core equation is r_t = R_phi(x,y<=t) - R_phi(x,y<=t-1), a temporal difference whose sum telescopes to the original sequence reward. The optimal-policy invariance argument is sound (with gamma = 1), and the authors correctly cite Ng et al. and Chan et al. The specific move—using an off-the-shelf reward model's prefix logits to get token-level rewards without retraining—is new relative to the cited literature, and it is a genuinely useful trick if the prefix scores are meaningful.\n\nWhat the paper does well: the method is simple, cheap, and easy to bolt onto PPO or RLOO. The presentation is direct, and the connection to potential-based shaping is acknowledged rather than hidden. Experiments cover three tasks and three base models, with multiple evaluation types. The appendix even tries to connect RED to DPO's implicit credit assignment, which is an interesting direction.\n\nWhere it gets soft. First, the empirical support is thinner than the \"consistent superiority\" claim. There are no error bars, no seeds, no code. Table 4 shows a counterexample: on SafeRLHF with LLaMA3, PPO-R.S-RED scores 14.242 vs. 14.870 for PPO-R.S, and RLOO-R.S-RED raises the cost score from 0.852 to 2.049 while the safe rate drops. That is not consistent superiority. Second, the reward-model evaluation is partly circular: the same reward model that defines the dense training signal is used to score the results, so reward-score gains are partly expected. Third, the central assumption—that a reward model trained only on complete sequences gives calibrated scores on prefixes—is only validated by a 20-annotator study asking whether highlighted tokens look reasonable. That is a coarse check, not a measurement of calibration or correlation with true token influence.\n\nThere is also a technical issue in Appendix A.2. The derivation of the DPO connection factorizes the optimal sequence-level policy into per-token Boltzmann factors with per-token partition functions. That factorization is not generally valid because the per-token normalizers depend on the prefix and do not multiply to the sequence partition function. This does not threaten the main invariance claim, which rests on potential-based shaping, but the appendix should be corrected.\n\nWho this is for: anyone building RLHF pipelines who wants cheap dense rewards. If the prefix-score assumption holds, this is a solid contribution. The flaws are addressable with seeds, error bars, code, a direct test of prefix-score validity, and more hedged claims about the multi-reward setting. I would send it to review rather than desk-reject.","headline":"RED is a clean, honest shaping trick—sound theory and a simple method, but the empirical case is under-validated and the appendix has one suspect derivation.","tokens_in":30951,"tokens_out":5430,"would_cite":true,"duration_ms":59639,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T50","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"RED assigns each generated token a reward by subtracting the reward model's score for the prefix so far from its score one step later, yielding fine-grained credit without retraining the reward model, at minimal extra cost, and with the…","keywords":["reward redistribution","token-level rewards","RLHF","credit assignment","sequence reward model","potential-based reward shaping","PPO","RLOO"],"falsifier":"A concrete test: compare the reward model's prefix scores $R_\\phi(x, y_{\\le t})$ with human ratings of partial responses across a held-out set. If the prefix scores are roughly flat, dominated by prompt length, or only weakly correlated with quality on prefixes, the difference rewards carry little signal, and RED's improvement over sparse-reward RLHF should disappear in a controlled comparison.","tokens_in":29872,"feed_emoji":"🎯","tokens_out":4493,"duration_ms":41927,"temperature":0.7,"pith_summary":"RED claims that the sparse, sequence-level reward used in RLHF can be converted into token-level rewards at negligible extra cost, by subtracting the reward model's score for the prefix so far from its score one token later. Because the reward model's value head already returns a scalar for any prefix, the method needs no retraining, no architectural change, and none of the extra reward-model training that earlier dense-reward methods require. The key identity is that these per-token rewards telescope back to the original sequence score, so the redistributed training objective has the same optimal policy as standard RLHF. On question answering, summarization, and safety-alignment tasks, RED improves both PPO and RLOO over sparse-reward baselines and over attention-based credit assignment. If correct, the method makes fine-grained credit assignment a drop-in upgrade for essentially any RLHF pipeline that already has a reward model.","feed_headline":"Dense token rewards without retraining the reward model","feed_subtitle":"Subtracting prefix scores turns one holistic score into per-token credit and improves RLHF on three tasks.","key_machinery":"The machinery is the Sequence-MDP view of language generation, used with a time-difference decomposition of the reward model's own scores. The reward model is treated as a potential function on prefixes; each token's reward is the change in potential it induces. This makes RED a form of potential-based reward shaping, and the paper proves that the advantage function is unchanged, so the optimal policy under RLHF is preserved. A convex combination hyperparameter $\\beta_c$ blends token-level and sequence-level rewards, and the method plugs into PPO and REINFORCE-style (RLOO) algorithms.","core_discovery":"The paper's central claim is that the sparse, sequence-level reward used in RLHF can be converted into token-level rewards by temporal differentiation: $r_t^{\\mathrm{RM}} = R_\\phi(x, y_{\\le t}) - R_\\phi(x, y_{\\le t-1})$. Because the reward model's value head already produces a scalar for every prefix, no retraining or architectural change is needed. The sum of these per-token rewards telescopes to the original sequence score, so the redistributed MDP is return-equivalent to the original one, and the optimal policy is preserved. Experiments on question answering (Nectar), summarization (TL;DR), and safety alignment (SafeRLHF) with PPO and RLOO show consistent improvement over sparse-reward baselines and over attention-based credit assignment (ABC).","pith_inferences":["A natural extension is step-level credit for reasoning tasks by segmenting a response into reasoning steps and applying the same difference rule to step-boundary prefixes; the paper notes token-level assignment is too fine for multi-step reasoning.","The difference reward is essentially a first-order sensitivity measure of the reward model; one could analyze its variance to detect reward-hacking-prone tokens or to debug reward models.","One could test RED with any reward model family; if prefix scores are poorly calibrated, RED should underperform, making prefix-score calibration a cheap predictor of whether RED will help a given pipeline.","The convex-combination hyperparameter $\\beta_c$ may interact with the KL penalty, and the sensitivity analysis in the paper only varies $\\beta_c$ on one task, so that interaction is worth mapping."],"forward_implications":["Any RLHF pipeline that already has a sequence reward model can obtain dense token rewards at negligible extra compute by reusing the same value head on prefixes.","RED is agnostic to the RL algorithm; it improves both PPO and RLOO, and can be applied separately to reward and cost models in multi-objective safety alignment.","Because the optimal policy is unchanged, RED's gains are training-efficiency gains, not a shift in the alignment objective.","The paper shows DPO implicitly performs a form of reward redistribution, suggesting dense-credit methods and preference optimization are connected through the same telescoping identity.","In multi-reward safety settings, redistributed cost signals reduce cost scores and increase safety rates relative to sparse baselines in most configurations."],"supporting_citations":[{"why":"Supplies the Sequence-MDP and return-equivalence theory that justifies redistribution preserving the optimal policy.","marker":"(Arjona-Medina et al., 2019)"},{"why":"Provides the potential-based reward shaping result used to prove the optimal policy is unchanged.","marker":"(Ng et al., 1999)"},{"why":"Provides the equivalence-class and optimal-policy analysis, and the DPO connection used in the proof.","marker":"(Rafailov et al., 2024b)"},{"why":"Introduces ABC, the main comparison method, and the convex combination formulation with $\\beta_c$.","marker":"(Chan et al., 2024)"},{"why":"Supplies the RLOO baseline and REINFORCE-style optimization used in the experiments.","marker":"(Ahmadian et al., 2024)"},{"why":"Sets up the SafeRLHF benchmark and the reward/cost model training used in the safety task.","marker":"(Dai et al., 2023)"}],"fun_headline_variants":["Per-token rewards without retraining the reward model","Subtract prefix scores to get token-level rewards","Fine-grained RLHF via reward redistribution","Token-level credit from off-the-shelf reward models","Sparse to dense rewards for RLHF"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a reward model trained on complete responses produces meaningful scores when its value head is applied to incomplete prefixes; if those prefix scores are noisy or uncalibrated, the redistributed token rewards can mislead training rather than help it.","fun_headline_variants_meta":{"raw":{"variants":["Per-token rewards without retraining the reward model","Subtract prefix scores to get token-level rewards","Fine-grained RLHF via reward redistribution","Token-level credit from off-the-shelf reward models","Sparse to dense rewards for RLHF"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000205,"raw_usage":{"total_tokens":1363,"prompt_tokens":885,"completion_tokens":478,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":408}},"tokens_in":501,"tokens_out":478,"duration_ms":5041,"temperature":1.0,"reasoning_tokens":408,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:44:44.495087+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: compare the reward model's prefix scores $R_\\phi(x, y_{\\le t})$ with human ratings of partial responses across a held-out set. If the prefix scores are roughly flat, dominated by prompt length, or only weakly correlated with quality on prefixes, the difference rewards carry little signal, and RED's improvement over sparse-reward RLHF should disappear in a controlled comparison.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Sequence-MDP and return-equivalence theory that justifies redistribution preserving the optimal policy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the RLOO baseline and REINFORCE-style optimization used in the experiments."}],"review_version":1}