Pith. sign in

REVIEW 4 major objections 6 minor 46 references

The paper argues that RLHF instability follows from driving token-level updates with a single sequence-level reward, and that a sentence-to-token reward decomposition stabilizes training while preserving preference alignment.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 13:54 UTC pith:7QAX6MQQ

load-bearing objection A genuinely new granularity-aware reframing of dense reward RLHF, with a plausible mechanism but under-powered stability evidence and an underdetermined influence estimate. the 4 major comments →

arxiv 2607.18258 v1 pith:7QAX6MQQ submitted 2026-05-15 cs.AI

S2T-RLHF: Hierarchical Credit Assignment for Stable Preference-Based RLHF

Wei Chen , Guanghui Zhu , Yafei Li , Limin Wang , Yihua Huang This is my paper
classification cs.AI
keywords RLHFcredit assignmentreward decompositionpreference-based rewardtraining stabilitysentence-level allocationtoken-level refinementbargaining solution
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that RLHF instability is not an implementation bug but a structural mismatch: a single sequence-level reward is propagated to token-level policy updates, making credit assignment within a response ambiguous and amplifying noise. It claims that the common fix of denser token-level rewards is incomplete, because when preference signals are noisy and only defined at the response level, over-refinement can amplify uncertainty. The proposed alternative is a granularity-aware principle: allocate reward by semantically meaningful units, sentences first, then refine within sentences in a bounded way. This two-stage decomposition, S2T-RLHF, is reported to improve training stability and robustness while maintaining competitive preference alignment across datasets and optimization settings.

Core claim

The paper's central claim is that RLHF instability is a structural consequence of driving token-level optimization with low-resolution sequence-level rewards. Because human preferences are expressed at the granularity of sentences and semantic units, directly deriving token-level rewards from a global scalar is ill-posed. S2T-RLHF therefore decomposes the fixed sequence-level reward in two stages: first allocating it across sentences via a scale-invariant bargaining mechanism over latent semantic influence vectors, then applying bounded, simplex-constrained token-level refinement inside each sentence. The paper argues that this hierarchy preserves total reward mass while breaking sign-aligne

What carries the argument

The load-bearing mechanism is the two-stage reward decomposition. Stage I estimates per-sentence latent 'semantic influence vectors' by ridge regression on deletion-based perturbations, builds a Gram matrix from these vectors, and solves a bargaining fixed-point equation (formally Mβ = β^{−1}, with a numerical safeguard) to allocate the sequence reward across sentences while conserving total reward mass. Stage II uses a lightweight Dirichlet Token Allocation Network that maps token hidden states to positive concentration parameters; the normalized expectation yields token weights within each sentence, and the PPO advantage is bounded by clipping so that token-level modulation cannot arbitrar

Load-bearing premise

Everything rests on the Stage-I 'semantic influence vectors' being meaningful: they are estimated from at most three reward differences against a 3584-dimensional embedding via ridge regression, so if the signal is mostly regularization noise, the sentence allocation and the semantic-bargaining explanation collapse.

What would settle it

Take the same responses and replace the estimated influence vectors with random unit vectors of the same norm before running Stage I; if the reported stability and win-rate advantages survive, then the semantic estimation is not load-bearing. A sharper check: compare sentence reward shares from the ridge-based vectors against shares from random vectors on the same responses—if the two allocations are statistically indistinguishable, the 'reward-sensitive direction' claim is unsupported.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If correct, dense-reward RLHF design should aim to match reward granularity to the semantic structure of language rather than to maximize credit-assignment precision.
  • Token-only refinement may improve late-stage optimization but increases early-stage instability; sentence-level allocation is the main stabilizer, and combining both gives the best stability-alignment trade-off.
  • S2T-RLHF shows smaller objective oscillation and lower policy drift than sequence-level RLHF while retaining a win advantage over baselines under both an LLM judge and a learned reward-model evaluator.
  • Stability gains come from structural regularization rather than added capacity: pretraining the token-allocation network did not improve stability and sometimes slightly worsened it.
  • Overly fine-grained decomposition methods exhibit reward collapse under aggressive learning rates, while S2T-RLHF remains comparatively stable.
  • The method adds moderate computational overhead, dominated by perturbation-based influence estimation, with the bargaining solver and token refinement being lightweight.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Inference: the granularity-alignment principle predicts that as reward-model noise increases, the optimal decomposition granularity should coarsen; this is a testable consequence the paper does not directly test.
  • Inference: the theory's reliance on conservation of reward mass and bounded modulation suggests that other allocation mechanisms with these two properties could yield similar stability, so the specific bargaining and Dirichlet choices may be replaceable.
  • Inference: the overhead analysis points to amortized or cached influence estimation as a practical path toward scaling the stability gains to larger models and longer generations.
  • Inference: if the instability analysis is right, similar hierarchical credit assignment may apply to step-level reasoning or code-generation tasks, though the paper itself notes that sentence units may be too coarse for tightly coupled symbolic reasoning.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes S2T-RLHF, a two-stage hierarchical reward decomposition for preference-based RLHF. In Stage I, a fixed sequence-level reward is allocated to sentences via a Nash-bargaining-style fixed-point procedure built from 'latent semantic influence vectors' estimated by ridge regression on deletion-based perturbations. In Stage II, each sentence reward is redistributed across tokens by a Dirichlet-expectation allocation network (DTAN) with a clipped mixing into the PPO advantage. The central claims are (i) that RLHF instability is structural, arising from propagating low-resolution sequence-level rewards to token-level updates, and (ii) that a granularity-aware, stability-oriented decomposition—sentences first, bounded token refinement second—improves training stability and robustness while preserving preference alignment. The paper includes theoretical remarks in Appendices D–E connecting the two-stage design to mitigation of advantage alignment, variance amplification, and length-dependent gradient scaling, plus experiments on HH-RLHF with zero-shot evaluation on safety benchmarks.

Significance. If the empirical and theoretical claims hold, the paper makes a genuinely useful conceptual contribution: it challenges the assumption that finer-grained credit assignment is always better, and it demonstrates a concrete, reward-model-agnostic decomposition that conserves reward mass and bounds token-level modulation. The method does not require reward-model retraining or token-level supervision, and the comparison against standard RLHF and two dense-reward baselines (ABC, SCAR) is an external, meaningful evaluation. The paper also contains self-critical limitation statements (Appendix D.3), honest runtime analysis (Appendix K), and mechanism-swap ablations (Appendix I.3) that separately test the allocation mechanisms. However, the current evidence is not strong enough to establish the headline stability claim: the key quantitative stability comparisons lack uncertainty quantification, and the semantic-influence estimation at the core of Stage I is numerically underdetermined. The central idea is plausible and the issues are fixable, but the manuscript as written overstates the strength of its conclusions.

major comments (4)
  1. [Equation (2), Appendix C.3] The abstract and Section 4.2 claim that S2T-RLHF 'improves training stability' and that Obj-Δ drops by 'an order of magnitude' relative to RLHF. The supporting evidence is statistically fragile. In Table 9, RLHF Obj-Δ is 0.01134 ± 0.03368 while S2T-RLHF is 0.00101 ± 0.01299; in Table 10, 0.01106 ± 0.02562 vs 0.00719 ± 0.01361. If the ± values are standard deviations across three seeds, the intervals overlap substantially, and the order-of-magnitude reduction is a point estimate. No per-seed trajectories, bootstrapped confidence intervals, or significance tests are reported. Similarly, the win-rate tables (Tables 1–4, 11–12) report no uncertainty; for example, the SCAR comparison (48.1 vs 43.1 and 45.8 vs 44.1) could easily be within noise. Because the stability claim is load-bearing, the authors should either provide proper uncertainty quantification (e.g., CIs, per-seed results, permuta
  2. [Appendix E, Propositions E.3–E.5] The Stage-I allocation rests on estimating a d=3584-dimensional 'latent semantic influence vector' v_i by ridge regression from at most K_p=3 deletion-based perturbations, with ridge λ=10^-3. A 3584-dimensional direction is numerically underdetermined from three (ΔR, δ) pairs; the resulting v_i—and hence M, u, and the sentence shares r_i in Eq. (8)—may be dominated by regularization artifacts. If v_i is noise-driven, the reported alignment gains cannot be attributed to semantic bargaining. The paper should validate the influence estimates (e.g., with more perturbations, dimensionality reduction, perturbation stability checks, or an ablation using random/zero influence vectors). Without such validation, the proposed mechanism's causal role in the empirical results is unsupported.
  3. [Corollary B.3] The theoretical mitigation results are closer to consequences of the method's definitions than to quantitative theorems. For example, Proposition E.3 relies on the unformalized Remark D.1 assumption that |V(s_t)| << |R(y)| for 'a substantial portion' of the trajectory; Propositions E.3–E.5 then argue that conservation of reward mass and bounded token modulation 'weaken', 'attenuate', or 'mitigate' the three instability sources without giving measurable bounds or a comparison of the resulting advantage variance/gradient norms. As stated, these are qualitative justifications rather than proofs. The authors should either formalize the assumptions and quantify the claimed reductions (e.g., an upper bound on Var(A_t^S2T) in terms of localized reward support) or explicitly label these remarks as intuitive/heuristic and not invite the reader to rely on them as theorems.
  4. [Preferences] Corollary B.3 claims that if M=V^T V is positive definite then the fixed-point equation M β = β^{-1} has a positive solution. But M is positive semidefinite by construction; it is positive definite only when the influence vectors are linearly independent, which is not guaranteed by the generative process in Eq. (2) and is especially questionable when the number of sentences K is small or when v_i are dominated by ridge regularization. The proof also asserts existence by coercivity of φ, but does not establish uniqueness or that the damped iteration in Eq. (7) converges to the fixed point. These gaps do not invalidate the method, but the theoretical claims should be corrected to either state sufficient conditions more carefully or weaken the statements.
minor comments (6)
  1. [Section 3.4, Eq. (7)] The tables report win/tie/lose percentages but no number of evaluation prompts, so the practical significance of differences such as 53.0 vs 36.5 cannot be assessed. Please report sample sizes and, ideally, binomial confidence intervals.
  2. [Table 11] The damped fixed-point iteration is described as a 'solver' but no convergence guarantee is given. Lemma B.4 only shows positivity preservation. If the iteration is not guaranteed to converge to a fixed point, the phrase 'After convergence' in Eq. (8) needs qualification; otherwise, report the convergence diagnostics (e.g., maximum residual, fraction of runs hitting the iteration limit).
  3. [Appendix I.2] The row labeled 'Toxicity' presumably refers to RealToxicityPrompts, while the text lists RealToxicityPrompts and ToxiGen separately. Please use consistent dataset names.
  4. [Notation] The text after Table 12 repeats a paragraph verbatim. Please remove the duplication.
  5. [Figures 2–4] The paper switches between R(x), R(seq), and R(s) for sequence-level and sentence-level rewards, sometimes within the same section (e.g., Eq. (3) vs Eq. (10) vs Appendix C.2.2). A single notation table or clearer subscript convention would improve readability.
  6. [Appendix G] The main-text trajectories show only smoothed curves. Please add either error bands across seeds or a note specifying that they are representative runs, since the quantitative stability claim is made over three seeds.

Circularity Check

1 steps flagged

Theoretical mitigation proofs are largely definitional, but the empirical alignment comparisons are external and non-circular.

specific steps
  1. self definitional [App. C.2.2–C.2.3 (Eqs. 26–28); App. D.2 (Remarks D.4–D.6); App. E (Props E.3–E.5)]
    "At =A (seq) · h (1−λ) +λ·clip wt ¯w i (Eq. 28) ... 'This construction preserves the expected advantage magnitude, Et[Araw t ] =A (seq)' (Eq. 27) ... 'Under S2T-RLHF , the sequence-level reward is first decomposed into sentence-level rewards {R(s)}S s=1 ... Consequently, the resulting update signal is no longer driven by a single sequence-wide scalar' (Prop. E.3)."

    Props E.3–E.5 are presented as an end-to-end theoretical justification for stability, but their premises are the method's own design definitions: Eq. 27 imposes E_t[A_t]=A(seq) by weight normalization, Eq. 28 imposes boundedness by clipping, Lemma E.1 imposes conservation by summing normalized utilities, and Lemma E.2 imposes the simplex by definition of c_{s,t}. The conclusions (alignment broken, variance reduced, length decoupled) are restatements of these construction constraints rather than independent derivations. Moreover, Remarks D.4–D.6 assume localized reward support r_t, while the implemented advantage in Eq. 28 still multiplies the same sequence-level A(seq) by non-negative weights, so the claimed 'no longer driven by a single sequence-wide scalar' is asserted by the conceptual

full rationale

The paper's empirical core—pairwise win rates against RLHF/ABC/SCAR under two evaluators, cross-dataset and model-family robustness checks, and stage/mechanism ablations—is genuinely external and not fitted to the reported preference outcomes, so the central alignment claim has independent support. There is no load-bearing self-citation chain: the cited prior work is external, and no uniqueness theorem is imported from the authors' own results. The only circular/definitional element is the Appendix E theoretical narrative: the mitigation propositions largely restate the design constraints of the advantage construction (conservation, bounded clipping, simplex weights) as conclusions, and the D.4–D.6 analysis assumes the localized-reward support that the conceptual decomposition defines rather than establishes for the implemented PPO signal. This is a localized partial circularity in the theoretical justification, not in the external empirical evaluation. Statistical concerns (overlapping seed-level intervals for Obj-Δ, underdetermined ridge regression for v_i, unformalized |V(s_t)| << |R(y)| assumption) are correctness risks rather than circularity and do not further raise the circularity score.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 2 invented entities

The method's central claim needs: a trusted encoder embedding; a reward model whose scalar scores are valid; influence vectors that genuinely reflect reward sensitivity despite severe underdetermination; sentences that are the right semantic units; and the unstated clipping/mixing constants of Eq. (28). The theory adds two unformalized distributional premises (Remark D.1's magnitude assumption, Remark D.3's regularity assumptions). None of these come with independent evidence in the paper.

free parameters (5)
  • Ridge regularization λ for influence estimation = 10⁻³
    Controls the latent influence vector v_i estimated from ≤3 perturbation samples per sentence; with severe underdetermination (d=3584) this prior largely determines v_i's direction and norm, which flows into M, u, and the sentence shares r_i. (§3.4 Eq. 2; Appendix C.3)
  • Perturbations per sentence Kp = 3 (max)
    The number of deletion variants per sentence; with λ=10⁻³, three samples cannot constrain a 3584-dim direction, so allocation quality depends on this ad hoc choice. (Appendix C.3)
  • Advantage mixing coefficient λ = not reported
    Eq. (28) blends the sequence-level advantage with the clipped token modulation and thereby sets how much token-level refinement is actually allowed; no value or clip bounds are given in Tables 5-8, though the method's stability properties hinge on it. (C.2.3)
  • DTAN concentration MLP f_φ = hidden 1024, GELU, input 3584
    Learned jointly with the policy during RLHF; determines within-sentence token weights. It is trained on the same reward signal it modulates, so its allocation is fitted rather than validated out-of-sample. (§3.5; Table 7)
  • Solver damping η / safeguard ε = 0.5 / 10⁻⁸
    Fixed-point and ε-floor settings of the bargaining solver; ε pins utilities of sentences whose Mβ entry is non-positive, affecting the allocation near the safeguard. (Table 6; C.1)
axioms (6)
  • domain assumption |V(s_t)| ≪ |R(y)| for a large portion of the trajectory, when the value function is imperfect
    Stated in Remark D.1 as 'reasonable to assume'; the entire advantage-alignment and length-scaling analysis (D.1, Props E.3/E.5) depends on it. If the value baseline is actually informative, the sign-alignment mechanism and its variance claims do not follow.
  • domain assumption Mild regularity: non-vanishing diagonal contributions and non-negative correlations of token gradients over time
    Remark D.3 invokes these to conclude the policy-gradient norm grows at least linearly with sequence length T; no distributional model is specified, so the claim is an assertion rather than a theorem.
  • domain assumption Sentences are the semantically natural granularity at which human preference is expressed
    Asserted in §1 and §3.3 ('sentences express coherent semantic intent'); the whole two-stage design presumes sentence units are the right intermediate resolution, and the paper itself concedes in D.3 that tightly coupled reasoning tasks (math proofs, code) may not fit.
  • ad hoc to paper Deletion of content words is a valid semantic perturbation whose reward/embedding deltas reveal causal sentence influence
    Eqs. (1)-(2) and Appendix C.3 build the influence vectors on single-token deletions; deletion is a proxy for counterfactual semantic variation and only 1-3 variants are used, so the causal reading of the regression is assumed rather than validated.
  • standard math M = VᵀV positive definiteness for existence of the Nash-equilibrium fixed point
    Corollary B.3 requires M ≻ 0 (linearly independent influence vectors); otherwise the existence argument fails and the ε-safeguard (with unexplained convergence of the damped iteration) is the only support. The paper notes 'at least one solution under mild regularity conditions' and relies on the safeguard.
  • domain assumption The reward model RM-Gemma-2B scoring is a meaningful supervision signal whose decomposition is sensible
    The whole method redistributes R(x); if R is misspecified or noisy the decomposition propagates the misspecification — a limitation the paper itself states in D.3.
invented entities (2)
  • latent semantic influence vector v_i no independent evidence
    purpose: Summarize, per sentence, the reward model's sensitivity to semantic perturbations, and define the bargaining matrix M=VᵀV
    An internal ridge-regression construct (Eq. 2) with no falsifiable handle outside the paper; its semantic interpretation is asserted, and with ≤3 samples in d=3584 it may be a regularization artifact.
  • Nash-style bargaining equilibrium (β*, u*) no independent evidence
    purpose: Produce scale-invariant, symmetric sentence reward shares (Eqs. 4-8)
    A modeling construct: the utilities are defined as linear functions of the bargaining coefficients themselves (ũ=Mβ), so the 'bargaining' is a fixed-point allocation rule rather than an empirical finding; no external benchmark tests the claim that it reflects semantic influence.

pith-pipeline@v1.3.0-alltime-deepseek · 26266 in / 25529 out tokens · 212359 ms · 2026-08-02T13:54:38.530046+00:00 · methodology

0 comments
read the original abstract

Reinforcement learning from human feedback (RLHF) with preference-based reward models often exhibits unstable training dynamics. A key contributing factor is that standard RLHF relies on a single sequence-level scalar reward, which is propagated to token-level policy updates and leaves credit assignment within a response inherently ambiguous. Recent work has attempted to address this issue by refining rewards into denser token-level supervision, often relying on the implicit assumption that finer-grained credit assignment improves optimization. We argue that this assumption is incomplete: when preference signals are noisy and only defined at the response level, overly fine-grained reward refinement can amplify reward uncertainty and destabilize learning. To address this problem, we propose a granularity-aware principle for hierarchical credit assignment, emphasizing stability-oriented reward design rather than maximal allocation precision. Under this principle, sentences serve as a natural intermediate granularity, balancing semantic coherence with robustness to token-level noise. Guided by this view, we introduce S2T-RLHF. This sentence-to-token reward decomposition framework first allocates sequence-level preference rewards across sentences and then applies bounded token-level refinement within each sentence, without reward-model retraining or token-level supervision. Experiments across multiple datasets and optimization settings show that S2T-RLHF improves training stability and robustness while maintaining competitive preference alignment.

Figures

Figures reproduced from arXiv: 2607.18258 by Guanghui Zhu, Limin Wang, Wei Chen, Yafei Li, Yihua Huang.

Figure 1
Figure 1. Figure 1: Overview of S2T-RLHF. A global sequence-level reward is first decomposed into sentence [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: KL divergence to the reference policy during training. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Reward trajectory during training [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Policy entropy evolution during training. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Early-stage training dynamics under learning rate [PITH_FULL_IMAGE:figures/full_fig_p032_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Early-stage training dynamics under learning rate [PITH_FULL_IMAGE:figures/full_fig_p033_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Early-stage training dynamics under learning rate [PITH_FULL_IMAGE:figures/full_fig_p034_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Early-stage training dynamics under KL constraint [PITH_FULL_IMAGE:figures/full_fig_p035_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Training-time reward trajectories under the optimization reward model (RM-Gemma-2B). [PITH_FULL_IMAGE:figures/full_fig_p039_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Illustrative training dynamics under fine-grained reward decomposition with a larger [PITH_FULL_IMAGE:figures/full_fig_p040_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 22 linked inside Pith

  1. [1]

    Deep reinforcement learning from human preferences.Advances in neural information processing systems, 30, 2017

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences.Advances in neural information processing systems, 30, 2017

    2017

  2. [2]

    Fine-tuning language models from human preferences

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019

    Pith/arXiv arXiv 1909

  3. [3]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    2022

  4. [4]

    Rlhf: A comprehensive survey for cultural, multimodal and low latency alignment methods.arXiv preprint arXiv:2511.03939, 2025

    Raghav Sharma, Manan Mehta, and Sai Tiger Raina. Rlhf: A comprehensive survey for cultural, multimodal and low latency alignment methods.arXiv preprint arXiv:2511.03939, 2025

    arXiv 2025

  5. [5]

    Ma-rlhf: Rein- forcement learning from human feedback with macro actions.arXiv preprint arXiv:2410.02743, 2024

    Yekun Chai, Haoran Sun, Huang Fang, Shuohuan Wang, Yu Sun, and Hua Wu. Ma-rlhf: Rein- forcement learning from human feedback with macro actions.arXiv preprint arXiv:2410.02743, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    Confronting reward model overoptimization with constrained RLHF

    Ted Moskovitz, Aaditya K Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca Dragan, and Stephen Marcus McAleer. Confronting reward model overoptimization with constrained RLHF. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  7. [7]

    Regularizing hidden states enables learning generalizable reward model for llms.Advances in Neural Information Processing Systems, 37:62279–62309, 2024

    Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. Regularizing hidden states enables learning generalizable reward model for llms.Advances in Neural Information Processing Systems, 37:62279–62309, 2024

    2024

  8. [8]

    Inform: Mitigating reward hacking in rlhf via information-theoretic reward modeling.Advances in Neural Information Processing Systems, 37:134387–134429, 2024

    Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao, Lefei Zhang, and Dacheng Tao. Inform: Mitigating reward hacking in rlhf via information-theoretic reward modeling.Advances in Neural Information Processing Systems, 37:134387–134429, 2024

    2024

  9. [9]

    DPO meets PPO: Reinforced token optimization for RLHF

    Han Zhong, Zikang Shan, Guhao Feng, Wei Xiong, Xinle Cheng, Li Zhao, Di He, Jiang Bian, and Liwei Wang. DPO meets PPO: Reinforced token optimization for RLHF. InForty-second International Conference on Machine Learning, 2025

    2025

  10. [10]

    Information-theoretic reward decomposition for generalizable rlhf.arXiv preprint arXiv:2504.06020, 2025

    Liyuan Mao, Haoran Xu, Amy Zhang, Weinan Zhang, and Chenjia Bai. Information-theoretic reward decomposition for generalizable rlhf.arXiv preprint arXiv:2504.06020, 2025

    arXiv 2025

  11. [11]

    Capo: Towards enhancing llm reasoning through verifiable generative credit assignment.arXiv e-prints, pages arXiv–2508, 2025

    Guofu Xie, Yunsheng Shi, Hongtao Tian, Ting Yao, and Xiao Zhang. Capo: Towards enhancing llm reasoning through verifiable generative credit assignment.arXiv e-prints, pages arXiv–2508, 2025

    2025

  12. [12]

    Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners.OpenAI blog, 1(8):9, 2019

    2019

  13. [13]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  14. [14]

    Finetuned language models are zero-shot learners.arXiv preprint arXiv:2109.01652, 2021

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners.arXiv preprint arXiv:2109.01652, 2021

    Pith/arXiv arXiv 2021

  15. [15]

    Learning to summarize with human feedback

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea V oss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008–3021, 2020

    2020

  16. [16]

    Spread preference annotation: Direct preference judgment for efficient llm alignment.arXiv preprint arXiv:2406.04412, 2024

    Dongyoung Kim, Kimin Lee, Jinwoo Shin, and Jaehyung Kim. Spread preference annotation: Direct preference judgment for efficient llm alignment.arXiv preprint arXiv:2406.04412, 2024. 10

    Pith/arXiv arXiv 2024

  17. [17]

    The alignment problem from a deep learning perspective.arXiv preprint arXiv:2209.00626, 2022

    Richard Ngo, Lawrence Chan, and Sören Mindermann. The alignment problem from a deep learning perspective.arXiv preprint arXiv:2209.00626, 2022

    Pith/arXiv arXiv 2022

  18. [18]

    Stepwise alignment for constrained language model policy optimization.Advances in Neural Information Processing Systems, 37:104471–104520, 2024

    Akifumi Wachi, Thien Tran, Rei Sato, Takumi Tanabe, and Youhei Akimoto. Stepwise alignment for constrained language model policy optimization.Advances in Neural Information Processing Systems, 37:104471–104520, 2024

    2024

  19. [19]

    Panacea: Pareto alignment via preference adaptation for llms

    Yifan Zhong, Chengdong Ma, Xiaoyuan Zhang, Ziran Yang, Haojun Chen, Qingfu Zhang, Siyuan Qi, and Yaodong Yang. Panacea: Pareto alignment via preference adaptation for llms. Advances in Neural Information Processing Systems, 37:75522–75558, 2024

    2024

  20. [20]

    Preference learning for AI alignment: a causal perspective

    Kasia Kobalczyk and Mihaela van der Schaar. Preference learning for AI alignment: a causal perspective. InForty-second International Conference on Machine Learning, 2025

    2025

  21. [21]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741, 2023

    2023

  22. [22]

    Contrastive preference learning: learning from human feedback without rl

    Joey Hejna, Rafael Rafailov, Harshit Sikchi, Chelsea Finn, Scott Niekum, W Bradley Knox, and Dorsa Sadigh. Contrastive preference learning: learning from human feedback without rl. arXiv preprint arXiv:2310.13639, 2023

    Pith/arXiv arXiv 2023

  23. [23]

    Orpo: Monolithic preference optimization without reference model.arXiv preprint arXiv:2403.07691, 2024

    Jiwoo Hong, Noah Lee, and James Thorne. Orpo: Monolithic preference optimization without reference model.arXiv preprint arXiv:2403.07691, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    Scaling laws for reward model overoptimization

    Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. InInternational Conference on Machine Learning, pages 10835–10866. PMLR, 2023

    2023

  26. [26]

    Learning to understand goal specifications by modelling reward.arXiv preprint arXiv:1806.01946, 2018

    Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Arian Hosseini, Pushmeet Kohli, and Edward Grefenstette. Learning to understand goal specifications by modelling reward.arXiv preprint arXiv:1806.01946, 2018

    Pith/arXiv arXiv 2018

  27. [27]

    Retroformer: Retrospective large language agents with policy gradient optimization.arXiv preprint arXiv:2308.02151, 2023

    Weiran Yao, Shelby Heinecke, Juan Carlos Niebles, Zhiwei Liu, Yihao Feng, Le Xue, Rithesh Murthy, Zeyuan Chen, Jianguo Zhang, Devansh Arpit, et al. Retroformer: Retrospective large language agents with policy gradient optimization.arXiv preprint arXiv:2308.02151, 2023

    Pith/arXiv arXiv 2023

  28. [28]

    Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms.arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  29. [29]

    Privately aligning language models with reinforcement learning.arXiv preprint arXiv:2310.16960, 2023

    Fan Wu, Huseyin A Inan, Arturs Backurs, Varun Chandrasekaran, Janardhan Kulkarni, and Robert Sim. Privately aligning language models with reinforcement learning.arXiv preprint arXiv:2310.16960, 2023

    Pith/arXiv arXiv 2023

  30. [30]

    Dense reward for free in reinforcement learning from human feedback

    Alex James Chan, Hao Sun, Samuel Holt, and Mihaela Van Der Schaar. Dense reward for free in reinforcement learning from human feedback. InInternational Conference on Machine Learning, pages 6136–6154. PMLR, 2024

    2024

  31. [31]

    Tlcr: Token-level continuous reward for fine-grained reinforcement learning from human feedback

    Eunseop Yoon, Hee Suk Yoon, SooHwan Eom, Gunsoo Han, Daniel Nam, Daejin Jo, Kyoung- Woon On, Mark Hasegawa-Johnson, Sungwoong Kim, and Chang Yoo. Tlcr: Token-level continuous reward for fine-grained reinforcement learning from human feedback. InFindings of the Association for Computational Linguistics: ACL 2024, pages 14969–14981, 2024

    2024

  32. [32]

    Scar: Shapley credit assignment for more efficient rlhf.arXiv preprint arXiv:2505.20417, 2025

    Meng Cao, Shuyuan Zhang, Xiao-Wen Chang, and Doina Precup. Scar: Shapley credit assignment for more efficient rlhf.arXiv preprint arXiv:2505.20417, 2025

    Pith/arXiv arXiv 2025

  33. [33]

    R3hf: Reward re- distribution for enhancing reinforcement learning from human feedback.arXiv preprint arXiv:2411.08302, 2024

    Jiahui Li, Tai-wei Chang, Fengda Zhang, Kun Kuang, and Long Chen. R3hf: Reward re- distribution for enhancing reinforcement learning from human feedback.arXiv preprint arXiv:2411.08302, 2024

    Pith/arXiv arXiv 2024

  34. [34]

    A critical look at tokenwise reward-guided text generation.arXiv preprint arXiv:2406.07780, 2024

    Ahmad Rashid, Ruotian Wu, Julia Grosse, Agustinus Kristiadi, and Pascal Poupart. A critical look at tokenwise reward-guided text generation.arXiv preprint arXiv:2406.07780, 2024. 11

    arXiv 2024

  35. [35]

    Judging llm-as-a-judgee with mt-bench and chatbot arena.Advances in neural information processing systems, 36:46595–46623, 2023

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judgee with mt-bench and chatbot arena.Advances in neural information processing systems, 36:46595–46623, 2023

    2023

  36. [36]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Pith/arXiv arXiv 2024

  37. [37]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Pith/arXiv arXiv 2023

  38. [38]

    Raft: Reward ranked finetuning for generative foundation model alignment.Transactions on Machine Learning Research, 2023

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment.Transactions on Machine Learning Research, 2023

    2023

  39. [39]

    Cambridge University Press, 1999

    Abhinay Muthoo.Bargaining theory with applications. Cambridge University Press, 1999

    1999

  40. [40]

    Multi-task learning as a bargaining game.arXiv preprint arXiv:2202.01017, 2022

    Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning as a bargaining game.arXiv preprint arXiv:2202.01017, 2022

    Pith/arXiv arXiv 2022

  41. [41]

    Fairness-aware meta-learning via nash bargaining.Advances in Neural Information Processing Systems, 37:83235–83267, 2024

    Yi Zeng, Xuelin Yang, Li Chen, Cristian Ferrer, Ming Jin, Michael Jordan, and Ruoxi Jia. Fairness-aware meta-learning via nash bargaining.Advances in Neural Information Processing Systems, 37:83235–83267, 2024

    2024

  42. [42]

    Hindsight-dice: Stable credit assignment for deep reinforcement learning.arXiv preprint arXiv:2307.11897, 2023

    Akash Velu, Skanda Vaidyanath, and Dilip Arumugam. Hindsight-dice: Stable credit assignment for deep reinforcement learning.arXiv preprint arXiv:2307.11897, 2023

    Pith/arXiv arXiv 2023

  43. [43]

    Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback.arXiv preprint arXiv:2204.05862, 2022

    Pith/arXiv arXiv 2022

  44. [44]

    Universal and transferable adversarial attacks on aligned language models.arXiv preprint arXiv:2307.15043, 2023

    Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks on aligned language models.arXiv preprint arXiv:2307.15043, 2023

    Pith/arXiv arXiv 2023

  45. [45]

    Real- toxicityprompts: Evaluating neural toxic degeneration in language models.arXiv preprint arXiv:2009.11462, 2020

    Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. Real- toxicityprompts: Evaluating neural toxic degeneration in language models.arXiv preprint arXiv:2009.11462, 2020

    Pith/arXiv arXiv 2009

  46. [46]

    smoother

    Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection.arXiv preprint arXiv:2203.09509, 2022. 12 A Preliminaries Reinforcement Learning from Human Feedback.Reinforcement learning from human feedback (RLHF) is a framework...

    Pith/arXiv arXiv 2022