Meta-Learned Reward Shaping for Reinforcement Learning from Human Feedback

Yunpeng Chu

REVIEW 4 major objections 6 minor 20 references

Meta-learned reward shaping improves every RLHF baseline it is tested on, the paper reports, lifting long-form instruction-following win rate to 90.8% while provably preserving the optimal policy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 03:08 UTC pith:EBG36OE2

load-bearing objection The idea is a useful one, but the invariance theorem is wrong for the actual algorithm, the auxiliary tasks are undefined, and the results can't be verified; this isn't ready for review. the 4 major comments →

arxiv 2607.26094 v1 pith:EBG36OE2 submitted 2026-07-28 cs.LG cs.CL

Meta-Learned Reward Shaping for Reinforcement Learning from Human Feedback

Yunpeng Chu This is my paper

classification cs.LG cs.CL

keywords reinforcement learning from human feedbackreward shapingmeta-learningpotential-based shapingpolicy invarianceLLM alignmentGRPOreward sparsity

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that a small, task-aware reward-shaping function, meta-learned once across 64 auxiliary tasks, can be added to any RLHF reward signal to improve alignment without changing the intended optimal policy. The claim is that this composite reward—base reward plus a potential-based shaping term—yields consistent gains over PPO, DPO, GRPO, and DAPO on a LLaMA-3-8B model: 90.8% length-controlled win rate on AlpacaEval 2.0, 9.14 on MT-Bench, and improvements on MATH and IFEval, alongside 41% lower training instability. A sympathetic reader would care because, if true, the result is a cheap, algorithm-agnostic plug-in for RLHF that reduces reward sparsity and overoptimization without retraining the reward model or changing the RL loop.

Core claim

The central claim is that the composite reward R_hat = R_base + alpha * Phi(x,y;phi), where Phi is meta-learned and constrained to a token-level potential-based decomposition, leaves the optimal policy unchanged (Theorem 1) while adding a task-specific learning signal. The meta-objective combines task discrimination (preferred vs. rejected response log-likelihood), entropy regularization (to prevent reward collapse), and a conservation loss that projects Phi toward the potential-based subspace. Empirically, the paper reports that the frozen, sub-1M-parameter shaping network consistently outperforms the four baselines across four benchmarks, and that the gains persist even when the base rewar

What carries the argument

The central mechanism is the potential-based decomposition of the shaping function: Phi(x,y;phi) = sum_{t=1}^{|y|} [ gamma * phi(x, y_<=t) - phi(x, y_<t) ], for a learned potential phi. Because the terms telescope over a trajectory, the total added reward depends only on initial and terminal states, so the optimal policy is unchanged. A conservation loss (Eq. 8) enforces that the learned Phi stays close to this potential-based form, and a frozen encoder extracts prompt/response embeddings from the reference model, keeping the shaping input distribution stable and bounding representation drift (Proposition 1).

Load-bearing premise

The entire result rests on the assumption that the 64 auxiliary meta-tasks are representative of the evaluation domains and disjoint from the test prompts—the paper never specifies what these tasks are or how separation was enforced, and if they overlap with AlpacaEval 2.0 or MT-Bench prompts the 90.8% win rate would reflect leakage rather than generalization.

What would settle it

Release the full list of 64 auxiliary tasks and check for overlap with the prompts in AlpacaEval 2.0, MT-Bench, MATH, and IFEval; if a significant fraction overlap, the central generalization claim fails. Alternatively, rerun the meta-learning phase with random or reversed preference labels on the same tasks and re-measure AlpacaEval 2.0: if the win rate stays near 90.8%, the gains come from the shaping architecture or training dynamics, not from task-aware meta-learning.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Any on-policy RLHF algorithm (PPO, GRPO, DAPO) can be augmented with the frozen shaping network and gain 3.9 to 6.6 points on AlpacaEval 2.0 without modifying the reward model or the algorithm.
The shaping function improves all tested benchmarks, including mathematical reasoning and verifiable instruction following, not just chatbot-style preference.
Training becomes more stable: 41% less training instability, 37% lower reward variance, and convergence to 90% of final reward within 150 steps versus 250 or more for baselines.
MeRLa composes with enhanced base rewards such as process reward models and rubric-based ensembles, adding gains even though the base reward is already richer.
Because the shaping function preserves the optimal policy, the reported gains are attributed to denser, task-aware signal rather than reward hacking; the paper reports 23% higher n-gram diversity and improved judge scores on honesty and safety.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the reported gains hold under independent reproduction with the meta-task set revealed, reward shaping could become a standard cheap add-on in RLHF pipelines, potentially reducing the need for larger reward models or more complex optimization algorithms.
A testable extension is to retrain the shaping function on random or mislabeled preference pairs: if the gains persist, the improvement may come from the architecture or training dynamics rather than task-aware meta-learning, which would change the interpretation of the results.
The frozen-encoder design suggests the shaping signal depends only on reference-model representations, so the method should transfer to larger base models with minimal retuning; verifying this would strengthen the generality claims.
The paper's observation that a small residual deviation from the potential-based form appears beneficial invites a follow-up study quantifying how much non-potential bias can be tolerated before degrading policy quality, connecting to the reward-hacking literature.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

The idea is a useful one, but the invariance theorem is wrong for the actual algorithm, the auxiliary tasks are undefined, and the results can't be verified; this isn't ready for review.

read the letter

First thing you should know: the paper's core sell is a lightweight shaping network that can be bolted onto any RLHF pipeline, and that's a legitimate motivation. But the guarantee that it preserves the optimal policy does not hold for the way the algorithm is implemented, and the experiments are too under-specified to trust yet.

What's genuinely new: MeRLa combines a small MLP shaping function on frozen embeddings, trained across auxiliary tasks with a conservation loss toward potential-based structure, and then adds it as a fixed bonus to the reward. That specific spot doesn't appear in the cited literature, and the compatibility results (improvements over PPO/GRPO/DAPO) are the right experiments to do if the method were sound.

Where it falls apart: Theorem 1 is the load-bearing claim. The shaping in Eq. (4) is a sum of per-token potential differences, but Algorithm 1 adds this as a single terminal reward after the whole response is generated. The telescoping from Ng et al. applies to per-transition shaping, not to a terminal reward; the residual term αγφ(final) depends on the entire response and generally changes the optimal policy. So the invariance guarantee doesn't apply, and the proof sketch's claim that the term 'does not depend on the policy' is wrong. That's not a minor gap. Second, the auxiliary task distribution T is the heart of the method, and we are never told what the 64 tasks are, how prompts or responses are sampled, or what separation exists from AlpacaEval/MT-Bench/MATH/IFEval. Without that, the strong numbers are more likely leakage than generalization. Third, the method is called meta-learning, but Phase I is multi-task training with a summed loss; there's no inner loop. Fourth, Proposition 2 is an inequality without proof, and the empirical 'validation' measures exactly the L_con that appears in the bound, which is circular. Fifth, there's no code or data, and the numbers themselves are suspicious: 90.8% LC AlpacaEval and 9.14 MT-Bench for an 8B model are unusually high and the protocol isn't detailed enough to rule out judge bias.

What deserves credit: the design choice of a frozen encoder is sensible, and the discussion of representation drift and entropy misalignment shows the authors anticipated some of the right concerns. The ablation structure is fine.

Who this is for: someone working on reward shaping for LLMs might want to read the method section and ablation list as a starting point, but not the theory.

My recommendation: do not send to peer review in this state. The central theoretical claim is incompatible with the algorithm, the task distribution is undisclosed, and there are no public artifacts. The authors need to fix the theory or change the algorithm, define the task split, and release code and data. Then it could be worth revisiting.

Referee Report

4 major / 6 minor

Summary. The paper proposes MeRLa, a two-phase method that meta-learns a low-parameter, frozen reward-shaping function Φ(x,y;φ) on 64 auxiliary tasks, then adds αΦ to a base RLHF reward during on-policy RLHF training. The central claims are: (i) the composite reward preserves the optimal policy because Φ is constrained to a potential-based form; (ii) the meta-objective gives formal control of representation drift and entropy-induced incentive misalignment; and (iii) on LLaMA-3-8B, MeRLa improves over PPO, DPO, GRPO, and DAPO across AlpacaEval 2.0, MT-Bench, MATH, and IFEval, with 41% less training instability. The theoretical guarantees and the large reported gains are the load-bearing elements of the paper.

Significance. If the theoretical guarantee and empirical numbers were both correct, MeRLa would be a practically useful plug-in: a <1M-parameter frozen shaping network, compatible with common RLHF backbones, that improves reward density and stability without requiring additional human labels. The paper also deserves credit for attempting to address representation drift, entropy-induced misalignment, and compatibility with process- and rubric-based rewards. However, the central policy-invariance theorem is not valid as stated for the implemented algorithm, the auxiliary-task distribution is unspecified, and no code or data are provided. The reported gains are therefore suggestive but not yet substantiated; the contribution, as presented, cannot be relied upon.

major comments (4)

[§5.1, Algorithm 1, Eq. (9)] Theorem 1 does not apply to the deployed algorithm. In Algorithm 1, lines 13–16, the shaping term is computed offline as a scalar Φ(x,y;φ*) and added to the terminal reward in Eq. (9); it is not delivered as a per-token transition reward. Potential-based invariance (Ng et al., 1999) requires the shaping term to be added to each transition reward in the MDP. Moreover, the algebraic claim in the proof sketch is wrong for γ≠1: the finite sum in Eq. (4) telescopes to γφ(s_T) − φ(s_0) + (γ−1)Σ_{t=1}^{T−1} φ(s_t), which depends on intermediate states, not only endpoints. For γ=1, the shaped reward becomes R_base + α(φ(y_T) − φ(x)); since φ(y_T) depends on the generated response, this generally changes the optimal policy under the response-level objective in Eq. (9). Thus the invariance guarantee that motivates the method is unsupported.
[§4.2, Algorithm 1 (Phase I)] The auxiliary task distribution T is never specified. The paper states M=64 meta-tasks but does not define the tasks, the prompt/response sampling procedure, or any separation between the meta-training distribution and the evaluation benchmarks (AlpacaEval 2.0, MT-Bench, MATH, IFEval). If the auxiliary tasks overlap with the test sets or with the preference pool used to construct the benchmarks, the reported 90.8% AlpacaEval LC win rate could reflect leakage rather than generalization. No code, data, or checkpoint is provided, so this central empirical claim cannot be independently verified. This is a reproducibility and leakage-risk issue, not merely a presentation matter.
[§5.3, Eq. (13)] Proposition 2 is circular as stated. Eq. (13) asserts Δ_incentive ≤ α·(L_con + ε_pb), but Δ_incentive is never defined independently of the conservation loss; ε_pb is also not formally defined (Φ* is introduced without explanation, and differs from Φ_pb in Eq. (8)). Without an independent definition of policy-ranking change and a theorem relating it to L_con, Eq. (13) is a restatement of a definition rather than a bound. The claimed validation in §7.2 — that L_con=0.0034 yields a maximum policy-ranking change of α·ε_max=0.001 — is asserted without derivation. The paper's claim to 'formally address' entropy-induced incentive misalignment is therefore not established.
[§4.2, Eq. (8); §6.5, Eq. (14)] Two additional load-bearing gaps. First, the conservation loss L_con in Eq. (8) requires the projection Φ_pb of Φ onto the potential-based space, but the projection space, norm, and computational procedure are not specified; as written, the loss is not implementable. Second, Eq. (14) claims that MeRLa 'cannot degrade performance in expectation' because Theorem 1 preserves the optimal policy. This is a non sequitur: policy invariance of an expected-reward objective does not imply that a finite-sample RL algorithm with KL penalty and nonzero conservation residual attains no-worse performance. The complementarity argument in §6.5 relies on this unjustified claim.

minor comments (6)

[§5.2] Typo: 'represenattions' should be 'representations'.
[§4.3] The sentence 'GRPO due to its computational efficiency (the critic model required)' appears to be missing 'no': it should read 'no critic model required'.
[Table 1] The table footnote mentions '† denotes our reimplementation,' but no entries in the table carry a dagger. Either add the marks or remove the footnote.
[Figure 3(b)] The caption says 'the task number being the most critical,' but the text reports that removing L_task causes a ~3.7% drop while removing the task sampler causes a ~5.5% drop. Please clarify which component is 'most critical.'
[Notation] The notation for residual conservation error is inconsistent: Eq. (8) uses a squared error, §5.1 uses ϵ=∥Φ−Φ_pb∥, and §5.3 uses ε_pb=∥Φ−Φ*∥. Unify these definitions.
[Limitations] The limitations section appropriately acknowledges the single-base-model evaluation and the residual-bias caveat, but these caveats are in tension with the abstract's unqualified 'consistent improvements' claim and with Eq. (14)'s non-degradation assertion.

Circularity Check

1 steps flagged

Central benchmark claims are externally grounded; the §7.2 'validation' of Proposition 2 is circular by construction.

specific steps

fitted input called prediction [§7.2 (Training Dynamics and Stability — Conservation Loss and Residual Bias), validating Proposition 2 (§5.3)]
"At the end of meta-learning, L_con = 0.0034, indicating that Φ closely approximates the potential-based form. The resulting residual base ϵ_con = 0.003 translates to a maximum policy ranking change of α·ϵ_max = 0.001, which is negligible. This empirically validates the incentive alignment bound (Proposition 2)"

Proposition 2 states Δ_incentive ≤ α·(L_con(ϕ)+ϵ_pb), so the bound's right-hand side is defined directly in terms of the conservation loss L_con. The §7.2 'validation' measures L_con after meta-learning and substitutes that same quantity back into the inequality, then declares the bound confirmed. No independent measurement of Δ_incentive—the actual maximum change in policy ranking—is reported. The 'prediction' is therefore satisfied by construction and is not an empirical test of the bound.

full rationale

The main empirical claim is not circular: Table 1 compares MeRLa against fixed, external benchmarks (AlpacaEval 2.0, MT-Bench, MATH, IFEval) and standard baselines (PPO, DPO, GRPO, DAPO) after a frozen shaping function is deployed, rather than fitting the benchmarks themselves. The meta-learning objective (Eq. 5) is optimized on auxiliary tasks and the resulting Φ is frozen before RLHF, so the benchmark numbers are not generated by plugging benchmark outcomes back into the training loss. There is also no load-bearing self-citation chain: the references cited for the invariance and stability arguments (Ng et al. 1999; Kumar et al. 2022; Pashakhanloo and Koulakov 2023) are external, not prior work by the same author. The one clear circular step is in §7.2, where Proposition 2's bound is 'empirically validated' by measuring L_con and substituting it into the very inequality that defines the claimed bound; this is a tautology rather than an independent check. Separately, the paper's own Limitations section admits that the policy-invariance guarantee 'holds exactly only when the conservation loss is zero,' and the proof sketch of Theorem 1 applies a telescoping-potential argument to a deployment that adds the shaping term at the response level (Eq. 9) rather than as per-token rewards; that is a correctness or applicability concern, not an identity-by-construction, so it is not counted as a circular step here. Overall, the central benchmark contribution has independent external content, but one supporting analytical 'validation' reduces by construction, yielding a score of 4.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central claim rests on hand-set constants (α, λ1, λ2, M) and on several unproven transfer steps: the Ng theorem is applied to a KL-regularized episodic objective; the residual of the soft conservation constraint is assumed negligible; the incentive-alignment bound is asserted; and the 'meta-task' distribution is assumed valid and disjoint from evaluation. The theoretical sections therefore do not add independent support beyond what is already standard in the literature.

free parameters (5)

shaping strength α = 0.3
Hand-set strength of the shaping term in Eq. (2); no sensitivity sweep reported.
entropy weight λ1 = 0.05
Hand-set weight of entropy regularization in Eq. (5).
conservation weight λ2 = 0.1
Hand-set weight of the potential-based conservation loss in Eq. (5).
number of meta-tasks M = 64
Ablation shows performance plateaus at M=64, so it is selected to maximize reported results.
discount factor γ in Eq. (4) = not reported
Appears in the potential-based decomposition; the proof requires γ=1 for clean telescoping to endpoints, while Theorem 1 claims all γ∈(0,1).

axioms (5)

standard math Ng et al. (1999) potential-based shaping preserves the optimal policy.
Invoked in §4.1 and Theorem 1; the classical theorem is correct in its original MDP setting.
domain assumption The KL-regularized RLHF objective (Eq. 9) inherits policy invariance from the classical unregularized MDP.
Theorem 1 transfers the Ng result to the βKL-regularized episodic LLM setting without proof; entropy-regularized objectives require separate treatment.
domain assumption The auxiliary task distribution T provides a valid, deployment-relevant training signal disjoint from the evaluation benchmarks.
T is never concretely defined; if it overlaps with AlpacaEval/MT-Bench/MATH/IFEval, the empirical claims are not a test of generalization.
ad hoc to paper The residual non-potential bias ε=||Φ−Φ_pb|| is small enough that the policy change is negligible in practice.
The paper states the invariance guarantee holds only at zero conservation loss, then asserts the residual is 'beneficial in practice' without analysis (§5.1, §7.2).
ad hoc to paper Proposition 2's inequality Δ_incentive ≤ α·(L_con + ε_pb) is a valid bound.
No derivation is given; the bound is asserted in §5.3 and then used to conclude entropy regularization is safe.

pith-pipeline@v1.3.0-alltime-deepseek · 10078 in / 16313 out tokens · 162649 ms · 2026-08-01T03:08:03.279262+00:00 · methodology

0 comments

read the original abstract

Reinforcement Learning from Human Feedback (RLHF) is the standard approach for aligning large language models with human preferences, but its quality is limited by static, task-agnostic reward models. This mismatch leads to sparse learning signals and suboptimal alignment. We introduce MeRLa (Meta-Learned Reward Shaping), a principled framework that meta-learns a task-aware shaping function $\Phi(x,y;\phi)$ across auxiliary tasks before RLHF training. The learned shaping produces a composite reward that preserves policy optimality while providing task-specific learning signals. Our meta-objective combines task discrimination, entropy regularization, and potential-based conservation for stable convergence. We provide theoretical guarantees for policy invariance, analyze representation drift sensitivity, and formally address incentive misalignment from entropy maximization. Experiments on LLaMA-3-8B across four benchmarks show consistent improvements over PPO, DPO, GRPO, and DAPO, achieving a 90.8% length-controlled win rate on AlpacaEval 2.0 and a score of 9.14 on MT-Bench, with 41% less training instability. MeRLa retains its benefits when combined with process-based and rubric-based enhanced rewards.

Figures

Figures reproduced from arXiv: 2607.26094 by Yunpeng Chu.

**Figure 3.** Figure 3: (a) Scaling behavior: MeRLa consistently outper [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

**Figure 2.** Figure 2: (a) Bar chart comparing methods across four bench [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 4.** Figure 4: (a) Distribution of reward values for the base re [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗

**Figure 5.** Figure 5: (a) Average reward during RLHF training. MeRLa [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

20 extracted references · 20 linked inside Pith

[1]

arXiv:2402.14740

Ahmadian,A.;Cremer,C.;Gallé,M.;etal.2024.BacktoBa- sics: Revisiting REINFORCE Style Optimization for Learn- ing from Human Feedback in LLMs. arXiv:2402.14740. Bai,Y.;Jones,A.;Ndousse,K.;etal.2022. TrainingaHelp- ful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv:2204.05862. Casper, S.; Davies, X.; Shi, C.; et al

Pith/arXiv arXiv 2024
[3]

arXiv:2310.01377

UltraFeed- back: Boosting Language Models with Scaled AI Feedback. arXiv:2310.01377. Dubois, Y.; Galambosi, B.; Liang, P.; et al

Pith/arXiv arXiv
[6]

arXiv:2403.13787

Reward- Bench: Evaluating reward models for Language Modeling. arXiv:2403.13787. Lee, H.; Phatale, S.; Mansoor, H.; et al

Pith/arXiv arXiv
[7]

RLHF: Scaling Reinforcement Learning from Human Feed- back with AI Feedback

RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feed- back with AI Feedback. arXiv:2309.00267. Lightman, H.; Kosaraju, V.; Burda, Y.; et al

Pith/arXiv arXiv
[8]

arXiv:2305.20050

Let’s verify step by step. arXiv:2305.20050. Ma, H.; Luo, Z.; Vo, T. V.; et al

Pith/arXiv arXiv
[9]

arXiv:2408.03029

Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learning. arXiv:2408.03029. Meng, Y.; Xia, M.; and Chen, D

Pith/arXiv arXiv
[10]

arXiv:2405.14734

SimPO: Sim- ple Preference Optimization with a Reference-free Reward. arXiv:2405.14734. Min,S.;Lewis,M.;Zettlemoyer,L.;andHajishirzi,H.2022. MetaICL: Learning to Learn in Context. InNorth American Chapter of the Association for Computational Linguistics (NAACL), 2791–2809. Ng,A.Y.;Harada,D.;andRussell,S.1999.PolicyInvariance under Reward Transformations: ...

Pith/arXiv arXiv 2022
[11]

InInter- national Conference on Machine Learning (ICML)

WARM: On the benefits of weight averaged reward models. InInter- national Conference on Machine Learning (ICML). Schulman,J.;Wolski,F.;Dhariwal,P.;etal.2017. Proximal Policy Optimization Algorithms. arXiv:1707.06347. Shao, Z.; Wang, P.; Zhu, Q.; et al

Pith/arXiv arXiv 2017
[12]

arXiv:2402.03300

DeepSeekMath: PushingtheLimitsofMathematicalReasoninginOpenLan- guage Models. arXiv:2402.03300. Uesato, J.; Kushman, N.; Kumar, R.; et al

Pith/arXiv arXiv
[13]

arXiv:2211.14275

Solv- ing Math Word Problems with Process- and outcome-based feedback. arXiv:2211.14275. Wang, B.; Zheng, R.; Chen, L.; et al

Pith/arXiv arXiv
[14]

arXiv:2401.06080

Secrets of RLHF in Large Language Models part II: Reward Model- ing. arXiv:2401.06080. Wang, T.; Yu, P.; Tan, X. E.; et al

Pith/arXiv arXiv
[15]

arXiv:2308.04592

Shepherd: A Critic for Language Model Generation. arXiv:2308.04592. Xie, T.; Zhao, S.; Wu, C. H.; et al

Pith/arXiv arXiv
[16]

arXiv:2309.11489

Text2Reward: Reward Shaping with Language Models for Reinforcement Learning. arXiv:2309.11489. Yu, Q.; Zhang, Z.; Zhu, R.; et al

Pith/arXiv arXiv
[18]

arXiv:2401.10020

Self-Rewarding Language Models. arXiv:2401.10020. Zheng, L.; Chiang, W.-L.; Sheng, Y.; et al

Pith/arXiv arXiv
[36]

arXiv:2311.07911

Zhou,J.;Lu,T.;Mishra,S.;etal.2023.Instruction-Following Evaluation for Large Language Models. arXiv:2311.07911. Zou, H.; Ren, T.; Yan, D.; et al

Pith/arXiv arXiv 2023
[2019]

arXiv:1901.09330

Reward Shaping via Meta-Learning. arXiv:1901.09330

Pith/arXiv arXiv 1901
[2022]

arXiv:2106.09685

LoRA: Low-rank Adaptation of Large Language Models. arXiv:2106.09685. Kumar, A.; Raghunathan, A.; Jones, R.; Ma, T.; and Liang, P

Pith/arXiv arXiv
[2023]

arXiv:2307.15217

Open Prob- lemsandFundamentalLimitationsofReinforcementLearn- ing from Human Feedback. arXiv:2307.15217. Cui, G.; Yuan, L.; Ding, N.; et al

Pith/arXiv arXiv
[2024]

arXiv:2404.04475

Length- Controlled AlpacaEval:ASimple Wayto DebiasAutomatic Evaluators. arXiv:2404.04475. Gao,L.;Schulman,J.;andHilton,J.2022. ScalingLawsfor Reward Model Overoptimization. arXiv:2210.10760. Grattafiori,A.;Dubey,A.;Jauhri,A.;etal.2024. TheLlama 3 Herd of Models. arXiv:2407.21783. Hendrycks,D.;Burns,C.;Kadavath,S.;etal.2021. Measur- ingMathematicalProblemS...

Pith/arXiv arXiv 2022
[2025]

arXiv:2503.14476

DAPO: An Open-SourceLLMReinforcementLearningSystematScale. arXiv:2503.14476. Yuan, W.; Pang, R. Y.; Cho, K.; et al

Pith/arXiv arXiv

[1] [1]

arXiv:2402.14740

Ahmadian,A.;Cremer,C.;Gallé,M.;etal.2024.BacktoBa- sics: Revisiting REINFORCE Style Optimization for Learn- ing from Human Feedback in LLMs. arXiv:2402.14740. Bai,Y.;Jones,A.;Ndousse,K.;etal.2022. TrainingaHelp- ful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv:2204.05862. Casper, S.; Davies, X.; Shi, C.; et al

Pith/arXiv arXiv 2024

[2] [3]

arXiv:2310.01377

UltraFeed- back: Boosting Language Models with Scaled AI Feedback. arXiv:2310.01377. Dubois, Y.; Galambosi, B.; Liang, P.; et al

Pith/arXiv arXiv

[3] [6]

arXiv:2403.13787

Reward- Bench: Evaluating reward models for Language Modeling. arXiv:2403.13787. Lee, H.; Phatale, S.; Mansoor, H.; et al

Pith/arXiv arXiv

[4] [7]

RLHF: Scaling Reinforcement Learning from Human Feed- back with AI Feedback

RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feed- back with AI Feedback. arXiv:2309.00267. Lightman, H.; Kosaraju, V.; Burda, Y.; et al

Pith/arXiv arXiv

[5] [8]

arXiv:2305.20050

Let’s verify step by step. arXiv:2305.20050. Ma, H.; Luo, Z.; Vo, T. V.; et al

Pith/arXiv arXiv

[6] [9]

arXiv:2408.03029

Highly Efficient Self-Adaptive Reward Shaping for Reinforcement Learning. arXiv:2408.03029. Meng, Y.; Xia, M.; and Chen, D

Pith/arXiv arXiv

[7] [10]

arXiv:2405.14734

SimPO: Sim- ple Preference Optimization with a Reference-free Reward. arXiv:2405.14734. Min,S.;Lewis,M.;Zettlemoyer,L.;andHajishirzi,H.2022. MetaICL: Learning to Learn in Context. InNorth American Chapter of the Association for Computational Linguistics (NAACL), 2791–2809. Ng,A.Y.;Harada,D.;andRussell,S.1999.PolicyInvariance under Reward Transformations: ...

Pith/arXiv arXiv 2022

[8] [11]

InInter- national Conference on Machine Learning (ICML)

WARM: On the benefits of weight averaged reward models. InInter- national Conference on Machine Learning (ICML). Schulman,J.;Wolski,F.;Dhariwal,P.;etal.2017. Proximal Policy Optimization Algorithms. arXiv:1707.06347. Shao, Z.; Wang, P.; Zhu, Q.; et al

Pith/arXiv arXiv 2017

[9] [12]

arXiv:2402.03300

DeepSeekMath: PushingtheLimitsofMathematicalReasoninginOpenLan- guage Models. arXiv:2402.03300. Uesato, J.; Kushman, N.; Kumar, R.; et al

Pith/arXiv arXiv

[10] [13]

arXiv:2211.14275

Solv- ing Math Word Problems with Process- and outcome-based feedback. arXiv:2211.14275. Wang, B.; Zheng, R.; Chen, L.; et al

Pith/arXiv arXiv

[11] [14]

arXiv:2401.06080

Secrets of RLHF in Large Language Models part II: Reward Model- ing. arXiv:2401.06080. Wang, T.; Yu, P.; Tan, X. E.; et al

Pith/arXiv arXiv

[12] [15]

arXiv:2308.04592

Shepherd: A Critic for Language Model Generation. arXiv:2308.04592. Xie, T.; Zhao, S.; Wu, C. H.; et al

Pith/arXiv arXiv

[13] [16]

arXiv:2309.11489

Text2Reward: Reward Shaping with Language Models for Reinforcement Learning. arXiv:2309.11489. Yu, Q.; Zhang, Z.; Zhu, R.; et al

Pith/arXiv arXiv

[14] [18]

arXiv:2401.10020

Self-Rewarding Language Models. arXiv:2401.10020. Zheng, L.; Chiang, W.-L.; Sheng, Y.; et al

Pith/arXiv arXiv

[15] [36]

arXiv:2311.07911

Zhou,J.;Lu,T.;Mishra,S.;etal.2023.Instruction-Following Evaluation for Large Language Models. arXiv:2311.07911. Zou, H.; Ren, T.; Yan, D.; et al

Pith/arXiv arXiv 2023

[16] [2019]

arXiv:1901.09330

Reward Shaping via Meta-Learning. arXiv:1901.09330

Pith/arXiv arXiv 1901

[17] [2022]

arXiv:2106.09685

LoRA: Low-rank Adaptation of Large Language Models. arXiv:2106.09685. Kumar, A.; Raghunathan, A.; Jones, R.; Ma, T.; and Liang, P

Pith/arXiv arXiv

[18] [2023]

arXiv:2307.15217

Open Prob- lemsandFundamentalLimitationsofReinforcementLearn- ing from Human Feedback. arXiv:2307.15217. Cui, G.; Yuan, L.; Ding, N.; et al

Pith/arXiv arXiv

[19] [2024]

arXiv:2404.04475

Length- Controlled AlpacaEval:ASimple Wayto DebiasAutomatic Evaluators. arXiv:2404.04475. Gao,L.;Schulman,J.;andHilton,J.2022. ScalingLawsfor Reward Model Overoptimization. arXiv:2210.10760. Grattafiori,A.;Dubey,A.;Jauhri,A.;etal.2024. TheLlama 3 Herd of Models. arXiv:2407.21783. Hendrycks,D.;Burns,C.;Kadavath,S.;etal.2021. Measur- ingMathematicalProblemS...

Pith/arXiv arXiv 2022

[20] [2025]

arXiv:2503.14476

DAPO: An Open-SourceLLMReinforcementLearningSystematScale. arXiv:2503.14476. Yuan, W.; Pang, R. Y.; Cho, K.; et al

Pith/arXiv arXiv