Pith. sign in

REVIEW 3 major objections 5 minor 54 references

Direct Advantage Regression: Aligning LLMs with Online AI Reward

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Direct Advantage Regression replaces online RLHF with one weighted supervised learning objective.

desk verdict Useful reward-vs-preference comparison and a clean derivation, but the harmlessness result appears to train toward harmfulness because the reward prompt is never inverted. read the letter →

arxiv 2504.14177 v1 pith:TKJLJY72 submitted 2025-04-19 cs.AI cs.CLcs.HC

Li He , He Zhao , Stephen Wan , Dadong Wang , Lina Yao , Tongliang Liu This is my paper · ORCID
classification cs.AIcs.CLcs.HC
keywords directadvantageregressiononlineAIfeedbackrewardweightedsupervisedfine-tuningdualKLregularizationalignmentfrompreferencesRLHFLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Direct Advantage Regression (DAR) is an alignment algorithm that replaces the two-stage RLHF pipeline—reward collection followed by policy optimization with a policy-gradient method—with a single weighted supervised fine-tuning step. Each on-policy response is assigned a weight equal to the product of an advantage-based reward term and a policy-ratio discount that jointly enforces two KL constraints, one against a fixed reference policy and one against the current sampling policy. The paper's central empirical claim is that this online, RL-free update outperforms both online direct-preference methods (DPO, IPO, SLiC with AI preference) and online RL methods (PPO, RLOO with AI reward), reaching 98.27 percent reference win rate on TL;DR, 92.67 percent on Helpfulness, and 85.84 percent on Harmlessness as judged by GPT-4-Turbo, while keeping response lengths bounded. A supporting claim is that scalar AI reward is higher-fidelity supervision than binary AI preference, showing consistently higher human-AI agreement across several annotator models. If correct, fine-grained AI supervision can replace both human labels and RL machinery in online alignment.

What carries the argument

The engine is the DAR regression weight in Equation (7): each on-policy response is up-weighted by the advantage exponential $\exp(A/(\alpha+\beta))$ and discounted by the ratio $(\pi_{\mathrm{ref}}(y|x)/\pi_t(y|x))^{\alpha/(\alpha+\beta)}$, and the product becomes the weight for ordinary next-token maximum-likelihood training on $\log\pi_\theta(y|x)$. The two KL coefficients $\alpha$ and $\beta$ play distinct roles: their sum $\alpha+\beta$ scales the temperature of the advantage weight, while the ratio $\alpha/(\alpha+\beta)$ sets how strongly the update stays near the fixed reference policy rather than the moving sampling policy. This construction converts the dual-constrained reward objective into a supervised loss, avoiding value networks and policy-gradient estimators.

What would settle it

Take a DAR-tuned model and the same 1,000-prompt test subsets, but have human raters judge the model's outputs against the reference instead of GPT-4-Turbo; a large drop from the reported 98.27/92.67/85.84 percent win rates would show that the AI-reward supervision, not human preference, drove the gains.

Watch

Extended reading notes

Core claim

The central claim is that LLM alignment can be solved as iterative weighted supervised regression on replies sampled from the current policy, with weights encoding both the advantage of a reply and a dual KL regularization. For the objective $\max_\pi \mathbb{E}_{x,y\sim\pi}[A(x,y)] - \alpha D_{\mathrm{KL}}(\pi\|\pi_{\mathrm{ref}}) - \beta D_{\mathrm{KL}}(\pi\|\pi_t)$, the paper proves (Theorem 4.1) that the optimum is $\pi^*(y|x) \propto \pi_{\mathrm{ref}}(y|x)^{\alpha/(\alpha+\beta)} \pi_t(y|x)^{\beta/(\alpha+\beta)} \exp(A(x,y)/(\alpha+\beta))$, and projecting onto a parametric policy by KL minimization yields the DAR update $\pi_{t+1} = \arg\max_\theta \mathbb{E}_{(x,y)\sim D_{\pi_t}}[(\pi_{\mathrm{ref}}(y|x)/\pi_t(y|x))^{\alpha/(\alpha+\beta)} \exp(A(x,y)/(\alpha+\beta)) \log\pi_\theta(y|x)]$. With Monte-Carlo advantage estimation, advantage normalization, and weight clipping, the paper reports that this update attains the highest reference win rate among all compared methods on TL;DR, Helpfulness, and Harmlessness, and avoids the length inflation seen in the SFT and online-preference baselines.

Load-bearing premise

The load-bearing premise is that the scalar score an LLM annotator gives a single answer is a faithful, bias-free proxy for human preference; if that score is skewed by length, style, position, or safety wording, DAR will simply amplify the skew.

Editorial extensions

If this is right

  • AI reward is a more informative supervision signal than AI preference: across Qwen2, Llama-3, Mistral, Gemma-2 and GPT-4 annotators, reward-based labels agree with human judgments more often than preference-based labels on all three datasets.
  • An RL-free learner can match or beat online RLHF: DAR's weighted SFT reaches higher GPT-4-Turbo reference win rates than both online PPO and RLOO while using the same scalar reward.
  • Dual KL regularization checks reward over-optimization: DAR keeps response lengths near reference levels, unlike SFT+best-of-n on TL;DR and online DAP methods on Helpfulness.
  • The same update works with a human-trained reward model: on the HelpSteer2 setting DAR obtains the best MT-Bench score among the trained methods compared.
  • Methods learning from AI reward, DAR included, need 3-5 times fewer online annotations than preference-based online DAP methods to reach their plateau.
  • If the dual-KL weighting is what stabilizes DAR, the same trick can be grafted onto offline preference losses such as DPO or IPO, giving them a similar monotone-improvement guarantee without changing their data requirements.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because DAR's weights are built from the scalar $r(x,y)$ directly, the same objective could be applied to any continuously scored feedback—graded rubrics, safety scores, or human Likert ratings—not just LLM-generated rewards; testing this would show whether the dual-KL weighting, rather than the AI annotator, is what drives DAR's gains.
  • The paper's Table 1 agreement figures (60–80 percent) imply that DAR's measured win rates may overstate progress toward human preference; a natural next experiment is the same three test sets judged by human raters rather than GPT-4-Turbo.
  • The dual-KL trick could be retrofitted to offline preference algorithms: applying the $\pi_{\mathrm{ref}}/\pi_t$ ratio weight inside an offline DPO or IPO loss would give such methods a monotone-improvement guarantee analogous to DAR's.
  • The paper's bias analysis for pairwise preference (second-position bias) suggests a similar audit is needed for the single-response reward prompts; checking whether reward scores drift with response length or formatting would expose whether DAR amplifies annotator artifacts.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Direct Advantage Regression (DAR), an online alignment algorithm for LLMs that uses scalar AI rewards to reweight a supervised fine-tuning objective. The update rule in Eq. (7) is derived from a dual-KL-constrained advantage maximization objective, with a closed-form optimal policy obtained via Gibbs' inequality, and is implemented as weighted SFT with Monte-Carlo baseline estimation, advantage normalization, and weight clipping. Experiments compare AI reward labels against AI preference labels for agreement with human preferences, and report reference win rates on TL;DR, Helpfulness, and Harmlessness, plus MT-Bench scores after fine-tuning on HelpSteer2 with a pretrained reward model. The paper claims that DAR outperforms online DPO/IPO/SLiC with AI preference and online PPO/RLOO with AI reward while avoiding reward over-optimization.

Significance. If the empirical claims hold, DAR is a simple, RL-free alternative to PPO/RLOO with a clean theoretical derivation: the proof of Theorem 4.1 in Appendix B is correct, and Eq. (7) genuinely follows from the stated objective rather than being fitted post hoc. The head-to-head comparison of AI reward versus AI preference as supervision signals (Table 1) is a useful contribution. However, the reported Harmlessness result appears to rely on an inverted reward direction, the main win-rate comparison conflates algorithm and supervision type, and the PPO baseline is admitted to be suboptimal. These issues place a substantial burden on the experimental claims, although the central algorithm idea remains defensible for the other two tasks.

major comments (3)
  1. [Algorithm 1, Table 8, Table 11, Section 6.2] The harmlessness reward prompt in Table 8 instructs the annotator to return a SCORE from 1 to 10 where 10 means 'extremely harmful,' while Algorithm 1 and Eq. (7) maximize the advantage computed directly from that score with no sign inversion. The evaluation prompt in Table 11 also instructs GPT-4-Turbo to select the 'most harmful' response. Under these printed definitions, the reported 85.84% Harmlessness win rate in Table 2 measures harmfulness, not harmlessness, and the training signal would actively maximize harmful behavior, contradicting the stated alignment goal. The high human-AI agreement for Harmlessness in Table 1 is likewise inconsistent with this orientation, since the human-preferred response in the Harmlessness dataset is the less harmful one. Please state explicitly whether a sign inversion (such as using 10 - SCORE or negating the advantage) was applied in the implementation; if not, the Harmlessness experiments need to be redone with the correct reward direction.
  2. [Section 6.2, Table 2] The main win-rate comparison between DAR with AI reward and DPO/IPO/SLiC with AI preference changes both the algorithm and the supervision type simultaneously, so the conclusion that 'AI reward is a better form of AI supervision' is not actually supported by the win-rate results. Table 1 does isolate the supervision signal through human-AI agreement, but the headline performance comparison in Table 2 cannot separate algorithmic advantage from supervision advantage. Please add an ablation that keeps either the algorithm or the supervision signal fixed, or rephrase the claims to describe the joint combination rather than attributing the gain to AI reward alone.
  3. [Appendix D.3.2, Table 2, Table 3] The PPO baseline is described as 'suboptimal' by the authors and is not run for the HelpSteer2 experiment in Table 3. The claim that DAR outperforms online RLHF therefore rests on a PPO implementation that the authors themselves judge to be inadequate, including the absence of a separate value model. Please either tune PPO more carefully (for example, with a separate value model and moving-average reference) or restrict the claim to comparisons against RLOO and SFT+Best-of-N.
minor comments (5)
  1. [Section 1] There is a typo in 'proper on-policy learnring approach'; it should be 'learning'.
  2. [Section 4.3] 'This approach further simplies the computational complexity' should be 'simplifies'.
  3. [Appendix E.2] The claim that the positional-bias difference is 'statistically significant' is not accompanied by a test statistic or p-value; please provide the test used.
  4. [Table 1] Human-AI agreement numbers are reported without confidence intervals or significance tests; given the small subset size (1k) and the relatively small gaps between AI reward and AI preference, some uncertainty quantification would strengthen the head-to-head claim.
  5. [Appendix D.4] The evaluation uses a 1,000-sample random subset for win rates, but no standard errors adjusted for clustering by prompt are reported; the provided standard deviations over seeds are helpful but do not capture within-prompt correlation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: DAR's update rule is derived from its own dual-KL objective via Gibbs' inequality, not fitted or self-cited.

full rationale

The theoretical chain is self-contained. Equation (5) states the dual-KL objective; Theorem 4.1 is proved in Appendix B by algebraic expansion (Eq. 10) and Gibbs' inequality (Eq. 11); Eq. (7) is then obtained from Eq. (6) by substituting the closed-form pi* and importance-weighting from pi* to the sampling policy pi_t. No parameter is fitted to the benchmark it predicts, and no load-bearing claim is justified by a self-citation: the AWR/EM lineage (Peng et al., 2019; Peters & Schaal, 2007) is prior external work cited as background, not an unverified uniqueness theorem. The score is therefore 0; the reader's 2.0 concern is empirical confounding, not derivation circularity. Two non-circularity issues are nevertheless flagged. First, Sections 6.4 and 6.5 ablation win rates are judged by Qwen2-72B-Instruct, the same model that supplies the online AI reward, so those ablation curves are confounded; the headline Table 2 win rates use GPT-4-Turbo, so the central comparisons are not internally forced. Second, the harmlessness reward prompt (Table 8) defines SCORE with '1 being completely harmless and 10 being extremely harmful RESPONSE', while Algorithm 1 maximizes the reward and Eq. (7) upweights high-A responses; the Table 11 evaluation likewise asks the judge to 'choose the most harmful response'. Under the printed definitions, DAR would optimize harmfulness on its third headline task, a sign-consistency/correctness problem rather than a circularity problem.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

DAR introduces no new entities. Its free parameters are the two KL coefficients (expressed as total regularization and alpha ratio), the weight clip threshold, and the sampling size, all chosen by win-rate ablations on 1k test sets. The axioms are standard mathematics plus two domain assumptions: the AWR off-policy approximation, and the validity of the AI reward as a human-preference proxy. The latter is the weakest link because the paper's evaluation is also AI-judged.

free parameters (4)
  • total KL regularization (alpha + beta) = 0.05
    Tuned via win-rate ablations on a 1k test set (Figure 4a); it sets the temperature of the advantage weight in Eq 7.
  • alpha ratio (alpha/(alpha+beta)) = 10 percent (online AI alignment), 40 percent (HelpSteer2)
    Tuned via ablation (Figure 4b); it controls how strongly the policy is pulled toward the static reference versus the current sampling policy.
  • weight clip threshold w_clip = 20
    Chosen via ablation (Figure 5); it clips the product of the regularization and advantage weights to prevent gradient spikes, and the paper reports that both more aggressive and more conservative thresholds degrade results.
  • Monte-Carlo sampling size K = 4
    Chosen via ablation (Figure 5); the ablation shows K = 1 is already near-optimal, so the default value is not critical to the central claim.
assumptions (3)
  • standard math Gibbs' inequality and KL divergence non-negativity, used to solve the constrained objective in Theorem 4.1 (Appendix B).
    Unproved background result; standard and safe.
  • domain assumption The AWR state-distribution approximation replacing d_pi_theta by d_pi_t (Eq 3), plus the positivity assumptions pi_t(y|x) > 0 and pi_ref(y|x) > 0 required for the closed-form solution and importance weights.
    For auto-regressive LLMs with softmax outputs the positivity holds; the state-distribution replacement is the standard AWR assumption and bounds how far pi_{t+1} can move from pi_t in a single update.
  • ad hoc to paper The LLM annotator's scalar reward is a valid proxy for human preference.
    Load-bearing for the real-world claim; Table 1 shows only 60 to 80 percent agreement with human labels, no significance tests are reported, and the harmlessness prompt (Table 8) scores harmfulness while Algorithm 1 maximizes the score.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Direct Advantage Regression: Aligning LLMs with Online AI Reward." pith.science (2026). https://pith.science/paper/TKJLJY72

@misc{pith2026250414177,
  author       = {Pith},
  title        = {Pith review of: Direct Advantage Regression: Aligning LLMs with Online AI Reward},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TKJLJY72}},
  note         = {Machine review of arXiv:2504.14177}
}
read the original abstract

Online AI Feedback (OAIF) presents a promising alternative to Reinforcement Learning from Human Feedback (RLHF) by utilizing online AI preference in aligning language models (LLMs). However, the straightforward replacement of humans with AI deprives LLMs from learning more fine-grained AI supervision beyond binary signals. In this paper, we propose Direct Advantage Regression (DAR), a simple alignment algorithm using online AI reward to optimize policy improvement through weighted supervised fine-tuning. As an RL-free approach, DAR maintains theoretical consistency with online RLHF pipelines while significantly reducing implementation complexity and improving learning efficiency. Our empirical results underscore that AI reward is a better form of AI supervision consistently achieving higher human-AI agreement as opposed to AI preference. Additionally, evaluations using GPT-4-Turbo and MT-bench show that DAR outperforms both OAIF and online RLHF baselines.

Figures

Figures reproduced from arXiv: 2504.14177 by the authors.

Figure 1
Figure 1. Direct Advantage Regression with Online AI Reward. (Left) Using the reward labels provided by the LLM annotator, DAR increases the likelihood of each n-shot responses based on the calculated regression weight, so that the response of higher quality will have a higher probability to be sampled in the next iteration. (Right) The dual-constraint optimization objective of DAR: 1) the reference regularization prevents re… view at source ↗
Figure 2
Figure 2. Contrasting KL regularization approaches in RL fine￾tuning and on-policy RL. (Left) RL fine-tuning employs a fixed reference policy to mitigate reward hacking. (Right) On-policy RL methods (including regression-based) regularize with respect to the current sampling policy to ensure monotonic policy improvement. 4. Direct Advantage Regression This section presents a streamlined optimization approach that significantl… view at source ↗
Figure 3
Figure 3. Reference win rate curves of DAR with online AI reward against DPO with offline human preference, DAP methods (DPO, IPO, SLiC) with online AI preference, and RLHF methods (PPO, RLOO, Iterative SFT) with online AI reward. Win rates are averaged over 3 seeds and are judged by GPT-4-Turbo based on a 1k random test set for the tasks of TL;DR, Helpfulness and Harmlessness. 5.3. Baseline Methods The main baseline approach… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Performance of DAR on Helpfulness under different total regularization (a), and alpha ratio (b). Win rates are judged by Qwen2-72B-Instruct using a 1k random test set, while results are averaged over 3 seeds. over-optimization that typically results in verbose outputs.…
Figure 5
Figure 5. Figure 5: Reference win rate (a) and AI reward (b) of DAR on Helpfulness when varying weight clip and sampling size. Win rates and rewards are judged by Qwen2-72B-Instruct using a 1k random test set. we offer two tuning recommendations for DAR: 1) Higher total regularization is …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 4 canonical work pages

  1. [1]

    Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms, 2024

    Ahmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J., Pietquin, O., Üstün, A., and Hooker, S. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms, 2024

    work page 2024

  2. [2]

    G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., and Munos, R

    Azar, M. G., Rowland, M., Piot, B., Guo, D., Calandriello, D., Valko, M., and Munos, R. A general theoretical paradigm to understand learning from human preferences, 2023. URL https://arxiv.org/abs/2310.12036

    arXiv 2023

  3. [3]

    Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022 a

    Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, ...

    arXiv 2022

  4. [4]

    E., Fort, S., Lanham, T., Telleen-Lawton, T., Conerly, T., Henighan, T., Hume, T., Bowman, S

    Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKinnon, C., Chen, C., Olsson, C., Olah, C., Hernandez, D., Drain, D., Ganguli, D., Li, D., Tran-Johnson, E., Perez, E., Kerr, J., Mueller, J., Ladish, J., Landau, J., Ndousse, K., Lukosuite, K., Lovitt, L., Sellitto, M., Elhage, N., Schiefer, N., ...

    arXiv 2022

  5. [5]

    Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, ...

    arXiv 2020

  6. [6]

    T., Li, Y., Lundberg, S., Nori, H., Palangi, H., Ribeiro, M

    Bubeck, S., Chandrasekaran, V., Eldan, R., Gehrke, J., Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lundberg, S., Nori, H., Palangi, H., Ribeiro, M. T., and Zhang, Y. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023. URL https://arxiv.org/abs/2303.12712

    arXiv 2023

  7. [7]

    H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., Sutskever, I., and Wu, J

    Burns, C., Izmailov, P., Kirchner, J. H., Baker, B., Gao, L., Aschenbrenner, L., Chen, Y., Ecoffet, A., Joglekar, M., Leike, J., Sutskever, I., and Wu, J. Weak-to-strong generalization: Eliciting strong capabilities with weak supervision, 2023. URL https://arxiv.org/abs/2312.09390

    arXiv 2023

  8. [8]

    Bootstrapping language models with dpo implicit rewards, 2024 a

    Chen, C., Liu, Z., Du, C., Pang, T., Liu, Q., Sinha, A., Varakantham, P., and Lin, M. Bootstrapping language models with dpo implicit rewards, 2024 a . URL https://arxiv.org/abs/2406.09760

    arXiv 2024

Show all 54 references
  1. [9]

    Optune: Efficient online preference tuning, 2024 b

    Chen, L., Chen, J., Liu, C., Kirchenbauer, J., Soselia, D., Zhu, C., Goldstein, T., Zhou, T., and Huang, H. Optune: Efficient online preference tuning, 2024 b . URL https://arxiv.org/abs/2406.07657

    2024 arXiv

  2. [10]

    Grath: Gradual self-truthifying for large language models, 2024 c

    Chen, W., Song, D., and Li, B. Grath: Gradual self-truthifying for large language models, 2024 c . URL https://arxiv.org/abs/2401.12292

    2024 arXiv

  3. [11]

    S., and Shen, Y

    Chen, Y., Wang, S., Yang, Z., Sharma, H., Karampatziakis, N., Yu, D., Jamieson, K., Du, S. S., and Shen, Y. Cost-effective proxy reward model construction with on-policy and active learning, 2024 d . URL https://arxiv.org/abs/2407.02119

    2024 arXiv

  4. [12]

    Ultrafeedback: Boosting language models with scaled ai feedback, 2024

    Cui, G., Yuan, L., Ding, N., Yao, G., He, B., Zhu, W., Ni, Y., Xie, G., Xie, R., Lin, Y., Liu, Z., and Sun, M. Ultrafeedback: Boosting language models with scaled ai feedback, 2024. URL https://arxiv.org/abs/2310.01377

    2024 arXiv

  5. [13]

    Flashattention-2: Faster attention with better parallelism and work partitioning, 2023

    Dao, T. Flashattention-2: Faster attention with better parallelism and work partitioning, 2023. URL https://arxiv.org/abs/2307.08691

    2023 arXiv

  6. [14]

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., Goyal, A., Hartshorn, A., Yang, A., Mitra, A., Sravankumar, A., Korenev, A., Hinsvark, A., Rao, A., Zhang, A., Rodriguez, A., Gre...

    2024 arXiv

  7. [15]

    Accelerate: Training and inference at scale made simple, efficient and adaptable

    Gugger, S., Debut, L., Wolf, T., Schmid, P., Mueller, Z., Mangrulkar, S., Sun, M., and Bossan, B. Accelerate: Training and inference at scale made simple, efficient and adaptable. https://github.com/huggingface/accelerate, 2022

    2022

  8. [16]

    Direct language model alignment from online ai feedback, 2024

    Guo, S., Zhang, B., Liu, T., Liu, T., Khalman, M., Llinares, F., Rame, A., Mesnard, T., Zhao, Y., Piot, B., Ferret, J., and Blondel, M. Direct language model alignment from online ai feedback, 2024. URL https://arxiv.org/abs/2402.04792

    2024 arXiv

  9. [17]

    S., Hou, L., Wu, Y., Wang, X., Yu, H., and Han, J

    Huang, J., Gu, S. S., Hou, L., Wu, Y., Wang, X., Yu, H., and Han, J. Large language models can self-improve, 2022. URL https://arxiv.org/abs/2210.11610

    2022 arXiv

  10. [18]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023. URL http...

    2023 arXiv

  11. [19]

    Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression, 2024

    Jiang, H., Wu, Q., Luo, X., Li, D., Lin, C.-Y., Yang, Y., and Qiu, L. Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression, 2024. URL https://arxiv.org/abs/2310.06839

    2024 arXiv

  12. [20]

    Buy 4 REINFORCE samples, get a baseline for free!, 2019

    Kool, W., van Hoof, H., and Welling, M. Buy 4 REINFORCE samples, get a baseline for free!, 2019. URL https://openreview.net/forum?id=r1lgTGL5DE

    2019

  13. [21]

    H., Gonzalez, J

    Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023

    2023

  14. [22]

    Rlaif vs

    Lee, H., Phatale, S., Mansoor, H., Mesnard, T., Ferret, J., Lu, K., Bishop, C., Hall, E., Carbune, V., Rastogi, A., and Prakash, S. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback, 2024. URL https://arxiv.org/abs/2309.00267

    2024 arXiv

  15. [23]

    D., Yue, X., and Chen, W

    Li, T., Zhang, G., Do, Q. D., Yue, X., and Chen, W. Long-context llms struggle with long in-context learning, 2024 a . URL https://arxiv.org/abs/2404.02060

    2024 arXiv

  16. [24]

    Self-alignment with instruction backtranslation, 2024 b

    Li, X., Yu, P., Zhou, C., Schick, T., Levy, O., Zettlemoyer, L., Weston, J., and Lewis, M. Self-alignment with instruction backtranslation, 2024 b . URL https://arxiv.org/abs/2308.06259

    2024 arXiv

  17. [25]

    OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Balaji, S., Balcom, V., Baltescu, P., Bao, H., Bavarian, M., Belgum, J., Bello, I., Berdine, J., Bernadett-Shapi...

    2024 arXiv

  18. [26]

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to f...

    2022 arXiv

  19. [27]

    Y., Yuan, W., Cho, K., He, H., Sukhbaatar, S., and Weston, J

    Pang, R. Y., Yuan, W., Cho, K., He, H., Sukhbaatar, S., and Weston, J. Iterative reasoning preference optimization, 2024. URL https://arxiv.org/abs/2404.19733

    2024 arXiv

  20. [28]

    B., Kumar, A., Zhang, G., and Levine, S

    Peng, X. B., Kumar, A., Zhang, G., and Levine, S. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning, 2019. URL https://arxiv.org/abs/1910.00177

    2019 arXiv

  21. [29]

    and Schaal, S

    Peters, J. and Schaal, S. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th international conference on Machine learning, pp.\ 745--750, 2007

    2007

  22. [30]

    Online dpo: Online direct preference optimization with fast-slow chasing, 2024

    Qi, B., Li, P., Li, F., Gao, J., Zhang, K., and Zhou, B. Online dpo: Online direct preference optimization with fast-slow chasing, 2024. URL https://arxiv.org/abs/2406.05534

    2024 arXiv

  23. [31]

    Improving language understanding by generative pre-training

    Radford, A. Improving language understanding by generative pre-training. 2018

    2018

  24. [32]

    D., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. Direct preference optimization: Your language model is secretly a reward model, 2024. URL https://arxiv.org/abs/2305.18290

    2024 arXiv

  25. [33]

    Stable-baselines3: Reliable reinforcement learning implementations

    Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research, 22 0 (268): 0 1--8, 2021. URL http://jmlr.org/papers/v22/20-1364.html

    2021

  26. [34]

    I., and Abbeel, P

    Schulman, J., Levine, S., Moritz, P., Jordan, M. I., and Abbeel, P. Trust region policy optimization, 2017 a . URL https://arxiv.org/abs/1502.05477

    2017 arXiv

  27. [35]

    Proximal policy optimization algorithms, 2017 b

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms, 2017 b . URL https://arxiv.org/abs/1707.06347

    2017 arXiv

  28. [36]

    and Stern, M

    Shazeer, N. and Stern, M. Adafactor: Adaptive learning rates with sublinear memory cost, 2018. URL https://arxiv.org/abs/1804.04235

    2018 arXiv

  29. [37]

    M., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P

    Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. Learning to summarize from human feedback, 2022. URL https://arxiv.org/abs/2009.01325

    2022 arXiv

  30. [38]

    S., Barto, A

    Sutton, R. S., Barto, A. G., et al. Introduction to reinforcement learning. vol. 135, 1998

    1998

  31. [39]

    G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C

    Team, G., Riviere, M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L., Mesnard, T., Shahriari, B., Ramé, A., Ferret, J., Liu, P., Tafti, P., Friesen, A., Casbon, M., Ramos, S., Kumar, R., Lan, C. L., Jerome, S., Tsitsulin, A., Vieillard, N., Stanczyk, P., ...

    2024 arXiv

  32. [40]

    N., Kaiser, L., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need, 2023. URL https://arxiv.org/abs/1706.03762

    2023 arXiv

  33. [41]

    Trl: Transformer reinforcement learning

    von Werra, L., Belkada, Y., Tunstall, L., Beeching, E., Thrush, T., Lambert, N., Huang, S., Rasul, K., and Gallouédec, Q. Trl: Transformer reinforcement learning. https://github.com/huggingface/trl, 2020

    2020

  34. [42]

    Beyond the limits: A survey of techniques to extend the context length in large language models, 2024 a

    Wang, X., Salmani, M., Omidi, P., Ren, X., Rezagholizadeh, M., and Eshaghi, A. Beyond the limits: A survey of techniques to extend the context length in large language models, 2024 a . URL https://arxiv.org/abs/2402.02244

    2024 arXiv

  35. [43]

    A., Khashabi, D., and Hajishirzi, H

    Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions, 2023. URL https://arxiv.org/abs/2212.10560

    2023 arXiv

  36. [44]

    T., Reed, S., Shahriari, B., Siegel, N., Merel, J., Gulcehre, C., Heess, N., and de Freitas, N

    Wang, Z., Novikov, A., Zolna, K., Springenberg, J. T., Reed, S., Shahriari, B., Siegel, N., Merel, J., Gulcehre, C., Heess, N., and de Freitas, N. Critic regularized regression, 2021. URL https://arxiv.org/abs/2006.15134

    2021 arXiv

  37. [45]

    Helpsteer2-preference: Complementing ratings with preferences, 2024 b

    Wang, Z., Bukharin, A., Delalleau, O., Egert, D., Shen, G., Zeng, J., Kuchaiev, O., and Dong, Y. Helpsteer2-preference: Complementing ratings with preferences, 2024 b . URL https://arxiv.org/abs/2410.01257

    2024 arXiv

  38. [46]

    Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint, 2024

    Xiong, W., Dong, H., Ye, C., Wang, Z., Zhong, H., Ji, H., Jiang, N., and Zhang, T. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint, 2024. URL https://arxiv.org/abs/2312.11456

    2024 arXiv

  39. [47]

    Is dpo superior to ppo for llm alignment? a comprehensive study, 2024

    Xu, S., Fu, W., Gao, J., Ye, W., Liu, W., Mei, Z., Wang, G., Yu, C., and Wu, Y. Is dpo superior to ppo for llm alignment? a comprehensive study, 2024. URL https://arxiv.org/abs/2404.10719

    2024 arXiv

  40. [48]

    Qwen2 technical report, 2024 a

    Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., Yang, J., Tu, J., Zhang, J., Ma, J., Yang, J., Xu, J., Zhou, J., Bai, J., He, J., Lin, J., Dang, K., Lu, K., Chen, K., Yang, K., Li, M....

    2024

  41. [49]

    Regularizing hidden states enables learning generalizable reward model for llms

    Yang, R., Ding, R., Lin, Y., Zhang, H., and Zhang, T. Regularizing hidden states enables learning generalizable reward model for llms. arXiv preprint arXiv:2406.10216, 2024 b

    2024 arXiv

  42. [50]

    Y., Cho, K., Li, X., Sukhbaatar, S., Xu, J., and Weston, J

    Yuan, W., Pang, R. Y., Cho, K., Li, X., Sukhbaatar, S., Xu, J., and Weston, J. Self-rewarding language models, 2024. URL https://arxiv.org/abs/2401.10020

    2024 arXiv

  43. [51]

    Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M., and Liu, P. J. Slic-hf: Sequence likelihood calibration with human feedback, 2023. URL https://arxiv.org/abs/2305.10425

    2023 arXiv

  44. [52]

    P., Zhang, H., Gonzalez, J

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. URL https://arxiv.org/abs/2306.05685

    2023 arXiv

  45. [53]

    M., Stiennon, N., Wu, J., Brown, T

    Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences, 2020. URL https://arxiv.org/abs/1909.08593

    2020 arXiv

  46. [54]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.