Pith. sign in

REVIEW 3 major objections 6 minor 33 references

Utility-inspired Reward Transformations Improve Reinforcement Learning Training of Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A utility-theory reward transformation makes RLHF-trained language models more harmless without sacrificing helpfulness.

desk verdict A simple, well-motivated reward transformation with plausible gains against reward hacking, but the empirical case rests on a single LLM judge and a validation-tuned configuration. read the letter →

arxiv 2501.06248 v2 pith:JX3YDFAO submitted 2025-01-08 cs.LG cs.AIcs.CLecon.GNq-fin.EC

Roberto-Rafael Maura-Rivero , Chirag Nagpal , Roma Patel , Francesco Visin This is my paper · ORCID
classification cs.LGcs.AIcs.CLecon.GNq-fin.EC
keywords rewardaggregationRLHFInadaconditionsutilitytheoryharmlessnesshelpfulnesshackinglanguagemodelalignment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the common practice of summing multiple reward signals in reinforcement learning from human feedback (RLHF) is flawed, because it treats a critically bad score on one dimension the same as a mildly bad score across all dimensions. To fix this, the authors introduce the Inada Reward Transformation (IRT), which applies a steep linear penalty to rewards below a per-dimension threshold and a concave compression (logarithmic at the chosen settings) to rewards above it. Training a 2B-parameter Gemma model with IRT applied to the harmlessness reward before adding the helpfulness reward yields a 75% strict win rate against linear aggregation on a harmlessness judge, while helpfulness stays at 52% preference. The paper presents this as a cheap, drop-in modification to existing RLHF pipelines.

What carries the argument

The central object is the Inada Reward Transformation (IRT), defined in Equation (2) as $IRT(r_i) = U_{CRRA}(r_i - \tau_i + 1)$ for $r_i > \tau_i$ and $\beta_i(r_i - \tau_i)$ for $r_i \le \tau_i$, where $U_{CRRA}$ is the Constant Relative Risk Aversion utility function. With $\gamma=1$, the above-threshold branch is $\ln(r_i - \tau_i + 1)$. The transformation is applied to each reward individually before summation; the paper primarily uses a Partial IRT on the harmlessness reward only. The work it does is to make the aggregated reward sharply contrast responses that fall below the safety threshold from those that merely achieve mediocre scores, while ignoring further gains once a response is already safely above threshold.

What would settle it

A human preference evaluation on the same test prompts, comparing the IRT-trained model against the linear-aggregation baseline: if human raters do not prefer the IRT model's responses as often as the LLM judge does (or prefer the baseline), the paper's central claim is contradicted.

Watch

Extended reading notes

Core claim

The central discovery is that transforming individual rewards before aggregation, rather than aggregating then transforming, fixes two failure modes of linear reward combination: insensitivity to critically low rewards and over-optimization of already-high rewards. Concretely, IRT replaces the harmlessness reward $r_{HA}$ with a piecewise function: above threshold $\tau$ it behaves like a constant relative risk aversion utility (at $\gamma=1$, a logarithm), and below $\tau$ it applies a linear penalty with slope $\beta$. With hyperparameters $\beta=2$, $\gamma=1$, $\tau=0$, the transformed harmlessness reward plus the untransformed helpfulness reward trains a model that a Gemma 2 27B judge rates as more harmless 75% of the time (strict wins) and at least as helpful 52% of the time, compared to the linear baseline. The authors attribute this to IRT eliminating reward-hacking artifacts like punting and boilerplate disclaimers.

Load-bearing premise

The claimed safety improvement is measured by a single LLM judge, the Gemma 2 27B instruction-tuned model, which stands in for human preferences; no human raters are involved. If that judge does not reflect what humans actually consider more or less harmful, the win rates do not establish a real safety gain.

Editorial extensions

If this is right

  • Applying IRT to the harmlessness reward (with $\beta=2$, $\gamma=1$, $\tau=0$) yields a strict win rate of 0.75 against linear aggregation on the harmlessness judge, while helpfulness preference remains at 0.52.
  • The transformation is a drop-in change: it modifies only the reward signal before summation, leaving the RL algorithm, reward models, and training data untouched, and adds negligible compute.
  • IRT can be applied partially (to any subset of rewards) or fully, and its three hyperparameters control the strictness of the threshold, the slope of the penalty below it, and the concavity above it.
  • Qualitative examples show IRT-trained models avoid reward-hacking artifacts such as punting, boilerplate medical disclaimers, and excessive catchphrases.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same piecewise concave-and-penalizing shape could be applied to other safety-critical reward dimensions, such as truthfulness or bias, with thresholds learned from data rather than grid-searched.
  • IRT is not tied to any particular RL algorithm; it should transfer to any paradigm that combines multiple reward scores, including best-of-$n$ sampling and direct preference optimization.
  • If the LLM judge's preferences correlate poorly with human judgments, the reported win rates could reflect stylistic differences (e.g., fewer disclaimers) rather than genuine safety improvements; a human rater study would settle this.
  • The optimal threshold $\tau^* = 0$ suggests the harmlessness reward model's scores are roughly calibrated around zero, so IRT mainly rescales the sign-dependent region; other reward models with different scales would require re-tuning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes the Inada Reward Transformation (IRT), a piecewise transformation of individual reward signals in multi-reward RLHF, motivated by Inada conditions from economic utility theory. IRT applies a linear penalty with slope β to rewards below a threshold τ and a concave CRRA transformation to rewards above τ, before aggregation. The authors evaluate a partial IRT that transforms only the harmlessness reward on a Gemma 2B model trained on the Anthropic HH dataset, using a Gemma 2 27B LLM as the judge. They report that the IRT-trained model wins against a linear-aggregation baseline on harmlessness (AR(HA) preference 0.61, win rate 0.75) while roughly maintaining helpfulness (AR(HE) preference 0.52, win rate 0.52). The paper includes ablations of the IRT parameters, qualitative examples, and full hyperparameter search tables in the appendix.

Significance. If the reported results are reliable, the paper makes a useful, low-cost contribution to reward aggregation in RLHF: a simple transformation that can be applied to any reward model without additional training, and that mitigates a specific form of reward hacking (e.g., spurious medical disclaimers) visible in the baseline. The paper is transparent about experimental details, providing prompts, hyperparameters, and compute requirements, which supports reproducibility. The main weakness is that the entire quantitative evaluation rests on a single LLM judge without validation against human preferences; the central claim about 'harmlessness' and 'helpfulness' is therefore only as strong as the judge's alignment with human notions. The paper also overstates the helpfulness result, as 0.52 is not materially different from chance. With human evaluation or judge-human agreement evidence, the contribution would be considerably more convincing.

major comments (3)
  1. [Section 4.3, Table 1] The entire quantitative evaluation relies on a single zero-shot autorater, Gemma 2 27B, to judge helpfulness and harmlessness, with no validation of this judge against human judgments. The paper's central claim is about improving 'harmlessness' and preserving 'helpfulness', which are human-defined properties. The statement in Section 4.3 that LLM-as-judge is 'standard evaluation measure' does not establish that the judge's preferences match human preferences in this safety-critical setting, especially since the reward models and the judge are all Gemma-based. I recommend adding a small human evaluation on a random subset, or at minimum reporting the autorater's agreement with the human preference labels available in the Anthropic HH test set. Without such evidence, the win rates in Table 1 cannot be interpreted as evidence about genuine harmlessness or helpfulness.
  2. [Section 4.4, Tables 1 and 2] The hyperparameters β=2, γ=1, τ=0 were selected as the best of 18 configurations on a validation set using the same Gemma judge, and the test results in Table 1 correspond to that single selected configuration. This creates a selection effect: the reported numbers are the maximum over a grid, not an independent evaluation of a prespecified method. The helpfulness preference is 0.52 ± 0.01, which is not convincingly above chance (roughly 1.8 standard errors), and the ablation in Table 2 shows that removing the curvature (γ=0, β=2, τ=0) yields higher helpfulness (0.59) with only a slightly lower harmlessness preference (0.57 vs 0.61). The claim in the abstract that models trained with IRT 'score as more helpful' is not supported by these numbers. The paper should either report the full distribution of test results across the grid (already in Appendix D) in a more prominent way, apply a model-selection correction, or temper the claim to 'without degrading helpfulness'.
  3. [Section 5.2, Table 2; Eq. (2)] The ablation demonstrates that the main driver of the harmlessness improvement is the penalty factor β (the slope of the linear branch below the threshold τ), not the Inada-inspired concave branch above the threshold. Setting γ=0, which removes the concave curvature, gives AR(HA) 0.57 and AR(HE) 0.59, compared with 0.61 and 0.52 for the full IRT. The paper does not compare against a simple piecewise-linear transformation with the same slope below τ and slope 1 above τ, nor against a linear aggregation with a higher weight on the harmlessness reward (e.g., 2*r_HA + r_HE). Such baselines are needed to determine whether the improvement stems from the economic-theoretic shape or merely from an increased penalty on low harmlessness rewards, which would make the contribution a form of loss aversion already known in the literature.
minor comments (6)
  1. [Section 2.4 and Eq. (2)] The paper calls IRT 'Inada-inspired', but the linear branch below τ violates Inada condition 2 (strict concavity) over that part of the domain. Please clarify that the Inada properties are only satisfied above the threshold.
  2. [Section 4.3, Metrics] The win rate WR = W/(W+L) excludes ties, so the headline 'win rate 0.75' in Table 1 is a conditional win rate over non-tied comparisons; the tie rates (e.g., 0.55 for AR(HA) in Table 2) should be displayed in Table 1 to avoid misinterpretation.
  3. [Section 4.4] The description of the grid-search objective 'maximized the average winrate, i.e., 1/2 (Helpfulness WR + Harmlessness WR)' is ambiguous: it is not clear whether the average is over the two datasets or over the two autoraters. The appendix tables (e.g., Table 9) show win rates for both AR(HA) and AR(HE); please specify the exact selection criterion.
  4. [Section 5.1] The sentence 'while retaining performance - and even slightly improving it - on the helpfulness score' is not consistent with the AR(HE) preference of 0.52, which is not a statistically meaningful improvement; please rephrase to 'without significant degradation'.
  5. [Section 9 (Limitations)] The limitations section discusses model scale and threshold selection but does not mention the absence of human evaluation or the reliance on a single autorater; these are the most consequential limitations of this study and should be acknowledged.
  6. [Tables 3 and 5] The qualitative examples convincingly illustrate reward hacking in the baseline, but they are a handful of cherry-picked cases; consider reporting a quantitative measure of such artifacts (e.g., frequency of out-of-context medical disclaimers in the full test set).

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the IRT form is an explicit design choice, hyperparameters are tuned on a held-out validation split, and the reported win rates are empirical test-set measurements rather than consequences of the transformation's definition.

full rationale

The claimed derivation chain is not circular. The Inada Reward Transformation in Eq. (2) is introduced as an explicit modification of the standard UCRRA utility function; the paper does not claim to derive the transformation from the experimental outcomes. The baseline is exactly the identity case beta=1, gamma=0, tau=0, so comparing IRT to the baseline is a comparison between two concrete reward transformations, not a tautology. Hyperparameters were selected by grid search on validation splits (Section 4.4), and the headline numbers in Table 1 are reported on the held-out test split of the Anthropic HH dataset, so the central result is not the same data used for fitting. The self-citations that appear (Eisenstein et al. 2023 for LLM-as-judge; Wang et al. 2024, Patel et al. 2021, Gemp et al. 2024 in related work) are contextual and not load-bearing: none is used to justify the IRT form or to rule out alternatives. The main weakness, reliance on a single Gemma 2 27B autorater with no human validation, is a measurement-validity concern rather than a circularity concern; it does not make the reported preferences equal to the training reward by construction. The paper's Limitations section candidly notes the small model scale and the need for better threshold selection, which further supports that the contribution is an empirical proposal rather than a repackaged input.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several domain assumptions (reward model validity, judge validity, reward scale comparability) and on three free hyperparameters selected by grid search. No new physical or conceptual entities are introduced.

free parameters (3)
  • beta (penalty factor for harmlessness IRT) = 2
    Selected by grid search on the validation set to maximize average win rate (Section 4.4).
  • gamma (diminishing returns curvature for harmlessness IRT) = 1
    Selected by grid search on the validation set to maximize average win rate (Section 4.4).
  • tau (reward threshold for harmlessness IRT) = 0
    Selected by grid search on the validation set to maximize average win rate (Section 4.4).
assumptions (4)
  • domain assumption The reward models trained on half of the helpfulness and harmlessness datasets provide valid scalar signals for those qualities.
    The entire RLHF pipeline optimizes these scores; if the reward models are miscalibrated, the transformation's threshold and penalty operate on unreliable inputs. Section 4.1.
  • domain assumption The Gemma 2 27B LLM autorater is a valid proxy for human judgments of helpfulness and harmlessness.
    Section 4.3 relies entirely on this judge for evaluation; no human evaluation is performed.
  • domain assumption Reward scores from the two reward models are on comparable scales, so summing transformed values is meaningful.
    The IRT applies a fixed threshold tau=0 to the harmlessness reward, implicitly assuming this value is appropriate for that reward's scale; no reward normalization is performed.
  • ad hoc to paper The piecewise IRT functional form (Eq. 2) with threshold, penalty, and log curvature is an appropriate model for reward aggregation.
    The specific form is proposed by the authors from economic theory; it is not derived from RLHF theory or from data. The parameters are then fitted to validation performance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Utility-inspired Reward Transformations Improve Reinforcement Learning Training of Language Models." pith.science (2026). https://pith.science/paper/JX3YDFAO

@misc{pith2026250106248,
  author       = {Pith},
  title        = {Pith review of: Utility-inspired Reward Transformations Improve Reinforcement Learning Training of Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JX3YDFAO}},
  note         = {Machine review of arXiv:2501.06248}
}
read the original abstract

Current methods that train large language models (LLMs) with reinforcement learning feedback, often resort to averaging outputs of multiple rewards functions during training. This overlooks crucial aspects of individual reward dimensions and inter-reward dependencies that can lead to sub-optimal outcomes in generations. In this work, we show how linear aggregation of rewards exhibits some vulnerabilities that can lead to undesired properties of generated text. We then propose a transformation of reward functions inspired by economic theory of utility functions (specifically Inada conditions), that enhances sensitivity to low reward values while diminishing sensitivity to already high values. We compare our approach to the existing baseline methods that linearly aggregate rewards and show how the Inada-inspired reward feedback is superior to traditional weighted averaging. We quantitatively and qualitatively analyse the difference in the methods, and see that models trained with Inada-transformations score as more helpful while being less harmful.

Figures

Figures reproduced from arXiv: 2501.06248 by the authors.

Figure 1
Figure 1. Linear reward aggregation (a) and (b) show two different responses with different helpful and harm￾ful ratings (green and red), but same aggregated reward (blue). Note that the response in (a) is rated satisfac￾torily helpful (above minimum helpfulness threshold, depicted as a green dotted line), but also dangerously harmful (below maximum harmfulness threshold, de￾picted as a dotted red line), while the one in (b) … view at source ↗
Figure 3
Figure 3. The Inada Reward Transformation (IRT). Rewards above the helpfulness threshold get discounted (intuitively, once the answer is helpful there is little gain making it more helpful), while rewards below the harmfulness threshold get further penalised. As a result, the aggregated reward in (a) is much lower than the one in (b), allowing to differentiate between the two cases. Partial IRT A Partial IRT only transform so… view at source ↗
Figure 2
Figure 2. Impact of the three hyperparameters of the Inada Reward Transformation. The reward thresh￾old (τ ) determines the point of application of the re￾ward transformations governed by the other two hyper￾parameters. A larger penalty factor (β) amplifies the negative impact of rewards below the threshold, while a higher diminishing returns (γ) de-emphasize gains in already satisfactory values [PITH_FULL_IMAGE:figures/full… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 8 canonical work pages

  1. [1]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862

    arXiv 2022

  2. [2]

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258

    arXiv 2021

  3. [3]

    Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165

    arXiv 2020

  4. [4]

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30

    2017

  5. [5]

    Vincent Conitzer, Rachel Freedman, Jobst Heitzig, Wesley H Holliday, Bob M Jacobs, Nathan Lambert, Milan Moss \'e , Eric Pacuit, Stuart Russell, Hailey Schoelkopf, et al. 2024. Social choice should guide ai alignment in dealing with diverse human feedback. arXiv preprint arXiv:2404.10271

    arXiv 2024

  6. [6]

    Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D'Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak Ramachandran, et al. 2023. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. arXiv preprint arXiv:2312.09244

    arXiv 2023

  7. [7]

    Kawin Ethayarajh and Dan Jurafsky. 2021. Attention flows are shapley value explanations. arXiv preprint arXiv:2105.14652

    work page Pith review arXiv 2021

  8. [8]

    Luise Ge, Daniel Halpern, Evi Micha, Ariel D Procaccia, Itai Shapira, Yevgeniy Vorobeychik, and Junlin Wu. 2024. Axioms for ai alignment from human feedback. arXiv preprint arXiv:2405.14758

    arXiv 2024

Show all 33 references
  1. [9]

    Gemma Team , Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295

    2024 arXiv

  2. [10]

    Ian Gemp, Yoram Bachrach, Marc Lanctot, Roma Patel, Vibhavari Dasagi, Luke Marris, Georgios Piliouras, Siqi Liu, and Karl Tuyls. 2024. States as strings as strategies: Steering language models with game-theoretic solvers. arXiv preprint arXiv:2402.01704

    2024 arXiv

  3. [11]

    Amelia Glaese, Nat McAleese, Maja Tr e bacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. 2022. Improving alignment of dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.14375

    2022 arXiv

  4. [12]

    Athul Paul Jacob, Yikang Shen, Gabriele Farina, and Jacob Andreas. 2023. The consensus game: Language model generation via equilibrium search. arXiv preprint arXiv:2310.09139

    2023 arXiv

  5. [13]

    Lars Ljungqvist and Thomas J Sargent. 2018. Recursive macroeconomic theory. MIT press

    2018

  6. [14]

    Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca D

    Ted Moskovitz, Aaditya K. Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca D. Dragan, and Stephen McAleer. 2023. https://arxiv.org/abs/2310.04373 Confronting reward model overoptimization with constrained rlhf . Preprint, arXiv:2310.04373

    2023 arXiv

  7. [15]

    R \'e mi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. 2023. Nash learning from human feedback. arXiv preprint arXiv:2312.00886

    2023 arXiv

  8. [16]

    Kanad Shrikar Pardeshi, Itai Shapira, Ariel D Procaccia, and Aarti Singh. 2024. Learning social welfare functions. arXiv preprint arXiv:2405.17700

    2024 arXiv

  9. [17]

    Roma Patel, Marta Garnelo, Ian Gemp, Chris Dyer, and Yoram Bachrach. 2021. Game-theoretic vocabulary selection via the shapley value and banzhaf index. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human L...

    2021

  10. [18]

    John W Pratt. 1978. Risk aversion in the small and in the large. In Uncertainty in economics, pages 59--79. Elsevier

    1978

  11. [19]

    Ariel D Procaccia, Aviv Zohar, Yoni Peleg, and Jeffrey S Rosenschein. 2009. The learnability of voting rules. Artificial Intelligence, 173(12-13):1133--1149

    2009

  12. [20]

    Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. 2024. Warm: On the benefits of weight averaged reward models

    2024

  13. [21]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

    2017 arXiv

  14. [22]

    Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. 2023. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716

    2023 arXiv

  15. [23]

    Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008--3021

    2020

  16. [24]

    Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. 2024. A minimaximalist approach to reinforcement learning from human feedback. arXiv preprint arXiv:2401.04056

    2024 arXiv

  17. [25]

    Alex Tamkin, Amanda Askell, Liane Lovitt, Esin Durmus, Nicholas Joseph, Shauna Kravec, Karina Nguyen, Jared Kaplan, and Deep Ganguli. 2023. Evaluating and mitigating discrimination in language model decisions. arXiv preprint arXiv:2312.03689

    2023 arXiv

  18. [26]

    Hirofumi Uzawa. 1961. On a two-sector model of economic growth. The Review of Economic Studies, 29(1):40--47

    1961

  19. [27]

    Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, and Yun-Hsuan Sung. 2024. Foundational autoraters: Taming large language models for better automatic evaluation. arXiv preprint arXiv:2407.10817

    2024 arXiv

  20. [28]

    Zihao Wang, Chirag Nagpal, Jonathan Berant, Jacob Eisenstein, Alex D'Amour, Sanmi Koyejo, and Victor Veitch. 2024. Transforming and combining rewards for aligning large language models. arXiv preprint arXiv:2402.00742

    2024 arXiv

  21. [29]

    Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359

    2021 arXiv

  22. [30]

    Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8:229--256

    1992

  23. [31]

    Smith, Mari Ostendorf, and Hannaneh Hajishirzi

    Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. 2023. https://arxiv.org/abs/2306.01693 Fine-grained human feedback gives better rewards for language model training . Preprint, arXiv:2306.01693

    2023 arXiv

  24. [32]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  25. [33]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.