Pith. sign in

REVIEW 4 major objections 4 minor 3 cited by

Curiosity-Driven Reinforcement Learning from Human Feedback

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding an intrinsic curiosity reward—the prediction error of a small forward model on token choices outside the top-$k$—lets RLHF produce more diverse outputs while keeping reward-model-judged alignment comparable to standard RLHF.

desk verdict The diversity half of the paper is plausible; the alignment half is under-evidenced, but the study is honest, broad, and worth a serious referee. read the letter →

arxiv 2501.11463 v2 pith:5W7QFDKB submitted 2025-01-20 cs.CL

Haoran Sun , Yekun Chai , Shuohuan Wang , Yu Sun , Hua Wu , Haifeng Wang This is my paper · ORCID
classification cs.CL
keywords RLHFoutputdiversityintrinsicrewardcuriosity-drivenexplorationpredictionerrorlargelanguagemodelsalignmenttextgeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reinforcement learning from human feedback (RLHF) aligns language models with preferences but tends to shrink the variety of outputs they produce. This paper asks whether an intrinsic curiosity reward, borrowed from exploration in reinforcement learning, can undo that shrinkage at the RLHF stage without sacrificing alignment. The proposed method, CD-RLHF, adds a reward proportional to how poorly a small forward model predicts the next state, but only when the policy chose a token outside its top-$k$ choices ($k=1$ in the experiments). Across summarization and instruction-following tasks, the paper reports higher lexical and semantic diversity on every model tested, with reward-model scores essentially level against vanilla RLHF, and says the gains carry over to out-of-distribution questions and story writing. If right, it means output diversity can be treated as a training-time objective alongside preference alignment, not only as a post-hoc trade-off.

What carries the argument

The load-bearing object is the Intrinsic Curiosity Module (ICM), a two-layer MLP feature encoder $\phi$ plus a two-layer MLP forward model $f$ that predicts $\phi(s_{t+1})$ from $\phi(s_t)$ concatenated with the action embedding $\psi(a_t)$. Its squared prediction error is the curiosity reward; a top-$k$ gate zeros that reward for probable continuations, and reward whitening rescales it along the trajectory. The module is trained separately from the policy by minimizing the same squared error, so familiar states become less rewarding as prediction improves. The mechanism's job is to turn each RLHF step into a per-token novelty signal that can be added to the sparse reward-model score and the KL penalty without changing the alignment objective itself.

What would settle it

Replace the ICM prediction error with random noise of the same scale, keeping the top-$k$ gate and reward whitening identical; if diversity rises as much as with the real signal, prediction error is not the active ingredient. A complementary check is to measure factual accuracy of the diverse outputs on a ground-truth task such as summarization factuality or closed-book QA, since useful diversity should not systematically increase errors.

Watch

Extended reading notes

Core claim

CD-RLHF extends the standard RLHF reward $r^{(e)} = R - \beta D_{\mathrm{KL}}(\pi_{\mathrm{policy}} \| \pi_{\mathrm{ref}})$ by adding an intrinsic term $\eta r^{(i)}$. The Intrinsic Curiosity Module (ICM) encodes the current state and action, predicts the next state's features with a small forward model, and takes the squared prediction error $\frac{1}{2}\|\hat{\phi}(s_{t+1}) - \phi(s_{t+1})\|_2^2$ as the curiosity signal. This reward is zeroed whenever the chosen token falls in the top-$k$ of the policy's output distribution, so exploration is aimed at less probable continuations. State features are produced from the reference model's last hidden layers and action features from the policy model's token embeddings, giving the prediction problem one coherent feature space. With this reward added to PPO, the paper reports consistent diversity gains over vanilla RLHF on TL;DR summarization and UltraFeedback instruction following, comparable reward-model scores, and better judge-model win rates on the out-of-distribution MT-Bench benchmark.

Load-bearing premise

The method assumes that a small predictor's failure to guess the next hidden state marks a continuation that is worth exploring, rather than merely unpredictable or noisy.

Editorial extensions

If this is right

  • Output diversity can be improved during RLHF training itself, rather than through decoding-time tricks or auxiliary sentence-level rewards.
  • The diversity gains survive outside the training distribution: on MT-Bench, CD-RLHF scores higher than RLHF by an LLM judge and still produces more diverse outputs.
  • A single hyperparameter $\eta$ and the top-$k$ gate give a practical dial: fewer intrinsic rewards (larger $k$) reduce diversity but keep alignment, while more frequent rewards eventually hurt alignment.
  • The alignment-diversity trade-off is shifted but not eliminated: CD-RLHF models remain less diverse than the SFT models they start from.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the intrinsic reward is gated to non-top-$k$ tokens, part of the diversity gain may come from upweighting low-probability tokens rather than from genuinely useful novelty; a direct way to test this is to compare CD-RLHF against a version with a random noise reward of matched scale.
  • The curiosity signal resembles an epistemic-uncertainty estimate, so the same ICM reward could double as a calibration measure; one could check whether the more diverse completions are also more accurate on factuality benchmarks.
  • The mechanism is not tied to PPO: the same reward could be plugged into DPO-style objectives or used at decoding time, which the paper leaves untested.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes CD-RLHF, an extension of PPO-based RLHF in which a token-level intrinsic reward, computed as the prediction error of an Intrinsic Curiosity Module (ICM), is added to the extrinsic reward-model score and KL penalty. The intrinsic reward is gated by a top-k mask so that it only applies when the sampled token is not among the k most probable tokens (k=1 in the main experiments). The authors evaluate the method on TL;DR summarization and UltraFeedback instruction following using Gemma-2B, Gemma-7B, Llama-3.2-1B, and Llama-3.2-3B, reporting consistent improvements on lexical and semantic diversity metrics, positive GPT-4 and human diversity win rates, and RM scores comparable to vanilla RLHF. They additionally report MT-Bench and story-generation results. The central claim is that CD-RLHF improves output diversity while maintaining alignment quality comparable to standard RLHF.

Significance. If the central claim holds, the method is a simple, practical addition to PPO-style RLHF that could mitigate the well-documented diversity collapse of aligned LLMs. The paper has clear strengths: it targets an important problem; it evaluates across four base models and two tasks; it includes multiple diversity metrics spanning lexical and semantic aspects; it provides GPT-4 and human diversity judgments; it includes ablations on the top-k gate and on reward frequency; and it releases code. These are substantial assets. However, the evidence as presented is not yet sufficient to support the strength of the abstract's claims. All headline numbers come from single training runs without error bars or significance tests, and the alignment-quality evidence relies on the same reward model used as the PPO training signal. The paper's own Limitations section concedes that the diversity-alignment trade-off still persists and that CD-RLHF remains below SFT in diversity, which tempers the framing but does not contradict the stated contribution.

major comments (4)
  1. [§4.1, Table 1] All results in Table 1 and throughout the paper are single-run point estimates: no error bars, confidence intervals, multiple seeds, or significance tests are reported for any diversity metric or RM score. The abstract's phrase "significant gains" is therefore not statistically established. Please provide at least three independent training runs per configuration and report means with standard deviations and appropriate paired tests for the headline comparisons, or else temper the language to "consistent improvements" in the current runs.
  2. [§4.1 evaluation description, Table 1, Table 6] The RM scores used to support the alignment-parity claim are produced by the same reward model that provides the PPO training signal for both RLHF and CD-RLHF (as stated in §4.1 and in the evaluation section). Parity on this metric does not establish that CD-RLHF preserves human-preferred quality, since the intrinsic reward upweights low-probability tokens that the training reward model may overvalue. Table 6 is the only independent quality check, and it covers a single model (Llama-3.2-3B) with results that are statistical ties on both datasets (27 win / 23 loss on TL;DR; 21 win / 22 loss / 7 tie on UltraFeedback). Please add per-model independent quality evaluation, for example with a held-out reward model or human preference judgments, and report the complete results rather than only the single-model table.
  3. [§3.2.2, Figure 5, Table 7] The top-k mask directly rewards non-greedy token choices, so the reported diversity gains could in principle arise from this anti-greedy upweighting rather than from the prediction-error novelty signal. No control is provided in which the same top-k mask is combined with a non-curiosity intrinsic reward (for example, a constant bonus, a uniform random bonus, or an entropy bonus restricted to non-top-k tokens). Without such a control, it is not established that the ICM's prediction error is the operative mechanism or that the "curiosity" component adds value over simply penalizing greedy choices. Please add such a baseline or clearly reinterpret the contribution as an anti-greedy exploration bonus.
  4. [§4.5, Table 2, Table 9] The MT-Bench results are reported as evidence of strong generalization, but Figure 7 shows that CD-RLHF's win rate against RLHF is only 22-32% with the majority of evaluations ending in ties; the text should be framed accordingly. More importantly, the case study in Table 9 contains a CD-RLHF response asserting that "a large-scale manufacturing plant in Tokyo can lead to a decrease in demand for steel products," which contradicts the background paragraph's stated causal relationship and the story. This example illustrates that the additional diversity can include factual errors that the training reward model may not penalize. The paper should discuss this risk explicitly and, ideally, add a factual-consistency or correctness metric to the alignment evaluation.
minor comments (4)
  1. [§4.1, A.3] The GPT-4 and human win-rate figures are based on 50 and 25 sampled instances per dataset, respectively, but no confidence intervals or exact counts are shown for Figure 3; please report the raw counts and intervals alongside the percentages.
  2. [§3.2.2] The reward whitening formula r(i) = (r(i) − µ)/σ^2 uses σ^2 without stating whether σ denotes standard deviation or variance; please define the notation precisely.
  3. [Appendix B.2] The human evaluation protocol does not state the number of annotators, their qualifications, or inter-annotator agreement; these details are needed to assess the reliability of the human diversity judgments.
  4. [§4.5, Figure 7] The phrase "win rates ... over RLHF" is misleading when CD-RLHF wins in only about a quarter to a third of comparisons and ties in the majority; please report the full win/tie/loss distribution and avoid the "win rate over" wording.

Circularity Check

1 steps flagged · score 2.0 of 10

Alignment RM scores are partially circular (same RM as training signal); diversity claim is independent.

  1. fitted input called prediction [Section 4.1, Evaluation (RM scores bullet); Table 1]
    "RM scores evaluate the alignment quality using the trained reward model, which is identical to the one used in PPO training."

    The headline claim of 'maintaining alignment with human preferences comparable to standard RLHF' is primarily supported by RM scores. By the paper's own statement, this RM is 'identical to the one used in PPO training,' so both CD-RLHF and RLHF are optimized against the same fitted reward model. Comparable RM scores therefore largely confirm that the intrinsic reward did not disrupt the training objective, rather than independently verifying human-preference quality. This is a fitted-input-called-prediction pattern: a model fitted to preferences is used as the alignment outcome measure. The circularity is partial because GPT-4 pairwise comparisons and MT-Bench provide some external evidence, and the diversity claim is unaffected.

full rationale

The central diversity claim is self-contained: CD-RLHF is trained with an intrinsic reward defined as ICM prediction error masked to non-top-k tokens, while diversity is measured by external n-gram, SelfBLEU, Sentence-BERT, GPT-4, and human judgments. None of these diversity metrics enters the training objective, so the reported diversity gains are not equivalent to the reward by construction. The ICM and top-k masking scheme are adapted from external prior work (Pathak et al. 2017; Burda et al. 2019b), not from self-citations, and no load-bearing self-citation or imported uniqueness theorem appears. The one partial circularity is the alignment evaluation: the headline RM scores come from the same reward model used as the PPO training signal, so 'comparable RM scores' largely restates that both policies were optimized against the same RM. This weakens the 'human preferences' portion of the abstract's claim, but the paper supplements with GPT-4 pairwise comparisons and MT-Bench, and the diversity half of the central claim remains independently evidenced. Overall circularity is minor and localized to one evaluation metric.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The ledger counts the new knobs the paper introduces on top of standard RLHF. The main free parameters are eta and k, both chosen by hand without a stated tuning procedure. The ICM itself is a new trainable component with no separate falsifiable prediction. The axioms are the standard RLHF proxy assumption plus specific assumptions about prediction error as novelty and about feature-space alignment, both of which the paper asserts rather than validates.

free parameters (3)
  • eta (intrinsic reward weight) = 0.04 for Gemma-2B/7B and Llama-3.2-1B UltraFeedback; 0.06 for Llama-3.2-1B TL;DR; 0.08 for Llama-3.2-3B
    Scales intrinsic rewards relative to extrinsic RM rewards; chosen per model and dataset by hand. The Limitations state the intrinsic reward scale is much larger than the extrinsic reward, so eta must be small.
  • k in top-k masking = 1 (top-1)
    Determines when intrinsic rewards are disabled. The ablation in Appendix C.2 shows k=3 and k=10 reduce diversity gains, so the choice affects the central result.
  • ICM hidden sizes = encoder hidden size = 2x actor hidden; forward model hidden size = actor intermediate size
    Architectural choice for the ICM with no sensitivity analysis.
assumptions (4)
  • domain assumption The reward model used in training is a faithful proxy for human preferences, so RM scores measure alignment quality.
    Used throughout Section 4.2 as the primary alignment metric; standard in RLHF but not validated against human judgments except a small GPT-4 check in Appendix C.1.
  • domain assumption Prediction error in the latent space of LLM hidden states is a valid measure of state novelty for language generation.
    Borrowed from Pathak et al. (2017) and Burda et al. (2019b), applied in Section 3.2.2 without independent validation for text.
  • ad hoc to paper Tokens outside the top-1 probability are the useful exploration targets; top-1 tokens are already known and need no intrinsic reward.
    Introduced in Section 3.2.2; the paper's own ablation in Section 4.4 shows the frequency of intrinsic rewards affects the diversity-quality balance.
  • ad hoc to paper Reference-model hidden states and actor-model token embeddings can be combined into one coherent feature space for prediction.
    Section 3.2.2 'Feature Space Alignment'; no analysis of representation mismatch is provided.
invented entities (1)
  • Intrinsic Curiosity Module (ICM)
    purpose: Predicts next hidden state from current state and action; prediction error is used as intrinsic reward.
    A trainable network introduced for this method; no falsifiable prediction outside the paper's own evaluation, and the paper notes its output scale is mismatched with extrinsic rewards.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Curiosity-Driven Reinforcement Learning from Human Feedback." pith.science (2026). https://pith.science/paper/5W7QFDKB

@misc{pith2026250111463,
  author       = {Pith},
  title        = {Pith review of: Curiosity-Driven Reinforcement Learning from Human Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5W7QFDKB}},
  note         = {Machine review of arXiv:2501.11463}
}
read the original abstract

Reinforcement learning from human feedback (RLHF) has proven effective in aligning large language models (LLMs) with human preferences, but often at the cost of reduced output diversity. This trade-off between diversity and alignment quality remains a significant challenge. Drawing inspiration from curiosity-driven exploration in reinforcement learning, we introduce curiosity-driven RLHF (CD-RLHF), a framework that incorporates intrinsic rewards for novel states, alongside traditional sparse extrinsic rewards, to optimize both output diversity and alignment quality. We demonstrate the effectiveness of CD-RLHF through extensive experiments on a range of tasks, including text summarization and instruction following. Our approach achieves significant gains in diversity on multiple diversity-oriented metrics while maintaining alignment with human preferences comparable to standard RLHF. We make our code publicly available at https://github.com/ernie-research/CD-RLHF.

Figures

Figures reproduced from arXiv: 2501.11463 by the authors.

Figure 1
Figure 1. Illustration of the CD-RLHF framework. The policy model generates a completion based on the given [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. The computation of ICM loss LICM and intrin￾sic reward r (i) t at time t, the state representation st and st+1 are derived from the reference model, while the action representation at is taken from the policy model. The intrinsic reward computation aligns with LICM, as the state prediction accuracy improves during ICM training, reducing the intrinsic reward over time. This follows the principle of curiosity￾driven R… view at source ↗
Figure 4
Figure 4. Output diversity and alignment performance [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Diversity and alignment quality changes with [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 7
Figure 7. Figure 7: Win rates of CD-RLHF against vanilla RLHF [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Avoidance Decoding for Diverse Multi-Branch Story Generation

    cs.CL 2025-09 conditional novelty 5.0 of 10

    Avoidance Decoding penalizes token choices that resemble previously generated story branches, using a hybrid concept-level and narrative-level similarity penalty, and reports large diversity gains across several LLMs.

  2. Quality-constrained Entropy Maximization Policy Optimization for LLM Diversity

    cs.CL 2026-02 conditional novelty 4.0 of 10

    Quality-constrained entropy maximization yields simple DPO-like objectives that increase LLM output diversity while preserving or slightly improving quality, with theoretical guarantees under tuned temperature conditions.

  3. Meta-Thinking in LLMs via Multi-Agent Reinforcement Learning: A Survey

    cs.AI 2025-04 conditional novelty 4.0 of 10

    The paper surveys existing work on LLM meta-thinking and argues that multi-agent reinforcement learning is a promising missing ingredient for building self-correcting language models.

Reference graph

Works this paper leans on

55 extracted references · 22 canonical work pages · cited by 3 Pith papers

  1. [1]

    Arash Ahmadian, Chris Cremer, Matthias Gall \'e , Marzieh Fadaee, Julia Kreutzer, Ahmet \"U st \"u n, and Sara Hooker. 2024. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. arXiv preprint arXiv:2402.14740

    arXiv 2024

  2. [2]

    Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan

    Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield - Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. 2021. https://arxiv.org/abs/21...

    arXiv 2021

  3. [3]

    Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Hatfield - Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson...

    arXiv doi:10.48550/arxiv.2204.05862 2022

  4. [4]

    Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. 2016. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29

    work page 2016

  5. [5]

    Herbie Bradley, Andrew Dai, Hannah Teufel, Jenny Zhang, Koen Oostermeijer, Marco Bellagente, Jeff Clune, Kenneth Stanley, Gr \'e gory Schott, and Joel Lehman. 2023. Quality-diversity through ai feedback. arXiv preprint arXiv:2310.13032

    arXiv 2023

  6. [6]

    Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A. Efros. 2019 a . https://openreview.net/forum?id=rJNwDjAqYX Large-scale study of curiosity-driven learning . In International Conference on Learning Representations

    work page 2019

  7. [7]

    Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. 2019 b . https://openreview.net/forum?id=H1lJJnR5Ym Exploration by random network distillation . In International Conference on Learning Representations

    work page 2019

  8. [8]

    Louis Castricato, Alexander Havrilla, Shahbuland Matiana, Michael Pieler, Anbang Ye, Ian Yang, Spencer Frazier, and Mark Riedl. 2022. Robust preference learning for storytelling via contrastive reinforcement learning. arXiv preprint arXiv:2210.07792

    work page Pith review arXiv 2022

Show all 55 references
  1. [9]

    Yekun Chai, Haoran Sun, Huang Fang, Shuohuan Wang, Yu Sun, and Hua Wu. 2024. Ma-rlhf: Reinforcement learning from human feedback with macro actions. arXiv preprint arXiv:2410.02743

    2024 arXiv

  2. [10]

    Yekun Chai, Shuohuan Wang, Chao Pang, Yu Sun, Hao Tian, and Hua Wu. 2023. https://doi.org/10.18653/v1/2023.findings-acl.676 ERNIE -code: Beyond E nglish-centric cross-lingual pretraining for programming languages . In Findings of the Association for Computational Linguistics: ...

    2023 doi

  3. [11]

    Alex James Chan, Hao Sun, Samuel Holt, and Mihaela van der Schaar. 2024. https://openreview.net/forum?id=eyxVRMrZ4m Dense reward for free in reinforcement learning from human feedback . In Forty-first International Conference on Machine Learning

    2024

  4. [12]

    Christiano, Jan Leike, Tom B

    Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. https://proceedings.neurips.cc/paper/2017/hash/d5e2c0adad503c91f91df240d0cd4e49-Abstract.html Deep reinforcement learning from human preferences . In Advances in Neural Information ...

    2017

  5. [13]

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377

    2023 arXiv

  6. [14]

    Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, KaShun SHUM, and Tong Zhang. 2023. https://openreview.net/forum?id=m7p5O7zblY RAFT : Reward ranked finetuning for generative foundation model alignment . Transactions on Machi...

    2023

  7. [15]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  8. [16]

    Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and R \'e mi Munos. 2023. A general theoretical paradigm to understand learning from human preferences. arXiv e-prints, pages arXiv--2310

    2023

  9. [17]

    Nick Haber, Damian Mrowca, Stephanie Wang, Li F Fei-Fei, and Daniel L Yamins. 2018. Learning to play with intrinsically-motivated, self-aware agents. Advances in neural information processing systems, 31

    2018

  10. [18]

    Glass, Akash Srivastava, and Pulkit Agrawal

    Zhang-Wei Hong, Idan Shenfeld, Tsun-Hsuan Wang, Yung-Sung Chuang, Aldo Pareja, James R. Glass, Akash Srivastava, and Pulkit Agrawal. 2024. https://openreview.net/forum?id=4KqkizXgXU Curiosity-driven red-teaming for large language models . In The Twelfth International Conferenc...

    2024

  11. [19]

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lo RA : Low-rank adaptation of large language models . In International Conference on Learning Representations

    2022

  12. [20]

    Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. 2024. https://openreview.net/forum?id=PXD3FAVHJT Understanding the effects of RLHF on LLM generalisation and diversity . In The Twelfth Internationa...

    2024

  13. [21]

    Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and William B Dolan. 2016. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Huma...

    2016

  14. [22]

    Lei Li, Yekun Chai, Shuohuan Wang, Yu Sun, Hao Tian, Ningyu Zhang, and Hua Wu. 2024. https://openreview.net/forum?id=d94x0gWTUX Tool-augmented reward modeling . In The Twelfth International Conference on Learning Representations

    2024

  15. [23]

    Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. 2023. Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models. In Forty-first International Conference on Machine Learning

    2023

  16. [24]

    Yiming Liang, Ge Zhang, Xingwei Qu, Tianyu Zheng, Jiawei Guo, Xinrun Du, Zhenzhu Yang, Jiaheng Liu, Chenghua Lin, Lei Ma, et al. 2024. I-sheep: Self-alignment of llm from scratch through an iterative self-enhancement paradigm. arXiv preprint arXiv:2408.08072

    2024 arXiv

  17. [25]

    Siyang Liu, Sahand Sabour, Yinhe Zheng, Pei Ke, Xiaoyan Zhu, and Minlie Huang. 2022. Rethinking and refining the distinct metric. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 762--770

    2022

  18. [26]

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. 2024. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173

    2024 arXiv

  19. [27]

    Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American C...

    2016

  20. [28]

    Bellemare, A \"a ron van den Oord, and R \'e mi Munos

    Georg Ostrovski, Marc G. Bellemare, A \"a ron van den Oord, and R \'e mi Munos. 2017. https://proceedings.mlr.press/v70/ostrovski17a.html Count-based exploration with neural density models . In Proceedings of the 34th International Conference on Machine Learning, volume 70 of ...

    2017

  21. [29]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leik...

    2022

  22. [30]

    Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. 2017. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pages 2778--2787. PMLR

    2017

  23. [31]

    Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022. https://doi.org/10.18653/v1/2022.emnlp-main.225 Red teaming language models with language models . In Proceedings of the 2022 Conference on...

    2022 doi

  24. [32]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36

    2024

  25. [33]

    Nils Reimers and Iryna Gurevych. 2019. https://doi.org/10.18653/v1/D19-1410 Sentence- BERT : Sentence embeddings using S iamese BERT -networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference...

    2019 doi

  26. [34]

    Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J \'e r \'e my Rapin, et al. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950

    2023 arXiv

  27. [35]

    J Schmidhuber. 1991 a . A possibility for implementing curiosity and boredom in model-building neural controllers. In Proc. of the International Conference on Simulation of Adaptive Behavior: From Animals to Animats, pages 222--227

    1991

  28. [36]

    J \"u rgen Schmidhuber. 1991 b . Curious model-building control systems. In Proc. international joint conference on neural networks, pages 1458--1463

    1991

  29. [37]

    Bradly C Stadie, Sergey Levine, and Pieter Abbeel. 2015. Incentivizing exploration in reinforcement learning with deep predictive models. arXiv preprint arXiv:1507.00814

    2015 arXiv

  30. [38]

    Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F

    Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. 2020. https://arxiv.org/abs/2009.01325 Learning to summarize from human feedback . CoRR, abs/2009.01325

    2020 arXiv

  31. [39]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295

    2024 arXiv

  32. [40]

    Guy Tevet and Jonathan Berant. 2021. https://doi.org/10.18653/v1/2021.eacl-main.25 Evaluating the evaluation of diversity in natural language generation . In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volum...

    2021 doi

  33. [41]

    Jonathan Uesato, Nate Kushman, Ramana Kumar, Francis Song, Noah Siegel, Lisa Wang, Antonia Creswell, Geoffrey Irving, and Irina Higgins. 2022. Solving math word problems with process-and outcome-based feedback. arXiv preprint arXiv:2211.14275

    2022 arXiv

  34. [42]

    Michael V \"o lske, Martin Potthast, Shahbaz Syed, and Benno Stein. 2017. Tl; dr: Mining reddit to learn automatic summarization. In Proceedings of the Workshop on New Frontiers in Summarization, pages 59--63

    2017

  35. [43]

    Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. 2024. https://openreview.net/forum?id=2cRzmWXK9N Beyond reverse KL : Generalizing direct preference optimization with diverse divergence constraints . In The Twelfth International Conference on Learning Representations

    2024

  36. [44]

    Fan Wu, Emily Black, and Varun Chandrasekaran. 2024. Generative monoculture in large language models. arXiv preprint arXiv:2407.02209

    2024 arXiv

  37. [45]

    Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. 2023. Fine-grained human feedback gives better rewards for language model training. Advances in Neural Information Processing Systems, 36:590...

    2023

  38. [46]

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244

    2023 arXiv

  39. [47]

    Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024. https://openreview.net/forum?id=51iwkioZpn Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation . In Fort...

    2024

  40. [48]

    Shuai Yang, Yuying Ge, Yang Li, Yukang Chen, Yixiao Ge, Ying Shan, and Yingcong Chen. 2024. Seed-story: Multimodal long story generation with large language model. arXiv preprint arXiv:2407.08683

    2024 arXiv

  41. [49]

    Zhewei Yao, Reza Yazdani Aminabadi, Olatunji Ruwase, Samyam Rajbhandari, Xiaoxia Wu, Ammar Ahmad Awan, Jeff Rasley, Minjia Zhang, Conglong Li, Connor Holmes, et al. 2023. Deepspeed-chat: Easy, fast and affordable rlhf training of chatgpt-like models at all scales. arXiv prepri...

    2023 arXiv

  42. [50]

    Eunseop Yoon, Hee Suk Yoon, SooHwan Eom, Gunsoo Han, Daniel Nam, Daejin Jo, Kyoung-Woon On, Mark Hasegawa-Johnson, Sungwoong Kim, and Chang Yoo. 2024. https://doi.org/10.18653/v1/2024.findings-acl.889 TLCR : Token-level continuous reward for fine-grained reinforcement learning...

    2024 doi

  43. [51]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36

    2024

  44. [52]

    Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. 2018. Texygen: A benchmarking platform for text generation models. SIGIR

    2018

  45. [53]

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593

    2019 arXiv

  46. [54]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  47. [55]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.