Pith. sign in

REVIEW 3 major objections 6 minor 16 references

SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read On-policy and off-policy preference data help different tasks, and SIMPLEMIX—an equal 50/50 mixture of the two—consistently outperforms using either source alone in DPO-based alignment.

desk verdict The mixing recipe works and is well-controlled, but the 'on-policy' label is misleading—this is really SFT-sample plus offline data—and the abstract's numbers need fixing. read the letter →

arxiv 2505.02363 v1 pith:BFOX3J77 submitted 2025-05-05 cs.CL

Tianjian Li , Daniel Khashabi This is my paper · ORCID
classification cs.CL
keywords preferenceoptimizationdirecton-policydataoff-policymixinglanguagemodelalignmenttask-dependentevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper attempts to settle a live dispute in language-model alignment: is preference data sampled from the model being trained (on-policy) strictly better than data collected from other models (off-policy), or is the advantage task-dependent? By varying only the data source while keeping the DPO algorithm fixed, the authors find that on-policy data brings most of its gains on objectively verifiable tasks such as math and coding, while off-policy data does better on open-ended tasks such as creative writing and personal recommendations. From this they propose SIMPLEMIX, which samples preference pairs with equal probability from the supervised-fine-tuned starting model $\pi_{\mathrm{SFT}}$ and from an off-policy dataset, then trains with the standard DPO loss. Across two base models and two preference datasets, the 50/50 mixture beats pure on-policy and pure off-policy DPO by an average of 6.03% on Alpaca Eval 2.0 and beats the more complex hybrid methods HyPO and DPO-Mix-P by an average of 3.05%.

What carries the argument

The mechanism that carries the argument is SIMPLEMIX's data distribution: for each example, the winning and losing responses are drawn with equal probability either from on-policy generations of the supervised-fine-tuned model $\pi_{\mathrm{SFT}}$ or from the off-policy dataset $D_{\mathrm{off}}$, and the pair is trained with the DPO loss. This makes SIMPLEMIX a 'hard' version of the DPO-Mix-P sampler, which draws from an interpolation between the current policy and the SFT policy. The other load-bearing piece is the paper's task-type hypothesis: on-policy data helps verifiable reasoning tasks while off-policy data helps open-ended tasks, and the paper shows in §3 that this difference is not explained by response length.

What would settle it

Run SIMPLEMIX with on-policy data sampled from the current policy $\pi_\theta$ instead of the frozen $\pi_{\mathrm{SFT}}$, keeping all other settings identical; if the 50/50 mixture then no longer beats both pure on- and pure off-policy DPO on math and coding, or if open-ended tasks stop favoring off-policy data, the paper's complementarity claim as stated would be falsified.

Watch

Extended reading notes

Core claim

The central claim is that on-policy and off-policy preference data are complementary in direct preference optimization (DPO), the standard algorithm that increases the model's log-probability of chosen responses relative to rejected ones. In controlled experiments that change only the data source, on-policy data—responses sampled from the frozen supervised-fine-tuned model and labeled by a reward model—improves performance most on tasks with objective answers, such as mathematical reasoning and coding, while off-policy data from existing collections like UltraFeedback and HelpSteer2 improves performance most on open-ended tasks like creative writing and recommendations. Because the two sources help different task types, the authors propose SIMPLEMIX: mix the two data sources with equal probability inside the same DPO objective, so the training distribution is $D = D_{\mathrm{on}} \cup D_{\mathrm{off}}$. This simple mixture outperforms both pure sources and the more complex hybrid methods HyPO and DPO-Mix-P, on both LM-as-a-judge evaluation (Alpaca Eval 2.0 length-controlled win rate) and reference-based benchmarks.

Load-bearing premise

The claim rests on defining on-policy data as responses sampled once from the frozen supervised-fine-tuned starting model rather than from the model being updated during training; if on-policy is understood as sampling from the current evolving policy, the reported complementarity and the margins over HyPO and DPO-Mix-P may not carry over.

Editorial extensions

If this is right

  • A balanced 50/50 mix of on- and off-policy preference data is a strong default for DPO-style alignment, beating pure on-policy, pure off-policy, and the more complex hybrid samplers tested across two base models and two datasets.
  • Data-origin decisions should be task-dependent: on-policy sampling earns its extra compute on verifiable reasoning tasks, while abundant off-policy data suffices or wins on open-ended tasks.
  • Off-policy data is not just a diversity booster; in the right mixture it improves the final model, and filtering it by reward-model quality pushes the improvement further, for example from 29.41 to 31.62 length-controlled win rate on Llama-3.1-8B-Instruct.
  • The on-policy advantage on math and coding is not a length artifact, since the gain persists across generation-length bins, so practitioners can attribute it to task type rather than output length.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the field's notion of on-policy were tied to the evolving policy, the 50/50 optimum might shift; a direct comparison of frozen-SFT sampling versus live-policy sampling would show whether SIMPLEMIX's simplicity carries over to true online DPO.
  • The task-dependent split suggests a finer-grained rule than a global 50/50 mix: tune the on/off sampling ratio per prompt category, which could improve on the flat mixture reported here.
  • The same mixing idea could be tested on other preference objectives such as IPO or KTO, since the mechanism is about where the preference pairs come from, not the specific loss.
  • Because off-policy quality filtering helps, a natural extension is to filter on-policy pairs with a held-out reward model as well, which would make the reported margin over hybrid baselines less sensitive to judge alignment.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies the interplay between on-policy and off-policy preference data in language model alignment. It defines on-policy data as responses sampled from either the SFT model or the current policy, and off-policy data as external preference datasets. Through DPO experiments on Llama-3.1-8B-Instruct and Llama-3.1-Tulu-3-8B-SFT with UltraFeedback and HelpSteer2, the paper reports that on-policy data is more effective for math and coding while off-policy data is more effective for open-ended tasks such as creative writing and recommendations. It then proposes SIMPLEMIX, which samples preference pairs equally from the SFT model and the off-policy dataset, and reports that this simple mixture outperforms single-source DPO and more complex hybrid methods (HyPO, DPO-Mix-P) on Alpaca Eval 2.0 and an eight-benchmark average. The paper also investigates diversity, mixing ratios, and off-policy data filtering.

Significance. If the claims hold, SIMPLEMIX provides a simple and effective default for preference optimization, and the task-dependent complementarity result is a practically useful guideline for data curation. The paper's strengths include controlled comparisons that vary only the data source while fixing the algorithm, the use of two SFT base models and two preference datasets, evaluation with both LM-as-a-judge and reference-based benchmarks, and a length-control analysis that rules out a trivial explanation of the reasoning-task gains. The central empirical finding—that a 50/50 mixture of SFT-generated and externally sourced preference pairs improves alignment—is consistent across the four experimental settings reported in Table 2. However, the headline 'on-policy' terminology rests on a nonstandard definition, and the task-complementarity analysis is based on very small per-category prompt subsets, so the scope of the conclusions is narrower than the abstract suggests.

major comments (3)
  1. [Section 2 (Definition of on-policy) and Sections 3–5] The paper defines on-policy data as responses sampled from either pi_SFT or pi_theta and acknowledges in footnote 1 that this departs from the standard RL convention where on-policy means samples from the current policy pi_theta. Throughout the experiments, the 'on-policy' component is always sampled from the frozen SFT model before training, so both data sources are actually off-policy with respect to the final model. Because the central complementarity claim and the comparison against HyPO and DPO-Mix-P (both of which use current-policy sampling) are framed in terms of 'on-policy' data, the findings may not transfer to the standard online setting. The paper should either rename this data source (e.g., 'SFT-generated data') throughout the abstract, main text, and conclusions, or add a condition that samples from the evolving policy to show that the results hold under the standard notion.
  2. [Section 3.2, Figure 2, and Appendix B] The per-category win rates underlying the complementarity claim are computed on very small prompt subsets: Appendix B shows that math and coding constitute only 4.35% and 7.08% of the 805 Alpaca Eval 2.0 prompts, corresponding to roughly 35 and 57 prompts, respectively. The paper reports differences of +5.72, +7.02, -2.85, and -3.29 percentage points for these categories, but Figure 2 shows no confidence intervals or significance tests. These differences may be within sampling noise. The authors should report bootstrap confidence intervals or significance tests for each per-category comparison, as they already do in Figure 3 for the length-stratified analysis.
  3. [Sections 5.2–5.3, Figures 4–6, and Table 2] The 0.5 mixing ratio and p=0.4 filtering threshold are selected by maximizing the length-controlled win rate on Alpaca Eval 2.0, which is the same benchmark used to report the headline improvement in Table 2. This selection on the evaluation set can inflate the reported gains and makes the '6.03% average improvement' claim difficult to interpret. The authors should disclose this selection procedure explicitly in the main text and provide evidence that the 0.5 ratio is not an artifact of tuning to this single benchmark, for example by evaluating on a held-out set of preference prompts or by reporting results for a fixed, pre-specified ratio.
minor comments (6)
  1. [Abstract] The text reads '6.03 over' where a percent sign appears to be missing; it should be '6.03%.' The same issue occurs in the introduction.
  2. [Section 3.1] The paper treats meta-llama/Llama-3.1-8B-Instruct as pi_SFT, but this model is an instruction-tuned chat model, not a pure SFT checkpoint. This should be justified, since the DPO reference model is the starting model and the 'on-policy' samples are drawn from this same model.
  3. [Section 5.1, Table 3] The column 'Avg Reward' in Table 3 is not defined in the text or caption; the reader cannot tell whether it is the average reward of the generated training responses or of the final model. Please clarify in the caption.
  4. [Figure 4 caption] The caption contains the typo 'Perfomance' instead of 'Performance.'
  5. [Appendix D and Tables 4–5] The benchmark name 'Winogrande' is misspelled as 'Winograde' in multiple places (Appendix D and the table captions).
  6. [Section 2, Set notation] In the definition of on-policy data, the notation D_on = {(x,y) | y ~ pi_theta or pi_SFT} would be clearer if it indicated which variable is the model and which is the policy; consider writing 'pi_theta' and 'pi_SFT' with explicit subscripts on the sampling distribution.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SIMPLEMIX's reported gains are empirical comparisons against external benchmarks, not derivations from its own inputs.

full rationale

The paper's central claims are empirical rather than derived. The complementarity finding in Section 3 is an experimental comparison of on-policy DPO and off-policy DPO on Alpaca Eval 2.0 prompt categories, with win rates measured against GPT-4-turbo. The SIMPLEMIX result in Section 4 is an empirical comparison of a 50/50 mixture of SFT-sampled and off-policy preference pairs against DPO, HyPO, and DPO-Mix-P on Alpaca Eval 2.0 and eight reference-based benchmarks. Nothing in the method's definition forces these outcomes: mixing two data sources could plausibly hurt, and the measured improvements are contingent benchmark results. No equation in the paper reduces to an earlier equation by construction, and no fitted parameter is renamed as a prediction in the headline result. The equal mixing ratio is stated as the method in Section 4.1 before the mixture-ratio ablation in Section 5.2, and the main Table 2 results use unfiltered data. The nonstandard definition of on-policy data as samples from either pi_SFT or pi_theta is explicitly disclosed in footnote 1; this is a scope or validity concern about whether the findings transfer to standard current-policy sampling, not a circular derivation from the paper's own outputs. The p=0.4 filtering threshold and the 0.5 mixing ratio are selected through ablations reported on Alpaca Eval 2.0, which is a benchmark-selection risk, but the paper does not present these configurations as independent predictions. Citations to the authors' prior work are not load-bearing: Lu et al. (2024) is used only as a prompting technique in the diversity ablation, and the paper's key comparisons rely on external baselines and benchmarks. Overall, the derivation chain is self-contained in the sense that the findings are measured outcomes, and no circularity step can be exhibited from the text.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim rests mainly on empirical choices: the definition of on-policy as SFT-policy sampling, the reward-model proxy, and evaluation via Alpaca Eval. The free parameters are mostly standard hyperparameters plus the 0.5 ratio and p=0.4 threshold that are tuned on the evaluation benchmark. No new theoretical entities are introduced.

free parameters (5)
  • Mixing ratio (on-policy fraction) = 0.5
    Selected in Section 5.2 (Figure 4) as the best ratio on Alpaca Eval 2.0, then used in all main SIMPLEMIX experiments; this is fit to the evaluation benchmark.
  • Off-policy filtering threshold p = 0.4
    Selected in Section 5.3 (Figures 5 and 6) to maximize Alpaca Eval LC for the quality heuristic; used in the 'On + Off Filtered' result.
  • On-policy sample count N = 4 (stated in Section 5.1; not restated in Section 4.1)
    Number of on-policy responses sampled per prompt; affects data quality and diversity. Not specified for the main HelpSteer2 experiments.
  • Sampling temperature and top-p = temperature 0.7, top-p 0.9
    Used for on-policy sampling; standard but chosen by hand, not derived.
  • DPO hyperparameters beta and peak learning rate = beta=0.1, learning rate 5e-7
    Chosen after an initial sweep (Appendix A) and fixed across all conditions; not specific to the central claim but part of the recipe.
assumptions (6)
  • domain assumption The Skywork reward model is a valid proxy for human preferences in labeling chosen and rejected responses and filtering data.
    Used throughout Sections 3, 4, and 5.3 to annotate on-policy pairs and select high-quality off-policy data. If this reward model is biased, the preference labels are systematically wrong.
  • domain assumption Alpaca Eval 2.0 length-controlled win rate is a valid measure of alignment.
    Primary evaluation in the paper; the authors themselves note in Section 7 that LLM-as-a-judge benchmarks can be hacked and are sensitive to formatting.
  • domain assumption The WildChat category labels applied to Alpaca Eval 2.0 prompts identify the four representative task types (math, coding, creative writing, recommendation).
    Used in Section 3.1 and Appendix B to test the task-complementarity hypothesis; classification by gpt-4o-mini may contain errors, and the math subset has about 35 prompts.
  • domain assumption Sampling from the frozen SFT policy qualifies as on-policy data.
    Definition in Section 2 and footnote 1; departs from standard RL usage where on-policy means sampling from the current policy. This assumption underpins the on/off-policy distinction and the comparison with HyPO and DPO-Mix-P.
  • standard math The Bradley-Terry preference model underlying DPO (Equation 1) is valid for these preference datasets.
    DPO's loss is derived from a Bradley-Terry reward model in Rafailov et al.; the paper does not re-derive it but relies on it for the loss used throughout.
  • domain assumption UltraFeedback and HelpSteer2 are representative off-policy preference datasets.
    Used for all main results; both datasets were collected from pools of models, and their quality and coverage affect the conclusions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning." pith.science (2026). https://pith.science/paper/BFOX3J77

@misc{pith2026250502363,
  author       = {Pith},
  title        = {Pith review of: SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BFOX3J77}},
  note         = {Machine review of arXiv:2505.02363}
}
read the original abstract

Aligning language models with human preferences relies on pairwise preference datasets. While some studies suggest that on-policy data consistently outperforms off -policy data for preference learning, others indicate that the advantages of on-policy data may be task-dependent, highlighting the need for a systematic exploration of their interplay. In this work, we show that on-policy and off-policy data offer complementary strengths in preference optimization: on-policy data is particularly effective for reasoning tasks like math and coding, while off-policy data performs better on open-ended tasks such as creative writing and making personal recommendations. Guided by these findings, we introduce SIMPLEMIX, an approach to combine the complementary strengths of on-policy and off-policy preference learning by simply mixing these two data sources. Our empirical results across diverse tasks and benchmarks demonstrate that SIMPLEMIX substantially improves language model alignment. Specifically, SIMPLEMIX improves upon on-policy DPO and off-policy DPO by an average of 6.03% on Alpaca Eval 2.0. Moreover, it outperforms prior approaches that are much more complex in combining on- and off-policy data, such as HyPO and DPO-Mix-P, by an average of 3.05%.

Figures

Figures reproduced from arXiv: 2505.02363 by the authors.

Figure 1
Figure 1. Left panel: Our work studies data origin in preference optimization of LMs. Middle panel: We show that on-policy data and off-policy data are complementary: on-policy data mostly improves the model’s performance on reasoning tasks that are objectively correct or incorrect (e.g., Math and Coding) while off-policy data improves on sub-tasks where humans might disagree with each other (e.g., creative writing and person… view at source ↗
Figure 2
Figure 2. Comparison of win rates (against GPT-4-turbo) across different prompt categories in Alpaca Eval 2.0 for (left) objective tasks that have a groundtruth answer and (right) open-ended tasks where humans have individual preferences. On-policy DPO improves performance in math and coding, while off-policy DPO demonstrates better performance in creative writing and making personal recommendations. 0-600 600-1200 1200-1800 … view at source ↗
Figure 3
Figure 3. Comparison of win rates (against GPT-4-turbo) by the length of generation. On-policy training does not significantly outperform off-policy training as generation length increases in both math and coding tasks (left) and creative writing as well as recommendation tasks (right). Error bars show 95% confidence intervals from bootstrapping. tions benefit more from on-policy data and that the improve￾ment in math and cod… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Perfomance on Alpaca Eval 2.0 for different on- to off-policy data ratio for performing DPO on top of πSFT = Meta-LLama-3.1-8B-Instruct (left) and Llama-3.1-Tulu-8B-SFT (right) on the Ultrafeedback (Cui et al., 2024) dataset. A balanced mixture (0.5 on-policy + 0.5 off…
Figure 6
Figure 6. Figure 6: Alpaca Eval 2.0 Length Controlled Win Rate for πSFT, Mixing On + Off-policy Data and Mixing in high-quality off￾policy data with on-policy data in DPO. Selecting high-quality off-policy data can improve performance using SIMPLEMIX. policy preference optimization is min…
Figure 7
Figure 7. Figure 7: Distribution of different categories of prompts in Alpaca Eval 2.0 (Dubois et al., 2023; 2024). C. Dataset Details Here we describe the datasets we used in §4: Ultrafeedback (Cui et al., 2024) and HelpSteer2 (Wang et al., 2024) in detail: • Ultrafeedback (Cui et al., 2…
Figure 8
Figure 8. Figure 8: Comparison of win rates (against GPT-4-turbo) across different prompt categories in Alpaca Eval 2.0 by training Llama-3.1-8B-Instruct and Tulu-3-8B-SFT with SIMPLEMIX. SIMPLEMIX offers a good balance between reasoning tasks and open-ended tasks. H. Diverse Prompt [PIT…
Figure 9
Figure 9. Figure 9: The prompt used to elicit diverse responses from a language model. I. Full Reward Distributions [PITH_FULL_IMAGE:figures/full_fig_p019_9.png]
Figure 10
Figure 10. Figure 10: Histogram of reward distribution for different sampling temperatures. Increasing the temperature results in generations of lower quality, as measured by our reward model. 20 15 10 5 0 5 10 Reward Value 0 1000 2000 3000 4000 Frequency temp = 2.0 Average: -10.56 20 15 1…
Figure 11
Figure 11. Figure 11: Histograms of the reward for sampling from Llama-3.1-8B-Instruct with different sampling schemes (prompting & and increasing the temperature). 20 [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 8 canonical work pages

  1. [5]

    Cited on pages 5 and 16

    URL https://openreview.net/forum? id=4hturzLcKX. Cited on pages 5 and 16. Dubois, Y ., Liang, P., and Hashimoto, T. Length-controlled alpacaeval: A simple debiasing of automatic evalu- ators. In First Conference on Language Modeling ,

    work page

  2. [6]

    Cited on pages 3, 5, and 16

    URL https://openreview.net/forum? id=CybBmzWBX0. Cited on pages 3, 5, and 16. Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Model alignment as prospect theoretic optimiza- tion. In International Conference on Machine Learning (ICML), 2024. Cited on page 18. Gao, L., Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C...

    arXiv 2024

  3. [8]

    A Systematic Examination of Preference Learning through the Lens of Instruction-Following

    URL https://api.semanticscholar. org/CorpusID:263830494. Cited on page 17. Kim, J., Goyal, A., Zhang, A., Xiong, B., Hou, R., Kam- badur, M., Mahajan, D., Hajishirzi, H., and Tan, L. A systematic examination of preference learning through the lens of instruction-following, 2024. URL https: //arxiv.org/abs/2412.15282. Cited on page 7. Knight, K., Badarau, ...

    work page Pith review arXiv 2024

  4. [9]

    Cited on page 8

    URL https://openreview.net/forum? id=4NJBV6Wp0h. Cited on page 8. Park, J., Jwa, S., Meiying, R., Kim, D., and Choi, S. OffsetBias: Leveraging debiased data for tuning eval- uators. In Al-Onaizan, Y ., Bansal, M., and Chen, Y .- N. (eds.), Findings of the Association for Computa- tional Linguistics: EMNLP 2024 , pp. 1043–1067, Mi- ami, Florida, USA, Novem...

    arXiv 2024

  5. [10]

    Cited on page 7

    URL https://openreview.net/forum? id=G8LaO1P0xv. Cited on page 7. Song, Y ., Zhou, Y ., Sekhari, A., Bagnell, D., Krishna- murthy, A., and Sun, W. Hybrid RL: Using both of- fline and online data can make RL efficient. In Interna- tional Conference on Learning Representations (ICLR),

    work page

  6. [11]

    Cited on page 7

    URL https://openreview.net/forum? id=yyBis80iUuU. Cited on page 7. Song, Y ., Bagnell, D., and Singh, A. Hybrid re- inforcement learning from offline observation alone. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), International Conference on Machine Learn- ing (ICML), volume 235 of Proceedi...

    arXiv

  7. [12]

    cc/paper_files/paper/2020/file/ 1f89885d556929e98d3ef9b86448f951-Paper

    URL https://proceedings.neurips. cc/paper_files/paper/2020/file/ 1f89885d556929e98d3ef9b86448f951-Paper. pdf. Cited on page 17. Tajwar, F., Singh, A., Sharma, A., Rafailov, R., Schnei- der, J., Xie, T., Ermon, S., Finn, C., and Ku- mar, A. Preference fine-tuning of LLMs should leverage suboptimal, on-policy data. In Forty- first International Conference o...

    work page 2020

  8. [13]

    Cited on pages 1 and 7

    URL https://openreview.net/forum? id=bWNPx6t0sF. Cited on pages 1 and 7. Talmor, A., Herzig, J., Lourie, N., and Berant, J. Com- monsenseQA: A question answering challenge tar- geting commonsense knowledge. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pp. 4149–4158,

    work page

Show all 16 references
  1. [15]

    Cited on page 7

    URL https://openreview.net/forum? id=bPuYxFBHyI. Cited on page 7. Tang, Y ., Guo, D. Z., Zheng, Z., Calandriello, D., Cao, Y ., Tarassov, E., Munos, R.,´Avila Pires, B., Valko, M., Cheng, Y ., and Dabney, W. Understanding the per- formance gap between online and offline alignm...

    2024 arXiv

  2. [16]

    alignment

    Cited on page 7. 14 SIMPLE MIX: Frustratingly Simple Off- and On-policy Data Mixing in Language Model Preference Learning Wang, Z., Dong, Y ., Delalleau, O., Zeng, J., Shen, G., Egert, D., Zhang, J. J., Sreedhar, M. N., and Kuchaiev, O. Help- steer2: Open-source dataset for tr...

    2024 arXiv

  3. [309]

    acl-long.309/

    URL https://aclanthology.org/2023. acl-long.309/. Cited on page 16. Ivison, H., Wang, Y ., Liu, J., Wu, Z., Pyatkin, V ., Lambert, N., Smith, N. A., Choi, Y ., and Hajishirzi, H. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback. In Adva...

    2023 arXiv

  4. [662]

    acl-long.662

    URL https://aclanthology.org/2024. acl-long.662. Cited on page 1. Almazrouei, E., Alobeidli, H., Alshamsi, A., Cappelli, A., Cojocaru, R., Debbah, M., Goffinet, E., Heslow, D., Lau- nay, J., Malartic, Q., Noune, B., Pannier, B., and Penedo, G. Falcon-40B: an open large languag...

    2024

  5. [2019]

    URL https: //aclanthology.org/N19-1421

    doi: 10.18653/v1/N19-1421. URL https: //aclanthology.org/N19-1421. Cited on pages 5 and 17. Tan, K., Fan, W., and Wei, Y . Hybrid reinforcement learning breaks sample size barriers in linear MDPs. In Advances in Neural Information Processing Systems (NeurIPS),

    doi

  6. [2020]

    Bose, A., Xiong, Z., Saha, A., Du, S

    Cited on pages 5 and 17. Bose, A., Xiong, Z., Saha, A., Du, S. S., and Fazel, M. Hybrid preference optimization for alignment: Provably faster convergence rates by combining offline preferences with online exploration, 2024. URL https://arxiv. org/abs/2412.10616. Cited on page...

    2024 arXiv

  7. [2023]

    Cited on page 7

    URL https://proceedings.mlr.press/ v202/ball23a.html. Cited on page 7. Biderman, S., Schoelkopf, H., Anthony, Q., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., and Van Der Wal, O. Pythia: a suite for an...

    2023

  8. [2024]

    Cited on page 17

    URL https://proceedings.mlr.press/ v238/gheshlaghi-azar24a.html. Cited on page 17. Bai, Y ., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with reinforcement learni...

    arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.