Pith. sign in

REVIEW 4 major objections 6 minor 35 references

HEAL: A Hypothesis-Based Preference-Aware Analysis Framework

T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that DPO, SimPO, and ORPO learn the proxy reward model's ranking only partially—suppressing negative responses works, but fine-grained preference strength is missed, and only SimPO keeps the ranking on out-of-distribution p

desk verdict Useful diagnostic framework, but the 'effectively capture preferences' claim collapses once the gold standard is the same proxy that wrote the training labels. read the letter →

arxiv 2508.19922 v1 pith:DJQD52LC submitted 2025-08-27 cs.CL

Yifu Huo , Chenglong Wang , Qiren Zhu , Shunjie Xing , Tong Xiao , Chunliang Zhang , Tongran Liu , Jinbo Zhu This is my paper
classification cs.CL
keywords preferenceoptimizationLLMalignmenthypothesisspacerankingaccuracystrengthcorrelationproxyrewardmodelDPOSimPO
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the usual way of evaluating preference-aligned LLMs—sampling one response and comparing it with a reference—misses how the model behaves across the full set of responses it could generate. It introduces HEAL, which treats alignment as a re-ranking problem: for each prompt, many candidate responses are ordered by the policy's generation likelihoods and by a gold-standard proxy reward model, and the two orderings are compared with ranking accuracy (Tau-b) and preference strength correlation (Pearson). Applying HEAL to DPO, SimPO, and ORPO, the paper concludes that these methods do capture proxy preferences, mostly by downweighting disliked responses, but the capture is incomplete: ranking accuracy stays modest, preference strength correlation is weak, learned preferences are specific to the proxy, and only SimPO generalizes to out-of-distribution data. The contribution is mainly diagnostic: a sampling-free, multi-candidate evaluation lens plus UniHypoBench, a benchmark with many candidate responses per prompt, for locating exactly where alignment methods fail.

What carries the argument

The ranked hypothesis space: for each prompt, the set of plausible responses is ordered by an indicator function, with generation likelihood for the policy and reward score for the gold-standard proxy. HEAL measures alignment as the agreement between these two orderings using ranking accuracy, a rescaling of Tau-b, and preference strength correlation, an expectation-based Pearson correlation between the two indicator values. UniHypoBench supplies the multi-candidate hypothesis spaces, with 2,985 prompts each containing at least eight responses generated by different LLMs, so the two metrics can be computed without sampling the aligned model.

What would settle it

Score every UniHypoBench prompt's candidate responses with held-out human annotations (or a second reward model never used to build training pairs) and compute HEAL's ranking accuracy for DPO, SimPO, and ORPO against that gold standard. If ranking accuracy stays near the 50 percent chance level while ArmoRM-based accuracy is high, the paper's central claim would hold only for one proxy, not for preference capture generally; if SimPO's out-of-distribution advantage disappears under a human gold standard, the generalization result is an artifact of proxy similarity rather than true preference le

Watch

Extended reading notes

Core claim

The paper's central claim is that current preference optimization methods effectively capture the preferences encoded by a proxy reward model while simultaneously suppressing negative samples. Using HEAL, the paper shows this capture is real but partial: DPO, SimPO, and ORPO raise ranking accuracy and preference strength correlation relative to the base model on in-distribution data, but ranking accuracy rarely passes 67%, preference strength correlation usually stays below 0.3, and performance drops markedly when the gold-standard proxy is changed. The exception is SimPO, which shows substantial generalization to out-of-distribution conditions. The paper interprets this pattern as evidence

Load-bearing premise

The load-bearing premise is that the proxy reward model's scores are a trustworthy gold standard for alignment quality; the same proxy annotated the training pairs, and the paper's own comparison shows much weaker agreement with human annotations, so the central conclusion is about capturing that proxy's preferences rather than human preferences.

Editorial extensions

If this is right

  • If HEAL's measurements are right, pairwise win-rate and RewardBench-style accuracy overstate alignment quality, because models can pass single-pair checks while failing on multi-candidate rankings and strength calibration.
  • SimPO's out-of-distribution robustness, if it holds, makes length-normalized reference-free objectives the most promising current direction for alignment that survives distribution shift.
  • Proxy-specific preference signatures imply that a single alignment score is not meaningful without specifying which preference distribution was used for training and evaluation.
  • The finding that methods suppress negatives without reaching a bimodal separation suggests future preference losses should target discriminative calibration, not just pairwise margins.
  • HEAL's deterministic, sampling-free procedure can serve as a low-cost complement or alternative to LLM-as-a-Judge for routine alignment checks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: Because the same ArmoRM proxy scored the training pairs and defines the gold standard, HEAL's 'capture' numbers are at least partly a measure of fit to the training signal; re-running HEAL against independently collected human rankings on UniHypoBench would separate proxy-fitting from genuine preference learning.
  • Editorial inference: The weak preference-strength correlations point to a concrete failure mode in applications that threshold on likelihood or reward—such as best-of-n reranking—where ordinal correctness on coarse pairs can coexist with badly miscalibrated scores.
  • Editorial inference: UniHypoBench's many-candidate structure could be reused as a training objective, not just an evaluation set; directly maximizing ranking accuracy or strength correlation may yield alignment methods that overcome the partial-capture limit the paper documents.
  • Editorial inference: If personalized alignment is the goal, the framework's Appendix A.3 assumption that annotators or proxy models 'accurately reflect the target preferences' is the hard part, since HEAL measures alignment only relative to whatever preference distribution the reference encodes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes HEAL, an evaluation framework that recasts preference alignment as re-ranking in a hypothesis space and introduces two metrics: ranking accuracy (RA, based on Kendall's tau-b) and preference strength correlation (PSC, based on Pearson correlation). To support the framework, the authors construct UniHypoBench, a multi-response benchmark. They evaluate DPO, SimPO, and ORPO on several LLaMA models, using ArmoRM as both the training-label annotation source and the primary gold-standard evaluator, GRM as a second proxy, and HelpSteer2 human annotations as an additional gold standard. The paper's central claim is that current preference learning methods 'effectively capture preferences provided by proxy models while simultaneously suppressing negative samples'; secondary claims concern model-specific preference patterns and SimPO's stronger out-of-distribution generalization.

Significance. The framework is a useful conceptual contribution: evaluating alignment through ranked hypothesis spaces rather than single sampled responses is a genuine improvement in diagnostic methodology, and the proposed RA/PSC pair addresses an under-explored distinction between ordinal preference capture and continuous preference-strength calibration. The UniHypoBench resource, with 10+ responses per prompt, is potentially valuable for future work. The experimental effort is substantial, spanning multiple models, proxies, and ablations, and the authors provide code and data. If the central empirical claim were supported, the paper would be a solid contribution to preference-learning evaluation. However, the same-proxy evaluation design substantially limits the claim that can be drawn, and several claims are stronger than the presented evidence. The framework itself remains defensible and useful, but the empirical conclusions need either re-scoping or additional independent validation.

major comments (4)
  1. [Section 4.1, 4.2; Tables 1 and 4] The primary evaluation is circular with respect to the central claim. Section 4.1 states that ArmoRM is used both to annotate the preference training data and as the gold-standard evaluator, 'ensuring consistency between training and evaluation preference distributions.' Since DPO, SimPO, and ORPO all directly optimize the trained policy's likelihoods toward ArmoRM-labeled preferred responses, high RA/PSC in the same-distribution rows of Table 1 can largely reflect fitting to the training labeler rather than 'effectively captur[ing] preferences.' Table 4 strengthens this concern: against the original HelpSteer2 human annotations, RA is near chance (44.79-48.18) and PSC is negative (-0.037 to -0.104); optimized models do not beat the SFT base. Table 1's GRM 'different preference distribution' rows also show no consistent improvement over the base model. The abstract and Section 4.2 should
  2. [Section 4.2, Tables 1 and 7] No error bars, confidence intervals, or significance tests are reported for RA or PSC, which are dataset-level means over prompts. Many differences in Table 1 are under one percentage point (e.g., LLaMA-3.2-3B base vs +DPO on UniHypo: RA 54.64 vs 54.72; PSC 0.152 vs 0.154), and Appendix B's additional-backbone results show no movement at all for several models (e.g., Mistral-7B-it-v0.3 rows are nearly identical across methods). The claim that 'preference optimization effectively captures preference information' across the tested models is not statistically grounded. Please report bootstrap confidence intervals, paired tests over prompts, or otherwise justify that the observed differences are not noise.
  3. [Section 4.3, Table 2] The text in Section 4.3 is contradicted by the table it cites. The text says 'the results reveal that performance shows no significant improvement even when evaluated on the model's own preference distribution' and 'we observe performance degradation in some cases, particularly for the SimPO-aligned model.' Table 2, however, shows SimPO improving RA from 55.93 to 61.50 and PSC from 0.164 to 0.301 (w/o length normalization), and from 54.12 to 59.70 and 0.100 to 0.252 (w/ length normalization). This inconsistency undermines the narrative in that subsection and needs to be corrected or the analysis re-described.
  4. [Section 3.2, Eqs. (6)-(7)] The proposed 'ranking accuracy' metric is defined as (tau_b + 1)/2. This mapping does not equal the proportion of concordant pairs when ties are present, because tau_b's denominator includes tie-correction terms. The statement in Definition 3 that the mapping 'is equal to assigning a zero-valued weight to the discordant pairs' is therefore inaccurate. Since RA is one of the two central metrics, the definition should be made mathematically precise, or the metric should be replaced with a true concordance proportion (or otherwise explicitly justified).
minor comments (6)
  1. [Table 4] The column header 'RS.' should presumably be 'RA.'; elsewhere the paper consistently uses RA for ranking accuracy.
  2. [Eqs. (4)-(5)] The notation for the hypothesis space is confusing: Yx is defined as a set but simultaneously assigned an ordering constraint. A ranked list or tuple notation (y1, y2, ...) with a separate definition of the ordering would be clearer.
  3. [Section 4.1] The text says 'We evaluated our approach using three models, including LLaMA-3.2-3B-Instruct and LLaMA-3-8B-Instruct,' but only two are named. Please either list all three or correct the count.
  4. [Appendix B / References] The citation for Mistral in Appendix B points to Jiang et al. (2023), 'From CLIP to DINO,' which is not the Mistral model paper. Please correct the reference.
  5. [General] Several table cells in Table 1 run together (e.g., '54.640.152'), making the table hard to read. Consistent spacing would help.
  6. [Section 5] The sentence 'our findings reveal that preference learning algorithms are notably adept at capturing most of these sub-dimensions' is difficult to reconcile with Table 3, where helpfulness RA is near chance and complexity/verbosity PSC are strongly negative. Please qualify this statement or revise it to match the data.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the proxy-specific claim is evaluated with an appropriate held-out same-scorer protocol, and external checks bound the result.

full rationale

HEAL is an evaluation framework rather than a derivation, and its central claim is explicitly scoped to preferences provided by proxy models, not human preferences in general. The experiments use ArmoRM as both the training-label source and the evaluation gold standard, but this is not circular: the evaluation metrics (RA and PSC, Eqs. 6-8) are computed on held-out hypothesis sets (UniHypoBench, HelpSteer2 validation, UltraFeedback validation) whose prompts and responses were not used in training. Thus the measured agreement reflects whether the policy generalizes the proxy's ranking function to new inputs, which is a legitimate and non-tautological empirical question. The metrics are defined independently of the DPO/SimPO/ORPO training losses; nothing in Eq. 6-8 is defined in terms of the training data. The paper also includes two external checks that are not circular: a different proxy (GRM) with a different preference distribution shows no consistent improvement, and original human annotations from HelpSteer2 (Table 4) show ranking accuracy near chance and negative preference strength correlation. The paper explicitly acknowledges that 'current methods primarily learn model-specific judging patterns rather than general preferences' and that the reference preference is 'not a universal ground truth' (Appendix A.3). These acknowledgments and external results bound the central claim and prevent the same-proxy evaluation from being a self-fulfilling definition. No load-bearing step reduces to a self-citation or to the definition of the target quantity.

Assumptions & free parameters 5 free parameters · 5 assumptions · 2 invented entities

The central claim rests on training hyperparameters chosen by grid search, several domain assumptions about the proxy reward model and the representation of hypothesis spaces, and the new benchmark artifact. The most important assumption is that ArmoRM scores are a valid gold standard, which the paper's own human-annotation results call into question.

free parameters (5)
  • DPO beta = grid searched in [0.01, 0.05, 0.1]
    Controls KL constraint in the DPO loss (Eq. 3); final value not reported.
  • ORPO lambda = grid searched in [0.1, 0.5, 1.0]
    Odds-ratio penalty weight; final value not reported.
  • SimPO beta and gamma = beta in [2.0, 10.0], gamma in [0.3, 1.0]
    Reference-free reward scale and target margin; grid searched, final values not reported.
  • Learning rate = searched in [3e-7, 7e-7, 1e-6]
    Standard training hyperparameter; final value not reported.
  • Sampling temperature and top-p for UniHypoBench = temperature 0.75, top-p 0.95
    Used to generate benchmark hypotheses; affects response diversity and downstream results.
assumptions (5)
  • domain assumption The proxy reward model ArmoRM-LLaMA-3-8B-v0.1 provides a valid gold-standard ranking for alignment quality.
    Section 4.1 uses ArmoRM as primary gold-standard; Table 4 shows weak agreement with human annotations, making this assumption questionable.
  • domain assumption Generation likelihoods (or length-normalized versions) are comparable across responses and reflect preference strength.
    Section 3.2 computes Pearson correlation between likelihoods and gold scores; likelihoods are length-dependent and may not be calibrated consistently across models.
  • domain assumption A finite set of sampled responses can represent the infinite hypothesis space defined in Eq. 4.
    UniHypoBench samples 8+ responses per prompt, but Eq. 4 describes an infinite ordered set; the paper does not justify the finite approximation.
  • standard math Kendall's tau-b and Pearson correlation are appropriate measures of preference alignment.
    Used as metrics without justification for why these are the correct or most informative measures for preference learning.
  • domain assumption Responses from diverse LLMs form a representative hypothesis space for evaluation.
    Appendix A.1 describes 8 LLMs and sampling settings, but no coverage analysis or diversity validation is provided.
invented entities (2)
  • HEAL framework independent evidence
    purpose: Evaluation paradigm with two metrics: ranking accuracy and preference strength correlation.
    The metrics are defined and can be applied to any generative or reward model; the paper claims the framework is sampling-free and API-free, providing an external handle for others to apply it.
  • UniHypoBench independent evidence
    purpose: Benchmark of 2,985 prompts with 8+ responses per prompt for hypothesis-space evaluation.
    The dataset is released via an anonymous repository, so others can use it to test their own models; it provides a falsifiable, reusable benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HEAL: A Hypothesis-Based Preference-Aware Analysis Framework." pith.science (2026). https://pith.science/paper/DJQD52LC

@misc{pith2026250819922,
  author       = {Pith},
  title        = {Pith review of: HEAL: A Hypothesis-Based Preference-Aware Analysis Framework},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DJQD52LC}},
  note         = {Machine review of arXiv:2508.19922}
}
read the original abstract

Preference optimization methods like DPO have achieved remarkable performance in LLM alignment. However, the evaluation for these methods relies on a single response and overlooks other potential outputs, which could also be generated in real-world applications within this hypothetical space. To address this issue, this paper presents a \textbf{H}ypothesis-based Pr\textbf{E}ference-aware \textbf{A}na\textbf{L}ysis Framework (HEAL), a novel evaluation paradigm that formulates preference alignment as a re-ranking process within hypothesis spaces. The framework incorporates two complementary metrics: ranking accuracy for evaluating ordinal consistency and preference strength correlation for assessing continuous alignment. To facilitate this framework, we develop UniHypoBench, a unified hypothesis benchmark constructed from diverse instruction-response pairs. Through extensive experiments based on HEAL, with a particular focus on the intrinsic mechanisms of preference learning, we demonstrate that current preference learning methods can effectively capture preferences provided by proxy models while simultaneously suppressing negative samples. These findings contribute to preference learning research through two significant avenues. Theoretically, we introduce hypothesis space analysis as an innovative paradigm for understanding preference alignment. Practically, HEAL offers researchers robust diagnostic tools for refining preference optimization methods, while our empirical results identify promising directions for developing more advanced alignment algorithms capable of comprehensive preference capture.

Figures

Figures reproduced from arXiv: 2508.19922 by the authors.

Figure 1
Figure 1. The overview of HEAL. We conceptualize preference learning as an alignment process between the [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Upset plots of preference intersections on the OOD test set (UniHypo). The upper bar chart displays [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Violin plots comparing generation likelihood distributions across different alignment methods (including [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Upset plots of generation likelihoods without length normalization. [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]
Figure 5
Figure 5. Figure 5: Upset plots of generation likelihoods with length normalization. [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 7 canonical work pages

  1. [1]

    Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, and Wanli Ouyang. 2024. https://arxiv.org/abs/2402.14762 Mt-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues . ArXiv preprint, abs/2402.14762

    arXiv 2024

  2. [2]

    Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, and 1 others. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862

    arXiv 2022

  3. [3]

    Zhang, Xinyi Chen, Qiuyi Zhang, Rajesh Ranganath, and Kyunghyun Cho

    Angelica Chen, Sadhika Malladi, Lily H. Zhang, Xinyi Chen, Qiuyi Zhang, Rajesh Ranganath, and Kyunghyun Cho. 2024. https://api.semanticscholar.org/CorpusID:270123277 Preference learning algorithms do not learn preference rankings . ArXiv, abs/2405.19534

    arXiv 2024

  4. [4]

    Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. https://arxiv.org/abs/2310.01377 Ultrafeedback: Boosting language models with high-quality feedback . ArXiv preprint, abs/2310.01377

    arXiv 2023

  5. [5]

    Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and 510 others

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony S. Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and 510 others. 2024. https://api.semanticscholar.org/CorpusID:271571434 ...

    arXiv 2024

  6. [6]

    Yann Dubois, Bal'azs Galambosi, Percy Liang, and Tatsunori Hashimoto. 2024. https://arxiv.org/abs/2404.04475 Length-controlled alpacaeval: A simple way to debias automatic evaluators . ArXiv preprint, abs/2404.04475

    arXiv 2024

  7. [7]

    Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. https://api.semanticscholar.org/CorpusID:267406810 Kto: Model alignment as prospect theoretic optimization . ArXiv, abs/2402.01306

    arXiv 2024

  8. [8]

    Leo Gao, John Schulman, and Jacob Hilton. 2022. https://api.semanticscholar.org/CorpusID:252992904 Scaling laws for reward model overoptimization . In International Conference on Machine Learning

    work page 2022

Show all 35 references
  1. [9]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Yuanzhuo Wang, and Jian Guo. 2024. https://api.semanticscholar.org/CorpusID:274234014 A survey on llm-as-a-judge . ArXiv, abs/2411.15594

    2024 arXiv

  2. [10]

    Jiwoo Hong, Noah Lee, and James Thorne. 2024. https://api.semanticscholar.org/CorpusID:268363309 Orpo: Monolithic preference optimization without reference model . ArXiv, abs/2403.07691

    2024 arXiv

  3. [11]

    Dongsheng Jiang, Yuchen Liu, Songlin Liu, Jin'e Zhao, Hao Zhang, Zhen Gao, Xiaopeng Zhang, Jin Li, and Hongkai Xiong. 2023. From clip to dino: Visual encoders shout in multi-modal large language models. arXiv preprint arXiv:2310.08825

    2023 arXiv

  4. [12]

    Gonzalez, Haotong Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Haotong Zhang, and Ion Stoica. 2023. https://api.semanticscholar.org/CorpusID:261697361 Efficient memory management for large language model serving with pagedattention . Procee...

    2023

  5. [13]

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, and 1 others. 2024. https://arxiv.org/abs/2403.13787 Rewardbench: Evaluating reward models for language modeling . ArXiv preprint, a...

    2024 arXiv

  6. [14]

    Alexander Lex, Nils Gehlenborg, Hendrik Strobelt, Romain Vuillemot, and Hanspeter Pfister. 2014. https://doi.org/10.1109/TVCG.2014.2346248 Upset: Visualization of intersecting sets . IEEE Transactions on Visualization and Computer Graphics (InfoVis), 20(12):1983--1992

    2014

  7. [15]

    Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph Gonzalez, and Ion Stoica. 2024. https://api.semanticscholar.org/CorpusID:270562889 From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline . ArXiv, abs/2406.11939

    2024 arXiv

  8. [16]

    Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. https://arxiv.org/abs/2405.14734 Simpo: Simple preference optimization with a reference-free reward . ArXiv preprint, abs/2405.14734

    2024 arXiv

  9. [17]

    Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan...

    2022 arXiv

  10. [18]

    Grace Proebsting and Adam Poliak. 2024. https://api.semanticscholar.org/CorpusID:273323367 Hypothesis-only biases in large language model-elicited natural language inference . ArXiv, abs/2410.08996

    2024 arXiv

  11. [19]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36

    2024

  12. [20]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. https://www.microsoft.com/en-us/research/publication/zero-memory-optimizations-toward-training-trillion-parameter-models/ Zero: Memory optimizations toward training trillion parameter models . ArXiv

    2020

  13. [21]

    Gemma Team Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L'eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram'e, Johan Ferret, Peter Liu, Pouya Dehghani Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charl...

    2024 arXiv

  14. [22]

    Ziegler, Ryan J

    Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan J. Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2020. https://arxiv.org/abs/2009.01325 Learning to summarize from human feedback . ArXiv preprint, abs/2009.01325

    2020 arXiv

  15. [23]

    Chenglong Wang, Yang Gan, Yifu Huo, Yongyu Mu, Qiaozhi He, Murun Yang, Bei Li, Tong Xiao, Chunliang Zhang, Tongran Liu, and 1 others. 2025 a . Gram: A generative foundation reward model for reward generalization. arXiv preprint arXiv:2506.14175

    2025

  16. [24]

    Chenglong Wang, Yang Gan, Yifu Huo, Yongyu Mu, Murun Yang, Qiaozhi He, Tong Xiao, Chunliang Zhang, Tongran Liu, and Jingbo Zhu. 2025 b . Rovrm: A robust visual reward model optimized via auxiliary textual preference data. In Proceedings of the AAAI Conference on Artificial Int...

    2025

  17. [25]

    Chenglong Wang, Hang Zhou, Kaiyan Chang, Bei Li, Yongyu Mu, Tong Xiao, Tongran Liu, and Jingbo Zhu. 2024 a . Hybrid alignment training for large language models. arXiv preprint arXiv:2406.15178

    2024 arXiv

  18. [26]

    Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. 2024 b . https://api.semanticscholar.org/CorpusID:270562658 Interpretable preferences via multi-objective reward modeling and mixture-of-experts . In Conference on Empirical Methods in Natural Language Processing

    2024

  19. [27]

    Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. 2024 c . https://api.semanticscholar.org/CorpusID:273025954 Helpsteer2-preference: Complementing ratings with preferences . ArXiv, abs/2410.01257

    2024 arXiv

  20. [28]

    Tong Xiao and Jingbo Zhu. 2025. Foundations of large language models. arXiv preprint arXiv:2501.09223

    2025 arXiv

  21. [29]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. https://api.semanticscholar.org/Cor...

    2025 arXiv

  22. [30]

    Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. 2024. https://api.semanticscholar.org/CorpusID:270521260 Regularizing hidden states enables learning generalizable reward model for llms . ArXiv, abs/2406.10216

    2024 arXiv

  23. [31]

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, and Yongqiang Ma. 2024. https://api.semanticscholar.org/CorpusID:268536974 Llamafactory: Unified efficient fine-tuning of 100+ language models . ArXiv, abs/2403.13372

    2024 arXiv

  24. [32]

    Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, L. Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. https://api.semanticscholar.org/CorpusID:258822910 Lima: Less is more for alignment . ArX...

    2023 arXiv

  25. [33]

    Hang Zhou, Chenglong Wang, Yimin Hu, Tong Xiao, Chunliang Zhang, and Jingbo Zhu. 2024. Prior constraints-based reward model training for aligning large language models. In China National Conference on Chinese Computational Linguistics, pages 555--570. Springer

    2024

  26. [34]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  27. [35]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.