Pith. sign in

REVIEW 5 major objections 5 minor 23 references

For small language models writing medication leaflets from structured data, the post-training alignment method—ORPO in-domain, GRPO for cross-regulatory transfer—matters more than model scale or medical pretraining, and aligned 7B models be

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 05:11 UTC pith:DXVOFP5C

load-bearing objection Crisp ORPO/PLW results and a useful new dataset, but the GRPO cross-dataset claim is inflated by test-set checkpoint selection and reward/evaluation overlap. the 5 major comments →

arxiv 2607.13430 v1 pith:DXVOFP5C submitted 2026-07-15 cs.CL

Exploring Post-Training Alignment of Small Language Models for Biomedical Data-to-Text Generation: A Case Study of Medication Leaflet

Xi Yang , Guodong Liu , Chuqin Li , Fan Wu , Ergin Soysal , Min Jiang , Xing He , Jiang Bian
show 5 more authors
Yi Guo Shams Zaman Thomas Fuchs Todd Sanger Yonghui Wu
This is my paper
classification cs.CL
keywords small language modelsdata-to-text generationmedication leaflet generationpost-training alignmentORPOGRPObiomedical natural language processingcross-dataset generalization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether the way a small language model is aligned after pretraining matters more than its size or medical pretraining when converting structured drug data into patient-friendly leaflets. The authors fine-tune Qwen2.5 models (0.5B to 7B) with four alignment methods—SFT, DPO, ORPO, GRPO—on EMA-derived leaflet sections, then test zero-shot on FDA-derived labels. They claim that aligned small models beat the proprietary GPT-5 baseline on automatic metrics; ORPO is the best in-domain strategy, while GRPO is the most robust when transferring across regulatory sources. They also report that full-parameter tuning and prompt loss weighting help, while medical pretraining and parameter-efficient QLoRA do not. A sympathetic reader would care because the results point to an affordable, deployable alternative to large proprietary models for regulated patient-facing text, provided the alignment recipe is chosen deliberately.

Core claim

On the BioLeaflets test set, ORPO—an alignment objective that combines supervised likelihood with an odds-ratio preference term—beats both SFT and DPO on every reported metric at every scale, with semantic similarity rising by about 0.02 to 0.03 over SFT. Under zero-shot transfer to the newly constructed FDALeaflets set, GRPO initialized from instruction-tuned Qwen without SFT achieves the highest semantic similarity (0.9407) and METEOR (0.5400), above GPT-5; the SFT-initialized GRPO variant, while best in-domain, loses adaptability. The paper concludes that alignment strategy is a dominant factor for biomedical data-to-text generation, exceeding model scale and medical pretraining.

What carries the argument

The load-bearing components are the two best-performing alignment objectives: ORPO (odds-ratio preference optimization), which jointly maximizes the likelihood of preferred outputs and widens the odds ratio against rejected outputs without a reference model, and GRPO (group relative policy optimization), which samples 16 candidate responses per input and optimizes token-level advantages using METEOR and semantic-similarity rewards. Prompt loss weighting (PLW), which assigns a fractional loss (0.155 here) to prompt tokens instead of masking them, further improves both SFT and ORPO. The input representation is XML-structured concept fields extracted from leaflet sections, using the same schema

Load-bearing premise

The ranking claims stand on the assumptions that the LLM-generated structured inputs of FDALeaflets are faithful, that the section-level train/test split does not leak sections from the same leaflet into both train and test, and that automatic and LLM-as-judge metrics track true clinical quality—none of which is validated by human evaluation.

What would settle it

Re-run the comparison with a leaflet-level split of BioLeaflets (grouping sections by source leaflet) and with clinician review of outputs for factual errors and completeness; if the ORPO-over-SFT or GRPO-over-ORPO margins disappear or reverse under human judgment, or if any training sections share a source leaflet with test sections, the ranking claims lose their support.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Adopting ORPO with prompt loss weighting should be the default for in-domain structured biomedical data-to-text when high-quality references and preference pairs exist; it beats SFT and DPO on every reported metric at 3B and 7B scales.
  • For deployment across regulatory contexts without retraining, an SFT-free GRPO model is the more robust choice, outperforming ORPO, SFT, DPO, and GPT-5 on FDA-label transfer under automatic metrics.
  • Full-parameter fine-tuning is necessary for this task; QLoRA degrades semantic similarity to 0.90–0.92, so parameter-efficient adaptation is not sufficient when structured inputs must be mapped to faithful narratives.
  • Medical-domain pretraining (Meditron3-Qwen2.5-7B) gives no measurable edge over general Qwen2.5-7B, indicating that alignment, not broad biomedical pretraining, drives the gains.
  • Small aligned models can beat a proprietary large model under automatic and LLM-judge metrics, suggesting task-specific alignment can substitute for raw scale in regulated text generation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The SFT-free GRPO advantage suggests that reward-driven learning selects content and structure that transfer across regulatory conventions; a testable corollary is that GRPO models will also degrade less on other distribution shifts (e.g., other label formats or languages) than SFT/ORPO models.
  • Because GPT-5 nearly closes the gap under LLM-as-judge (lowest hallucination, highest fluency among baselines), human clinical evaluation could shift the practical ranking; this is worth testing before deployment decisions.
  • The finding that broad medical pretraining did not help implies that gains may require pretraining on medication- or regulatory-specific corpora; a direct extension would compare such specialized pretraining against alignment-only recipes on the same leaflet generation task.
  • The section-level random split may inflate in-domain numbers by allowing section-level memorization; a leaflet-level split re-evaluation would reveal whether the ORPO/GRPO ranking holds when generalization is measured across whole leaflets rather than individual sections.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper presents a comparative study of post-training alignment methods (SFT, DPO, ORPO, GRPO) applied to Qwen2.5 small language models for generating medication leaflets from structured regulatory data. Using the existing BioLeaflets dataset (EMA-derived) and a newly introduced FDALeaflets dataset (FDA-derived), the authors evaluate in-domain performance and cross-regulatory zero-shot generalization against T5 and GPT-5 baselines. The central claims are that ORPO consistently outperforms SFT and DPO across all metrics, and that GRPO provides the most robust cross-dataset performance, even surpassing GPT-5 on automatic metrics.

Significance. If the empirical claims hold, the paper would provide a practical guide for selecting alignment strategies in regulated biomedical data-to-text generation, a task with direct deployment relevance. The release of the FDALeaflets dataset and the systematic comparison across model scales and initialization strategies are valuable contributions. However, the central comparative claims are currently supported by an evaluation protocol that treats GRPO differently from the other methods, and by reward metrics that coincide with the reported evaluation metrics. These issues undermine the reliability of the headline conclusions until addressed.

major comments (5)
  1. [Experimental Settings; Cross-Dataset Generalizability Evaluation; Figure 1; Table 4] GRPO is not evaluated under the same protocol as SFT, DPO, and ORPO. The text states that GRPO checkpoints are saved every 1,000 steps and evaluated on the test set, with Figure 1 labeling the best result per metric. Table 4 then selects the 'best-performing model from each alignment method' on the in-domain test set. This gives GRPO the advantage of test-set-based checkpoint selection, whereas the other methods are trained for fixed epochs and evaluated once. The reported GRPO numbers are therefore maxima over several test-set evaluations, which can inflate both in-domain and cross-dataset performance. Please re-run the comparison using a validation split for model selection, or report GRPO at a fixed step count comparable to the epoch budgets of the other methods.
  2. [Results; Table 3 vs. Table 1] The abstract and Results state that ORPO 'consistently outperforms both SFT and DPO across all evaluation metrics.' This is contradicted by the SacreBLEU columns. For example, Qwen-2.5-7B-SFT-PLW achieves SacreBLEU 73.39 while Qwen-2.5-7B-ORPO-PLW achieves 68.08; for 0.5B, SFT-PLW reaches 71.46 versus 62.26 for ORPO-PLW. The in-domain advantage of ORPO is metric-dependent, and the claim should be revised accordingly.
  3. [Methods, GRPO paragraph] The GRPO reward functions are METEOR and semantic similarity (STS), and the reported advantages of GRPO are measured on exactly these metrics. This is partly circular: the method is optimized to maximize the same scores used to declare its superiority. Please report results on held-out metrics that were not used as rewards (e.g., SacreBLEU, ROUGE-L, MoverScore) as primary evidence, and discuss the extent to which the GRPO gains reflect reward overfitting rather than general generation quality.
  4. [Dataset section] Two data-construction issues need to be addressed. First, BioLeaflets sections are randomly split into train/test without leaflet-level grouping; sections from the same leaflet may appear in both partitions, causing possible information leakage and inflated in-domain scores. Second, FDALeaflets is constructed by using Qwen-235B-Thinking to generate structured inputs from FDA label text, with no human validation of the extracted structure. The cross-dataset evaluation is therefore conditional on the quality of these synthetic structured inputs. Please quantify potential overlap in the BioLeaflets split and provide a human-validated sample of FDALeaflets.
  5. [Non-preferred Sample Synthesis; Evaluation] Qwen3-235B-Thinking is used both to select the rejected responses for DPO/ORPO and as the LLM judge in Table 5. This shared provenance can bias the qualitative evaluation in favor of Qwen-family models. The automatic metrics are less affected, but the LLM-as-judge results are not independent. Please use an independent judge model or a human evaluation to confirm the qualitative findings.
minor comments (5)
  1. [Table 4] The table title says 'Cross-dataset evaluation on the BioLeaflets Test Set' but the evaluation is on FDALeaflets. Please correct.
  2. [Tables 1-4] 'METERO' is a typo for METEOR throughout the tables.
  3. [Table 5] The naming is inconsistent: the new dataset is called FDALeaflets in the text but 'OpenFDA' in the table header. Please align terminology.
  4. [Results] No confidence intervals or significance tests are reported. Given the small test sizes (742 samples; 400 FDALeaflets samples), many reported differences may be within noise. Please add statistical tests or bootstrap intervals.
  5. [Figure 1] The figure labels only the best result per model per metric, which obscures the trajectory and prevents readers from seeing variance across checkpoints. Showing all checkpoints would help assess training stability and the effect of checkpoint selection.

Circularity Check

3 steps flagged

GRPO's reported advantage is partly built into its METEOR/STS reward functions, test-set checkpoint selection, and the self-judging preference loop.

specific steps
  1. self definitional [Methods, Post-training Alignment Methods (GRPO); Experimental Settings and Evaluation (Evaluation); Results (Table 4)]
    "We apply GRPO to the biomedical D2T generation task by implementing two task-derived reward functions: (1) METEOR,... and (2) semantic similarity computed as cosine similarity between paragraph embeddings. ... The Qwen2.5-7B-Inst-GRPO model achieves the highest semantic similarity (0.9407) and METEOR (0.5400), substantially outperforming both GPT-5 (STS of 0.9295; METEOR of 0.5328) and models trained with SFT, DPO, or ORPO."

    GRPO's training reward is defined as METEOR plus semantic similarity (STS), and the paper's headline claim that 'GRPO yields the most robust cross-dataset performance' is evaluated with exactly these two metrics. SFT, DPO, and ORPO never optimize these metric-based rewards. The observed GRPO advantage on METEOR and STS is therefore partly self-fulfilling: the success criterion was inserted into the training objective. The circularity is only partial because the cross-dataset evaluation is on held-out FDALeaflets, so the result is not fully tautological.

  2. fitted input called prediction [Experimental Settings and Evaluation (Experiment Settings); Results (Figure 1); Methods (Cross-Dataset Generalizability Evaluation)]
    "In GRPO experiments, we train models with total 10,000 steps (~ 1.5 epoch) and checkpoints are saved every 1,000 training steps and evaluated on the test dataset to monitor training dynamics. ... achieving a peak semantic similarity (STS) score of 0.9711 and a METEOR score of 0.6967 at 9,000 steps. ... we select the best-performing model from each alignment method (SFT, DPO, ORPO, and GRPO)."

    The reported GRPO in-domain numbers are maxima over checkpoints evaluated on the test set, so the checkpoint itself is selected on the test set. The same test-selected checkpoint is then used as the 'best-performing model from each alignment method' for the cross-dataset comparison, while SFT, DPO, and ORPO are trained for fixed epochs and evaluated once with no analogous test-set checkpoint selection. The GRPO advantage is therefore partly an artifact of test-set selection rather than an unbiased property of the method; the 'prediction' of GRPO robustness is statistically forced by choosing the best test-set checkpoint.

  3. self definitional [Methods, Non-preferred Sample Synthesis; Experimental Settings and Evaluation (Evaluation); Results (Table 5)]
    "We then employ the Qwen3-235B-Thinking model as an automated judge to evaluate the generated against the gold-standard reference. The model selects the most appropriate non-preferred (rejected) response... Further, we evaluate model outputs using an LLM-as-judge approach (Qwen3-235B-Thinking) across four dimensions: adequacy, hallucination presence, entity inclusion, and fluency."

    The same Qwen3-235B-Thinking model is used both to construct the preference pairs that train DPO/ORPO and to score the final outputs of those models. Thus the LLM-as-judge results in Table 5 are not an independent evaluation; they measure alignment to the very judge that shaped the training signal. This is circular for the claim that preference-based and RL methods 'generally outperform' supervised approaches on LLM-judge dimensions, although the automatic metrics in Tables 1-4 are unaffected by this particular loop.

full rationale

The paper is an empirical comparison rather than a formal derivation, so circularity appears as results that are partly built into the experimental protocol. The most load-bearing issue is GRPO: its reward functions are exactly METEOR and STS, which are also the metrics used to declare GRPO the most robust cross-dataset method. In addition, GRPO checkpoints are selected on the test set ('evaluated on the test dataset'), and the same test-selected checkpoint is used as the 'best-performing model from each alignment method' for the cross-dataset table, whereas SFT, DPO, and ORPO receive no such selection. This makes the reported GRPO advantage partially an artifact of test-set checkpoint selection. A third, supporting circularity is that the Qwen3-235B-Thinking model both selects the rejected responses for preference training and acts as the LLM judge for final outputs, so the qualitative robustness claims in Table 5 are partly self-referential. These issues are not fully tautological because FDALeaflets is held out and the automatic metrics are externally defined, but they substantially weaken the central claim that GRPO is the most robust alignment method. The paper itself acknowledges the absence of human evaluation, which would have provided an independent check.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central claims rest on a handful of tuned hyperparameters (β, PLW weight, GRPO reward combination) and on domain assumptions about the reliability of LLM-generated FDA structured data, leaflet-level independence in the data split, and the validity of using the same LLM to build and grade preference data.

free parameters (3)
  • DPO/ORPO preference scaling β = 0.3
    Selected from {0.1, 0.3, 0.6} based on preliminary validation; affects all preference-based results.
  • PLW prompt-loss weight = 0.155
    Computed from the Rg=0.51 heuristic; applied in SFT and ORPO and materially changes results.
  • GRPO reward combination weights (METEOR vs STS) = unspecified
    The paper says combining METEOR and STS gives the most stable improvements but does not report the combination formula or weights.
axioms (4)
  • domain assumption Sections from the same medication leaflet are statistically independent; a section-level random split does not leak leaflet-level information.
    Dataset section: 'The sections are randomly split into a training set of 6,640 samples and a test set of 742 samples.' No leaflet-level grouping is described.
  • domain assumption The Qwen-235B-Thinking model faithfully converts FDA SPL text into structured XML concepts consistent with BioLeaflets.
    Dataset section: 'leverage the Qwen-235B-thinking model to generate the structured information based on the text to create the new dataset – FDALeaflets.'
  • domain assumption LLM-as-judge scores from Qwen3-235B-Thinking reflect clinical adequacy and hallucination without bias from using the same model to construct preference pairs.
    Evaluation section and Non-preferred Sample Synthesis both use Qwen3-235B-Thinking.
  • domain assumption GPT-5 zero-shot prompting is a fair baseline for 'proprietary LLM' despite no fine-tuning or reported prompt optimization.
    Baselines section: 'For comparison, we include ... GPT-5.' No prompt details are given.

pith-pipeline@v1.3.0-alltime-deepseek · 13370 in / 14234 out tokens · 132373 ms · 2026-08-02T05:11:22.790333+00:00 · methodology

0 comments
read the original abstract

Translating complex biomedical data into patient-friendly narratives is central to modern biomedical informatics. This study presents a comparative analysis of training small language models (SLMs) in specialized biomedical datato-text generation tasks. We explore widely adopted post-training methods including supervised fine-tuning (SFT), direct preference optimization (DPO), odds ratio preference optimization (ORPO), and group relative policy optimization (GRPO) with Qwen-based SLMs on a medicine package leaflets dataset. To assess cross-dataset generalizability, we also curated drug label data from openFDA. We evaluate models using both standard lexical overlap metrics like ROUGE as well as semantic similarity measures. Across our experiments, the results show that (1) the aligned SLMs outperform proprietary models like GPT-5; (2) ORPO outperforms the SFTbaselines; (3) GRPO yields the most robust cross-dataset performance among the alignment methods tested as well as GPT-5.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 5 canonical work pages

  1. [1]

    Evidence-Based Design of Prescription Medication Information: An Updated Scoping Review

    Russell AM, Lovett R, Vogeley A, et al. Evidence-Based Design of Prescription Medication Information: An Updated Scoping Review. Drug Saf. 2025;48(6):607-641. doi:10.1007/s40264-025-01527-8

    work page doi:10.1007/s40264-025-01527-8 2025

  2. [2]

    A systematic review of quantitative and qualitative research on the role and effectiveness of written information available to patients about individual medicines

    T R, A B, P K, J G, D N, Al E. A systematic review of quantitative and qualitative research on the role and effectiveness of written information available to patients about individual medicines. Health Technology Assessment. 2007;11(5). doi:10.3310/hta11050

    doi:10.3310/hta11050 2007

  3. [3]

    Benefits and challenges of electronic package leaflet (ePL) - review of ePL pilots in hospital settings in Europe

    Skogman-Lindqvist C, Lapatto-Reiniluoto O, Sirviö M, Sivén M. Benefits and challenges of electronic package leaflet (ePL) - review of ePL pilots in hospital settings in Europe. European Journal of Pharmaceutical Sciences. 2023;191:106605. doi:10.1016/j.ejps.2023.106605

    arXiv 2023

  4. [4]

    FDA drug labeling: rich resources to facilitate precision medicine, drug safety, and regulatory science

    Fang H, Harris SC, Liu Z, et al. FDA drug labeling: rich resources to facilitate precision medicine, drug safety, and regulatory science. Drug Discovery Today. 2016;21(10):1566-1570. doi:10.1016/j.drudis.2016.06.006

    work page doi:10.1016/j.drudis.2016.06.006 2016

  5. [5]

    Factors that Facilitate and Hinder the Comprehension of Patient Information Leaflets (PILs): A Brief Scoping Review

    Medina-Córdoba M, Cadavid S, Pérez-Acosta AM, Amaya-Giraldo V. Factors that Facilitate and Hinder the Comprehension of Patient Information Leaflets (PILs): A Brief Scoping Review. Front Pharmacol. 2021;12. doi:10.3389/fphar.2021.740334

    arXiv 2021

  6. [6]

    Neural text generation in regulatory medical writing

    Meyer C, Adkins D, Pal K, Galici R, Garcia-Agundez A, Eickhoff C. Neural text generation in regulatory medical writing. Front Pharmacol. 2023;14. doi:10.3389/fphar.2023.1086913

    arXiv 2023

  7. [7]

    Biomedical Data-to-Text Generation via Fine-Tuning Transformers

    Yermakov R, Drago N, Ziletti A. Biomedical Data-to-Text Generation via Fine-Tuning Transformers. In: Belz A, Fan A, Reiter E, Sripada Y, eds. Proceedings of the 14th International Conference on Natural Language Generation. Association for Computational Linguistics; 2021:364-370. doi:10.18653/v1/2021.inlg-1.40

    work page doi:10.18653/v1/2021.inlg-1.40 2021

  8. [8]

    Natural Language Processing for Digital Health in the Era of Large Language Models

    Sarker A, Zhang R, Wang Y, et al. Natural Language Processing for Digital Health in the Era of Large Language Models. Yearb Med Inform. 2024;33(01):229-240. doi:10.1055/s-0044-1800750

    work page doi:10.1055/s-0044-1800750 2024

  9. [9]

    A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness

    Wang F, Zhang Z, Zhang X, et al. A Comprehensive Survey of Small Language Models in the Era of Large Language Models: Techniques, Enhancements, Applications, Collaboration with LLMs, and Trustworthiness. ACM Trans Intell Syst Technol. 2025;16(6):145:1-145:87. doi:10.1145/3768165

    doi:10.1145/3768165 2025

  10. [10]

    Qwen2.5 Technical Report

    Qwen, Yang A, Yang B, et al. Qwen2.5 Technical Report. arXiv. Preprint posted online January 3, 2025:arXiv:2412.15115. doi:10.48550/arXiv.2412.15115

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.15115 2025

  11. [11]

    Training language models to follow instructions with human feedback

    Ouyang L, Wu J, Jiang X, et al. Training language models to follow instructions with human feedback. arXiv. Preprint posted online March 4, 2022:arXiv:2203.02155. doi:10.48550/arXiv.2203.02155

    Pith/arXiv arXiv doi:10.48550/arxiv.2203.02155 2022

  12. [12]

    Direct Preference Optimization: Your Language Model is Secretly a Reward Model

    Rafailov R, Sharma A, Mitchell E, Ermon S, Manning CD, Finn C. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. arXiv. Preprint posted online July 29, 2024:arXiv:2305.18290. doi:10.48550/arXiv.2305.18290

    Pith/arXiv arXiv doi:10.48550/arxiv.2305.18290 2024

  13. [13]

    ORPO: Monolithic Preference Optimization without Reference Model

    Hong J, Lee N, Thorne J. ORPO: Monolithic Preference Optimization without Reference Model. In: Al-Onaizan Y, Bansal M, Chen YN, eds. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics; 2024:11170-11189. doi:10.18653/v1/2024.emnlp-main.626

    doi:10.18653/v1/2024.emnlp-main.626 2024

  14. [14]

    Group Sequence Policy Optimization

    Zheng C, Liu S, Li M, et al. Group Sequence Policy Optimization. arXiv. Preprint posted online July 28, 2025:arXiv:2507.18071. doi:10.48550/arXiv.2507.18071

    Pith/arXiv arXiv doi:10.48550/arxiv.2507.18071 2025

  15. [15]

    OpenFDA: an innovative platform providing access to a wealth of FDA’s publicly available data

    Kass-Hout TA, Xu Z, Mohebbi M, et al. OpenFDA: an innovative platform providing access to a wealth of FDA’s publicly available data. J Am Med Inform Assoc. 2016;23(3):596-600. doi:10.1093/jamia/ocv153

    work page doi:10.1093/jamia/ocv153 2016

  16. [16]

    Instruction Fine-Tuning: Does Prompt Loss Matter? arXiv

    Huerta-Enochian M, Ko SY. Instruction Fine-Tuning: Does Prompt Loss Matter? arXiv. Preprint posted online October 14, 2024:arXiv:2401.13586. doi:10.48550/arXiv.2401.13586

    Pith/arXiv arXiv doi:10.48550/arxiv.2401.13586 2024

  17. [17]

    Meteor: an automatic metric for MT evaluation with high levels of correlation with human judgments

    Lavie A, Agarwal A. Meteor: an automatic metric for MT evaluation with high levels of correlation with human judgments. In: Proceedings of the Second Workshop on Statistical Machine Translation. StatMT ’07. Association for Computational Linguistics; 2007:228-231. Accessed February 26, 2026. https://dl.acm.org/doi/10.5555/1626355.1626389

    arXiv 2007

  18. [18]

    Understanding R1-Zero-Like Training: A Critical Perspective

    Liu Z, Chen C, Li W, et al. Understanding R1-Zero-Like Training: A Critical Perspective. arXiv. Preprint posted online October 6, 2025:arXiv:2503.20783. doi:10.48550/arXiv.2503.20783

    Pith/arXiv arXiv doi:10.48550/arxiv.2503.20783 2025

  19. [19]

    RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models

    Khaki S, Li J, Ma L, Yang L, Ramachandra P. RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Models. arXiv. Preprint posted online March 30, 2024:arXiv:2402.10038. doi:10.48550/arXiv.2402.10038

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.10038 2024

  20. [20]

    Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment

    Xu L, Xie H, Qin SZJ, Tao X, Wang FL. Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment. arXiv. Preprint posted online December 19, 2023:arXiv:2312.12148. doi:10.48550/arXiv.2312.12148

    Pith/arXiv arXiv doi:10.48550/arxiv.2312.12148 2023

  21. [21]

    LoRA: Low-Rank Adaptation of Large Language Models

    Hu EJ, Shen Y, Wallis P, et al. LoRA: Low-Rank Adaptation of Large Language Models. arXiv. Preprint posted online October 16, 2021:arXiv:2106.09685. doi:10.48550/arXiv.2106.09685

    Pith/arXiv arXiv doi:10.48550/arxiv.2106.09685 2021

  22. [22]

    QLoRA: Efficient Finetuning of Quantized LLMs

    Dettmers T, Pagnoni A, Holtzman A, Zettlemoyer L. QLoRA: Efficient Finetuning of Quantized LLMs. arXiv. Preprint posted online May 23, 2023:arXiv:2305.14314. doi:10.48550/arXiv.2305.14314

    Pith/arXiv arXiv doi:10.48550/arxiv.2305.14314 2023

  23. [23]

    Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization

    Razin N, Malladi S, Bhaskar A, Chen D, Arora S, Hanin B. Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization. arXiv. Preprint posted online April 27, 2025:arXiv:2410.08847. doi:10.48550/arXiv.2410.08847

    Pith/arXiv arXiv doi:10.48550/arxiv.2410.08847 2025