Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineering

Ahmad Faraz Khan; Ali Anwar; Anas Mohamed; Ayaan Ahmad; Azal Ahmad Khan; Saman Bahzad Khan; Shuwen Ge; Xinran Wang

REVIEW 5 major objections 6 minor 12 references

Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineering

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Sem-DPO claims that weighting each DPO loss term by the embedding similarity between the original and the chosen prompt keeps optimized prompts close to the user's meaning, while preserving and in reported experiments improving…

desk verdict A plausible and cheap DPO reweighting for prompt optimization, but the 'provable bound' is a triangle inequality, the abstract's numbers don't match the appendix, and the method needs a rewrite before being taken seriously. read the letter →

arxiv 2507.20133 v2 pith:OL5JERAC submitted 2025-07-27 cs.CL cs.AIcs.LG

Anas Mohamed , Azal Ahmad Khan , Xinran Wang , Ahmad Faraz Khan , Shuwen Ge , Saman Bahzad Khan , Ayaan Ahmad , Ali Anwar This is my paper · ORCID

classification cs.CLcs.AIcs.LG

keywords directpreferenceoptimizationpromptengineeringtext-to-imagegenerationsemanticconsistencydrifthumanalignmentCLIPscoreweighting

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Direct Preference Optimization (DPO), a method that aligns a language model directly from pairwise preference data instead of training a separate reward model, has become a lightweight way to automatically rewrite text-to-image prompts, but its token-level objective lets winning prompts drift away from the user's intended meaning while still scoring higher on human preference. This paper introduces Sem-DPO, which multiplies each DPO loss term by an exponential weight $W_\alpha(x,y_w)=\exp(-\alpha\, d_{\mathrm{cos}}(e_\phi(x), e_\phi(y_w)))$, so preference pairs that stray semantically are down-weighted. The paper's central theoretical claim is that this weighting gives the first provable bound on semantic drift: the distance between the original prompt and the final generated image is at most the prompt-level semantic drift plus a small generator consistency error. On three text-to-image benchmarks and two language models, Sem-DPO reports 8–12% higher CLIP similarity and 5–9% higher human-preference scores than DPO. If the bound holds, semantic fidelity can be controlled without sacrificing the simplicity or efficiency of DPO.

What carries the argument

The load-bearing object is the semantic consistency weight $W_\alpha(x,y_w) = \exp(-\alpha\, d_{\mathrm{cos}}(e_\phi(x), e_\phi(y_w)))$, computed offline with a frozen text embedding model $e_\phi$ and folded into the DPO loss as a per-sample multiplier. It down-weights any preference pair whose preferred prompt is semantically far from the original prompt, so the policy gradient no longer rewards stylistic overfitting. The companion theoretical mechanism is the norm triangle inequality used in Proposition 2, which splits image-level drift into prompt-level semantic drift plus the generator's T2I consistency error $\varepsilon$; Sem-DPO targets the first term while $\varepsilon$ is treated as fixed. Proposition 1 shows the exponential kernel approximates hard filtering with bounded error, which is why the smooth weight is safe to use.

What would settle it

Measure the empirical T2I consistency error on a held-out set of prompts: render each prompt with the same Stable Diffusion v1.4 generator, embed prompt and image with a fixed CLIP-style model, and record $\|e(y) - E_{\mathrm{Img}}(G(y))\|$; if typical values are of the same size as the semantic drift Sem-DPO claims to remove, then the Proposition 2 bound does not guarantee closeness in practice. A second check is to compare Sem-DPO against DPO on the final image embeddings for the same prompts and see whether the distribution of image-level drift actually shifts.

Watch

Extended reading notes

Core claim

The discovery is that the semantic drift that plagues DPO-based prompt optimization can be written as a triangle-inequality decomposition, $d_{\mathrm{T2I}}(x,y) \le d_{\mathrm{Semantic}}(x,y) + \varepsilon$, where $d_{\mathrm{T2I}}(x,y) = \|e(x) - E_{\mathrm{Img}}(G(y))\|$ is the distance between the original prompt and the image generated from the optimized prompt, $d_{\mathrm{Semantic}}(x,y) = \|e(x)-e(y)\|$ is the drift between original and optimized prompt embeddings, and $\varepsilon$ bounds the text-to-image generator's consistency error. Sem-DPO is the mechanism that makes the first term small: it weights the DPO loss by $W_\alpha(x,y_w)=\exp(-\alpha\, d_{\mathrm{cos}}(e_\phi(x), e_\phi(y_w)))$, suppressing training pairs whose chosen prompt has drifted. The paper further shows this exponential weight is a smooth approximation to hard threshold filtering, with deviation bounded by $M(1-e^{-\alpha\tau})$. The intended reading is that the irreducible generator error $\varepsilon$ is the only remaining source of drift once prompt-level drift is controlled.

Load-bearing premise

The whole bound rests on the assumption that the text-to-image model always produces an image whose embedding lies within a small fixed distance $\varepsilon$ of its prompt's embedding; if that per-prompt faithfulness assumption fails or $\varepsilon$ is large in practice, the promised neighborhood around the original text is not actually small.

Editorial extensions

If this is right

Automatic prompt rewriting for text-to-image models can improve human-preference metrics without silently changing the user's request, removing the main failure mode that makes DPO risky in creative tooling.
The bound $d_{\mathrm{T2I}}(x,y) \le d_{\mathrm{Semantic}}(x,y) + \varepsilon$ means the text-to-image generator itself sets a floor on semantic fidelity; improving generator faithfulness will directly tighten the drift guarantee for any prompt optimizer.
Because the semantic weights are computed offline, any DPO-style preference-tuning pipeline can adopt the same weighting at negligible extra training cost, making semantic awareness a default rather than a specialized extension.
The reported trade-off with $\alpha$ gives practitioners a dial: lower values maximize meaning preservation, mid-range values (about 4) give the best CLIP scores, while very high values over-suppress gradients and degrade all metrics.
The majority-win pattern over DPO, SFT, and human inputs on CLIP, HPSv2.1, and PickScore suggests that flat baselines augmented with semantic weighting are the comparative setting future prompt-optimization work should use.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The triangle-inequality bound is a general template: any preference-optimization method that controls prompt-level semantic drift, not just Sem-DPO, inherits the same guarantee, because the proof never uses the specific form of the Sem-DPO weight.
Sem-DPO weights only the chosen prompt's distance from the original; weighting the gap between chosen and rejected distances (so a pair is down-weighted only if the chosen prompt drifts more than the rejected one) is a natural extension that would prevent reliance on preference pairs where both candidates are off-meaning.
The numerical guarantee is embedding-dependent: swapping the frozen text encoder $e_\phi$ for a different one (for instance an embedding aligned with the image encoder) would change both measured drift and the effective generator error $\varepsilon$, so the inequality is general but the size of the promised neighborhood is not.
Because large $\alpha$ shrinks the effective batch, an adaptive per-batch $\alpha$ that maintains a target effective sample size could extend the preference–semantics frontier instead of letting all metrics fall at high regularization strength.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

The thing you should know: Sem-DPO is a simple, cheap modification of DPO — multiply each preference term by exp(-alpha * cosine distance between the original prompt and the chosen prompt) — and the appendix tables do show it mostly beating plain DPO on CLIP and HPS across three datasets. But the central claim, a 'provably bounded neighborhood' for semantic drift, is not actually proven, and the abstract's effect sizes don't match the paper's own tables.

What's genuinely new and good: the problem is real. DPO for prompt optimization does drift semantically while chasing preference scores, and the paper gives a clean statement of the failure mode. The weighting is offline, cheap, and easy to drop into any DPO pipeline, which makes it a useful practical baseline. The head-to-head win-rate analysis in Figure 4 is a reasonable way to present the improvement. Proposition 1, bounding the gap to hard filtering by M(1-e^{-alpha tau}), is standard but correct.

The soft spots are substantial. Proposition 2 is a triangle inequality: it bounds the image drift by the prompt drift plus the generator's consistency error. But the paper never shows Sem-DPO controls prompt drift — the weighting only down-weights training pairs, and there's no theorem that the trained policy's outputs stay near the original prompt. So the bound is vacuous unless you already assume the thing you want to prove. Assumption 1, that the T2I generator's consistency error is bounded by a small epsilon, is asserted and never measured. The abstract says 8-12% higher CLIP and 5-9% higher HPS/PickScore than DPO; Tables 1 and 2 show relative CLIP gains over DPO are roughly 2-7%, and PickScore gains are under 4%. That is a real discrepancy. The main experiments fix alpha=8 while the hyperparameter analysis recommends alpha=4 for the best trade-off. There are no error bars, no code, no comparison against the closest reweighted or semantic DPO variants (Weighted-PO, WPO, DPO-Kernels, DSPO) even though those are cited. The limitations section honestly notes manual tuning, but that undercuts the 'provably' language.

None of this means the idea is worthless. The direction is plausible, and the experiments, as reported, mostly favor Sem-DPO. But the current paper is not a reliable contribution: the headline guarantee is not derived, and the headline numbers are inconsistent with the appendix. With honest numbers, a real bound (or no bound claim), and a few missing baselines, a corrected version could be worth taking seriously.

This is a paper for people working on prompt optimization or preference tuning for text-to-image models. I'd send this version to peer review with the expectation of major revision — there is enough substance in the idea and the experiments to justify referee time — but I would not accept it as is.

Best, [you]

Referee Report

5 major / 6 minor

Summary. The paper proposes Sem-DPO, a variant of Direct Preference Optimization for automatic prompt engineering that reweights each DPO sample by an exponential function of the cosine distance between the original prompt and the chosen prompt, W(x,y_w)=exp(-alpha*d(e_phi(x),e_phi(y_w))). The authors claim (i) this weighting is a smooth, bounded approximation to hard semantic filtering; (ii) it yields a provably bounded neighborhood for optimized prompts via Proposition 2; and (iii) it improves CLIP similarity and human-preference scores over DPO on three text-to-image benchmarks with two language models. The empirical appendix tables show consistent but modest gains in most settings, but the abstract's quantitative claims are not supported by those tables, and the theoretical guarantee is not actually derived from the training objective.

Significance. If the central claims held, Sem-DPO would be a simple, low-overhead improvement to DPO for prompt optimization, and the first analytical control of semantic drift in this setting would be a useful conceptual contribution. The paper deserves credit for a simple and computationally cheap modification, for reporting head-to-head comparisons, and for including full result tables in the appendix. However, the main theoretical result is a tautological triangle inequality that does not connect to the trained policy, the evaluation metric and the training weight share the same embedding family, and the headline improvement percentages contradict the appendix tables. The empirical results in Tables 1-2 are not strong enough to compensate for the unsupported central claim, so the paper is not publishable in its current form.

major comments (5)

[§4, Proposition 2] The abstract's central claim that Sem-DPO 'keeps learned prompts within a provably bounded neighborhood' is not established. Proposition 2 proves only d_T2I-Drift(x,y) <= d_Semantic-Drift(x,y) + epsilon, which is a direct application of the triangle inequality plus Assumption 1. The proof does not show that Sem-DPO controls d_Semantic-Drift for the outputs of the trained policy: the training loss down-weights individual training samples whose chosen prompt is far from x, but no theorem bounds d(x, pi_theta(.|x)) or the drift of the loss minimizer's output distribution. Without such a bound, the inequality is consistent with arbitrarily large drift and does not justify the phrase 'provably bounded neighborhood.'
[§4, Assumption 1] Assumption 1 is load-bearing for Proposition 2 but is not verified. It asserts ||e(y)-E_Img(G(y))|| <= epsilon for every prompt y on the deployed Stable Diffusion v1.4 pipeline, yet the paper provides no empirical estimate of epsilon. If epsilon is comparable to the drifts being bounded, the final inequality is vacuous; the paper also never attempts to measure d_Semantic-Drift after training. A reader cannot tell whether the claimed guarantee is informative.
[Abstract vs. Appendix Tables 1 and 2] The abstract reports '8-12% higher CLIP similarity and 5-9% higher human-preference scores (HPSv2.1, PickScore) than DPO,' but the appendix tables support much smaller gains. From Table 1, the relative CLIP gains over DPO are 2.3% (COCO), 6.2% (DiffusionDB), and 7.1% (Lexica), while PickScore gains are 2.8%, 0.95%, and 0.86%. Only HPSv2.1 gains fall in the claimed 5-9% range. The abstract's numbers should be corrected or the underlying experiments should be reported consistently.
[§5.1, §5.2, Limitations] There is a direct contradiction between the configuration used for the main experiments and the configuration recommended by the paper's own hyperparameter analysis. Section 5.1 states alpha was set to 8 after manual tuning and all main experiments use alpha=8, while Section 5.2 (Figure 5) concludes that alpha≈4 is optimal for meaning preservation and maximizing human preference. The Limitations section also acknowledges that alpha=8 was manually chosen. Since the reported results depend on an off-recommendation hyperparameter, the paper should explain why alpha=8 was used and provide main-result sensitivity across alpha.
[§3.2, §5.2, Metrics] The semantic consistency weight W(x,y_w)=exp(-alpha*d(e_phi(x),e_phi(y_w))) uses the same embedding family e_phi as the CLIP Score evaluation metric, and the theoretical drift measure in Proposition 2 also uses the same embedding space. The method is therefore optimized for the same proxy on which it is primarily evaluated; the paper does not report whether the semantic gains persist under a different text embedding or on a non-CLIP semantic metric. This circularity weakens the interpretation that Sem-DPO improves semantic fidelity rather than merely aligning with the chosen embedding.

minor comments (6)

[§3.2, Eq. (3)-(4)] The displayed Sem-DPO objective appears to have a typo: the second term inside the logarithm is written as 'pi_theta(y_l|x)/pi_ref(y_l|x)' without the 'log' operator, unlike the standard DPO term in Eq. (1). The correct form should be log(pi_theta(y_l|x)/pi_ref(y_l|x)).
[§2, Related Works] The text describes KTO as 'semantic-aware Kernel-based Preference Optimization,' but KTO is Kahneman-Tversky Optimization and is not a kernel-based method. This is a factual error in a related-work description.
[Figure 3] Several labels in Figure 3 are garbled, e.g., 'Human/uni00A0Input', 'Sem/uni00ADDPO', and 'Qwen/uni00AD1.5b/uni00A0/uni00AD/uni00A0COCO'. These should be fixed for readability.
[§5.1] There is a typo: 'Stable Difussion v1.4' should be 'Stable Diffusion v1.4'.
[§4, Proposition 1] Proposition 1 assumes the DPO loss term is bounded, |l(Delta)| <= M, but l(Delta) = -log sigma(Delta) is unbounded as Delta -> -infinity. The paper should justify this uniform bound or replace it with a more realistic assumption.
[Appendix Tables 1-2] The appendix tables report only point estimates; no standard deviations, confidence intervals, or significance tests are provided, even though Figure 4 reports win/tie/loss percentages. This makes it difficult to assess whether the modest PickScore gains are statistically meaningful.

Circularity Check

1 steps flagged · score 6.0 of 10

The 'provable bound' in Proposition 2 is the triangle inequality restated; it never uses the Sem-DPO loss, so the abstract's bounded-neighborhood guarantee is a tautology plus an unverified epsilon.

renaming known result [Section 4, Proposition 2 and the abstract's 'provably bounded neighborhood' claim]
"By applying the triangle inequality property of vector norms, we can establish an upper bound: ||e(x)-EImg(G(y))|| <= ||e(x)-e(y)|| + ||e(y)-EImg(G(y))|| ... dT2I-Drift(x,y) <= dSemantic-Drift(x,y) + dT2I(y). ... showing that Sem-DPO keeps learned prompts within a provably bounded neighborhood of the original text."

The paper's headline guarantee is merely the vector-norm triangle inequality written in new notation: the left side is the image drift, and the first term on the right is exactly the embedding distance d(x,y) that already defines the Sem-DPO weight W=exp(-alpha*d(x,yw)). The proof never invokes the Sem-DPO loss, the weighting, or the trained policy; it holds for any prompt y and any generator G. Thus the 'analytical bound on semantic drift' is equivalent to its mathematical input by construction, not a property established for Sem-DPO. The remaining term dSemantic-Drift is the quantity that would need to be controlled for a bounded neighborhood, and no result in the paper bounds it; Assumption 1's epsilon is asserted, not estimated.

full rationale

The paper's theoretical centerpiece reduces by construction to a known mathematical identity: Proposition 2 is the triangle inequality restated as a 'semantic drift bound,' with no dependence on the Sem-DPO objective. This makes the central 'provably bounded neighborhood' claim partly circular and mostly unsupported. I did not find load-bearing self-citations, and the empirical comparisons are against external benchmarks with transparent weighting, so the empirical part retains independent content. The score reflects that one central 'prediction' (the analytical guarantee) reduces to a tautology and an unverified assumption, while the method itself is not a self-citation chain or a fitted-parameter prediction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

All free parameters are hyperparameters of the loss. The main assumptions are the bounded generator consistency error, the use of cosine distance in a frozen embedding as a semantic proxy, and boundedness of the DPO loss. No new entities are postulated.

free parameters (2)

alpha (semantic weighting strength) = 8 in main experiments; 4 recommended in Section 5.2
Controls W=exp(-alpha*d). Main experiments use alpha=8 after manual tuning, but the hyperparameter analysis says alpha=4 is optimal, so the reported numbers do not match the recommended setting.
beta (DPO temperature) = 0.05
Standard DPO inverse temperature, set manually. It scales the log-odds in the loss but is not newly fitted by this work.

assumptions (4)

domain assumption Assumption 1: ||e(y) - EImg(G(y))|| <= epsilon for any prompt y and the T2I generator G
Stated in Section 4, Proposition 2. The image-level semantic drift bound depends on an epsilon-small generator consistency error; the paper provides no measurement of epsilon.
standard math DPO reward reparameterization under Bradley-Terry
Equation (1) takes the DPO derivation from Rafailov et al. (2023) as given; this is standard background.
domain assumption Cosine distance in a frozen CLIP embedding space is a faithful proxy for semantic meaning
Used in Definition 1, the weight W, and the CLIP evaluation. The authors acknowledge in the Ethical Considerations section that this may fail in some domains.
ad hoc to paper Bounded DPO loss |l(delta)| <= M
Proposition 1 assumes a bound M on the per-example DPO loss; without further constraints on the policy and reference model, -log sigma(delta) is not globally bounded.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineering." pith.science (2026). https://pith.science/paper/OL5JERAC

@misc{pith2026250720133,
  author       = {Pith},
  title        = {Pith review of: Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OL5JERAC}},
  note         = {Machine review of arXiv:2507.20133}
}

read the original abstract

Generative AI can now synthesize strikingly realistic images from text, yet output quality remains highly sensitive to how prompts are phrased. Direct Preference Optimization (DPO) offers a lightweight, off-policy alternative to RL for automatic prompt engineering, but its token-level regularization leaves semantic inconsistency unchecked as prompts that win higher preference scores can still drift away from the user's intended meaning. We introduce Sem-DPO, a variant of DPO that preserves semantic consistency yet retains its simplicity and efficiency. Sem-DPO adjusts the DPO loss using a weight based on how different the winning prompt is from the original, reducing the impact of training examples that are semantically misaligned. We provide the first analytical bound on semantic drift for preference-tuned prompt generators, showing that Sem-DPO keeps learned prompts within a provably bounded neighborhood of the original text. On three standard text-to-image prompt-optimization benchmarks and two language models, Sem-DPO achieves 8-12% higher CLIP similarity and 5-9% higher human-preference scores (HPSv2.1, PickScore) than DPO, while also outperforming state-of-the-art baselines. These findings suggest that strong flat baselines augmented with semantic weighting should become the new standard for prompt-optimization studies and lay the groundwork for broader, semantics-aware preference optimization in language models.

Figures

Figures reproduced from arXiv: 2507.20133 by the authors.

**Figure 1.** Comparison of prompt optimization methods for Human Preference metrics and semantic (CLIP) alignment. Legend shown above. SFT and DPO focus on maximizing HP but these methods lead to reducing semantic consistency. but often reduces semantic consistency, where outputs stray from the original prompt’s meaning (as measured by CLIP). This suggests that the DPOoptimized prompts, though more aligned with human prefere… view at source ↗

**Figure 2.** Concept Figure. DPO steers prompts into the preference region, while the ideal target lies at the intersection of preference and semantic regions. SemDPO is designed to reach that overlap (p ∗ DPO) that score high on preference metrics, they suffer from a critical flaw. Since DPO operates at a token level without an intrinsic understanding of meaning, its optimization path often exits the semantic consistency space… view at source ↗

**Figure 3.** Semantic–preference landscape of prompt-optimization methods across models and datasets. Scatter plots chart CLIP Score against human preference (PickScore and HPSv2.1) on three datasets with Qwen-1.5b (top) and GPT-2 (bottom) generators. Points nearer the upper-right indicate prompts excelling in both metrics. against standard and state-of-the-art prompt optimization methods, leveraging both automated alignment m… view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Head-to-head comparison of Sem-DPO against three baselines (SFT, Base DPO, and Human Input) across [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: Impact of semantic-weighting coefficient α on alignment metrics. Each point corresponds to a different α value (shown in labels), tracing the tradeoff between CLIP Score and human-preference metrics: PickScore (left) and HPS v2.1 (right). Hyperparameter Analysis. In p…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 2 canonical work pages

[5]

arXiv preprint arXiv:2406.18346

Ai alignment through reinforcement learning from human feedback? contradictions and limita- tions. arXiv preprint arXiv:2406.18346. Junru Lu, Siyu An, Min Zhang, Yulan He, Di Yin, and Xing Sun. 2024. Fipo: Free-form instruction- oriented prompt optimization with preference dataset and modular fine-tuning schema. arXiv preprint arXiv:2402.11811. Motoki Omu...

arXiv 2024
[6]

arXiv preprint arXiv:2411.07595

Entropy controllable direct preference opti- mization. arXiv preprint arXiv:2411.07595. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al

arXiv
[7]

Advances in Neural Information Processing Systems, 35:27730–27744

Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744. Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. 2024. Disentangling length from qual- ity in direct preference optimization. arXiv preprint arXiv:2403.19159. Rafael Rafailov, Archit Sharma, Eric Mitchell, S...

arXiv 2024
[8]

arXiv preprint arXiv:2305.18290

Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290. Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, C...

arXiv 2022
[9]

arXiv preprint arXiv:2411.04712

See-dpo: Self entropy enhanced direct prefer- ence optimization. arXiv preprint arXiv:2411.04712. Jiaming Song, Chenlin Meng, and Stefano Ermon. 2020. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. Haoran Sun, Lixin Liu, Junjie Li, Fengyu Wang, Bao- hua Dong, Ran Lin, and Ruohui Huang. 2024. Conifer: Improving complex constrained in...

arXiv 2020
[10]

arXiv preprint arXiv:2503.00030

Game-theoretic regularized self-play align- ment of large language models. arXiv preprint arXiv:2503.00030. Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik

arXiv
[11]

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 8228–8238

Diffusion model alignment using direct prefer- ence optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 8228–8238. Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. 2023. Beyond reverse kl: Generaliz- ing direct preference optimization with diverse diver- gence constraints. arXiv prepr...

arXiv 2023
[12]

Orthogonal Finetuning for Direct Preference Optimization

Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36. Chenxu Yang, Ruipeng Jia, Naibin Gu, Zheng Lin, Siyuan Chen, Chao Pang, Weichong Yin, Yu Sun, Hua Wu, and Weiping Wang. 2024a. Orthogonal finetuning for direct preference optimization. arXiv preprint arXiv:2409.14836....

work page Pith review arXiv 2024

Show all 12 references

[2022]

arXiv preprint arXiv:2204.05862

Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. Miaomiao Cai, Simiao Li, Wei Li, Xudong Huang, Hanting Chen, Jie Hu, and Yunhe Wang. 2025. Dspo: Direct semantic preference optimization for real-world i...

2025 arXiv
[2023]

rlhf: Scaling reinforcement learn- ing from human feedback with ai feedback

Rlaif vs. rlhf: Scaling reinforcement learn- ing from human feedback with ai feedback. arXiv preprint arXiv:2309.00267. Adam Dahlgren Lindström, Leila Methnani, Lea Krause, Petter Ericson, Íñigo Martínez de Rituerto de Troya, Dimitri Coelho Mollo, and Roel Dobbe

arXiv
[2024]

arXiv preprint arXiv:2406.13542

Self-play with execution feedback: Improving instruction-following capabilities of large language models. arXiv preprint arXiv:2406.13542. Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimizat...

2024 arXiv
[2025]

arXiv preprint arXiv:2504.02882

Diatool-dpo: Multi-turn direct preference opti- mization for tool-augmented large language models. arXiv preprint arXiv:2504.02882. Saeed Khaki, JinJin Li, Lan Ma, Liu Yang, and Prathap Ramachandra. 2024. Rs-dpo: A hybrid rejection sam- pling and direct preference optimization...

2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [5]

arXiv preprint arXiv:2406.18346

Ai alignment through reinforcement learning from human feedback? contradictions and limita- tions. arXiv preprint arXiv:2406.18346. Junru Lu, Siyu An, Min Zhang, Yulan He, Di Yin, and Xing Sun. 2024. Fipo: Free-form instruction- oriented prompt optimization with preference dataset and modular fine-tuning schema. arXiv preprint arXiv:2402.11811. Motoki Omu...

arXiv 2024

[2] [6]

arXiv preprint arXiv:2411.07595

Entropy controllable direct preference opti- mization. arXiv preprint arXiv:2411.07595. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al

arXiv

[3] [7]

Advances in Neural Information Processing Systems, 35:27730–27744

Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744. Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. 2024. Disentangling length from qual- ity in direct preference optimization. arXiv preprint arXiv:2403.19159. Rafael Rafailov, Archit Sharma, Eric Mitchell, S...

arXiv 2024

[4] [8]

arXiv preprint arXiv:2305.18290

Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290. Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. 2022. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, C...

arXiv 2022

[5] [9]

arXiv preprint arXiv:2411.04712

See-dpo: Self entropy enhanced direct prefer- ence optimization. arXiv preprint arXiv:2411.04712. Jiaming Song, Chenlin Meng, and Stefano Ermon. 2020. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502. Haoran Sun, Lixin Liu, Junjie Li, Fengyu Wang, Bao- hua Dong, Ran Lin, and Ruohui Huang. 2024. Conifer: Improving complex constrained in...

arXiv 2020

[6] [10]

arXiv preprint arXiv:2503.00030

Game-theoretic regularized self-play align- ment of large language models. arXiv preprint arXiv:2503.00030. Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik

arXiv

[7] [11]

In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 8228–8238

Diffusion model alignment using direct prefer- ence optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pages 8228–8238. Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. 2023. Beyond reverse kl: Generaliz- ing direct preference optimization with diverse diver- gence constraints. arXiv prepr...

arXiv 2023

[8] [12]

Orthogonal Finetuning for Direct Preference Optimization

Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36. Chenxu Yang, Ruipeng Jia, Naibin Gu, Zheng Lin, Siyuan Chen, Chao Pang, Weichong Yin, Yu Sun, Hua Wu, and Weiping Wang. 2024a. Orthogonal finetuning for direct preference optimization. arXiv preprint arXiv:2409.14836....

work page Pith review arXiv 2024