{"id":"fc711fbe-d073-4b58-9de5-be2988924dd5","arxiv_id":"2506.07492","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"EXPO is a pair of explicit preference-optimization losses that provably avoid DPO's uniform-regularization and poor-interpolation failure modes and outperform DPO on Anthropic HH and IMDb.","lead":"This paper introduces EXPO, two new loss functions for fine-tuning LLMs with human preferences that, unlike DPO, do not rely on an implicit reward model. The paper proves that DPO-style losses cannot simultaneously preserve strong reference behavior on some prompts and improve on others, while EXPO can, and shows higher win rates on real alignment benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3.1's no-free-lunch claim depends on Definition D.1's single-pair, equal-preference-per-prompt structure; the proof's 'identical arguments' step (Eq. 25-26) appears to conflate marginal and per-prompt optimization, so the claimed impossibility may not generalize to real preference data.","rationale":"The reader's weakest assumption identifies exactly the stylized data model (Definition D.1) as the delicate place, which matches my concern about transfer to real datasets. However, the reader couches it as a scope/transfer limitation, while my reading of the formal proof identifies a sharper, internal plausibility gap: the derivation of Theorem 3.1 appears to move from equality of the marginal preference probability across prompts to equality of the per-prompt minimizers in u, which is not justified when the response pairs and pi_ref(y|x) mass per response differ across prompts. This is not an accusation of error; it is a request for a concrete check. The empirical synthetic preservation test (Figure 5) only instantiates one pair per prompt, which is consistent with Definition D.1 and therefore cannot distinguish the general claim from the stylized construction. The real-world results (Figure 6, Table 2) do show positive EXPO performance and are honest evidence, but they do not test the structural impossibility theorem. The paper does creditably acknowledge the one-pair-per-prompt simplification in Definition D.1 ('this specification can naturally be relaxed... unnecessarily complicates the presentation'), which is a limitation statement I weigh explicitly. Given that the headline theoretical separation is the primary novelty and its proof has a potentially load-bearing gap, the CONDITIONAL verdict from the reader is appropriate; my concern would push toward requiring the authors to either repair the proof (or provide the missing per-prompt argument) or restrict the claims to the stylized setting. I do not see grounds for REJECT because the EXPO losses themselves, Proposition 4.1 (the gradient-estimation identity), and the synthetic demonstrations are independently checkable and appear sound; the concern is about the scope of the impossibility theorem, not about the internal validity of the EXPO construction.","tokens_in":25944,"tokens_out":2266,"duration_ms":24284,"concrete_test":"Reconstruct the counterexample behind Theorem 3.1 without Definition D.1's single-pair restriction: use at least two response pairs per prompt (so the per-prompt loss involves a sum over pairs) with the good and bad prompts sharing an identical pairwise preference probability p*(y1>y2|x)=0.8 but with different response spaces and pi_ref mass ratios, e.g., pi_ref(y_g1|x_g)=pi_ref(y_g2|x_g)=0.5 and pi_ref(y_b1|x_b)=0.9, pi_ref(y_b2|x_b)=0.1. Numerically minimize the DPO loss (7) in the tabular softmax parameterization. If the resulting minimizer satisfies dist on bad < dist(pi_ref,pi*) while dist on good = 0, Theorem 3.1 as stated fails; if it cannot, the proof's per-prompt independence claim is vindicated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central theoretical separation (Theorem 3.1 vs Proposition 4.2) rests on Definition D.1: exactly one response pair per prompt and identical pairwise preference probability across a good and a bad prompt (Definition D.1, items 2 and 4). In the proof, the loss decouples into per-prompt sums (Eq. 24), then the authors argue that if improving the bad prompt requires a nonzero optimal u(y1,y2,xbad), the same fixed (psi, mu, lambda) forces a nonzero u(y1,y2,xgood) because the per-prompt minimizers in Eq. 25 and Eq. 26 are 'identical arguments.' But this step conflates a marginal equivalence (the preference probability p*(y1>y2|x) is the same scalar for the two prompts) with full equivalence of the per-prompt objective. The per-prompt optimization variables are different: u(y1,y2,xgood) involves pi_theta(y|xgood)/pi_ref(y|xgood), while u(y1,y2,xbad) involves pi_theta(y|xbad)/pi_ref(y|xbad). Prompts may have disjoint response spaces (as in the synthetic preservation test), differ in the response pair {y1,y2}, and differ in pi_ref's relative mass on the two responses. A loss that at its optimum pulls u_bad away from zero can simultaneously keep u_good at zero because the induced per-prompt losses are not the same function of the single scalar u; the shared scalar p* alone does not identify the per-prompt subproblems. The formal proof as written asserts the contradiction rather than deriving it from the stated assumptions, so Theorem 3.1 is not established for realistic multi-pair, heterogeneous-preference datasets. If this concern lands, the abstract's broad claim that 'we prove that DPO-based objectives are subject to sub-optimal regularization and counter-intuitive interpolation behaviors' is supported only for a stylized two-prompt, one-pair-per-prompt construction, and the claimed practical force over DPO/IPO/f-DPO is weakened.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EXPO, a family of offline preference optimization losses that avoid the reparameterization trick used by DPO and related methods. The authors define a broad class of QPO losses (covering DPO, IPO, GPO, and f-DPO) and prove, under a stylized data model (Definition D.1), that any QPO loss cannot preserve an optimal policy on 'good' prompts while improving on 'bad' prompts (Theorem 3.1), and that no QPO loss satisfies the strong interpolation criterion (Theorem 3.6). They then propose two EXPO objectives (compositional and regression-based), prove that they satisfy the desired preservation and interpolation properties (Propositions 4.1–4.3), and provide synthetic and real-world experiments (Anthropic HH, IMDb, AlpacaEval 2) showing that EXPO outperforms DPO and IPO in win rates.","tokens_in":26390,"tokens_out":13244,"duration_ms":160224,"significance":"If the impossibility results are correct, they would establish fundamental structural limitations for a large family of popular offline preference optimization methods, beyond known issues such as lack of diversity or sensitivity to sampling. The EXPO losses are simple, computationally tractable, and do not rely on implicit reward models, making them attractive alternatives. The paper provides explicit proofs and reproducible code (stated in Appendix C), and the empirical evaluation on real datasets is a strength. However, the theoretical claims are currently proved only under restrictive assumptions that may not hold for typical preference datasets, and one of the central proofs (Theorem 3.6) is incomplete for the full QPO family as stated.","major_comments":[{"comment":"The proof relies crucially on Definition D.1, item 4, which requires the ground-truth preference probability p*(y1≻y2|x) to be identical for a good and a bad prompt. The proof then treats the per-prompt loss as a function of a scalar u and concludes that the minimizers for the good and bad prompts are identical. This is valid only if the policy class is unrestricted (so that u can be set independently per prompt) and if the per-prompt loss has a unique minimizer; neither condition is stated. More importantly, the theorem does not support the abstract's broad claim that 'DPO-based objectives are subject to sub-optimal regularization' for realistic datasets, where preference probabilities vary across prompts and multiple response pairs per prompt are common. Please either extend the theorem to more general data distributions or explicitly qualify the scope of the claim.","section":"Appendix D.1 (Theorem 3.1)"},{"comment":"The proof of Theorem 3.6 is only sketched for the case where the QPO loss depends on the log-ratio of policy and reference probabilities (i.e., μ=log). The argument introduces ρ as the log-ratio, but the general QPO loss in (10) uses an arbitrary monotonic μ, so the quantity being minimized is not necessarily a function of ρ. The step 'the only way for π̂θ to be independent of πref is if lim_{λ→0} B(λ)=±∞' is asserted without a rigorous justification, and the conclusion that only the WIC is achievable does not follow for arbitrary ψ, μ. Since Theorem 3.6 is a central impossibility result for the entire QPO family, it requires a complete proof or a restriction to a more specific subclass.","section":"Appendix D.4 (Theorem 3.6)"},{"comment":"The statement that 'minimizers of any valid f-DPO loss will be completely independent of πref for all λ∈(0,∞)' is derived from the assumption that f′ has an infinite limit at 0. Not all f-divergences used in f-DPO satisfy this property (e.g., those with bounded derivatives on their domain). The claim should be qualified to the specific divergence families for which the argument holds, or the proof should be generalized.","section":"Section 3.2, Appendix E.3 (f-DPO claim)"},{"comment":"Proposition 4.2 states that the EXPO minimizer preserves π*=πref on good prompts while improving bad prompts. The proof assumes that the policy class can represent π* on good prompts independently of the updates needed for bad prompts. In a shared-parameter model with limited capacity, the preservation guarantee is only approximate and the two objectives may interfere. The paper should explicitly state this limitation, as it affects the practical relevance of the theoretical separation.","section":"Section 4.3 (Proposition 4.2)"}],"minor_comments":[{"comment":"The word 'Perservation' in the list of desiderata is a typo; it should be 'Preservation'.","section":"Section 4.1"},{"comment":"The word 'Secion' in the sentence 'We also adopt the ℓc_EXPO loss from Secion 4.1' is a typo.","section":"Appendix C.1"},{"comment":"The caption repeats 'within dbad_x' twice; the second occurrence should refer to 'dgood_x' when describing the unwanted gap.","section":"Figure 1 caption"},{"comment":"The Strong Interpolation Criteria uses 'arg minπθ' without addressing the possibility of multiple minimizers. Please specify how ties are resolved, since the proofs (e.g., Theorem 3.1) implicitly require uniqueness.","section":"Definition 3.2"},{"comment":"The win-rate results in Figure 6 and Table 2 are reported without confidence intervals or statistical significance tests. Given the relatively small differences (e.g., DPO vs EXPO on AlpacaEval LC), error bars would strengthen the empirical claims.","section":"Section 5 (real-world experiments)"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important question and the proposed EXPO method is a credible contribution. However, the theoretical centerpiece currently overclaims relative to what is proven. The proof of Theorem 3.1 depends on strong equal-preference-probability assumptions, and Theorem 3.6 is not actually proved for the full QPO family as stated. The authors should either tighten the theory or revise the claims. The empirical results are suggestive but lack error bars, and the comparison to the strongest baselines (e.g., SimPO, KTO) is limited to one dataset. I would condition acceptance on a careful revision that aligns the claims with the proofs."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper is worth refereeing. The formal criteria (SIC/WIC) and the preservation result are new and useful framing; the EXPO losses are simple, well-motivated, and beat DPO/IPO on the reported benchmarks. The stress-test worry about Theorem 3.1's 'identical arguments' step does not land: under Definition D.1, the per-prompt loss genuinely is a function of the single scalar u, so the minimization argument is sound. The real soft spot is elsewhere: the preservation theorem only covers a stylized data model (one pair per prompt, equal p* across a matched good/bad pair), and the abstract's 'we prove that DPO-based objectives are subject to...' is broader than the theorem. The paper is transparent about the single-pair assumption in a parenthetical, but the framing encourages over-reading.\n\nWhat is genuinely new: the SIC/WIC distinction and the impossibility result that no QPO loss can satisfy SIC are clean, and the regularizer-preservation tradeoff is a real phenomenon. Propositions 4.2 and 4.3 show the EXPO losses satisfy the desiderata by construction; the derivation of the regression loss via Proposition 4.1 is self-contained and correct.\n\nSoft spots in proportion: (1) The preservation theorem's assumptions are narrow; the generalization argument in the footnote ('can naturally be relaxed') is asserted but not shown, so the practical force over DPO/IPO/f-DPO for real datasets is weaker than the abstract suggests. (2) The empirical section reports win rates without error bars or seeds; one-epoch training on Anthropic HH and IMDb may be noisy. A commit hash for the code would make the reproducibility claim checkable. (3) The compositional EXPO loss is essentially supervised KL plus KL-to-reference, and the regression loss is structurally close to IPO; the paper is upfront about this, and the distinct derivation and lambda-in-[0,1] behavior justify the contribution, but the 'explicit vs implicit reward' framing is stronger than the actual distance.\n\nWho this is for: anyone working on preference optimization or DPO variants. The SIC/WIC criteria and the preservation example are likely to be reused. I'd cite it. It deserves a serious referee; I'd accept it and ask for softened abstraction claims, error bars, and a commit hash.","headline":"Solid, honest paper with a real formal contribution (SIC/WIC) and positive wins; the preservation theorem is narrower than the abstract admits, but the proof is correct under its stated assumptions.","tokens_in":26949,"tokens_out":5399,"would_cite":true,"duration_ms":60817,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proves that DPO-style preference losses cannot preserve already-optimal behavior while improving weak regions, and introduces explicit losses—EXPO—that provably can.","keywords":["preference optimization","DPO","implicit reward","RLHF","LLM alignment","regularization","interpolation","EXPO"],"falsifier":"Run the paper's synthetic bandit test with three responses and ground-truth preferences; if, at $\\lambda\\to 0$, DPO or IPO converge to the full optimal policy $\\pi^*$ rather than its mode $\\pi_\\delta$, then Propositions 3.4 and 3.5 are wrong. Alternatively, on a dataset matching Definition D.1, find a QPO loss whose minimizer improves bad prompts while leaving $\\pi^*$ untouched on good prompts, which would refute Theorem 3.1.","tokens_in":25764,"feed_emoji":"🎯","tokens_out":9142,"duration_ms":96909,"temperature":0.7,"pith_summary":"Direct preference optimization (DPO) and its offshoots replace the two-stage RLHF pipeline with a single loss that embeds an implicit reward. The paper claims that this reparameterization leaves a hidden structural cost: any loss in the broad quasi-convex preference optimization (QPO) family, covering DPO, IPO, GPO, and f-DPO, applies the same regularization everywhere, so improving prompts where the reference model is weak necessarily degrades prompts where it is already optimal, and as the regularization weight $\\lambda\\to 0$ these losses converge to a degenerate policy concentrated on the mode of the optimal policy rather than to the optimal policy itself. The paper then constructs explicit preference optimization (EXPO) losses, $\\ell^c_{\\mathrm{EXPO}}$ and $\\ell^r_{\\mathrm{EXPO}}$, that make the regularization explicit instead of implicit, and proves they preserve optimal behavior on good prompts and interpolate cleanly. A reader should care because the same structural flaw is shared by the most common alignment methods, and the proposed alternative is a drop-in objective that also wins more head-to-head comparisons in the paper's experiments.","feed_headline":"DPO cannot improve weak prompts without hurting strong ones","feed_subtitle":"A new explicit-preference loss, EXPO, provably preserves good behavior and beats DPO and IPO on real alignment data.","key_machinery":"The load-bearing object is the QPO loss family, defined by a quasi-convex shape function $\\psi$ and a monotone transform $\\mu$ of policy-to-reference ratios; it subsumes DPO, IPO, GPO, and f-DPO, and the paper uses it to pin the structural failures on the shared implicit-reward reparameterization. The constructive machinery is EXPO's explicit objective: $\\ell^c_{\\mathrm{EXPO}}$ combines a supervised KL divergence between the ground-truth and induced preference distributions with an unsupervised KL to the reference policy, while $\\ell^r_{\\mathrm{EXPO}}$ regresses the induced preference probability onto a weighted average of reference and ground-truth preference probabilities. Both are computable from observed data without an implicit reward, and their optima recover the BT-optimal policy when the reference is already optimal.","core_discovery":"The paper's central discovery is a separation between the QPO family and EXPO. For any quasi-convex preference loss of the form (10), Theorem 3.1 shows that, under a stylized dataset in which the reference policy $\\pi_{\\mathrm{ref}}$ already equals the BT-optimal policy $\\pi^*$ on a `good` prompt partition, any minimizer that improves over $\\pi_{\\mathrm{ref}}$ on `bad` prompts must also move away from $\\pi^*$ on good prompts. Theorem 3.6 shows that no such loss can satisfy the strong interpolation criteria, because the reparameterized losses can only reach the degenerate mode $\\pi_\\delta$ as $\\lambda\\to 0$. The proposed EXPO losses instead minimize either a KL divergence between the true and induced preference distributions plus a KL to $\\pi_{\\mathrm{ref}}$ (compositional), or a squared regression of the induced preference probability onto a weighted average of the reference and ground-truth preference probabilities (regression); Proposition 4.2 and Proposition 4.3 establish preservation and strong interpolation for both.","pith_inferences":["Editorial extension: the paper's SIC/WIC distinction offers a cheap screening test for any new preference loss; train it on a small bandit problem with known ground truth and inspect the $\\lambda\\to 0$ limit before scaling up.","Editorial extension: because both EXPO losses operate on preference probabilities rather than log reward ratios, they could be adapted to non-Bradley-Terry labels such as ordinal or multi-graded preferences, an extension the paper does not explore.","Editorial extension: the preservation result predicts that DPO-family models will show measurable degradation, after alignment, on prompts where the reference model is already strong; stratifying real-data win rates by initial reference quality would test this."],"forward_implications":["If the theorems are correct, tuning $\\lambda$ inside DPO, IPO, GPO, or f-DPO cannot resolve the trade-off between improving weak prompts and preserving strong prompts; a loss outside the QPO family is required.","If EXPO objectives behave as claimed, practitioners can train preference-aligned LLMs with a single explicit loss and unbiased SGD gradients, without reward model training or RLHF reparameterizations.","If the real-world results replicate, EXPO's compositional variant can incorporate unlabeled prompt data, giving offline methods access to a source of signal that was previously used mainly by online RLHF.","If EXPO satisfies the strong interpolation criterion, users can set $\\lambda$ to control a meaningful trade-off between faithful preference matching and reference-model conservatism, instead of moving toward a degenerate mode."],"supporting_citations":[{"why":"Defines the DPO loss through the implicit-reward reparameterization; this is the primary object whose limitations Theorems 3.1 and 3.6 target.","marker":"Rafailov et al. (2024)"},{"why":"Defines the IPO loss used as another QPO special case and supplies the bandit experimental setup used for interpolation tests.","marker":"Azar et al. (2024)"},{"why":"Defines GPO, included in the QPO family that the impossibility theorems cover.","marker":"Tang et al. (2024)"},{"why":"Defines f-DPO, another QPO family member, and reports that reverse-KL (DPO) performs best among f-divergences, which the paper uses to justify baseline choices.","marker":"Wang et al. (2024a)"},{"why":"Provides the Anthropic HH preference dataset used in real-world win-rate experiments and cites the empirical sampling regime with one response pair per prompt.","marker":"Bai et al. (2022a)"},{"why":"Supplies the red-teaming companion data to the HH dataset, also used in the real-world evaluation.","marker":"Ganguli et al. (2022)"},{"why":"Shows that common learning constraints obscure DPO's link to RLHF, motivating EXPO's independence from implicit-reward reparameterizations.","marker":"Kong et al. (2025)"},{"why":"Supplies the quasi-convexity definition that defines the QPO loss family.","marker":"Greenberg and Pierskalla (1971)"}],"fun_headline_variants":["EXPO: explicit preference loss, no implicit reward model needed","Proven: DPO hurts strong prompts; EXPO avoids the trap","Explicit preference optimization beats DPO without hidden rewards","No reparameterization trick: EXPO provably preserves strong prompts","EXPO outperforms DPO and IPO while keeping good prompts intact"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything about the preservation and interpolation theorems hangs on a stylized data model: one response pair per prompt, a clean partition into good and bad prompts, an already-optimal reference on good prompts, and equal preference probabilities across the two groups; real preference data can violate any of these.","fun_headline_variants_meta":{"raw":{"variants":["EXPO: explicit preference loss, no implicit reward model needed","Proven: DPO hurts strong prompts; EXPO avoids the trap","Explicit preference optimization beats DPO without hidden rewards","No reparameterization trick: EXPO provably preserves strong prompts","EXPO outperforms DPO and IPO while keeping good prompts intact"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000216,"raw_usage":{"total_tokens":1465,"prompt_tokens":1011,"completion_tokens":454,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":366}},"tokens_in":627,"tokens_out":454,"duration_ms":5909,"temperature":1.0,"reasoning_tokens":366,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:34:02.396703+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the paper's synthetic bandit test with three responses and ground-truth preferences; if, at $\\lambda\\to 0$, DPO or IPO converge to the full optimal policy $\\pi^*$ rather than its mode $\\pi_\\delta$, then Propositions 3.4 and 3.5 are wrong. Alternatively, on a dataset matching Definition D.1, find a QPO loss whose minimizer improves bad prompts while leaving $\\pi^*$ untouched on good prompts, which would refute Theorem 3.1.","supporting_citations":[{"cited_title":"Common learning constraints alter interpretations of direct preference optimization","cited_arxiv_id":null,"evidence_quote":"Shows that common learning constraints obscure DPO's link to RLHF, motivating EXPO's independence from implicit-reward reparameterizations."},{"cited_title":"and Pierskalla, W","cited_arxiv_id":null,"evidence_quote":"Supplies the quasi-convexity definition that defines the QPO loss family."}],"review_version":1}