Pith. sign in

REVIEW 4 major objections 6 minor 69 references

This paper claims that preference-over-preference annotations—ordinal comparisons of which preference reflects a stronger distinction—can replace scalar preference scores as a source of adaptive margins in RLHF, and that DPO-PoP, which infe

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 14:47 UTC pith:3QNL2SYJ

load-bearing objection Genuinely new PoP margin construction, but the main empirical claim is confounded by a doubled training budget that the paper acknowledges but never controls. the 4 major comments →

arxiv 2509.22851 v4 pith:3QNL2SYJ submitted 2025-09-26 cs.LG cs.AIcs.CL

Adaptive Margin RLHF via Preference over Preferences

Yaswanth Chittepu , Prasann Singhal , Greg Durrett , Scott Niekum This is my paper
classification cs.LG cs.AIcs.CL
keywords preference over preferencesRLHFdirect preference optimizationadaptive marginsreward modelingpreference strengthordinal annotationalignment
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes a new form of supervision for RLHF alignment: instead of asking annotators to score responses or preferences, ask them to compare two preferences and say which one reflects a stronger contrast. It uses these preference-over-preference (PoP) labels to set an adaptive margin for each preference datapoint inside the direct preference optimization (DPO) loss: the weaker preference's reward gap becomes a lower bound on the stronger preference's margin. On the UltraFeedback dataset, the random-sampling variant (DPO-PoP-random) outperforms vanilla DPO, DPO with a fixed margin, and DPO with ground-truth margins on RewardBench, win rate, and AlpacaEval; the iterative-sampling variant achieves the highest in-domain preference classification accuracy. The paper also documents a tradeoff: aggressively modeling weak preferences improves in-domain classification but harms generative quality. The practical upshot is that ordinal 'which preference is stronger?' labels, which are easier to give reliably than numeric scores, can power margin-aware alignment without scalar ratings.

Core claim

The central claim is that preference strength can be supervised ordinally and used directly as a margin in DPO. Given two preference pairs (A better than B and C better than D), a PoP label saying the first preference is stronger is interpreted as the reward gap r(A)-r(B) exceeding r(C)-r(D). DPO-PoP enforces this by subtracting the weaker preference's implicit reward gap, computed with a stop-gradient on a slowly-updated target policy and clipped to a fixed range, from the stronger preference's gap inside a log-sigmoid. The paper reports that models trained this way, especially with randomly sampled PoP pairs, beat DPO variants that have access to true margin values, suggesting that ordinal

What carries the argument

The central object is the preference-over-preference (PoP) comparison and its translation into a margin inequality. A PoP label (A≻B)≻(C≻D) states that the reward gap for A-vs-B is larger than for C-vs-D. DPO-PoP converts this into a supervised loss by treating the weaker preference's implicit reward gap as a lower-bound margin for the stronger preference, using a stop-gradient so the weaker pair is not inverted, clipping the margin to a bounded interval for stability, and computing the margin with a Polyak-averaged target policy. This machinery replaces scalar margin sources with ordinal ones.

Load-bearing premise

The iterative sampling variant assumes that, before collecting a PoP label, one can select comparison preferences in which the anchor preference is already known to be stronger; without that oracle knowledge, its reported accuracy gains are not practically reproducible, and the approach's real-world relevance rests on whether random sampling with human or LLM labels retains the benefit.

What would settle it

Train DPO-PoP-random on PoP labels collected from human annotators who never see ground-truth preference scores, and compare it to DPO-margin-gt on a held-out reward benchmark and a generative win-rate evaluation; if the resulting model does not at least match DPO-margin-gt's generative win rate, the claim that PoP supervision beats scalar ground-truth margins would be contradicted. Similarly, implement iterative sampling without oracle margin knowledge by first asking annotators for PoP labels and then selecting pairs based on those labels; if the classification accuracy gain of DPO-PoP-iter

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, DPO-PoP-random achieves higher generative win rates and RewardBench overall scores than vanilla DPO, fixed-margin DPO, and DPO with ground-truth margins (e.g., 62.39% vs 61.25% win rate against vanilla DPO; RewardBench 78.87 vs 77.45 on Llama-3.2-3b).
  • DPO-PoP-iter, which uses iterative sampling to represent every preference equally against weaker ones, achieves the highest in-domain preference classification accuracy on UltraFeedback (79.97% vs 71.85% for ground-truth-margin DPO), but at a measurable cost in generative quality.
  • PoP annotations collected from an LLM annotator with random sampling also improve generative win rate over baselines (65% vs 58% for ground-truth-margin DPO), showing the approach does not require oracle margin scores to help.
  • The paper's generalization bound for adaptive-margin classifiers implies that the distribution of margins in the training set affects the bound: random sampling, which over-represents strong preferences, yields a tighter bound than iterative sampling, which over-represents weak preferences.
  • Practitioners face a stable discriminative-generative tradeoff: iterative sampling is preferable when the goal is in-domain preference classification, while random sampling is preferable when generative quality and robustness on out-of-domain benchmarks are the priority.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural extension not developed in the paper is to drop the same PoP-to-margin construction into other direct alignment objectives (e.g., IPO or SLiC-style losses) and into the reward-modeling stage of RLHF, since the margin inequality in the loss is not specific to DPO.
  • If PoP labels prove easier for humans to provide reliably than scalar ratings, the approach could make adaptive-margin alignment practical in human-annotation pipelines; this is testable by measuring annotation reliability and downstream performance on human PoP labels rather than LLM labels.
  • The documented discriminative-generative tradeoff suggests that in-domain preference classification accuracy is a misleading proxy for alignment quality; out-of-domain reward benchmarks appear to better capture what practitioners mean by 'better alignment.'
  • Because random sampling weights stronger preferences more heavily, DPO-PoP-random can be viewed as a bootstrapped, margin-aware version of scaled Bradley-Terry; a promising test is whether explicitly importance-sampling preferences by inferred strength reproduces or exceeds the same gains with fewer labels.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes preference-over-preference (PoP) supervision---an ordinal signal indicating which of two preference pairs reflects a stronger distinction---and uses it to define adaptive per-datapoint margins for DPO. The resulting DPO-PoP loss (Eq. 7) computes a margin from the weaker preference's implicit reward difference, using a stop-gradient on a Polyak-averaged target policy and clipping to [0, Mmax]. Experiments on UltraFeedback use synthetic PoP labels constructed from ground-truth margins and, in a separate section, labels from GPT-4.1-mini. The paper reports that DPO-PoP variants improve test classification accuracy, RewardBench, AlpacaEval-2, and UltraRM win rates over vanilla DPO, fixed-margin DPO, and DPO with ground-truth margins. It also documents a discriminative/generative tradeoff between two sampling strategies and provides a Rademacher generalization bound for adaptive-margin linear classifiers (Appendix H).

Significance. If the empirical attribution is correct, the contribution is practically interesting: it offers a way to obtain margin information from ordinal comparisons rather than scalar ratings, which may be easier for annotators, and it provides a concrete mechanism for using such information in a DPO-style objective. The synthetic experiments are run with multiple seeds and confidence intervals, and the paper is transparent enough to acknowledge the doubled training budget in Appendix G. The generalization bound in Appendix H is a useful formalization of why the distribution of margins matters. However, the central empirical claim is currently not established because the DPO-PoP training budget is not matched to the baselines, and the iterative sampling strategy presupposes oracle access to margin orderings. Both issues are fixable but load-bearing.

major comments (4)
  1. [§4.1.1, Eq. (7), Appendix G] The reported gains are confounded by an unequal training budget. With k=2, DPO-PoP trains on |DPoP|=2|Dpref| examples, and each PoP example in Eq. (7) exposes the model to two preference comparisons (the stronger and the weaker pair), so the effective number of preference-pair updates is roughly twice that of every DPO baseline. The Fig. 5 caption acknowledges 'the effective training budget is doubled', but no budget-matched control is run. Tables 1–4 and the LLM-annotator Tables 5–8 therefore do not establish that PoP supervision itself, rather than additional data and gradient steps, produces the improvements. A control that trains DPO/DPO-margin-gt on the same number of preference-pair examples (e.g., two epochs or duplicated data) is needed; alternatively, a k=1 PoP setting matched in preference-pair count would address the concern.
  2. [§4.1.1, iterative sampling] The described construction of the iterative PoP dataset is not implementable without oracle margins. The text says: 'We only choose k preference pairs in which our chosen preference is judged to be stronger than the comparative preference.' For a human or LLM annotator, the label is unknown until after the comparison, so this selection cannot be made in advance unless ground-truth margins are available. Consequently, the large in-domain accuracy gain of DPO-PoP-iter over DPO-margin-gt (Table 1: 79.97 vs. 71.85) rests on an impractical oracle. The paper should either provide a feasible selection procedure (e.g., using a reward model to rank preferences first) or explicitly reposition DPO-PoP-iter as a synthetic-only analysis with limited practical relevance.
  3. [§4.2, Tables 5–8] The LLM-annotator experiments are missing basic statistical reporting and inherit the same budget confound. Tables 5–8 report no confidence intervals and no stated number of seeds, so the large win-rate gap in Table 7 (65% vs. 58%) and the RewardBench differences in Table 6 cannot be distinguished from noise. Additionally, DPO-PoP-random is again trained with k=2, i.e., on twice the number of preference-pair examples as the baselines. Multi-seed results with intervals and a budget-matched baseline are required before the realism claim (Q3) can be evaluated.
  4. [Appendix H, Theorem 1] The theoretical bound in Eq. (19) treats margins m_i as fixed, exogenous quantities, whereas in DPO-PoP the margin comes from a target policy and is clipped. The paper does not show that the bound transfers to that adaptive setting. This weakens the 'theoretical prediction' used in Appendix I to explain the random-vs-iterative tradeoff; as written, the bound is an argument for adaptive-margin classifiers with pre-specified margins, not a proof about DPO-PoP. I recommend stating this explicitly so the theory is not overclaimed.
minor comments (6)
  1. [Eq. (4)] The margin-based reward modeling loss has an unmatched parenthesis in the log-sigmoid expression: `logσ(r_φ(x,y+)−r_φ(x,y−)−m(x,y+,y−)` is missing a closing parenthesis for the σ argument.
  2. [§1] Typo: 'Comparitive annotation' should be 'Comparative annotation'.
  3. [§4.1.1] The statement 'after generating the PoP dataset, we discard the original scalar scores' is potentially confusing in the synthetic setting, where the PoP labels are generated from those scores. Clarify that scalar scores are discarded only at training time, not during dataset construction.
  4. [§3, Eq. (7)] The target policy π̂_θ is introduced, but the Polyak averaging update rule is not specified. Please state the update (e.g., θ̂ ← τθ + (1−τ)θ̂) and the value of τ used.
  5. [Appendix J] Case study outputs contain several typos and OCR-like artifacts, e.g., 'depictsthe', 'authorś', and 'Torchyś'. Please proofread the appendix outputs.
  6. [General] The code link is listed as 'removed for review'; if the paper is published, the reproducibility statement should include a permanent link or repository identifier.

Circularity Check

0 steps flagged

No significant circularity: DPO-PoP uses a detached, clipped bootstrap margin and evaluates on held-out data; self-citations are background only.

full rationale

The paper's derivation chain is self-contained. DPO-PoP (Eq. 7) optimizes a standard DPO-style loss in which the margin is the clipped, stop-gradded implicit reward difference of the weaker preference computed from a slowly-updated target policy. This is a bootstrap target, not a fitted parameter: the margin is detached and clipped, and the gradient flows only through the stronger preference term, so the model does not directly set its own loss. PoP labels are used only to order preference pairs; the continuous margin magnitude comes from the model itself, which the paper explicitly acknowledges in Section 5 ('bootstrapped version of the Scaled Bradley-Terry loss'). Synthetic PoP construction (Section 4.1.1) uses ground-truth margins to generate ordinal labels, but those scalar scores are discarded ('we discard the original scalar scores and do not use them at any stage of model training'), and all headline evaluations are on held-out test splits or external benchmarks (RewardBench, AlpacaEval-2), so no prediction reduces to a training input. Self-citations [6,25] are background references (DAA naming, safety-constrained RLHF) and are not load-bearing for the PoP claim. The doubled effective training set for k=2 (Appendix G) is a potential confound but is not a circularity: it changes the amount of data, not the logical relation between inputs and predictions. The iterative-sampling implementation concern is practical, not circular, because the paper proposes post-hoc selection after annotation rather than claiming oracle access in the loss itself. No equation in the paper is equivalent to its own input by construction.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The method depends on several hyperparameters and assumptions. Most notably, the PoP dataset size multiplier k is a hand-chosen hyperparameter (default 2) that controls the effective training budget; the margin clipping constant Mmax and the margin-difference exclusion threshold are also set without principled justification. The theoretical bound in Appendix H assumes fixed, known margins, which is not the case in the actual algorithm. The core assumptions are the Bradley-Terry preference model, the validity of reward differences as preference strength, and the accuracy of DPO implicit rewards as estimates of true rewards.

free parameters (4)
  • k, PoP dataset multiplier = 2 (default), swept 1-16 in Appendix E
    Controls the size of the PoP dataset relative to the preference dataset; k=2 doubles the training data and is chosen based on downstream performance in Appendix E.
  • Mmax, margin clipping threshold = 10
    Chosen for optimization stability; no ablation reported.
  • Margin-difference exclusion threshold = 1
    Pairs with ground-truth margin difference below 1 are discarded (Section 4.1.1); this is a data-selection hyperparameter not justified by theory.
  • Polyak averaging coefficient (target policy update rate) = not reported
    The paper states margins are computed from a slowly-updated target policy via Polyak averaging but does not specify the averaging coefficient, leaving the algorithm under-specified.
axioms (5)
  • standard math Bradley-Terry model of preferences
    Section 2.1, Eq. 1; the entire reward modeling and DPO derivation assumes the BT likelihood.
  • domain assumption Preference strength can be measured by reward difference
    Section 3: PoP label r(A)-r(B) > r(C)-r(D) requires that reward difference is a monotone proxy for perceived strength; this is assumed, not tested.
  • domain assumption DPO implicit rewards approximate true rewards
    Section 2.2 and Eq. 6: margins are computed from beta log(pi_theta/pi_ref) on the weak example; if implicit rewards are miscalibrated, the margin targets are wrong.
  • domain assumption i.i.d. sample and known margins in theory
    Appendix H, Eq. 12-13: the generalization bound assumes margins m_i are known constants, but in DPO-PoP they are estimated online from the target policy.
  • ad hoc to paper The stop-gradient margin is a valid reference
    Section 3, Eq. 5-7: the stop-gradient prevents gradient gaming but creates a dynamic target that depends on the model itself; this is an algorithmic design choice not grounded in a principled objective.

pith-pipeline@v1.3.0-alltime-deepseek · 23110 in / 16291 out tokens · 103117 ms · 2026-08-04T14:47:08.819174+00:00 · methodology

0 comments
read the original abstract

Margin-based optimization is fundamental to improving generalization and robustness in classification tasks. In the context of reward model learning from preferences within Reinforcement Learning from Human Feedback (RLHF), existing methods typically rely on no margins, fixed margins, or margins that are simplistic functions of preference ratings. However, such formulations often fail to account for the varying strengths of different preferences or they rely on noisy margin information derived from preference ratings. Furthermore, many existing methods that use adaptive margins assume access to accurate preference scores, which can be difficult for humans to provide reliably. We propose leveraging preferences over preferences, that is, annotations indicating which of two preferences reflects a stronger distinction, to infer adaptive margins on a per-datapoint basis. Such preference-over-preference annotations are general and can be incorporated into both standard RLHF reward modeling objectives and direct alignment losses. As a concrete instantiation, we introduce DPO-PoP, an extension to Direct Preference Optimization (DPO) that incorporates adaptive margins from preference-over-preference supervision, enabling improved discriminative and generative performance. Additionally, we show a tradeoff between discriminative and generative performance and propose two sampling strategies for gathering preference-over-preference labels to navigate it.

Figures

Figures reproduced from arXiv: 2509.22851 by Greg Durrett, Prasann Singhal, Scott Niekum, Yaswanth Chittepu.

Figure 1
Figure 1. Figure 1: A pictorial illustration of the PoP framework. A [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Cumulative Accuracy vs Margin for the different DPO variants considered. Lower [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Spearman and Pearson correlations (left), and test classification accuracy (right) of DPO [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Win rates (left) and median advantage (right) of DPO-PoP models trained with varying [PITH_FULL_IMAGE:figures/full_fig_p018_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Training curves for test classification accuracy, UltraRM-winrate, and KL with respect to [PITH_FULL_IMAGE:figures/full_fig_p019_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

69 extracted references · 1 canonical work pages

  1. [1]

    Amini, T

    A. Amini, T. Vieira, and R. Cotterell. Direct preference optimization with an offset.arXiv preprint arXiv:2402.10571, 2024

    Pith/arXiv arXiv 2024

  2. [2]

    M. G. Azar, M. Rowland, B. Piot, D. Guo, D. Calandriello, M. Valko, and R. Munos. A general theoretical paradigm to understand learning from human preferences, 2023. URL https://arxiv.org/abs/2310.12036

    Pith/arXiv arXiv 2023

  3. [3]

    Bousquet, S

    O. Bousquet, S. Boucheron, and G. Lugosi. Introduction to statistical learning theory. In Advanced Lectures on Machine Learning, 2004. URL https://api.semanticscholar. org/CorpusID:669378

    2004

  4. [4]

    R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons.Biometrika, 39(3/4):324–345, 1952

    1952

  5. [5]

    Burton, M

    N. Burton, M. Burton, D. Rigby, C. A. Sutherland, and G. Rhodes. Best-worst scaling improves measurement of first impressions.Cognitive research: principles and implications, 4(1):36, 2019

    2019

  6. [6]

    Chittepu, B

    Y . Chittepu, B. Metevier, W. Schwarzer, A. Hoag, S. Niekum, and P. S. Thomas. Reinforcement learning from human feedback with high-confidence safety constraints, 2025. URL https: //arxiv.org/abs/2506.08266

    Pith/arXiv arXiv 2025

  7. [7]

    Cortes and V

    C. Cortes and V . Vapnik. Support-vector networks.Machine learning, 20:273–297, 1995

    1995

  8. [8]

    G. Cui, L. Yuan, N. Ding, G. Yao, B. He, W. Zhu, Y . Ni, G. Xie, R. Xie, Y . Lin, Z. Liu, and M. Sun. Ultrafeedback: Boosting language models with scaled ai feedback, 2024. URL https://arxiv.org/abs/2310.01377

    Pith/arXiv arXiv 2024

  9. [9]

    J. Dai, X. Pan, R. Sun, J. Ji, X. Xu, M. Liu, Y . Wang, and Y . Yang. Safe rlhf: Safe reinforcement learning from human feedback, 2023. URLhttps://arxiv.org/abs/2310.12773

    Pith/arXiv arXiv 2023

  10. [10]

    Dubois, B

    Y . Dubois, B. Galambosi, P. Liang, and T. B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators, 2025. URLhttps://arxiv.org/abs/2404.04475

    Pith/arXiv arXiv 2025

  11. [11]

    Freund, R

    Y . Freund, R. E. Schapire, et al. Experiments with a new boosting algorithm. Inicml, volume 96, pages 148–156. Citeseer, 1996

    1996

  12. [12]

    Grattafiori et al

    A. Grattafiori et al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407. 21783

    2024

  13. [13]

    M. L. Ha and V . Blanz. Deep ranking with adaptive margin triplet loss.arXiv preprint arXiv:2107.06187, 2021

    Pith/arXiv arXiv 2021

  14. [14]

    J. C. Handley. Comparative analysis of bradley-terry and thurstone-mosteller paired comparison models for image quality assessment. InPICS, volume 1, pages 108–112, 2001

    2001

  15. [15]

    Herbrich and J

    R. Herbrich and J. Weston. Adaptive margin support vector machines for classification. In1999 Ninth International Conference on Artificial Neural Networks ICANN 99. (Conf. Publ. No. 470), volume 2, pages 880–885 vol.2, 1999. doi: 10.1049/cp:19991223

    work page doi:10.1049/cp:19991223 1999

  16. [16]

    Huang, Y

    Y . Huang, Y . Wang, Y . Tai, X. Liu, P. Shen, S. Li, J. Li, and F. Huang. Curricularface: adaptive curriculum learning loss for deep face recognition. Inproceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5901–5910, 2020

    2020

  17. [17]

    Jaques, A

    N. Jaques, A. Ghandeharioun, J. H. Shen, C. Ferguson, À. Lapedriza, N. J. Jones, S. S. Gu, and R. W. Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog.ArXiv, abs/1907.00456, 2019. URL https://api.semanticscholar. org/CorpusID:195766797

    Pith/arXiv arXiv 1907

  18. [18]

    Kiritchenko and S

    S. Kiritchenko and S. M. Mohammad. Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation.arXiv preprint arXiv:1712.01765, 2017. 11

    Pith/arXiv arXiv 2017

  19. [19]

    Lambert, V

    N. Lambert, V . Pyatkin, J. Morrison, L. Miranda, B. Y . Lin, K. Chandu, N. Dziri, S. Kumar, T. Zick, Y . Choi, N. A. Smith, and H. Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024. URLhttps://arxiv.org/abs/2403.13787

    Pith/arXiv arXiv 2024

  20. [20]

    Ledoux and M

    M. Ledoux and M. Talagrand.Probability in Banach Spaces: Isoperimetry and Processes. Springer, 1991. URLhttps://api.semanticscholar.org/CorpusID:118526268

    1991

  21. [21]

    Ouyang, J

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022

    2022

  22. [22]

    X. B. Peng, A. Kumar, G. Zhang, and S. Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning, 2019. URL https://arxiv.org/abs/1910. 00177

    2019

  23. [23]

    Peters and S

    J. Peters and S. Schaal. Reinforcement learning by reward-weighted regression for operational space control. InProceedings of the 24th international conference on Machine learning, pages 745–750, 2007

    2007

  24. [24]

    B. Qin, D. Feng, and X. Yang. Towards understanding the influence of reward margin on preference model performance, 2024. URLhttps://arxiv.org/abs/2404.04932

    Pith/arXiv arXiv 2024

  25. [25]

    Rafailov, Y

    R. Rafailov, Y . Chittepu, R. Park, H. Sikchi, J. Hejna, B. Knox, C. Finn, and S. Niekum. Scaling laws for reward model overoptimization in direct alignment algorithms, 2024. URL https://arxiv.org/abs/2406.02900

    Pith/arXiv arXiv 2024

  26. [26]

    Rafailov, A

    R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct preference optimization: Your language model is secretly a reward model, 2024. URLhttps://arxiv. org/abs/2305.18290

    Pith/arXiv arXiv 2024

  27. [27]

    R. E. Schapire, Y . Freund, P. Bartlett, and W. S. Lee. Boosting the margin: A new explanation for the effectiveness of voting methods.The annals of statistics, pages 1651–1686, 1998

    1998

  28. [28]

    Shalev-Shwartz and S

    S. Shalev-Shwartz and S. Ben-David.Understanding machine learning: From theory to algorithms. Cambridge university press, 2014

    2014

  29. [29]

    Stiennon, L

    N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, C. V oss, A. Radford, D. Amodei, and P. Christiano. Learning to summarize from human feedback, 2022. URL https://arxiv. org/abs/2009.01325

    Pith/arXiv arXiv 2022

  30. [30]

    L. L. Thurstone. A law of comparative judgment. InScaling, pages 81–92. Routledge, 2017

    2017

  31. [31]

    Touvron, L

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Pith/arXiv arXiv 2023

  32. [32]

    Wadhwa, J

    M. Wadhwa, J. Chen, J. J. Li, and G. Durrett. Using natural language explanations to rescale human judgments, 2024. URLhttps://arxiv.org/abs/2305.14770

    Pith/arXiv arXiv 2024

  33. [33]

    B. Wang, R. Zheng, L. Chen, Y . Liu, S. Dou, C. Huang, W. Shen, S. Jin, E. Zhou, C. Shi, S. Gao, N. Xu, Y . Zhou, X. Fan, Z. Xi, J. Zhao, X. Wang, T. Ji, H. Yan, L. Shen, Z. Chen, T. Gui, Q. Zhang, X. Qiu, X. Huang, Z. Wu, and Y .-G. Jiang. Secrets of rlhf in large language models part ii: Reward modeling, 2024. URLhttps://arxiv.org/abs/2401.06080

    Pith/arXiv arXiv 2024

  34. [34]

    S. Wang, Z. Zhang, R. Zhao, F. Tan, and C. T. Nguyen. Reward difference optimization for sample reweighting in offline rlhf, 2024. URLhttps://arxiv.org/abs/2408.09385

    Pith/arXiv arXiv 2024

  35. [35]

    Z. Wang, Y . Dong, J. Zeng, V . Adams, M. N. Sreedhar, D. Egert, O. Delalleau, J. P. Scowcroft, N. Kant, A. Swope, and O. Kuchaiev. Helpsteer: Multi-attribute helpfulness dataset for steerlm,

  36. [36]

    Z. Wang, A. Bukharin, O. Delalleau, D. Egert, G. Shen, J. Zeng, O. Kuchaiev, and Y . Dong. Helpsteer2-preference: Complementing ratings with preferences, 2025. URL https://arxiv. org/abs/2410.01257. 12

    Pith/arXiv arXiv 2025

  37. [37]

    J. Wu, X. Wang, Z. Yang, J. Wu, J. Gao, B. Ding, X. Wang, and X. He.α-dpo: Adaptive reward margin is what direct preference optimization needs, 2024. URL https://arxiv.org/abs/ 2410.10148

    Pith/arXiv arXiv 2024

  38. [38]

    J. Wu, Y . Xie, Z. Yang, J. Wu, J. Gao, B. Ding, X. Wang, and X. He.β-dpo: Direct preference optimization with dynamicβ, 2024. URLhttps://arxiv.org/abs/2407.08639

    Pith/arXiv arXiv 2024

  39. [39]

    Zhang, R

    X. Zhang, R. Zhao, Y . Qiao, X. Wang, and H. Li. Adacos: Adaptively scaling cosine logits for effectively learning deep face representations, 2019. URL https://arxiv.org/abs/1905. 00292

    2019

  40. [40]

    Y . Zhao, R. Joshi, T. Liu, M. Khalman, M. Saleh, and P. J. Liu. Slic-hf: Sequence likelihood calibration with human feedback.arXiv preprint arXiv:2305.10425, 2023. A Large Language Model Usage Large Language Models (LLMs) were used solely for grammatical editing and improving writing flow. The research methodology, experimental design, data analysis, and...

    Pith/arXiv arXiv 2023

  41. [42]

    scores are in Tables 11 and 12 respectively. D.2 Generative Performance The results displaying the win rate of the model responses as judged by UltraRM [8] and AlpacaEval 2.0 win rates [10] are in Tables 13 and 14 respectively. 14 Algorithm Pearson Correlation Spearman Correlation Accuracy Vanilla DPO 0.3151 0.3244 0.69 DPO-margin-1 0.3161 0.3243 0.69 DPO...

    arXiv 2004

  42. [43]

    Torchy’s Tacos - Known for their creative tacos featuring unique flavors such as the Trailer Park (with green chile, cheese, bacon, sour cream) and the Taco Loco (chipotle ranch avocado crema)

  43. [44]

    Veracruz All Natural - This family-owned brand offers authentic Mexican dishes like migas (scrambled tortilla strips), enchiladas verdes, and homemade tamales

  44. [45]

    La Condesa - A colorful cart specializing in seafood-focused eats including shrimp po’ boys, fish tortas, ceviche burritos, and more

  45. [46]

    Franklin Barbecue (Food Truck Operations) – Although primarily famous for their brick-and-mortar location at East Cesar Chavez St., Franklin Food Trucks occasionally pop up around Austin offering mouth-watering barbecue platters, ribs, brisket sandwiches, etc

  46. [47]

    Holy Roller Fried Pickle Cart - Don’t miss out on these crispy fried pickled dill slices that make perfect snacks for any occasion!

  47. [48]

    Chi’Lantro - Savor Korean-inspired fusion cuisine by sampling items ranging from bulgogi beef bowls or kimchi quesadillas to spicy rice cakes topped with cilantro slaw

  48. [49]

    The Frutería - Stocked with fresh fruits and natural juices, this vendor serves both locals and tourists fruit-based delights, various smoothies & agua frescas

  49. [50]

    Home Slice Pizza - Famous for their New York-style pizza cut into generous square portions while serving flavorful toppings coupled with fantastic service. 9.Lone Star Shrimp Boil - Catchy name aside, taste buds will surely thank you too because they exclusively offer handcrafted Cajun boil packs loaded with succulent Louisiana prawns alongside corn and s...

  50. [51]

    Veracruz All Natural - This vendor specializes in Mexican cuisine, famous for their migas tacos made with crispy tortilla strips filled with shredded chicken or cheese

  51. [52]

    Trailer Park

    Torchyś Tacos - A well-known taco chain originating from this city offering innovative flavors like the "Trailer Park" (fry sauce, queso fresco, pico de gallo) and "Shack Shack," topped with bacon and grilled onions

  52. [53]

    Tacodeli - Offering Tex-Mex-style breakfast items such as breakfast burritos stuffed with scrambled eggs, salsa, avocado, jalapeños, and choice of meat – carne guisada, sausage, or chorizo

  53. [54]

    It comes in various toppings including classic pepperoni but also more unique options such as cathead (a combination of prosciutto and arugula)

    Home Slice Pizza - Known for their New York-style pizza served by the slice. It comes in various toppings including classic pepperoni but also more unique options such as cathead (a combination of prosciutto and arugula)

  54. [55]

    GŔaj Mahal - Primarily an Indian restaurant that serves delicious Punjabi and North Indian dishes through a mobile ordering system designed specifically for their carts

  55. [56]

    La Condesa - Specializing in authentic Oaxacan cuisine featuring regional favorites like tlayudas (crunchy fried grasshopper-like flatbread), mole negro, chiles rellenos, elote, and other local specialties

  56. [57]

    Torchy’s Trailer Park – Another outpost of Torchyś fame; they serve creative tacos paired with homemade pickles among other fun sides

  57. [58]

    These are just a taste of what Austin has to offer when it comes to street food culture

    ChiĹantro Mediterranean Grill - Serves up fresh salads, wraps, falafel bowls along with Korean BBQ fusion items like bulgogi beef on rice, making it perfect for adventurous eaters. These are just a taste of what Austin has to offer when it comes to street food culture. Enjoy exploring these vendors during your visit! J.2.3 DPO-margin-gt UltraRM reward of ...

  58. [59]

    Torchy’s Tacos - Thispopular taco joint offers creative tacos with unique flavor combinations. 2. Franklin Barbecue - Although not strictly a street vendor, this famous barbecue spot has multiple locations throughout the city. 3. La Condesa - A trendy Mexican restaurant that also serves delicious taquitos and other street-style eats. 4. Veracruz All Natur...

    2021

  59. [60]

    Torchyś Tacos - This local favorite specializes in creative tacos with unique flavor combinations like Trailer Park (fried green tomato, bacon, remoulade) and The Atomic Nacho Tower

  60. [61]

    Veracruz All Natural - For authentic Mexican cuisine, head to this vendor serving up delicious migas dishes, as well as other traditional favorites such as tortas and enchiladas

  61. [62]

    La Condesa taqueria - A sister location of Taco Joint on South Congress Avenue, offering handmade-made flour or corn tortillas filled with meats, seafood options and vegetarian delights

  62. [63]

    ChiĹantro - Offering Asian-inspired fusion eats, from bulgogi beef wraps to Korean-style fried rice bowls. 5. TacoJoint - Run by owner Robb White, owners serve generous portions packed full of savory meat choices along side fresh homemade chips made daily!

  63. [64]

    Big As Yo´Face

    Chuyś - Originating out of Austin, they offer eclectic Tex-Mex flavors featuring their signature "Big As Yo´Face" burrito – youĺl find plenty more quirky offerings there too!

  64. [65]

    Mela Indian Cuisine - Serves flavorful North Indian delicacies that transport your taste buds through streets of Old Delhi onto Rainey Street

  65. [66]

    Frank Lloyd Wright Barbecue & Catering Co - Specializes in slow-smoked brisket served at various venues around town including some pop-up events where it can be found parked nearby Capitol Dome during festivals etc.. These are just a few examples; however, several new eateries emerge each year so staying updated would also provide additional delightful fi...

  66. [67]

    Torchyś Tacos - Known for their creative tacos filled with unique ingredients like queso fresco, pickled carrots, onions, house-made hot sauce, cilantro lime crema, and more

  67. [68]

    Veracruz All Natural - Specializing in migas (mashed tortilla chips) topped with scrambled eggs, diced chicken cooked in its own lard, onions, tomatoes, jalapeños, refried beans, sour cream or Mexican crema & shredded cheese; as well as their famous shrimp dishes

  68. [69]

    Lion Burgers,

    Tacodeli - This local favorite offers breakfast favorites such as avocado toast, burritos, quesadillas, and egg sandwiches along with other tasty options made from scratch daily using fresh locally sourced produce when possible! 4.Money Lion Vegan Food Truck - A family-owned vegan truck offering flavorful plant-based versions of classic Tex-Mex dishes inc...

  69. [2023]

    URLhttps://arxiv.org/abs/2311.09528

    Pith/arXiv arXiv