Pith. sign in

REVIEW 4 major objections 4 minor 42 references

Reward-model training improves when augmentation targets its lowest-margin, most ambiguous preference pairs, not all pairs equally.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 22:10 UTC pith:ZGUIERMN

load-bearing objection Plausible margin-aware augmentation idea, but the theorem assumes away the very low-margin samples the algorithm actually uses, and the single-run, placeholder-laden evidence doesn't match the abstract's promises. the 4 major comments →

arxiv 2602.17658 v3 pith:ZGUIERMN submitted 2026-02-19 cs.LG cs.AIcs.ITmath.IT

MARS: Margin and Semantic-Aware Data Augmentation for Reward Modeling

Payel Bhattacharjee , Osvaldo Simeone , Ravi Tandon This is my paper
classification cs.LG cs.AIcs.ITmath.IT
keywords reward modelingdata augmentationmargin-aware samplingpreference optimizationBradley-Terry modelempirical Fisher informationRLHFparaphrasing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that synthetic data augmentation for reward modeling should be spent where the reward model is most uncertain: on preference pairs with near-zero reward margins. Rather than augmenting uniformly or only on high-confidence pairs, the proposed MARS framework re-allocates an augmentation budget adaptively across epochs, paraphrasing the chosen and rejected responses of low-margin examples most heavily. If correct, this gives a simple, model-agnostic lever to improve both reward-model accuracy and the downstream policies that are aligned with those rewards. The theoretical part claims a provable reason: low-margin samples dominate the average curvature (empirical Fisher information) of the Bradley-Terry loss, so concentrating augmentation there improves conditioning across all parameter directions. The empirical work reports consistent gains in pairwise accuracy, margin signal-to-noise ratio, and aligned-policy win-rates across three preference datasets and two alignment backbones.

Core claim

MARS treats the reward model's margin — the difference between its scores for the chosen and rejected response — as a difficulty signal, and concentrates a fixed augmentation budget on pairs where that margin is small. At each epoch, a softmax over negative absolute margins yields an augmentation probability per pair, the budget is split as B^t · q_i^t, and paraphrases of both responses generate new synthetic preference pairs. The paper proves (Theorem 1) that, under margin-separation and feature-diversity assumptions, the mixture of original data and such hard-sample augmentation has empirical Fisher information that dominates the original data's Fisher information by a factor [α + (1−α)γ_c

What carries the argument

The central object is the reward margin Δ_i^t = r_θ(x_i, y_i^+) − r_θ(x_i, y_i^−), used to define augmentation probabilities q_i^t = exp(−τ|Δ_i^t|)/Σ_j exp(−τ|Δ_j^t|). This margin signal selects which preference pairs get paraphrased and how many paraphrases each receives, coupling data generation to the model's current failure regions. The theory rests on the logistic curvature factor c(Δ) = σ(Δ)(1−σ(Δ)) inside the empirical Fisher matrix; the paper shows low-margin samples maximize this factor and uses matrix inequalities to establish curvature dominance for the augmented mixture.

Load-bearing premise

Low reward margins mark genuinely ambiguous but label-preserving comparisons, so that paraphrasing those pairs produces valid preference supervision rather than amplifying mislabeled or semantically drifted data.

What would settle it

Run MARS on a preference dataset where the labels of low-margin pairs are flipped or corrupted; if gains vanish or reverse relative to uniform augmentation, the method's benefit depends on low margins being trustworthy ambiguity signals. Conversely, a human evaluation showing that paraphrases of low-margin pairs frequently violate or reverse the original preference would directly undermine the mechanism.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Reward-model training can be improved by re-weighting a fixed augmentation budget toward low-margin pairs, with no change to the underlying model architecture.
  • The margin-aware strategy consistently improves pairwise accuracy and margin signal-to-noise ratio over uniform augmentation and West-of-N baselines.
  • Policies aligned with MARS-trained reward models show higher win-rates (judge-evaluated) than those aligned with baseline-trained reward models, across TinyLlama and Llama-3.2 backbones.
  • The theory suggests the mechanism is geometric: low-margin samples increase the average curvature of the BT loss, which should improve conditioning and stability of parameter updates.
  • Because the framework decouples where to augment from how to augment, the margin-based selection can be paired with other augmentation operators besides paraphrasing.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The margin signal conflates genuine label ambiguity with the reward model's own mis-ranking; if low margins mostly reflect reward-model error rather than true preference difficulty, MARS would systematically amplify noisy labels exactly where it concentrates supervision. A straightforward test is human evaluation of paraphrase label fidelity on low-margin pairs.
  • The theory's Assumption 2 — that augmented-data feature covariance dominates original-data covariance in every direction — is strong and unlikely to hold for paraphrases of the same prompt; if it fails, the PSD-dominance conclusion weakens to per-direction or average statements, though the empirical gains might persist.
  • The method should be sensitive to the temperature τ and the paraphraser's diversity; an extension would be to make the budget allocation depend not only on |Δ| but also on an external measure of semantic drift, which the paper invokes but does not operationalize in the main loop.
  • MARS could plausibly be applied to the policy optimization stage itself, using margin-like signals from the current policy to decide which synthetic rollouts to emphasize, rather than only to reward-model training.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes MARS, an adaptive data-augmentation loop for reward-model training under the Bradley-Terry preference model. At each epoch, the current reward model scores all preference tuples, computes the margin Δ_i = r(y^+) − r(y^−), and allocates a paraphrasing budget to each tuple with probability proportional to exp(−τ|Δ_i|), so that low-margin ("ambiguous") pairs receive the largest number of synthetic paraphrases of the chosen and rejected responses. The augmented set is mixed with the original preferences and the reward model is retrained. The paper reports pairwise-accuracy and margin-SNR gains on HH-RLHF, UltraFeedback, and PKU-SafeRLHF with DeBERTa-v3-base, and downstream PPO win-rates for TinyLlama and Llama-3.2 policies. The central theoretical claim is Theorem 1, which states that under Assumptions 1 and 2 the mixture curvature IR(θ) dominates α+(1−α)γ_curv times IP(θ), i.e., adding low-margin samples provably increases average loss curvature and conditioning.

Significance. If the claims were correct, margin-aware augmentation would be a cheap, model-agnostic route to better reward models and downstream alignment. The paper is clearly written, includes a detailed algorithm box, provides an explicit proof in Appendix A.1, and reports experiments on three standard preference datasets. These are real strengths. However, the paper's principal novelty — a provable curvature guarantee for MARS — collapses because the theorem is conditioned on an assumption that the algorithm itself contradicts. The empirical support is also thinner than the abstract claims: one backbone, one seed, and no statistical assessment. The result may still be interesting as a heuristic, but the paper as written does not support its stated contributions.

major comments (4)
  1. [§4, Eq. (14) vs. Algorithm 1] Theorem 1 (Eq. 16) is stated under Assumption 1 (Eq. 14), which requires that original samples z~P have |Δ_θ(z)| ≥ γ_org while augmented samples z~Q have |Δ_θ(z)| ≤ γ_aug < γ_org. This is exactly opposite to MARS: Algorithm 1 (lines 6–9) computes margins on the original dataset D and assigns the largest paraphrasing budget to the smallest |Δ_i|; Q is therefore constructed from the very low-margin tuples that already belong to P. The paper itself concedes this in the paragraph after Assumption 1: 'this is a simplification of the real system in which low-margin examples are present in the original dataset. In fact, it is exactly the low-margin examples that are prioritized by MARS for augmentation.' Consequently the proof in Appendix A.1 establishes a PSD domination for a hypothetical Q that MARS does not generate, and the theorem does not apply to the proposed method. The 'provable curvat
  2. [§4, Eq. (15)] Assumption 2 requires E_Q[ψψ^T] ⪰ β E_P[ψψ^T] for some β>0 in every feature direction. Since Q is generated by T5 paraphrases of the same original responses (often the same prompt), there is no reason to expect the augmented covariance to dominate the original covariance uniformly; the assumption is both unstated and unverified. The theorem's quantitative content (γ_curv = β c(γ_aug)/c(γ_org)) depends on β being sufficiently large, yet no estimate or validation of β is given. This is a load-bearing gap: without evidence for Eq. (15), even a corrected theorem would not imply a uniform curvature improvement for MARS.
  3. [§5, Figures 4–6 and Table 2] The body-level claim of 'consistent gains' rests on a single backbone (DeBERTa-v3-base), single runs, and no error bars or significance tests. Table 2 reports win-rates like 52:48 and 54:46, which are within typical run-to-run noise for PPO, yet the text concludes the gains are 'robust to scale and architecture.' The abstract further claims evaluation 'across multiple preference datasets, reward-model backbones, downstream alignment settings, and benchmarks including RewardBench' — RewardBench is not used, and only one reward-model backbone is trained. These overstatements need to be corrected in any revision; the reported improvements may be real but are not established at the current level of evidence.
  4. [§3.2 / Abstract] The title and abstract describe MARS as 'Margin and Semantic-Aware,' and the abstract says semantic distance is used 'as a second layer for refinement.' However, Algorithm 1 and the description in §3.2 use only the margin Δ_i to allocate budgets; no semantic-distance measure, threshold, or filtering step is defined or implemented. Either the semantic mechanism should be specified and evaluated, or the claims should be scaled back to margin-aware only.
minor comments (4)
  1. [Throughout] The manuscript contains recurring typos (e.g., 'curveture' in §3.1 and §4; 'accross' in §5.1) and inconsistent dataset names (e.g., 'PKU-SelfRLHF' in Figure 4 caption vs 'PKU-SafeRLHF' elsewhere). The paper title in the header ('MARS: Margin-Aware Reward-Modeling with Self-Refinement') differs from the abstract title ('MARS: Margin and Semantic-Aware Data Augmentation for Reward Modeling').
  2. [Figure 4] The bar labels in Figure 4 are difficult to read and the numerical values are repeated in a way that obscures the differences (e.g., the same four numbers appear in multiple sets of bars). Please use clearer annotations and report the exact values in a table.
  3. [§5 setup] The experimental section states that a fixed 1,000-prompt subset is used for RM training on each dataset, but it does not state how many synthetic pairs were generated (B^t), how τ was chosen, or whether the split is stratified. These details are needed to reproduce the main table.
  4. [Appendix A.2] The appendix says paraphraser outputs are 'filtered to remove degenerate outputs and excessively short responses,' but no threshold or filtering rule is specified, and no manual or automated quality check for label preservation is reported.

Circularity Check

1 steps flagged

Theorem 1's curvature guarantee is enforced by Assumption 1, which contradicts how MARS actually constructs Q.

specific steps
  1. self definitional [Section 4, Assumption 1 (Eq. 14) and following paragraph; Algorithm 1 (lines 6–10)]
    "Assumption 1 ... |∆θ(z)| ≥ γorg, z∼P (human labels), |∆θ(z)| ≤ γaug, z∼Q (augmented data). ... It is noted that this is a simplification of the real system in which low-margin examples are present in the original dataset. In fact, it is exactly the low-margin examples that are prioritized by MARS for augmentation (see Algorithm 1)."

    The proof of Theorem 1 derives IR(θ) ⪰ [α+(1−α)γcurv]IP(θ) by using Assumption 1: P has margins above γorg and the augmented distribution Q has margins below γaug, so the logistic curvature c(Δ)=σ(Δ)(1−σ(Δ)) is larger on Q. But Algorithm 1 assigns augmentation budget B^t q_i^t with q_i^t ∝ exp(−τ|Δ_i^t|), i.e., it paraphrases the lowest-margin samples that are already in D∼P. Thus the actual P contains margins near zero and the actual Q is generated from those same low-margin points; Assumption 1's separation does not describe the system. The theorem's conclusion is therefore not a prediction about MARS's Q but a restatement of the assumed margin property.

full rationale

The empirical part of the paper is self-contained and benchmarked externally (pairwise accuracy, SNR, and win-rates against Uniform Augmentation and WoN), with no load-bearing self-citations or uniqueness imports; those comparisons do not reduce to the fitted inputs. The theoretical circularity is confined to Section 4: Theorem 1, the paper's claimed 'principled justification' for margin-aware augmentation, assumes the very property that distinguishes the augmented distribution (low margins), and the paper itself admits that this assumption contradicts the algorithm. Since c(Δ)=σ(Δ)(1−σ(Δ)) is by definition maximized near Δ=0, the theorem's inequality is effectively a formalized restatement of Assumption 1 rather than an independent derivation. Figure 5's 'validation' is likewise a monotonicity check of the logistic function. This does not invalidate the empirical comparisons, but it removes the theoretical grounding claimed in the abstract and Section 4, giving partial circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The paper adds no new theoretical entities. It does rely on two ad hoc assumptions that carry the theoretical result (margins separated by difficulty; feature-covariance dominance), plus the domain assumptions that synthetic paraphrases preserve preference labels and that low margins indicate genuine ambiguity. Free parameters (τ, budget, epochs, paraphrase split, decoding thresholds) are all unreported, so the method is under-specified at the level of actual numbers.

free parameters (5)
  • τ (softmax sharpness) = unreported (stated τ ∈ (0,1])
    Eq. (5): q_i^t = exp(−τ|Δ_i^t|)/Σ_j exp(−τ|Δ_j^t|); controls how aggressively the augmentation budget concentrates on low-margin samples; value never given; note that exp(−τ|Δ|) does not match the theoretically motivated curvature weight c(Δ)=σ(Δ)(1−σ(Δ)).
  • B^t (total augmentation budget per epoch) = unreported
    Algorithm 1 line 8: each sample receives B^t·q_i^t generated samples; total budget chosen by hand; not reported.
  • T (number of refinement epochs) = unreported
    Algorithm 1: outer loop t=1..T; not reported.
  • n_i^+ / n_i^- split = unreported
    Eq. (6): n_i^+ + n_i^- = B^t q_i^t; how the budget is divided between chosen and rejected paraphrases is unspecified, though it changes the number of generated pairs (n^+ +1)(n^- +1)−1.
  • Paraphraser decoding/filtering thresholds = unreported
    Appendix A.2: 'beam search with moderate stochasticity' and 'filtered to remove degenerate outputs and excessively short responses' — no beam size, temperature, or length thresholds.
axioms (5)
  • standard math Bradley–Terry preference model (Eq. 1): p(y+ ≻ y− | x; θ) = σ(rθ(x,y+) − rθ(x,y−))
    Used throughout (Eqs. 1–2, 8–11); standard in RLHF reward modeling; also Plackett–Luce is cited but unused.
  • ad hoc to paper Assumption 1: |Δθ(z)| ≥ γ_org for human-labeled P and |Δθ(z)| ≤ γ_aug < γ_org for augmented Q (Eq. 14)
    Carries the theorem: it stipulates the very property MARS is designed to produce (Q low-margin). The paper itself notes: 'this is a simplification of the real system in which low-margin examples are present in the original dataset. In fact, it is exactly the low-margin examples that are prioritized by MARS for augmentation' (§4).
  • ad hoc to paper Assumption 2: E_Q[ψψ^T] ⪰ β E_P[ψψ^T] for some β>0 (Eq. 15)
    Unverified coverage condition needed for γ_curv>1 and the PSD bound; paraphrases of fixed prompts plausibly fail to dominate all feature directions of the original data.
  • domain assumption Synthetic paraphrases preserve the original preference orientation
    §3.2 and §5: generated pairs are added to the training set as valid (chosen ≻ rejected) supervision; no validation of label fidelity; the method amplifies these labels for the most heavily augmented samples.
  • domain assumption Low margin ⇔ genuine ambiguity/informativeness rather than reward-model mis-ranking
    §3.1: margins near zero are described as 'hard, low-confidence comparisons', yet the same sentence allows they may be 'incorrectly ranked'; if the latter, augmentation entrenches errors.

pith-pipeline@v1.3.0-alltime-deepseek · 112424 in / 25857 out tokens · 246509 ms · 2026-08-02T22:10:25.807793+00:00 · methodology

0 comments
read the original abstract

Reward modeling is central to alignment pipelines such as RLHF, RLAIF, and PPO-based policy optimization, yet its reliability is constrained by limited and heterogeneous human preference data that are expensive to collect at scale. While synthetic augmentation can expand preference supervision, existing methods often augment uniformly or at the representation level, without targeting examples where the reward model is uncertain or prone to mis-ranking. In this paper, we introduce MARS (Margin and Semantic-Aware Data Augmentation for Reward Modeling), an adaptive augmentation framework that prioritizes low-margin preference pairs and uses semantic distance as a second layer for refinement to enhance the contrast between the chosen and rejected responses. Across multiple preference datasets, reward-model backbones, downstream alignment settings, and benchmarks including RewardBench and AlpacaEval, MARS improves both reward-model quality and alignment performance over existing baselines. Our results show that reward-model augmentation is most effective when guided by both model margins and semantic structure.

Figures

Figures reproduced from arXiv: 2602.17658 by Osvaldo Simeone, Payel Bhattacharjee, Ravi Tandon.

Figure 1
Figure 1. Figure 1: Comparison between MARS (this paper) and the existing methods including no augmentation, Uniform Augmentation, West￾of-N (WoN) (Pace et al., 2024) with PKU-SafeRLHF (Ji et al., 2024) dataset and DeBERTa-v3-base model on different evaluation metrices: (1) SNR: the ratio of the mean and standard deviation of the obtained margin, (2) Pairwise Accuracy: reward for the chosen response is higher than the rejecte… view at source ↗
Figure 2
Figure 2. Figure 2: Adaptive data augmentation-based Reward Modeling in MARS: At every epoch t, the reward model (RM) from previous stage r t−1 θ calculates the margin of all the samples in preference dataset D, and samples with lower |∆t i | get more budget for augmented samples. Then, updated dataset (preference dataset and the synthetic dataset) is used to train the reward model r t−1 θ to get model r t θ . ous correlation… view at source ↗
Figure 3
Figure 3. Figure 3: Proposed workflow: Adaptive augmentation and refinement workflow. At every epoch t, the reward model (RM) from previous stage r t−1 θ calculates the margin that enables the calculation of selection/augmentation probability q t i , and given a fixed budget B t , for every i th-sample augmented samples are generated (such that n + i + n − i = B t · q t i ). Then, based on the calculated margin of all the sam… view at source ↗
Figure 4
Figure 4. Figure 4: Pairwise accuracy of DeBERTa-v3-base reward models under different training strategies. Results are reported on the Anthropic HH-RLHF (Bai et al., 2022), UltraFeedback (Cui et al., 2023), and PKU-SelfRLHF (Ji et al., 2024) test datasets. We compared training without augmentation, Uniform Augmentation, West-of-N (WoN) (Pace et al., 2024) and MARS (this paper). Min. Eigenvalue ( Mean curvature weight ωmin ) … view at source ↗
Figure 5
Figure 5. Figure 5: Small-margin (hard) preference pairs exhibit higher curvature. Left: Minimum eigenvalue of the bin-averaged empirical Fisher matrix, λmin 1 |B| P z∈B Ib(z)  , across equal￾count bins sorted by |∆θ(z)|. Right: Mean curvature weight Ez∼B[σ(∆θ(z))(1 − σ(∆θ(z)))]. Hard samples (small |∆|) in￾duce substantially higher curvature than confident pairs. Assumption 2 (Feature diversity). There exists β > 0 such th… view at source ↗
Figure 6
Figure 6. Figure 6: Margin signal-to-noise ratio (SNR defined as the ratio of mean and standard deviation of the margin) for DeBERTa-v3-base reward models under different training strategies including No Augmentation, Uniform Augmentation, WoN and MARS. For the HH-RLHF (Bai et al., 2022), PKU-SafeRLHF (Ji et al., 2024) and UltraFeedback (Cui et al., 2023) test datasets. Aligned Model Dataset (RM Training and Testing) MARS vs … view at source ↗
Figure 7
Figure 7. Figure 7: Minimum eigenvalue and mean curvature weight analysis for PKU-SafeRLHF dataset on pretrained reward-model-deberta-v3- large-v2 model. performed with a learning rate of 1 × 10−5 . Parameter-efficient fine-tuning is implemented via LoRA with rank r = 16 and scaling factor α = 32. For all datasets, the maximum prompt length is set to 512 tokens and the maximum generation length to 192 tokens. Policy optimizat… view at source ↗
Figure 8
Figure 8. Figure 8: Text completions on PKU-SafeRLHF dataset generated by TinyLlama models aligned with different reward-model training strategies. Compared to uniform augmentation and WoN baselines, MARS produces responses that are more safety-aware, contextually grounded, and coherent, particularly on ambiguous or safety-critical prompts. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Representative text completions on PKU-SafeRLHF generated by TinyLlama models aligned with different reward-model training strategies. Compared to uniform and WoN baselines, MARS produces responses that are more safety-aware, contextually grounded, and coherent, particularly on ambiguous or safety-critical prompts. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Representative text completions on UltraFeedback generated by TinyLlama models aligned with different reward-model training strategies. Compared to uniform and WoN baselines, MARS produces responses that are more safety-aware, contextually grounded, and coherent, particularly on ambiguous or safety-critical prompts. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

42 extracted references · 13 linked inside Pith

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Analysis of llms for educational question classification and generation

    Al Faraby, S., Romadhony, A., et al. Analysis of llms for educational question classification and generation. Computers and Education: Artificial Intelligence, 7: 0 100298, 2024

    2024

  3. [3]

    K., and Kochmar, E

    Alhafni, B., Vajjala, S., Bann \`o , S., Maurya, K. K., and Kochmar, E. Llms in education: Novel perspectives, challenges, and opportunities. arXiv preprint arXiv:2409.11917, 2024

    Pith/arXiv arXiv 2024

  4. [4]

    Natural gradient works efficiently in learning

    Amari, S.-I. Natural gradient works efficiently in learning. Neural computation, 10 0 (2): 0 251--276, 1998

    1998

  5. [5]

    Improving the scaling laws of synthetic data with deliberate practice

    Askari-Hemmat, R., Pezeshki, M., Dohmatob, E., Bordes, F., Astolfi, P., Hall, M., Verbeek, J., Drozdzal, M., and Romero-Soriano, A. Improving the scaling laws of synthetic data with deliberate practice. arXiv preprint arXiv:2502.15588, 2025

    Pith/arXiv arXiv 2025

  6. [6]

    Training a helpful and harmless assistant with reinforcement learning from human feedback

    Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862, 2022

    Pith/arXiv arXiv 2022

  7. [7]

    and Hero, A

    Berisha, V. and Hero, A. O. Empirical non-parametric estimation of the fisher information. IEEE Signal Processing Letters, 22 0 (7): 0 988--992, 2014

    2014

  8. [8]

    Bishop, C. M. and Nasrabadi, N. M. Pattern recognition and machine learning, volume 4. Springer, 2006

    2006

  9. [9]

    Bradley, R. A. and Terry, M. E. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952

    1952

  10. [10]

    Unsupervised learning of visual features by contrasting cluster assignments

    Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., and Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33: 0 9912--9924, 2020

    2020

  11. [11]

    Evaluating the feasibility of chatgpt in healthcare: an analysis of multiple clinical and research scenarios

    Cascella, M., Montomoli, J., Bellini, V., and Bignami, E. Evaluating the feasibility of chatgpt in healthcare: an analysis of multiple clinical and research scenarios. Journal of medical systems, 47 0 (1): 0 33, 2023

    2023

  12. [12]

    F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D

    Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 30, 2017

    2017

  13. [13]

    Ultrafeedback: Boosting language models with high-quality feedback, 2024

    Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y., Xie, G., Liu, Z., and Sun, M. Ultrafeedback: Boosting language models with high-quality feedback, 2024. In URL https://openreview. net/forum, 2023

    2024

  14. [14]

    Raft: Reward ranked finetuning for generative foundation model alignment

    Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, T. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767, 2023

    Pith/arXiv arXiv 2023

  15. [15]

    Simcse: Simple contrastive learning of sentence embeddings

    Gao, T., Yao, X., and Chen, D. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821, 2021

    Pith/arXiv arXiv 2021

  16. [16]

    Bonbon alignment for large language models and the sweetness of best-of-n sampling

    Gui, L., G \^a rbacea, C., and Veitch, V. Bonbon alignment for large language models and the sweetness of best-of-n sampling. Advances in Neural Information Processing Systems, 37: 0 2851--2885, 2024

    2024

  17. [17]

    P., Chiang, M

    Guo, Y., Tian, P., Kalpathy-Cramer, J., Ostmo, S., Campbell, J. P., Chiang, M. F., Erdogmus, D., Dy, J. G., and Ioannidis, S. Experimental design under the bradley-terry model. In IJCAI, pp.\ 2198--2204, 2018

    2018

  18. [18]

    The elements of statistical learning: data mining, inference, and prediction, 2009

    Hastie, T. The elements of statistical learning: data mining, inference, and prediction, 2009

    2009

  19. [19]

    Deberta: Decoding-enhanced bert with disentangled attention

    He, P., Liu, X., Gao, J., and Chen, W. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654, 2020

    Pith/arXiv arXiv 2006

  20. [20]

    Best-of-n jailbreaking

    Hughes, J., Price, S., Lynch, A., Schaeffer, R., Barez, F., Koyejo, S., Sleight, H., Jones, E., Perez, E., and Sharma, M. Best-of-n jailbreaking. arXiv preprint arXiv:2412.03556, 2024

    Pith/arXiv arXiv 2024

  21. [21]

    Pku-saferlhf: A safety alignment preference dataset for llama family models

    Ji, J., Hong, D., Zhang, B., Chen, B., Dai, J., Zheng, B., Qiu, T., Li, B., and Yang, Y. Pku-saferlhf: A safety alignment preference dataset for llama family models. arXiv e-prints, pp.\ arXiv--2406, 2024

    2024

  22. [22]

    E., Vuruma, S

    Lakkaraju, K., Jones, S. E., Vuruma, S. K. R., Pallagani, V., Muppasani, B. C., and Srivastava, B. Llms for financial advisement: A fairness and efficacy study in personal decision making. In Proceedings of the Fourth ACM International Conference on AI in Finance, pp.\ 100--107, 2023

    2023

  23. [23]

    R., Mesnard, T., Ferret, J., Bishop, C., Hall, E., Carbune, V., and Rastogi, A

    Lee, H., Phatale, S., Mansoor, H., Lu, K. R., Mesnard, T., Ferret, J., Bishop, C., Hall, E., Carbune, V., and Rastogi, A. Rlaif: Scaling reinforcement learning from human feedback with ai feedback. 2023

    2023

  24. [24]

    Y.-Y., Lee, A.-H., Lo, K., Chang, J

    Liao, Z., Antoniak, M., Cheong, I., Cheng, E. Y.-Y., Lee, A.-H., Lo, K., Chang, J. C., and Zhang, A. X. Llms as research tools: A large scale survey of researchers' usage and perceptions. arXiv preprint arXiv:2411.05025, 2024

    Pith/arXiv arXiv 2024

  25. [25]

    Rrm: Robust reward model training mitigates reward hacking

    Liu, T., Xiong, W., Ren, J., Chen, L., Wu, J., Joshi, R., Gao, Y., Shen, J., Qin, Z., Yu, T., et al. Rrm: Robust reward model training mitigates reward hacking. arXiv preprint arXiv:2409.13156, 2024

    Pith/arXiv arXiv 2024

  26. [26]

    Luce, R. D. et al. Individual choice behavior, volume 4. Wiley New York, 1959

    1959

  27. [27]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 0 27730--27744, 2022

    2022

  28. [28]

    West-of-n: Synthetic preferences for self-improving reward models

    Pace, A., Mallinson, J., Malmi, E., Krause, S., and Severyn, A. West-of-n: Synthetic preferences for self-improving reward models. arXiv e-prints, pp.\ arXiv--2401, 2024

    2024

  29. [29]

    Plackett, R. L. The analysis of permutations. Journal of the Royal Statistical Society Series C: Applied Statistics, 24 0 (2): 0 193--202, 1975

    1975

  30. [30]

    D., Ermon, S., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36: 0 53728--53741, 2023

    2023

  31. [31]

    Towards scientific intelligence: A survey of llm-based scientific agents

    Ren, S., Jian, P., Ren, Z., Leng, C., Xie, C., and Zhang, J. Towards scientific intelligence: A survey of llm-based scientific agents. arXiv preprint arXiv:2503.24047, 2025

    arXiv 2025

  32. [32]

    Trust region policy optimization

    Schulman, J., Levine, S., Abbeel, P., Jordan, M., and Moritz, P. Trust region policy optimization. In International conference on machine learning, pp.\ 1889--1897. PMLR, 2015

    2015

  33. [33]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    Pith/arXiv arXiv 2017

  34. [34]

    Scott, W. A. Maximum likelihood estimation using the empirical fisher information matrix. Journal of Statistical Computation and Simulation, 72 0 (8): 0 599--611, 2002

    2002

  35. [35]

    G., Dadashi, R., Hussenot, L., Ferret, J., Vieillard, N., Ram \'e , A., Shariari, B., Perrin, S., Friesen, A., Cideron, G., et al

    Sessa, P. G., Dadashi, R., Hussenot, L., Ferret, J., Vieillard, N., Ram \'e , A., Shariari, B., Perrin, S., Friesen, A., Cideron, G., et al. Bond: Aligning llms with best-of-n distillation. arXiv preprint arXiv:2407.14622, 2024

    Pith/arXiv arXiv 2024

  36. [36]

    Large language model alignment: A survey

    Shen, T., Jin, R., Huang, Y., Liu, C., Dong, W., Guo, Z., Wu, X., Liu, Y., and Xiong, D. Large language model alignment: A survey. arXiv preprint arXiv:2309.15025, 2023

    Pith/arXiv arXiv 2023

  37. [37]

    Team, Q. et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2 0 (3), 2024

    Pith/arXiv arXiv 2024

  38. [38]

    A differentiable reformulation for e-optimal design of experiments in nonlinear dynamic biosystems

    Telen, D., Van Riet, N., Logist, F., and Van Impe, J. A differentiable reformulation for e-optimal design of experiments in nonlinear dynamic biosystems. Mathematical Biosciences, 264: 0 1--7, 2015

    2015

  39. [39]

    An improved empirical fisher approximation for natural gradient descent

    Wu, X., Yu, W., Zhang, C., and Woodland, P. An improved empirical fisher approximation for natural gradient descent. Advances in Neural Information Processing Systems, 37: 0 134151--134194, 2024

    2024

  40. [40]

    Q., Salamatian, S., Sun, Z., Suresh, A

    Yang, J. Q., Salamatian, S., Sun, Z., Suresh, A. T., and Beirami, A. Asymptotics of language model alignment. In 2024 IEEE International Symposium on Information Theory (ISIT), pp.\ 2027--2032. IEEE, 2024

    2024

  41. [41]

    F., Lu, W., Thirunavukarasu, A

    Yang, R., Tan, T. F., Lu, W., Thirunavukarasu, A. J., Ting, D. S. W., and Liu, N. Large language models in health care: Development, applications, and challenges. Health Care Science, 2 0 (4): 0 255--263, 2023

    2023

  42. [42]

    Revolutionizing finance with llms: An overview of applications and insights

    Zhao, H., Liu, Z., Wu, Z., Li, Y., Yang, T., Shu, P., Xu, S., Dai, H., Zhao, L., Jiang, H., et al. Revolutionizing finance with llms: An overview of applications and insights. arXiv preprint arXiv:2401.11641, 2024

    arXiv 2024