Pith. sign in

REVIEW 4 major objections 6 minor 53 references

Aligning LLMs with Domain Invariant Reward Models

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that training a reward model to align source and target embeddings with an adversarial Wasserstein loss while optimizing source preferences transfers human preferences to unlabeled target domains, and demonstrates gains…

desk verdict A solid WDGRL-for-reward-models application with honest empirics; the theory is decorative and App-4 is oversold, but the central transfer claim holds up. read the letter →

arxiv 2501.00911 v1 pith:CCICQZAD submitted 2025-01-01 cs.LG

David Wu , Sanjiban Choudhury This is my paper · ORCID
classification cs.LG
keywords rewardmodelingdomainadaptationWassersteindistanceRLHFdomain-invariantrepresentationpreferencetransferLLMalignmentunlabeledtargetdata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that human preference signals are domain-agnostic enough that a reward model can transfer them from a labeled source domain to an unlabeled target domain, provided the model is forced to align source and target embeddings. The proposed method, DIAL, trains a base LLM with two heads: a critic head adversarially minimizing the Wasserstein distance between source and target embeddings, and a reward head optimizing a Bradley-Terry preference loss on source labels. Across four settings—cross-lingual, clean-to-noisy, few-shot-to-full, and simple-to-complex—this dual loss improves target reward accuracy or correlation over source-only training and over regularization baselines, sometimes matching oracle models trained with target labels. A theorem bounds target error by source error plus a Wasserstein term, giving the alignment objective a theoretical rationale. If correct, this gives a recipe for aligning LLMs in domains where preference annotation is expensive or impossible.

What carries the argument

The mechanism is the 1-Wasserstein distance, a measure of the minimum cost to transport one distribution onto another, computed through its Kantorovich-Rubinstein dual as the largest gap a 1-Lipschitz function can give between expectations. DIAL instantiates it as a critic head trained adversarially against the base model: the critic maximizes the expected source-minus-target score, a gradient penalty enforces the Lipschitz constraint, and the base model minimizes that distance to align embeddings. A separate linear reward head minimizes the Bradley-Terry source loss, and the two losses are optimized together so that aligned embeddings also separate preferred from rejected responses. Theorem 1 states that target error is bounded by source error plus $2 K L_\sigma W_1(\mu_S, \mu_T)$, making the paper's case that the domain loss is the right transfer objective.

What would settle it

Run DIAL on a synthetic target whose preferred category is inverted relative to source, e.g., odd-one-out with source (in: fruit, out: vegetable) and target (in: vegetable, out: fruit). If the Wasserstein alignment collapses fruit and vegetable embeddings, the reward head can satisfy both losses by learning 'not fruit' as a spurious reward, and target accuracy should drop to chance (0.2); observing that would refute the claim that alignment alone transfers meaningful preferences.

Watch

Extended reading notes

Core claim

The central claim is that a reward model can learn domain-agnostic human preferences by jointly making source and target prompt-response embeddings indistinguishable under an adversarially trained Wasserstein critic and separating chosen from rejected responses on the labeled source domain. DIAL implements this as a min-max game on the Wasserstein distance with gradient penalty, plus a Bradley-Terry loss. The paper reports that this beats source-only and regularization baselines in all four transfer settings, reaches oracle accuracy on several cross-lingual and clean-to-noisy splits, nearly matches the target-trained oracle in few-shot transfer, and helps PPO policies adapt to distribution shift during RLHF. The authors conclude that the domain loss carries the preference signal across domains because the aligned embeddings cluster source-positive with target-positive and source-negative with target-negative.

Load-bearing premise

The load-bearing premise is that making source and target prompt-response embeddings look alike preserves the specific signal that tells chosen from rejected responses; the paper explicitly concedes that alignment can instead succeed by removing that variation and exploiting a spurious source-specific reward.

Editorial extensions

If this is right

  • If DIAL works as claimed, reward models for low-resource languages can be trained with English preference labels plus unlabeled target text, and on several language/split combinations DIAL matches oracle models that used target labels.
  • If the RLHF results hold, on-policy preference annotation during PPO can be partially replaced by using the current policy's responses as unlabeled target data for periodic reward adaptation.
  • The bound $\epsilon_T(r,f) \leq \epsilon_S(r,f) + 2KL_\sigma W_1(\mu_S,\mu_T)$ implies that improving source accuracy and reducing the Wasserstein distance between source and target embeddings both directly reduce worst-case target error.
  • DIAL is claimed to be a general transfer method, with consistent gains over source-only and two regularization baselines in all four settings, so the same recipe should transfer to other unlabeled target domains with correlated human preferences.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the same dual-loss idea should be testable on preference inversions; if the target's chosen/rejected order flips relative to the source, marginal Wasserstein alignment would be expected to hurt accuracy because it removes the very variation that carries the reward signal.
  • Editorial extension: the paper's Theorem 1 bounds error using the Wasserstein distance on input space, while DIAL minimizes a Wasserstein distance on embeddings; closing that gap would tell whether the alignment objective is genuinely the quantity in the bound.
  • Editorial extension: the RLHF distribution-shift experiment suggests DIAL could serve as a lightweight online reward-adaptation loop for any off-policy alignment pipeline, not just safety refusal tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes DIAL, a framework for training reward models on labeled source-domain preference data plus unlabeled target-domain prompt-response pairs. The model uses a base LLM with two heads: a critic head adversarially trained to minimize the Wasserstein distance between source and target embeddings (with a gradient penalty), and a linear reward head trained with a Bradley-Terry loss on the source preferences. The central claim is that this dual objective learns reward models that transfer to target domains with no preference labels, and the paper evaluates this across four settings: cross-lingual (English to Korean/Thai/Chinese), clean-to-noisy, few-shot-to-full, and simple-to-complex. The paper also provides a theoretical bound on target error, a spurious-reward toy experiment, an RLHF distribution-shift analysis, and scaling experiments.

Significance. If the empirical gains are robust, DIAL offers a simple and general recipe for transferring reward models to unlabeled domains, which is practically valuable for RLHF in low-resource settings. The paper's strengths include held-out target evaluations with standard errors over three seeds, comparisons against two regularization baselines and oracle upper bounds, a released code/data repository, and a synthetic odd-one-out experiment that provides a falsifiable test of the transfer mechanism and shows average gains (0.559 vs 0.318). However, the theoretical result is not connected to the optimized objective, and the evaluation in the simple-to-complex setting is weaker than the abstract suggests; these issues need to be addressed before the central claim can be fully accepted.

major comments (4)
  1. [Sec. 2.3, Eq. (7)] Theorem 1 bounds target error in terms of the input-space Wasserstein distance W1(µS, µT) over (x, y), but the DIAL domain loss in Eqs. (2)-(4) minimizes a Wasserstein distance between learned embeddings, i.e., W1(θ#µS, θ#µT). Since a map can contract distances, the minimized embedding-space distance does not upper-bound the input-space quantity appearing in Eq. (7); the theorem as stated is therefore not a justification of the optimized objective. The authors should either provide a bound on target error in terms of the embedding-space Wasserstein distance (with an appropriate representation-error or inverse-Lipschitz term), or explicitly present Theorem 1 as heuristic motivation rather than as a guarantee for DIAL.
  2. [Sec. 2.2 / Sec. 6] The domain loss aligns marginal distributions of (x, y) embeddings, while the preference structure is a conditional property within each domain. The paper's own Sec. 6 concedes that the reward model can align source and target marginals while learning a source-specific spurious reward. To substantiate the claim that alignment transfers the preference signal rather than acting as an unlabeled-data regularizer, a control experiment is needed; for example, shuffling the source preference labels should eliminate DIAL's target gains if the mechanism is preference transfer, whereas a pure regularization effect would persist. Such an ablation, or an analysis of which embedding dimensions the reward head relies on before and after alignment, is necessary to support the paper's central conclusion.
  3. [Sec. 3.5, Table 4] In the simple-to-complex setting, Src-Pref-SFT attains Spearman 0.562±0.023 versus DIAL's 0.556±0.011, and DIAL's Pearson improvement over Src-Pref-SFT (0.577±0.011 vs 0.567±0.018) is within one standard error. The abstract's '0.508 → 0.556' and the conclusion's 'significant gains in target performance' therefore hold only against the Src-Pref baseline, not against the strongest regularized baseline. This should be stated explicitly in the abstract and conclusion, or the claim that DIAL 'outperforms all baselines' across all four settings should be qualified for this setting.
  4. [Appendix A.3 / A.4, Sec. 3.1] The appendix states that the clean-to-noisy setting used weight decay 0.01 and the few-shot-to-full setting used learning rate 1e−5 because these choices were 'helpful' or 'ensuring stability'. The paper does not state whether these hyperparameters were selected on a held-out validation split of the target domain or on the reported test sets. If the latter, the held-out claim is compromised and the reported advantages over baselines could reflect per-setting tuning. Please document the validation protocol for each setting and report sensitivity of the results to the two hyperparameters λ and λ_gp that appear in Eq. (4).
minor comments (6)
  1. [Abstract / Tables 1-4] The aggregate improvements in the abstract (0.621 → 0.661, 0.671 → 0.703, 0.845 → 0.920, 0.508 → 0.556) are not derived from any table; please state the averaging formula or cite the corresponding table rows, particularly since Table 1 reports per-language and per-subreddit numbers that vary substantially.
  2. [Algorithm 1] The pseudocode line `critic_loss = - wd + gp_loss` does not include the weight λ_gp that appears in Eq. (4); clarify whether λ_gp = 1 is assumed in the algorithm description or include the weight explicitly.
  3. [Sec. 3.8 vs Appendix A.6] Section 3.8 says the odd-one-out evaluation uses 100 datapoints, while Appendix A.6 describes 1000 train, validation, and test examples; please reconcile these numbers.
  4. [Sec. 2.3 / Appendix B] The main theorem is numbered Theorem 1 in Section 2.3 but Theorem 2 in Appendix B; unify the numbering across the paper.
  5. [Eq. (2)] Equation (2) uses the notation Dsrc for an expectation over (x_s, y_s), but Dsrc was defined in Section 2 as a set of preference triplets (x, y+, y−); clarify which marginal of the source distribution is used for the critic loss.
  6. [Related Work] The related work section lists Winata et al. (2022a) and (2022b) with overlapping titles and appears to cite the same paper twice; please merge these entries or clearly distinguish them if they are different works.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the target evaluations are held-out and the Wasserstein bound is a standard result, not a restatement of the objective.

full rationale

The empirical claims are not circular: DIAL trains on labeled source pairs and unlabeled target pairs, while all reported target numbers (cross-lingual 0.621->0.661; clean-to-noisy 0.671->0.703; few-shot 0.845->0.920; simple-to-complex 0.508->0.556) are measured on held-out target labels that are never used for training, so no 'prediction' is a fitted parameter renamed. Theorem 1 (Eq. 7) is a standard Wasserstein DA bound derived from the Lipschitz property of the disagreement function and Kantorovich-Rubinstein duality; it is not built from the DIAL loss. The paper's statement that 'the two terms on the right hand side correspond to the source and domain loss in DIAL' glosses over the mismatch between the input-space W1 in the theorem and the embedding-space W1 minimized in Eqs. 2-4, which is a validity gap rather than a circular reduction. There are no load-bearing author self-citations, and the Sec. 6 admission that the reward model 'can find a shortcut... without actually transferring the meaningful domain-agnostic concepts' is an honest limitation, not evidence that the result was assumed.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on four assumptions: (1) semantic alignment of source and target embeddings preserves preference structure, which the authors concede can fail via shortcuts (Sec 6); (2) a finite Lipschitz constant for the reward; (3) standard optimal transport duality; (4) translation and rewriting preserve preference labels. No free parameters are fitted to target test data; the listed hyperparameters are per-experiment stability choices. The two-head architecture reuses a standard WGAN critic and a standard linear reward head, so no new entities are postulated.

free parameters (4)
  • Domain loss weight lambda = 0.01
    WDGRL default from prior literature (App A.1), not fitted to target test data.
  • Gradient penalty weight lambda_gp = 1.0
    Standard WGAN-GP value (App A.1), held fixed across experiments.
  • Few-shot-to-full DA learning rate = 1e-5
    Set per experiment 'to ensure stability' (App A.4); a hand-chosen hyperparameter, not a fitted constant of the claim.
  • Clean-to-noisy DA head weight decay = 0.01
    Added only for the DA head in this setting 'to ensure stability' (App A.3); a task-specific hand choice.
assumptions (4)
  • domain assumption Preference-relevant signal survives distribution alignment: aligning source and target embedding distributions preserves the separating structure between chosen and rejected responses.
    Core premise of Sec 2.1-2.2; the authors themselves flag the shortcut failure mode in Sec 6.
  • domain assumption The learned reward function r is K-Lipschitz for some finite K.
    Theorem 1 (Sec 2.3) requires it; K is never estimated, so the bound is qualitative.
  • standard math The critic converges to a valid estimate of the Wasserstein distance via Kantorovich-Rubinstein duality and WGAN training dynamics.
    Dual form Eq. 1 (Villani 2009) and adversarial game Eq. 4; convergence and the Lipschitz constraint are assumed to hold in practice.
  • domain assumption Machine translation (NLLB) and LLM rewriting preserve the preference order of the original data.
    Cross-lingual test sets are NLLB translations (App A.2) and clean-to-noisy source is Gemma-2-9b-it rewrites (App A.3); label preservation is assumed, not validated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Aligning LLMs with Domain Invariant Reward Models." pith.science (2026). https://pith.science/paper/CCICQZAD

@misc{pith2026250100911,
  author       = {Pith},
  title        = {Pith review of: Aligning LLMs with Domain Invariant Reward Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CCICQZAD}},
  note         = {Machine review of arXiv:2501.00911}
}
abstract

Aligning large language models (LLMs) to human preferences is challenging in domains where preference data is unavailable. We address the problem of learning reward models for such target domains by leveraging feedback collected from simpler source domains, where human preferences are easier to obtain. Our key insight is that, while domains may differ significantly, human preferences convey \emph{domain-agnostic} concepts that can be effectively captured by a reward model. We propose \method, a framework that trains domain-invariant reward models by optimizing a dual loss: a domain loss that minimizes the divergence between source and target distribution, and a source loss that optimizes preferences on the source domain. We show \method is a general approach that we evaluate and analyze across 4 distinct settings: (1) Cross-lingual transfer (accuracy: $0.621 \rightarrow 0.661$), (2) Clean-to-noisy (accuracy: $0.671 \rightarrow 0.703$), (3) Few-shot-to-full transfer (accuracy: $0.845 \rightarrow 0.920$), and (4) Simple-to-complex tasks transfer (correlation: $0.508 \rightarrow 0.556$). Our code, models and data are available at \url{https://github.com/portal-cornell/dial}.

Figures

Figures reproduced from arXiv: 2501.00911 by the authors.

Figure 1
Figure 1. DIAL trains domain-invariant reward model for target domains with no labeled preference data. DIAL leverages labeled source data and unlabeled target data to train reward models on a dual loss: a domain loss that minimizes the divergence between source and target distribution, and a source loss that optimizes preferences on the source domain. We show DIAL is a general approach that we evaluate and analyze across 4 d… view at source ↗
Figure 2
Figure 2. DIAL overview. DIAL takes labeled source and unlabeled target data and trains a domain-invariant reward model. The model takes prompt (x) and response (y), passes it through a base language model (θ) with two heads: a domain critic head (ψ) and a reward head (ϕ). The critic head is trained adversarially to minimize the Wasserstein distance between source and target embeddings while the reward head optimizes preferen… view at source ↗
Figure 3
Figure 3. Reward model embeddings learned by DIAL and Src-Pref across training iterations on (a) Cross-lingual Transfer and (b) Few-shot-to-full Transfer. Src-Pref separates source embeddings, but not target embeddings, resulting in poor transfer. DIAL learns embeddings that cluster (source positive, target positive) and (source negative, target negative) allowing for better reward transfer. aligning embeddings between the tw… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Scaling with Data on legaladvice-Korean split. (a) DIAL performance with varying source-target data mix (b) DIAL scaling with unlabeled target data vs Src-Tgt-Pref scaling with labeled target data. Resuls on 3 seeds. fectly align with source and transfer rewards. 3.8 H…
Figure 5
Figure 5. Figure 5: Spurious reward. Accuracy results on odd￾one-out (100 datapoints) over 3 seeds. DIAL learns the correct reward, while Src-Pref learns spurious reward of “not source” which performs similar to random. To introduce spurious correlations, we create a synthetic task of cho…
Figure 7
Figure 7. Figure 7: Decision boundaries for zero-shot training [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 18 canonical work pages

  1. [1]

    Cem Anil, Yuhuai Wu, Anders Johan Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Venkatesh Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. 2022. https://openreview.net/forum?id=zSkYVeX7bC4 Exploring length generalization in large language models . In Advances in Neural Information Processing Systems

    work page 2022

  2. [2]

    https://claude.ai/ Claude

    Anthropic. https://claude.ai/ Claude

    work page

  3. [3]

    Mart \'i n Arjovsky, Soumith Chintala, and L \'e on Bottou. 2017. https://api.semanticscholar.org/CorpusID:13943041 Wasserstein gan . ArXiv, abs/1701.07875

    arXiv 2017

  4. [4]

    Ralph Allan Bradley and Milton E. Terry. 1952. https://api.semanticscholar.org/CorpusID:125209808 Rank analysis of incomplete block designs: I. the method of paired comparisons . Biometrika, 39:324

    1952

  5. [5]

    Gonzalez, and Ion Stoica

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. https://arxiv.org/abs/2403.04132 Chatbot arena: An open platform for evaluating llms by human preference . Preprint, arXiv:2403.04132

    arXiv 2024

  6. [7]

    Scott Crossley, Perpetual Baffour, Jules King, Lauryn Burleigh, Walter Reade, and Maggie Demkin. 2024. https://kaggle.com/competitions/learning-agency-lab-automated-essay-scoring-2 Learning agency lab - automated essay scoring 2.0. kaggle

    work page 2024

  7. [8]

    Bharath Bhushan Damodaran, Benjamin Kellenberger, R \'e mi Flamary, Devis Tuia, and Nicolas Courty. 2018. https://api.semanticscholar.org/CorpusID:4331539 Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation . In European Conference on Computer Vision

    work page 2018

  8. [9]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . arXiv preprint arXiv:2407.21783

    arXiv 2024

Show all 53 references
  1. [10]

    Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. 2022. https://proceedings.mlr.press/v162/ethayarajh22a.html Understanding dataset difficulty with V -usable information . In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of...

    2022

  2. [11]

    Alex Franklin, Maggie, Meg Benner, Natalie Rambis, Perpetual Baffour, Ryan Holbrook, Scott Crossley, and ulrichboser. 2022. https://kaggle.com/competitions/feedback-prize-effectiveness Feedback prize - predicting effective arguments

    2022

  3. [12]

    Ustinova, Hana Ajakan, Pascal Germain, H

    Yaroslav Ganin, E. Ustinova, Hana Ajakan, Pascal Germain, H. Larochelle, François Laviolette, Mario Marchand, and Victor S. Lempitsky. 2015. https://api.semanticscholar.org/CorpusID:2871880 Domain-adversarial training of neural networks . In Journal of machine learning research

    2015

  4. [13]

    GemmaTeam. 2024 a . https://api.semanticscholar.org/CorpusID:270843326 Gemma 2: Improving open language models at a practical size . ArXiv, abs/2408.00118

    2024 arXiv

  5. [14]

    GemmaTeam. 2024 b . https://api.semanticscholar.org/CorpusID:268379206 Gemma: Open models based on gemini research and technology . ArXiv, abs/2403.08295

    2024 arXiv

  6. [15]

    Courville

    Ishaan Gulrajani, Faruk Ahmed, Mart \' n Arjovsky, Vincent Dumoulin, and Aaron C. Courville. 2017. https://arxiv.org/abs/1704.00028 Improved training of wasserstein gans . CoRR, abs/1704.00028

    2017 arXiv

  7. [16]

    Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, et al. 2024. Direct language model alignment from online ai feedback. arXiv preprint arXiv:2402.04792

    2024 arXiv

  8. [17]

    Peter Hase, Mohit Bansal, Peter Clark, and Sarah Wiegreffe. 2024. https://api.semanticscholar.org/CorpusID:266977266 The unreasonable effectiveness of easy training data for hard tasks . ArXiv, abs/2401.06751

    2024 arXiv

  9. [18]

    Dan Hendrycks and Kevin Gimpel. 2016. https://api.semanticscholar.org/CorpusID:125617073 Gaussian error linear units (gelus) . arXiv: Learning

    2016

  10. [19]

    Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen

    J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021. https://api.semanticscholar.org/CorpusID:235458009 Lora: Low-rank adaptation of large language models . ArXiv, abs/2106.09685

    2021 arXiv

  11. [20]

    Haoyang Huang, Tianyi Tang, Dongdong Zhang, Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. https://openreview.net/forum?id=E4ebDehO3O Not all languages are created equal in LLM s: Improving multilingual capability by cross-lingual-thought prompting . In The 2023 Conference ...

    2023

  12. [21]

    HuggingFace. 2023. https://huggingface.co/docs/trl/en/index Huggingface transformers reinforcement learning

    2023

  13. [22]

    Constantinos Karouzos, Georgios Paraskevopoulos, and Alexandros Potamianos. 2021. https://aclanthology.org/2021.naacl-main.203/ Udalm: Unsupervised domain adaptation through language modeling . arXiv preprint arXiv:2104.07078

    2021 arXiv

  14. [23]

    Dongyoung Kim, Kimin Lee, Jinwoo Shin, and Jaehyung Kim. 2024. https://api.semanticscholar.org/CorpusID:270357971 Aligning large language models with self-generated preference data . ArXiv, abs/2406.04412

    2024 arXiv

  15. [24]

    Sungdong Kim, Sanghwan Bae, Jamin Shin, Soyoung Kang, Donghyun Kwak, Kang Min Yoo, and Minjoon Seo. 2023. https://openreview.net/forum?id=8gYRHspcxK Aligning large language models through synthetic feedback . In The 2023 Conference on Empirical Methods in Natural Language Processing

    2023

  16. [25]

    Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. 2023. https://api.semanticscholar.org/CorpusID:263830929 Understanding the effects of rlhf on llm generalisation and diversity . ArXiv, abs/2310.06452

    2023 arXiv

  17. [26]

    Smith, and Hannaneh Hajishirzi

    Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2024. https://arxiv.org/abs/2403.13787 Rewardbench: Evaluating reward models for language mo...

    2024 arXiv

  18. [27]

    Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. 2018. https://arxiv.org/abs/1811.07871 Scalable agent alignment via reward modeling: a research direction . arXiv preprint arXiv:1811.07871

    2018 arXiv

  19. [28]

    Jinlong Li, Runsheng Xu, Jin Ma, Qin Zou, Jiaqi Ma, and Hongkai Yu. 2023. https://api.semanticscholar.org/CorpusID:259983180 Domain adaptation based object detection for autonomous driving in foggy and rainy weather . IEEE Transactions on Intelligent Vehicles

    2023

  20. [29]

    Xiaochen Li, Zheng-Xin Yong, and Stephen H. Bach. 2024. https://api.semanticscholar.org/CorpusID:270703187 Preference tuning for toxicity mitigation generalizes across languages . ArXiv, abs/2406.16235

    2024 arXiv

  21. [30]

    Ilya Loshchilov and Frank Hutter. 2017. https://api.semanticscholar.org/CorpusID:3312944 Fixing weight decay regularization in adam . ArXiv, abs/1711.05101

    2017 arXiv

  22. [31]

    NllbTeam. 2022. https://api.semanticscholar.org/CorpusID:250425961 No language left behind: Scaling human-centered machine translation . ArXiv, abs/2207.04672

    2022 arXiv

  23. [32]

    https://chatgpt.com/ Chatgpt

    OpenAI. https://chatgpt.com/ Chatgpt

  24. [33]

    Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan...

    2022 arXiv

  25. [34]

    Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2024. https://arxiv.org/abs/2308.01263 Xstest: A test suite for identifying exaggerated safety behaviours in large language models . Preprint, arXiv:2308.01263

    2024 arXiv

  26. [35]

    John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. https://arxiv.org/abs/1707.06347 Proximal policy optimization algorithms . Preprint, arXiv:1707.06347

    2017 arXiv

  27. [36]

    Bernstein, and Diyi Yang

    Omar Shaikh, Michelle Lam, Joey Hejna, Yijia Shao, Michael S. Bernstein, and Diyi Yang. 2024. https://api.semanticscholar.org/CorpusID:270222089 Show, don't tell: Aligning language models with demonstrated feedback . ArXiv, abs/2406.00888

    2024 arXiv

  28. [37]

    Jian Shen, Yanru Qu, Weinan Zhang, and Yong Yu. 2018. https://arxiv.org/abs/1707.01217 Wasserstein distance guided representation learning for domain adaptation . Preprint, arXiv:1707.01217

    2018 arXiv

  29. [38]

    Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, and Chuang Gan. 2024. https://api.semanticscholar.org/CorpusID:268385111 Easy-to-hard generalization: Scalable alignment beyond human supervision . ArXiv, abs/2403.09472

    2024 arXiv

  30. [39]

    Eshaan Tanwar, Subhabrata Dutta, Manish Borthakur, and Tanmoy Chakraborty. 2023. https://arxiv.org/abs/2305.05940 Multilingual llms are better cross-lingual in-context learners with alignment . Preprint, arXiv:2305.05940

    2023 arXiv

  31. [40]

    Jeremy Tien, Jerry Zhi-Yang He, Zackory Erickson, Anca D Dragan, and Daniel S Brown. 2022. https://arxiv.org/abs/2204.06601 Causal confusion and reward misidentification in preference-based reward learning . arXiv preprint arXiv:2204.06601

    2022 arXiv

  32. [41]

    Chernova, and Dhruv Batra

    Joanne Truong, S. Chernova, and Dhruv Batra. 2020. https://api.semanticscholar.org/CorpusID:227162315 Bi-directional domain adaptation for sim2real transfer of embodied navigation agents . IEEE Robotics and Automation Letters, 6:2634--2641

    2020

  33. [42]

    Zhang, Kate Saenko, and Trevor Darrell

    Eric Tzeng, Judy Hoffman, N. Zhang, Kate Saenko, and Trevor Darrell. 2014. https://api.semanticscholar.org/CorpusID:17169365 Deep domain confusion: Maximizing for domain invariance . ArXiv, abs/1412.3474

    2014 arXiv

  34. [43]

    C \'e dric Villani et al. 2009. https://link.springer.com/book/10.1007/978-3-540-71050-9 Optimal transport: old and new , volume 338. Springer

    2009 doi

  35. [44]

    Genta Winata, Shijie Wu, Mayank Kulkarni, Thamar Solorio, and Daniel Preo t iuc-Pietro. 2022 a . https://aclanthology.org/2022.aacl-main.59/ Cross-lingual few-shot learning on unseen languages . In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Associatio...

    2022

  36. [45]

    Genta Indra Winata, Shijie Wu, Mayank Kulkarni, Thamar Solorio, and Daniel Preoțiuc-Petro. 2022 b . https://aclanthology.org/2022.aacl-main.59.pdf Cross-lingual few-shot learning on unseen languages . In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Asso...

    2022

  37. [46]

    Zhaofeng Wu, Ananth Balashankar, Yoon Kim, Jacob Eisenstein, and Ahmad Beirami. 2024. https://api.semanticscholar.org/CorpusID:269214448 Reuse your rewards: Reward model transfer for zero-shot cross-lingual alignment . ArXiv, abs/2404.12318

    2024 arXiv

  38. [47]

    Guohai Xu, Jiayi Liu, Ming Yan, Haotian Xu, Jinghui Si, Zhuoran Zhou, Peng Yi, Xing Gao, Jitao Sang, Rong Zhang, Ji Zhang, Chao Peng, Fei Huang, and Jingren Zhou. 2023. https://arxiv.org/abs/2307.09705 Cvalues: Measuring the values of chinese large language models from safety ...

    2023 arXiv

  39. [48]

    Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. 2024. https://api.semanticscholar.org/CorpusID:270521260 Regularizing hidden states enables learning generalizable reward model for llms . ArXiv, abs/2406.10216

    2024 arXiv

  40. [49]

    Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. 2024. https://arxiv.org/abs/2408.15240 Generative verifiers: Reward modeling as next-token prediction . arXiv preprint arXiv:2408.15240

    2024 arXiv

  41. [50]

    Denny Zhou, Nathanael Scharli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, and Ed Huai hsin Chi. 2022. https://api.semanticscholar.org/CorpusID:248986239 Least-to-most prompting enables complex reasoning in large language models ....

    2022 arXiv

  42. [51]

    Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. 2017. https://api.semanticscholar.org/CorpusID:206770979 Unpaired image-to-image translation using cycle-consistent adversarial networks . 2017 IEEE International Conference on Computer Vision (ICCV), pages 2242--2251

    2017

  43. [52]

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593

    2019 arXiv

  44. [53]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  45. [54]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.