{"id":"23b6e2ab-7dc2-44ae-a402-bda6d99c6bb0","arxiv_id":"2501.00911","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Applying Wasserstein-distance domain adaptation, a known technique, to reward models lets preference signals learned on labeled source data transfer to unlabeled target domains, with consistent but modest gains across four settings.","lead":"DIAL is a training method that transfers AI reward-model knowledge from domains with human preference labels (like English) to domains without them (like Korean), by pulling the two domains' data representations together during training. A smart generalist should care because preference labels are scarce in most languages and tasks, so a working transfer recipe would lower the cost of aligning AI systems.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Marginal embedding alignment (Eqs. 2-4) does not constrain the conditional preference structure within each domain; DIAL's gains could come from regularization rather than preference transfer, and a source-label-shuffle ablation would test this.","rationale":"The reader's weakest_assumption identifies the same load-bearing point: aligning source and target marginal distributions over prompt-response embeddings does not, by itself, preserve the signal that separates chosen from rejected responses. I agree with that assessment and with the CONDITIONAL verdict. The paper has genuine strengths: clear method description, four reasonably held-out evaluations with standard errors, a thoughtful limitation section that explicitly acknowledges the shortcut risk, and a synthetic odd-one-out experiment that partially addresses spurious rewards. None of these, however, rules out the possibility that the observed target improvements come from the unlabeled target data acting as an implicit regularizer or from target-specific surface cues rather than from transfer of domain-agnostic preference concepts. The theory, as written, does not bridge this gap because it bounds target error by the input-space Wasserstein distance while the algorithm minimizes an embedding-space Wasserstein distance; that is a supporting weakness rather than the primary one. The label-shuffle ablation I propose is the most direct way to separate 'preference transfer' from 'representation regularization': if shuffling source labels does not degrade target accuracy, the central claim is not supported; if it does degrade target accuracy, the mechanism is validated and the paper's main empirical conclusion stands. I therefore recommend keeping the reader's CONDITIONAL verdict rather than moving to ACCEPT, since the missing control is cheap, central, and currently absent.","tokens_in":20658,"tokens_out":3965,"duration_ms":41522,"concrete_test":"Run a label-shuffle ablation on at least two settings, e.g., legaladvice-Korean (App. 1) and CValues Split A (App. 3): train DIAL with source (chosen, rejected) labels randomly permuted (or sign-flipped) under identical hyperparameters, and compare target accuracy to intact-label DIAL and Src-Pref. If shuffled-label DIAL matches intact-label DIAL within one standard error, the gain is not preference transfer. If shuffled-label DIAL drops toward Src-Pref or random, the concern is resolved. Also report the achieved embedding Wasserstein distance in both conditions to confirm the domain loss is actually minimized.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the adversarial Wasserstein alignment in Eqs. (2)-(4) preserves the source preference signal while making source and target prompt-response marginals indistinguishable. Nothing in the objective enforces this: the domain loss is a function of the joint marginal over (x, y) only, while the preference signal is a conditional property within each domain. A critic with a gradient penalty can be driven to zero by collapsing or rotating away the embedding dimensions on which the Bradley-Terry loss depends; the linear reward head then has no route to transfer those dimensions to the target. The authors concede this exact failure mode in Sec. 6: 'the reward model can find a shortcut to successfully align the source and target embedding representations and do well on source, without actually transferring the meaningful domain-agnostic concepts.' The reported target gains could therefore be produced by the extra unlabeled data acting as a regularizer or by target-specific surface cues, rather than by the claimed preference transfer. Theorem 1 (Eq. 7) does not close the gap: it bounds target error by W1 of the input-space distributions mu_S and mu_T, whereas DIAL minimizes W1 between learned embeddings. Minimizing an embedding-space distance does not upper-bound the input-space distance appearing in Eq. (7), so the theorem is not a justification of the optimized objective. What is missing is a control showing that the preference labels, not just the marginal alignment, drive DIAL's target accuracy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DIAL, a framework for training reward models on labeled source-domain preference data plus unlabeled target-domain prompt-response pairs. The model uses a base LLM with two heads: a critic head adversarially trained to minimize the Wasserstein distance between source and target embeddings (with a gradient penalty), and a linear reward head trained with a Bradley-Terry loss on the source preferences. The central claim is that this dual objective learns reward models that transfer to target domains with no preference labels, and the paper evaluates this across four settings: cross-lingual (English to Korean/Thai/Chinese), clean-to-noisy, few-shot-to-full, and simple-to-complex. The paper also provides a theoretical bound on target error, a spurious-reward toy experiment, an RLHF distribution-shift analysis, and scaling experiments.","tokens_in":20782,"tokens_out":7666,"duration_ms":69346,"significance":"If the empirical gains are robust, DIAL offers a simple and general recipe for transferring reward models to unlabeled domains, which is practically valuable for RLHF in low-resource settings. The paper's strengths include held-out target evaluations with standard errors over three seeds, comparisons against two regularization baselines and oracle upper bounds, a released code/data repository, and a synthetic odd-one-out experiment that provides a falsifiable test of the transfer mechanism and shows average gains (0.559 vs 0.318). However, the theoretical result is not connected to the optimized objective, and the evaluation in the simple-to-complex setting is weaker than the abstract suggests; these issues need to be addressed before the central claim can be fully accepted.","major_comments":[{"comment":"Theorem 1 bounds target error in terms of the input-space Wasserstein distance W1(µS, µT) over (x, y), but the DIAL domain loss in Eqs. (2)-(4) minimizes a Wasserstein distance between learned embeddings, i.e., W1(θ#µS, θ#µT). Since a map can contract distances, the minimized embedding-space distance does not upper-bound the input-space quantity appearing in Eq. (7); the theorem as stated is therefore not a justification of the optimized objective. The authors should either provide a bound on target error in terms of the embedding-space Wasserstein distance (with an appropriate representation-error or inverse-Lipschitz term), or explicitly present Theorem 1 as heuristic motivation rather than as a guarantee for DIAL.","section":"Sec. 2.3, Eq. (7)"},{"comment":"The domain loss aligns marginal distributions of (x, y) embeddings, while the preference structure is a conditional property within each domain. The paper's own Sec. 6 concedes that the reward model can align source and target marginals while learning a source-specific spurious reward. To substantiate the claim that alignment transfers the preference signal rather than acting as an unlabeled-data regularizer, a control experiment is needed; for example, shuffling the source preference labels should eliminate DIAL's target gains if the mechanism is preference transfer, whereas a pure regularization effect would persist. Such an ablation, or an analysis of which embedding dimensions the reward head relies on before and after alignment, is necessary to support the paper's central conclusion.","section":"Sec. 2.2 / Sec. 6"},{"comment":"In the simple-to-complex setting, Src-Pref-SFT attains Spearman 0.562±0.023 versus DIAL's 0.556±0.011, and DIAL's Pearson improvement over Src-Pref-SFT (0.577±0.011 vs 0.567±0.018) is within one standard error. The abstract's '0.508 → 0.556' and the conclusion's 'significant gains in target performance' therefore hold only against the Src-Pref baseline, not against the strongest regularized baseline. This should be stated explicitly in the abstract and conclusion, or the claim that DIAL 'outperforms all baselines' across all four settings should be qualified for this setting.","section":"Sec. 3.5, Table 4"},{"comment":"The appendix states that the clean-to-noisy setting used weight decay 0.01 and the few-shot-to-full setting used learning rate 1e−5 because these choices were 'helpful' or 'ensuring stability'. The paper does not state whether these hyperparameters were selected on a held-out validation split of the target domain or on the reported test sets. If the latter, the held-out claim is compromised and the reported advantages over baselines could reflect per-setting tuning. Please document the validation protocol for each setting and report sensitivity of the results to the two hyperparameters λ and λ_gp that appear in Eq. (4).","section":"Appendix A.3 / A.4, Sec. 3.1"}],"minor_comments":[{"comment":"The aggregate improvements in the abstract (0.621 → 0.661, 0.671 → 0.703, 0.845 → 0.920, 0.508 → 0.556) are not derived from any table; please state the averaging formula or cite the corresponding table rows, particularly since Table 1 reports per-language and per-subreddit numbers that vary substantially.","section":"Abstract / Tables 1-4"},{"comment":"The pseudocode line `critic_loss = - wd + gp_loss` does not include the weight λ_gp that appears in Eq. (4); clarify whether λ_gp = 1 is assumed in the algorithm description or include the weight explicitly.","section":"Algorithm 1"},{"comment":"Section 3.8 says the odd-one-out evaluation uses 100 datapoints, while Appendix A.6 describes 1000 train, validation, and test examples; please reconcile these numbers.","section":"Sec. 3.8 vs Appendix A.6"},{"comment":"The main theorem is numbered Theorem 1 in Section 2.3 but Theorem 2 in Appendix B; unify the numbering across the paper.","section":"Sec. 2.3 / Appendix B"},{"comment":"Equation (2) uses the notation Dsrc for an expectation over (x_s, y_s), but Dsrc was defined in Section 2 as a set of preference triplets (x, y+, y−); clarify which marginal of the source distribution is used for the critic loss.","section":"Eq. (2)"},{"comment":"The related work section lists Winata et al. (2022a) and (2022b) with overlapping titles and appears to cite the same paper twice; please merge these entries or clearly distinguish them if they are different works.","section":"Related Work"}],"recommendation":"major_revision","confidential_remarks":"The empirical study is careful in several respects: three seeds, standard errors, oracle upper bounds, and a released repository. The main risk is that the theoretical section currently appears to justify the algorithm but does not, because the input-space Wasserstein bound in Eq. (7) is not connected to the embedding-space objective that DIAL minimizes. The label-shuffle ablation suggested in the report is the most direct way to test the preference-transfer mechanism; if the authors can provide it, along with a clear validation protocol for hyperparameters, I would be willing to revisit the manuscript favorably. I also recommend asking the authors to soften the abstract's simple-to-complex claim, since the best baseline is not beaten on Spearman."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is WDGRL (Shen et al. 2018) applied to LLM reward models, and the paper mostly says so. The empirical package is the real contribution: held-out evaluations across four transfer settings, standard errors over three seeds, consistent wins over source-only and target-regularization baselines, and a limitations section that names the main mechanism risk. For someone working on RLHF generalization it deserves referee time, not desk rejection.\n\nWhat is new and good: attaching a Wasserstein domain critic to a reward head and showing that unlabeled target data helps across cross-lingual, clean-to-noisy, few-shot-to-full, and simple-to-complex transfer. The gains are modest (a few accuracy points, about 0.05 correlation), but consistent, and the few-shot CValues result is strong. The odd-one-out toy and the online RLHF adaptation experiment are sensible extras.\n\nSoft spots: the theory in Sec. 2.3 does not connect to the algorithm. Theorem 1 bounds target error using W1 over input-space distributions; DIAL minimizes W1 between learned embeddings. No bridge is established, so the theorem reads as a generic domain-adaptation bound rather than a justification of this objective. That should be fixed or softened. App 4 is oversold: the '0.508 → 0.556' framing implies DIAL wins on correlation, but Src-Pref-SFT has higher Spearman and the Pearson gap is within one standard error. The text does disclose this, but the abstract/headline framing should match. The stress-test worry about marginal alignment is fair: aligning marginals does not by itself preserve the conditional preference structure, and the authors concede the shortcut failure mode in Sec. 6. The wins over Src-Pref-Tgt-NTP suggest target data alone is not enough, and the odd-one-out result supports genuine preference transfer, but a source-label-shuffle or reward-head-scratch ablation would pin the mechanism down. That is a missing control, not a fatal flaw. Also minor: cross-lingual and clean-to-noisy transfer assume NLLB translation and Gemma rewriting preserve preference labels, which is never validated; and the per-application stability tuning is a bit ad hoc.\n\nWho it's for: an application paper, not a new-method paper. A reader interested in reward-model transfer or domain adaptation for LLMs gets a clean, honest empirical study with one instructive theoretical gap and one overstated conclusion.\n\nRecommendation: send to peer review. Ask for a revised theory section that either bridges embedding-space W1 to input-space W1 or drops the claim that Theorem 1 justifies DIAL, and ask them to qualify the App-4 claim. I would cite the empirical results with the mechanism caveat, not the theory.","headline":"A solid WDGRL-for-reward-models application with honest empirics; the theory is decorative and App-4 is oversold, but the central transfer claim holds up.","tokens_in":21523,"tokens_out":6212,"would_cite":true,"duration_ms":57093,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that training a reward model to align source and target embeddings with an adversarial Wasserstein loss while optimizing source preferences transfers human preferences to unlabeled target domains, and demonstrates gains…","keywords":["reward modeling","domain adaptation","Wasserstein distance","RLHF","domain-invariant representation","preference transfer","LLM alignment","unlabeled target data"],"falsifier":"Run DIAL on a synthetic target whose preferred category is inverted relative to source, e.g., odd-one-out with source (in: fruit, out: vegetable) and target (in: vegetable, out: fruit). If the Wasserstein alignment collapses fruit and vegetable embeddings, the reward head can satisfy both losses by learning 'not fruit' as a spurious reward, and target accuracy should drop to chance (0.2); observing that would refute the claim that alignment alone transfers meaningful preferences.","tokens_in":20282,"feed_emoji":"🎯","tokens_out":6558,"duration_ms":57446,"temperature":0.7,"pith_summary":"The paper tries to establish that human preference signals are domain-agnostic enough that a reward model can transfer them from a labeled source domain to an unlabeled target domain, provided the model is forced to align source and target embeddings. The proposed method, DIAL, trains a base LLM with two heads: a critic head adversarially minimizing the Wasserstein distance between source and target embeddings, and a reward head optimizing a Bradley-Terry preference loss on source labels. Across four settings—cross-lingual, clean-to-noisy, few-shot-to-full, and simple-to-complex—this dual loss improves target reward accuracy or correlation over source-only training and over regularization baselines, sometimes matching oracle models trained with target labels. A theorem bounds target error by source error plus a Wasserstein term, giving the alignment objective a theoretical rationale. If correct, this gives a recipe for aligning LLMs in domains where preference annotation is expensive or impossible.","feed_headline":"No-target-labels reward training wins across 4 transfer settings","feed_subtitle":"Adversarial embedding alignment carries human preferences across languages, noise, and task complexity.","key_machinery":"The mechanism is the 1-Wasserstein distance, a measure of the minimum cost to transport one distribution onto another, computed through its Kantorovich-Rubinstein dual as the largest gap a 1-Lipschitz function can give between expectations. DIAL instantiates it as a critic head trained adversarially against the base model: the critic maximizes the expected source-minus-target score, a gradient penalty enforces the Lipschitz constraint, and the base model minimizes that distance to align embeddings. A separate linear reward head minimizes the Bradley-Terry source loss, and the two losses are optimized together so that aligned embeddings also separate preferred from rejected responses. Theorem 1 states that target error is bounded by source error plus $2 K L_\\sigma W_1(\\mu_S, \\mu_T)$, making the paper's case that the domain loss is the right transfer objective.","core_discovery":"The central claim is that a reward model can learn domain-agnostic human preferences by jointly making source and target prompt-response embeddings indistinguishable under an adversarially trained Wasserstein critic and separating chosen from rejected responses on the labeled source domain. DIAL implements this as a min-max game on the Wasserstein distance with gradient penalty, plus a Bradley-Terry loss. The paper reports that this beats source-only and regularization baselines in all four transfer settings, reaches oracle accuracy on several cross-lingual and clean-to-noisy splits, nearly matches the target-trained oracle in few-shot transfer, and helps PPO policies adapt to distribution shift during RLHF. The authors conclude that the domain loss carries the preference signal across domains because the aligned embeddings cluster source-positive with target-positive and source-negative with target-negative.","pith_inferences":["Editorial extension: the same dual-loss idea should be testable on preference inversions; if the target's chosen/rejected order flips relative to the source, marginal Wasserstein alignment would be expected to hurt accuracy because it removes the very variation that carries the reward signal.","Editorial extension: the paper's Theorem 1 bounds error using the Wasserstein distance on input space, while DIAL minimizes a Wasserstein distance on embeddings; closing that gap would tell whether the alignment objective is genuinely the quantity in the bound.","Editorial extension: the RLHF distribution-shift experiment suggests DIAL could serve as a lightweight online reward-adaptation loop for any off-policy alignment pipeline, not just safety refusal tasks."],"forward_implications":["If DIAL works as claimed, reward models for low-resource languages can be trained with English preference labels plus unlabeled target text, and on several language/split combinations DIAL matches oracle models that used target labels.","If the RLHF results hold, on-policy preference annotation during PPO can be partially replaced by using the current policy's responses as unlabeled target data for periodic reward adaptation.","The bound $\\epsilon_T(r,f) \\leq \\epsilon_S(r,f) + 2KL_\\sigma W_1(\\mu_S,\\mu_T)$ implies that improving source accuracy and reducing the Wasserstein distance between source and target embeddings both directly reduce worst-case target error.","DIAL is claimed to be a general transfer method, with consistent gains over source-only and two regularization baselines in all four settings, so the same recipe should transfer to other unlabeled target domains with correlated human preferences."],"supporting_citations":[{"why":"Supplies the Wasserstein GAN formulation and critic-based estimate of the 1-Wasserstein distance used as the domain loss.","marker":"Arjovsky et al., 2017"},{"why":"Provides the gradient penalty that enforces the Lipschitz constraint on the critic in DIAL.","marker":"Gulrajani et al., 2017"},{"why":"Gives the pairwise preference model used as the source reward loss.","marker":"Bradley and Terry, 1952"},{"why":"Source for optimal transport theory and the Kantorovich-Rubinstein duality that the Theorem 1 bound relies on.","marker":"Villani et al., 2009"},{"why":"The Stanford Human Preferences dataset that supplies source and target data for the cross-lingual and clean-to-noisy evaluations.","marker":"Ethayarajh et al., 2022"},{"why":"The Src-Pref-SFT baseline DIAL is compared against.","marker":"Yang et al., 2024"},{"why":"The Src-Pref-Tgt-NTP unsupervised domain adaptation baseline DIAL is compared against.","marker":"Karouzos et al., 2021"},{"why":"Establishes the RLHF reward modeling setup and the linear reward head used in DIAL.","marker":"Ouyang et al., 2022"},{"why":"The CValues safety preference dataset used in the few-shot-to-full transfer evaluation.","marker":"Xu et al., 2023"}],"fun_headline_variants":["Domain-invariant rewards transfer preferences without target labels","No target labels? Align reward models across domains instead","Adversarial embedding alignment carries preferences across tasks","Reward models learn universal preferences from source domains","Source-only rewards beat target-trained in 4 transfer settings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that making source and target prompt-response embeddings look alike preserves the specific signal that tells chosen from rejected responses; the paper explicitly concedes that alignment can instead succeed by removing that variation and exploiting a spurious source-specific reward.","fun_headline_variants_meta":{"raw":{"variants":["Domain-invariant rewards transfer preferences without target labels","No target labels? Align reward models across domains instead","Adversarial embedding alignment carries preferences across tasks","Reward models learn universal preferences from source domains","Source-only rewards beat target-trained in 4 transfer settings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000582,"raw_usage":{"total_tokens":2734,"prompt_tokens":933,"completion_tokens":1801,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":1726}},"tokens_in":549,"tokens_out":1801,"duration_ms":14071,"temperature":1.0,"reasoning_tokens":1726,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:40:09.311728+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DIAL on a synthetic target whose preferred category is inverted relative to source, e.g., odd-one-out with source (in: fruit, out: vegetable) and target (in: vegetable, out: fruit). If the Wasserstein alignment collapses fruit and vegetable embeddings, the reward head can satisfy both losses by learning 'not fruit' as a spurious reward, and target accuracy should drop to chance (0.2); observing that would refute the claim that alignment alone transfers meaningful preferences.","supporting_citations":[{"cited_title":"UDALM: Unsupervised Domain Adaptation through Language Modeling","cited_arxiv_id":"2104.07078","evidence_quote":"The Src-Pref-Tgt-NTP unsupervised domain adaptation baseline DIAL is compared against."}],"review_version":1}