Preference Learning for AI Alignment: a Causal Perspective

Katarzyna Kobalczyk; Mihaela van der Schaar

arxiv: 2506.05967 · v1 · submitted 2025-06-06 · 💻 cs.AI · cs.LG· stat.ML

Preference Learning for AI Alignment: a Causal Perspective

Katarzyna Kobalczyk , Mihaela van der Schaar This is my paper

Pith reviewed 2026-05-19 11:18 UTC · model grok-4.3

classification 💻 cs.AI cs.LGstat.ML

keywords preference learningreward modellingcausal inferenceAI alignmentlarge language modelsgeneralizationconfounding

0 comments

The pith

Framing reward modelling from preference data in a causal paradigm identifies misidentification, heterogeneity and confounding as barriers to reliable generalisation in LLM alignment.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes treating reward modelling from human preference data as a causal inference task rather than a purely statistical one. This lets the authors use established causal tools to name three recurring problems that block good performance on new prompts: mistaking non-causal correlations for the true drivers of preference, variation in preferences across users, and hidden user-specific factors that distort the observed data. A sympathetic reader cares because current preference-based training often fails to produce models that behave as intended outside the training distribution, and causal framing supplies concrete assumptions and interventions that could reduce those failures. The authors contrast everyday data-collection habits with the conditions required for causal identification and show illustrative cases where naive models break while causally-aware ones hold up better.

Core claim

Reward modelling from preference data can be reframed inside a causal paradigm; doing so surfaces the problems of causal misidentification, preference heterogeneity and confounding by user-specific factors, states the identification assumptions needed for reliable generalisation, and shows that causally motivated adjustments can reduce the failure modes exhibited by standard reward models.

What carries the argument

Causal identification strategies that contrast common preference-data collection practices against the assumptions required for generalisation from observational data.

Load-bearing premise

The challenges of misidentification, heterogeneity and confounding in preference data are mainly solvable by applying causal identification techniques rather than by other modelling or data changes.

What would settle it

A controlled experiment on held-out prompt-response pairs in which a reward model fitted under explicit causal assumptions generalises measurably better than a standard model trained on the same preference data, or fails to do so.

Figures

Figures reproduced from arXiv: 2506.05967 by Katarzyna Kobalczyk, Mihaela van der Schaar.

**Figure 1.** Figure 1: The causal model of preferences. Given prompt X and the two responses Y , Y ′ users assigns them unobservable rewards R, R′ determining the preference label L. We can think of the observed tuples (X, Y, Y ′ ) as treatments assigned to human labellers tasked with selecting a response that they prefer and the observed labels L as outcomes. Treatments are assigned according to some (often unknown) propensiti… view at source ↗

**Figure 2.** Figure 2: Confounding due to user-specific objectives. a) The user-specific contextual variable C can act as a confounder. If the prompts X are written by the users themselves, C affects the assigned rewards R, R′ and partially determines the treatment (X, Y, Y ′ ). b) Even if C is not confounding, it may influence the user specific rewards, introducing individual-level variation in treatment effects. Confoundi… view at source ↗

**Figure 3.** Figure 3: The latent treatment model. The effect of observed texts on R can be compressed into a set of latent variables Z partitioned into two kinds: Z X–the artifacts of X and Z T –the latent treatments determined jointly by X and Y . We assume that features of texts that affect the rewards can be effectively summarised into a set of latent features Z = {Z1, . . . , Zn} ∈ Z split into two parts: Z X–artefacts of t… view at source ↗

**Figure 4.** Figure 4: A reward model relying on the set of true causal [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

**Figure 5.** Figure 5: Comparison of model architectures. type(X) while retaining causal features relevant to the outcome. The training objective for this model is: min θ,w0,w1 max ϕ LR(θ, w0, w1) − λLadv(θ, ϕ), (5) where LR(θ, w0, w1) is the standard BTL loss for the reward function rθ,w0,w1 , the second term Ladv(θ, ϕ) is the binary cross-entropy loss between the true c’s and their logprobabilities predicted by hϕ ◦gθ, and … view at source ↗

**Figure 6.** Figure 6: Test time accuracy vs. confounding. The label “consistent” indicates whether type(X) = C. The causallyinspired multihead architecture with additional adversarial balancing significantly reduces overfitting and improves generalisation to the inconsistent OOD examples. 4.2. Case Study Results We analyse the results of the experiments summarised in [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗

**Figure 7.** Figure 7: The impact of ρ on determining αˆ and the classification accuracy. The decision boundary determining the preference label L is dependent on the ground-truth value of α. It is defined by a straight line with a slope of − α 1−α and the intercept at the origin. In a noise-free setting, examples (x, y, y′ , ℓ) for which δ falls below the decision boundary are labelled with ℓ = 0 (i.e, the first option (x, y) w… view at source ↗

read the original abstract

Reward modelling from preference data is a crucial step in aligning large language models (LLMs) with human values, requiring robust generalisation to novel prompt-response pairs. In this work, we propose to frame this problem in a causal paradigm, providing the rich toolbox of causality to identify the persistent challenges, such as causal misidentification, preference heterogeneity, and confounding due to user-specific factors. Inheriting from the literature of causal inference, we identify key assumptions necessary for reliable generalisation and contrast them with common data collection practices. We illustrate failure modes of naive reward models and demonstrate how causally-inspired approaches can improve model robustness. Finally, we outline desiderata for future research and practices, advocating targeted interventions to address inherent limitations of observational data.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

This paper recasts reward modeling from preferences as a causal inference problem to flag issues like confounding and heterogeneity, but it stays at the level of framing and desiderata without new methods or evidence.

read the letter

The core takeaway is that treating preference data for LLM reward models through a causal lens surfaces familiar problems—misidentification of what drives preferences, user-specific confounding, and heterogeneity—and contrasts them with how data is usually gathered in RLHF. That framing is the main contribution here. It draws on standard causal inference ideas to argue that observational preference pairs often violate the conditions needed for reliable generalization to new prompts. The paper does a clear job laying out those mismatches and sketching why naive models can fail in predictable ways under those violations. The discussion of common data practices versus causal assumptions is straightforward and could help people designing collection pipelines think more carefully. What is missing is any actual demonstration that causal fixes deliver better robustness. There are no derivations of new identification strategies, no experiments on synthetic or real preference data, and no comparisons showing gains over existing reward models. The illustrations appear to rest on simplified cases, so it is unclear how well they scale to the high-dimensional, noisy settings of actual LLM training. The paper closes by listing desiderata rather than resolving any of them. This is useful reading for alignment researchers already comfortable with causal methods who want a structured way to think about data limitations. It is less valuable for someone looking for a ready-to-use technique or validated improvement. The work shows honest engagement with the literature and the practical constraints of preference learning, so it is worth sending out for peer review to see whether the framing can be developed into something more concrete.

Referee Report

2 major / 2 minor

Summary. The manuscript proposes framing reward modeling from preference data in LLM alignment as a causal inference problem. It uses this lens to identify challenges such as causal misidentification, preference heterogeneity, and confounding from user-specific factors; contrasts key causal assumptions with standard RLHF data collection practices; illustrates failure modes of naive models; shows how causally-inspired approaches can improve robustness; and outlines desiderata for future work including targeted interventions.

Significance. If the proposed causal framing and fixes hold under realistic conditions, the work offers a structured toolbox for diagnosing and mitigating generalization failures in preference-based reward models, which is a timely contribution to AI alignment research. Strengths include the explicit mapping of causal assumptions to common observational practices and the emphasis on interventions rather than purely passive data collection.

major comments (2)

[Illustrations of failure modes and causal approaches] The demonstrations that causally-inspired reward models improve robustness (likely in the illustrations section) rely on simplified causal graphs or synthetic preference data. It remains unclear whether the identification strategies remain valid when preference data is high-dimensional, collected via standard pairwise comparisons without explicit interventions, and subject to unmeasured user-specific factors, as is typical in RLHF pipelines. This directly affects the central claim of reliable generalization.
[Key assumptions and contrast with data practices] § on key assumptions: the contrast between causal assumptions and common data collection practices is conceptually useful but lacks a concrete test or counterexample showing how violation of a specific assumption (e.g., no unmeasured confounding) produces measurable misalignment in an LLM reward model. Without this, the practical payoff of the causal toolbox is hard to evaluate.

minor comments (2)

Notation for latent user-specific factors and preference heterogeneity could be standardized early in the paper to prevent confusion with standard RLHF terminology.
A few citations to recent empirical work on preference heterogeneity in LLMs would help ground the conceptual discussion.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their thoughtful and constructive comments. We address each major comment below and indicate the revisions we intend to make to strengthen the manuscript.

read point-by-point responses

Referee: [Illustrations of failure modes and causal approaches] The demonstrations that causally-inspired reward models improve robustness (likely in the illustrations section) rely on simplified causal graphs or synthetic preference data. It remains unclear whether the identification strategies remain valid when preference data is high-dimensional, collected via standard pairwise comparisons without explicit interventions, and subject to unmeasured user-specific factors, as is typical in RLHF pipelines. This directly affects the central claim of reliable generalization.

Authors: We thank the referee for this observation. Our illustrations deliberately employ simplified synthetic settings and causal graphs to isolate the mechanisms of causal misidentification and the benefits of causally-informed modeling. These examples are not intended as empirical validation on production-scale RLHF data but as pedagogical demonstrations of the framework. The identification strategies themselves derive from standard results in causal inference that are applicable to high-dimensional observational data when the stated assumptions hold. In the revised manuscript we will add a dedicated discussion subsection that explicitly maps the framework to high-dimensional pairwise preference data, addresses the absence of explicit interventions, and outlines sensitivity analyses for unmeasured user-specific confounding. revision: yes
Referee: [Key assumptions and contrast with data practices] § on key assumptions: the contrast between causal assumptions and common data collection practices is conceptually useful but lacks a concrete test or counterexample showing how violation of a specific assumption (e.g., no unmeasured confounding) produces measurable misalignment in an LLM reward model. Without this, the practical payoff of the causal toolbox is hard to evaluate.

Authors: We agree that a more explicit quantitative counterexample would help readers assess the practical value of the proposed toolbox. The existing illustrations of failure modes already demonstrate qualitative misalignment under violated assumptions, but they remain synthetic. In the revision we will insert a focused simulation study that quantifies the effect of unmeasured confounding (e.g., via user-specific latent factors) on reward-model accuracy and downstream policy performance, using a data-generating process that mimics standard RLHF pairwise collection. revision: yes

Circularity Check

0 steps flagged

Conceptual framing draws on external causal literature without self-referential reduction

full rationale

The paper offers a high-level proposal to recast reward modeling from preferences as a causal inference task, identifying challenges such as misidentification, heterogeneity, and confounding by reference to standard causal assumptions and external literature. No equations, fitted parameters, or first-principles derivations are presented that could reduce to the paper's own inputs by construction. Illustrations of failure modes and suggested fixes rely on general causal identification strategies rather than any self-citation chain or ansatz smuggled from prior work by the same authors. The central contribution is therefore a framing exercise that remains independent of its own outputs and does not exhibit any of the enumerated circularity patterns.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Based on abstract only; the central proposal rests on the applicability of causal assumptions to observational preference data without new evidence supplied.

axioms (1)

domain assumption Key assumptions from causal inference literature are necessary for reliable generalization of reward models
Abstract states that inheriting from causal inference literature allows identification of key assumptions necessary for reliable generalization.

pith-pipeline@v0.9.0 · 5653 in / 1121 out tokens · 44160 ms · 2026-05-19T11:18:22.203936+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

20 extracted references · 20 canonical work pages · 2 internal anchors

[1]

Bose, A., Xiong, Z., Chi, Y ., Du, S

URL https://openreview.net/forum? id=BJg866NFvB. Bose, A., Xiong, Z., Chi, Y ., Du, S. S., Xiao, L., and Fazel, M. LoRe: Personalizing LLMs via Low-Rank Reward Modeling, 2025. URL https://arxiv.org/abs/ 2504.14439. eprint: 2504.14439. Bradley, R. A. and Terry, M. E. Rank Analysis of Incom- plete Block Designs: I. The Method of Paired Com- parisons. Biomet...

work page arXiv 2025
[2]

In: ACL’22

URL https://openreview.net/forum? id=dz79MhQXWvg. Butcher, B. Aligning Large Language Models with Coun- terfactual DPO, 2024. URL https://arxiv.org/ abs/2401.09566. eprint: 2401.09566. 9 Preference Learning for AI Alignment: a Causal Perspective Cao, B., Lin, H., Han, X., Liu, F., and Sun, L. Can Prompt Probe Pretrained Language Models? Understanding the ...

work page doi:10.18653/v1/2022.acl-long 2024
[3]

acl-long.398/

URL https://aclanthology.org/2022. acl-long.398/. Chen, L., Zhu, C., Chen, J., Soselia, D., Zhou, T., Gold- stein, T., Huang, H., Shoeybi, M., and Catanzaro, B. ODIN: Disentangled Reward Mitigates Hacking in RLHF. In Proceedings of the 41st International Conference on Machine Learning , pp. 7935–7952. PMLR, July

work page 2022
[4]

UltraFeedback: Boosting Language Models with Scaled AI Feedback

URL https://proceedings.mlr.press/ v235/chen24bn.html. ISSN: 2640-3498. Coste, T., Anwar, U., Kirk, R., and Krueger, D. Reward Model Ensembles Help Mitigate Overoptimization. In The Twelfth International Conference on Learning Rep- resentations, 2024. URL https://openreview. net/forum?id=dcjtMYkpXx. Crump, R., Hotz, V . J., Imbens, G., and Mitnik, O. Movi...

work page internal anchor Pith review Pith/arXiv arXiv doi:10.1007/s10618-021-00759-3 2024
[5]

URL https://openreview.net/forum? id=Ha2MnQM9Ph. Hahn, J. On the Role of the Propensity Score in Efficient Semiparametric Estimation of Average Treatment Effects. Econometrica, 66(2):315–331, 1998. ISSN 00129682, 14680262. doi: 10.2307/2998560. URL http://www. jstor.org/stable/2998560. Publisher: [Wiley, Econometric Society]. Hendrycks, D. and Gimpel, K. ...

work page doi:10.2307/2998560 1998
[6]

Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Sch¨olkopf, B., and Bachem, O

URL https://openreview.net/forum? id=88AS5MQnmC. Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Sch¨olkopf, B., and Bachem, O. Challenging Common As- sumptions in the Unsupervised Learning of Disentangled Representations. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, volu...

work page
[7]

Lopez, M

URL https://proceedings.mlr.press/ v97/locatello19a.html. Lopez, M. J. and Gutman, R. Estimation of Causal Ef- fects with Multiple Treatments: A Review and New Ideas. Statistical Science, 32(3):432 – 454, 2017. doi: 10.1214/17-STS612. URL https://doi.org/10. 1214/17-STS612. Publisher: Institute of Mathemati- cal Statistics. Luce, R. D. Individual choice b...

work page doi:10.1214/17-sts612 2017
[8]

Muldrew, W., Hayes, P., Zhang, M., and Barber, D

URL https://openreview.net/forum? id=TADTT9ughN. Muldrew, W., Hayes, P., Zhang, M., and Barber, D. Ac- tive Preference Learning for Large Language Models. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceed...

work page
[9]

Training language models to follow instructions with human feedback

URL https://proceedings.mlr.press/ v235/muldrew24a.html. Oberst, M. and Sontag, D. Counterfactual Off-Policy Eval- uation with Gumbel-Max Structural Causal Models. In Proceedings of the 36th International Conference on Machine Learning, pp. 4881–4890. PMLR, May 2019. URL https://proceedings.mlr.press/v97/ oberst19a.html. ISSN: 2640-3498. Ouyang, L., Wu, J...

work page internal anchor Pith review Pith/arXiv arXiv doi:10.18653/v1/2024 2019
[10]

and Rubin, Donald B

ISSN 0006-3444. doi: 10.1093/biomet/70.1.41. URL https://doi.org/10.1093/biomet/70. 1.41. eprint: https://academic.oup.com/biomet/article- pdf/70/1/41/662954/70-1-41.pdf. Sch¨olkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbren- ner, N., Goyal, A., and Bengio, Y . Toward Causal Repre- sentation Learning. Proceedings of the IEEE, 109(5):612– 634, May...

work page doi:10.1093/biomet/70.1.41 2021
[11]

Siththaranjan, A., Laidlaw, C., and Hadfield-Menell, D

URL https://openreview.net/forum? id=sNtDKdcI1f. Siththaranjan, A., Laidlaw, C., and Hadfield-Menell, D. Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF. In The Twelfth International Conference on Learning Representations,

work page
[12]

Skalse, J

URL https://openreview.net/forum? id=0tWTxYYPnW. Skalse, J. M. V ., Howe, N. H. R., Krasheninnikov, D., and Krueger, D. Defining and Characterizing Reward Gam- ing. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/ forum?id=yb3HOXO3lX2. Splawa-Neyman, J., Dabro...

work page 2022
[13]

ISSN 08834237, 21688745

Statistical Science , 5(4):465–472, 1990. ISSN 08834237, 21688745. URL http://www.jstor. org/stable/2245382. Publisher: Institute of Math- ematical Statistics. Tien, J., He, J. Z.-Y ., Erickson, Z., Dragan, A., and Brown, D. S. Causal Confusion and Reward Misidentification in Preference-Based Reward Learning. In The Eleventh International Conference on Le...

work page arXiv 1990
[14]

Vig, J., Gehrmann, S., Belinkov, Y ., Qian, S., Nevo, D., Singer, Y ., and Shieber, S

URL https://openreview.net/forum? id=R0Xxvr_X3ZA. Vig, J., Gehrmann, S., Belinkov, Y ., Qian, S., Nevo, D., Singer, Y ., and Shieber, S. Investigating Gender Bias in Language Models Using Causal Mediation Analysis. In Advances in Neural Information Processing Systems, volume 33, pp. 12388–12401. Curran Asso- ciates, Inc., 2020. URL https://proceedings. ne...

work page doi:10.18653/v1/2023.findings-emnlp 2020
[15]

findings-emnlp.1013/

URL https://aclanthology.org/2023. findings-emnlp.1013/. Wang, H., Xiong, W., Xie, T., Zhao, H., and Zhang, T. Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts. In Al-Onaizan, Y ., Bansal, M., and Chen, Y .-N. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12...

work page 2023
[16]

findings-emnlp.620

URL https://aclanthology.org/2024. findings-emnlp.620. Wu, A., Kuang, K., Xiong, R., Li, B., and Wu, F. Sta- ble Estimation of Heterogeneous Treatment Effects. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th In- ternational Conference on Machine Learning , volume 202 of Proceedings of Mac...

work page doi:10.18653/v1/2024.naacl-long.262 2024
[17]

Consistency in the observational space implies consistency in the latent space, i.e. for an individual with prompt- response assignment (X, Y, Y ′) whose latent factors are (Z X , ZT , Z′T ) ≡ (gX(X), gT (X, Y ), gT (X, Y ′)), we observe the associated potential outcome, i.e. L = L(Z X , ZT , Z′T )

work page
[18]

Unconfoundedness in the observational implies unconfoundedness in the latent space, i.e.L(Z X = zX; Z T = zT , Z′T = z′T ) ≡ L(zX; zT , z′T ) is independent of (Z X , ZT , Z′T )

work page
[19]

Extended Discussion C.1

E [L(x; y, y′)] = E h L(zX; zT , z′T ) i Thus, it follows that: E [L(x; y, y′)] = E h L(zX , zT , z′T ) i (by sufficiency) = E h L(zX , zT , z′T )|Z X = zX , ZT = zT , Z′T = z′Ti (by latent overlap & unconfoundedness) = E h L|Z X = zX , ZT = zT , Z′T = z′Ti (by latent consistency) C. Extended Discussion C.1. Interpretability of Latent Factors Aside from e...

work page 1959
[20]

Without such form of control, intervention-based preference learning becomes infeasible, as models would lack the ability to systematically vary latent factors

to specify desired properties of text . Without such form of control, intervention-based preference learning becomes infeasible, as models would lack the ability to systematically vary latent factors. Language models should be designed to generate responses that explicitly vary along key latent dimensions, such as response verbosity or style, rather than ...

work page 1983

[1] [1]

Bose, A., Xiong, Z., Chi, Y ., Du, S

URL https://openreview.net/forum? id=BJg866NFvB. Bose, A., Xiong, Z., Chi, Y ., Du, S. S., Xiao, L., and Fazel, M. LoRe: Personalizing LLMs via Low-Rank Reward Modeling, 2025. URL https://arxiv.org/abs/ 2504.14439. eprint: 2504.14439. Bradley, R. A. and Terry, M. E. Rank Analysis of Incom- plete Block Designs: I. The Method of Paired Com- parisons. Biomet...

work page arXiv 2025

[2] [2]

In: ACL’22

URL https://openreview.net/forum? id=dz79MhQXWvg. Butcher, B. Aligning Large Language Models with Coun- terfactual DPO, 2024. URL https://arxiv.org/ abs/2401.09566. eprint: 2401.09566. 9 Preference Learning for AI Alignment: a Causal Perspective Cao, B., Lin, H., Han, X., Liu, F., and Sun, L. Can Prompt Probe Pretrained Language Models? Understanding the ...

work page doi:10.18653/v1/2022.acl-long 2024

[3] [3]

acl-long.398/

URL https://aclanthology.org/2022. acl-long.398/. Chen, L., Zhu, C., Chen, J., Soselia, D., Zhou, T., Gold- stein, T., Huang, H., Shoeybi, M., and Catanzaro, B. ODIN: Disentangled Reward Mitigates Hacking in RLHF. In Proceedings of the 41st International Conference on Machine Learning , pp. 7935–7952. PMLR, July

work page 2022

[4] [4]

UltraFeedback: Boosting Language Models with Scaled AI Feedback

URL https://proceedings.mlr.press/ v235/chen24bn.html. ISSN: 2640-3498. Coste, T., Anwar, U., Kirk, R., and Krueger, D. Reward Model Ensembles Help Mitigate Overoptimization. In The Twelfth International Conference on Learning Rep- resentations, 2024. URL https://openreview. net/forum?id=dcjtMYkpXx. Crump, R., Hotz, V . J., Imbens, G., and Mitnik, O. Movi...

work page internal anchor Pith review Pith/arXiv arXiv doi:10.1007/s10618-021-00759-3 2024

[5] [5]

URL https://openreview.net/forum? id=Ha2MnQM9Ph. Hahn, J. On the Role of the Propensity Score in Efficient Semiparametric Estimation of Average Treatment Effects. Econometrica, 66(2):315–331, 1998. ISSN 00129682, 14680262. doi: 10.2307/2998560. URL http://www. jstor.org/stable/2998560. Publisher: [Wiley, Econometric Society]. Hendrycks, D. and Gimpel, K. ...

work page doi:10.2307/2998560 1998

[6] [6]

Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Sch¨olkopf, B., and Bachem, O

URL https://openreview.net/forum? id=88AS5MQnmC. Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Sch¨olkopf, B., and Bachem, O. Challenging Common As- sumptions in the Unsupervised Learning of Disentangled Representations. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International Conference on Machine Learning, volu...

work page

[7] [7]

Lopez, M

URL https://proceedings.mlr.press/ v97/locatello19a.html. Lopez, M. J. and Gutman, R. Estimation of Causal Ef- fects with Multiple Treatments: A Review and New Ideas. Statistical Science, 32(3):432 – 454, 2017. doi: 10.1214/17-STS612. URL https://doi.org/10. 1214/17-STS612. Publisher: Institute of Mathemati- cal Statistics. Luce, R. D. Individual choice b...

work page doi:10.1214/17-sts612 2017

[8] [8]

Muldrew, W., Hayes, P., Zhang, M., and Barber, D

URL https://openreview.net/forum? id=TADTT9ughN. Muldrew, W., Hayes, P., Zhang, M., and Barber, D. Ac- tive Preference Learning for Large Language Models. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceed...

work page

[9] [9]

Training language models to follow instructions with human feedback

URL https://proceedings.mlr.press/ v235/muldrew24a.html. Oberst, M. and Sontag, D. Counterfactual Off-Policy Eval- uation with Gumbel-Max Structural Causal Models. In Proceedings of the 36th International Conference on Machine Learning, pp. 4881–4890. PMLR, May 2019. URL https://proceedings.mlr.press/v97/ oberst19a.html. ISSN: 2640-3498. Ouyang, L., Wu, J...

work page internal anchor Pith review Pith/arXiv arXiv doi:10.18653/v1/2024 2019

[10] [10]

and Rubin, Donald B

ISSN 0006-3444. doi: 10.1093/biomet/70.1.41. URL https://doi.org/10.1093/biomet/70. 1.41. eprint: https://academic.oup.com/biomet/article- pdf/70/1/41/662954/70-1-41.pdf. Sch¨olkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbren- ner, N., Goyal, A., and Bengio, Y . Toward Causal Repre- sentation Learning. Proceedings of the IEEE, 109(5):612– 634, May...

work page doi:10.1093/biomet/70.1.41 2021

[11] [11]

Siththaranjan, A., Laidlaw, C., and Hadfield-Menell, D

URL https://openreview.net/forum? id=sNtDKdcI1f. Siththaranjan, A., Laidlaw, C., and Hadfield-Menell, D. Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF. In The Twelfth International Conference on Learning Representations,

work page

[12] [12]

Skalse, J

URL https://openreview.net/forum? id=0tWTxYYPnW. Skalse, J. M. V ., Howe, N. H. R., Krasheninnikov, D., and Krueger, D. Defining and Characterizing Reward Gam- ing. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/ forum?id=yb3HOXO3lX2. Splawa-Neyman, J., Dabro...

work page 2022

[13] [13]

ISSN 08834237, 21688745

Statistical Science , 5(4):465–472, 1990. ISSN 08834237, 21688745. URL http://www.jstor. org/stable/2245382. Publisher: Institute of Math- ematical Statistics. Tien, J., He, J. Z.-Y ., Erickson, Z., Dragan, A., and Brown, D. S. Causal Confusion and Reward Misidentification in Preference-Based Reward Learning. In The Eleventh International Conference on Le...

work page arXiv 1990

[14] [14]

Vig, J., Gehrmann, S., Belinkov, Y ., Qian, S., Nevo, D., Singer, Y ., and Shieber, S

URL https://openreview.net/forum? id=R0Xxvr_X3ZA. Vig, J., Gehrmann, S., Belinkov, Y ., Qian, S., Nevo, D., Singer, Y ., and Shieber, S. Investigating Gender Bias in Language Models Using Causal Mediation Analysis. In Advances in Neural Information Processing Systems, volume 33, pp. 12388–12401. Curran Asso- ciates, Inc., 2020. URL https://proceedings. ne...

work page doi:10.18653/v1/2023.findings-emnlp 2020

[15] [15]

findings-emnlp.1013/

URL https://aclanthology.org/2023. findings-emnlp.1013/. Wang, H., Xiong, W., Xie, T., Zhao, H., and Zhang, T. Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts. In Al-Onaizan, Y ., Bansal, M., and Chen, Y .-N. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12...

work page 2023

[16] [16]

findings-emnlp.620

URL https://aclanthology.org/2024. findings-emnlp.620. Wu, A., Kuang, K., Xiong, R., Li, B., and Wu, F. Sta- ble Estimation of Heterogeneous Treatment Effects. In Krause, A., Brunskill, E., Cho, K., Engelhardt, B., Sabato, S., and Scarlett, J. (eds.), Proceedings of the 40th In- ternational Conference on Machine Learning , volume 202 of Proceedings of Mac...

work page doi:10.18653/v1/2024.naacl-long.262 2024

[17] [17]

Consistency in the observational space implies consistency in the latent space, i.e. for an individual with prompt- response assignment (X, Y, Y ′) whose latent factors are (Z X , ZT , Z′T ) ≡ (gX(X), gT (X, Y ), gT (X, Y ′)), we observe the associated potential outcome, i.e. L = L(Z X , ZT , Z′T )

work page

[18] [18]

Unconfoundedness in the observational implies unconfoundedness in the latent space, i.e.L(Z X = zX; Z T = zT , Z′T = z′T ) ≡ L(zX; zT , z′T ) is independent of (Z X , ZT , Z′T )

work page

[19] [19]

Extended Discussion C.1

E [L(x; y, y′)] = E h L(zX; zT , z′T ) i Thus, it follows that: E [L(x; y, y′)] = E h L(zX , zT , z′T ) i (by sufficiency) = E h L(zX , zT , z′T )|Z X = zX , ZT = zT , Z′T = z′Ti (by latent overlap & unconfoundedness) = E h L|Z X = zX , ZT = zT , Z′T = z′Ti (by latent consistency) C. Extended Discussion C.1. Interpretability of Latent Factors Aside from e...

work page 1959

[20] [20]

Without such form of control, intervention-based preference learning becomes infeasible, as models would lack the ability to systematically vary latent factors

to specify desired properties of text . Without such form of control, intervention-based preference learning becomes infeasible, as models would lack the ability to systematically vary latent factors. Language models should be designed to generate responses that explicitly vary along key latent dimensions, such as response verbosity or style, rather than ...

work page 1983