REVIEW 3 major objections 4 minor 1 cited by
Interpreting Language Reward Models via Contrastive Explanations
T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read The paper proposes a post hoc, model-agnostic method to explain any binary reward-model comparison by generating attribute-conditioned LLM perturbations, classifying them as counterfactuals or semifactuals, and aggregating them into…
desk verdict A practical, clearly presented adaptation of contrastive explanations to reward models, with real qualitative payoff; the headline empirical claim is under-validated because attribute purity of perturbations is never checked. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the two-step attribute-conditioned perturbation prompt. In Step 1, the LLM is asked to identify, in each original response, the words relevant to each of fifteen high-level evaluation attributes (for example harmlessness, verbosity, relevance). In Step 2, the same LLM rewrites the chosen response to be worse (or the rejected response to be better) along one attribute, restricting edits to the words identified in Step 1 to keep perturbations local. The resulting perturbed responses are scored by the reward model and sorted into counterfactuals and semifactuals by whether the comparison preference flips; the preference-flip rate per attribute, aggregated over many comparisons, is the statistic that carries the global sensitivity analysis.
What would settle it
Take a sample of the 15-attribute perturbation sets generated by the method, and have independent human annotators (or a separate judge model) label, for each perturbed response, which attributes actually changed relative to the original. If a large share of responses labelled harmlessness or helpfulness also change other attributes, the preference-flip rates and global sensitivity rankings would not cleanly attribute causes, and the central claim would need qualification.
Extended reading notes
Core claim
The paper's central claim is that contrastive explanations of a reward model's binary comparison $(x, y_+, y_-)$ can be generated purely from black-box access to the model, using an external LLM to produce controlled perturbations. For a comparison with $r(x, y_+) > r(x, y_-)$, a perturbation $y'_+$ of the chosen response is a counterfactual when $r(x, y'_+) < r(x, y_-)$; a perturbation $y'_-$ of the rejected response is a counterfactual when $r(x, y'_-) > r(x, y_+)$. Perturbations that do not flip the preference are semifactuals. The authors show that generating perturbations via a two-step, attribute-conditioned prompting strategy yields both counterfactuals and semifactuals with high coverage on the HelpSteer2 and HH-RLHF datasets across three open reward models, and that aggregating preference-flip rates by attribute produces a ranking of attribute sensitivity that distinguishes reward models, including the finding that one model trained on the harmless dataset is markedly more sensitive to harmlessness, sensitivity, and neutrality than the others.
Load-bearing premise
The whole analysis presumes that each LLM-generated perturbation changes only the intended evaluation attribute, while leaving the response's topic, style, and other qualities fixed, so that an observed preference flip can be credited to that attribute.
Editorial extensions
If this is right
- Any black-box reward model can be audited locally without access to weights, gradients, or training data, because the method only queries the reward model for scores.
- Attribute-level preference-flip rates give a global sensitivity ranking, enabling direct comparisons between reward models trained on different data and the discovery of failure modes such as preferring disrespectful rewrites.
- Representative comparisons can be selected automatically by matching each comparison's local attribute ranking to the global sensitivity pattern, producing human-readable exemplars of model behaviour.
- The same perturbation sets can be reused across multiple reward models, so behaviour differences can be attributed to the models themselves rather than to different inputs.
Reading between the lines
- The method's usefulness depends on attribute purity: if the LLM's edits change more than the named attribute, the preference-flip rates conflate distinct causes; the paper validates locality with edit and semantic distances but does not directly measure attribute purity, so this is the natural next check.
- If attribute purity holds, the same pipeline could generate attribute-labelled preference data automatically, providing a lower-cost route to fine-grained reward-model training sets in the direction the authors sketch in their conclusion.
- The counterfactual/semifactual split could be refined further: semifactuals that sit between the two original rewards carry different information than those that overshoot, a distinction the paper notes as a limitation and future work.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a post hoc, model-agnostic framework for explaining binary preference comparisons made by reward models (RMs). For a given prompt x and a chosen/rejected response pair (y+, y-), the method uses a two-step LLM prompting strategy to generate 15 perturbed responses per original response, each targeting a specified high-level attribute (e.g., harmlessness, helpfulness) and aiming to make the chosen response worse or the rejected response better. The perturbed responses are scored by the RM and classified as counterfactuals (CF; preference flips) or semifactuals (SF; preference unchanged). The local explanations are aggregated into global attribute-level sensitivity via preference flip rates (PFR), and the paper presents a workflow for finding representative example comparisons. Quantitative evaluation on three datasets and three open-source RMs compares CF/SF coverage, syntactic/semantic distances, and diversity against Polyjuice and random LLM perturbation baselines. Qualitative case studies analyze OpenAssistant models and two 8B-parameter RMs, exposing behaviors such as assigning higher rewards to responses containing disrespectful language.
Significance. The framework is novel, simple, and practical: it is model-agnostic, requires only black-box access to the RM, and the appendices provide all prompts, making the method easy to reproduce. The global sensitivity analysis is a sensible way to aggregate local explanations, and the finding that v2 (trained on the harmless branch) is most sensitive to harmlessness on that dataset is a useful sanity check. The paper also demonstrates a valuable application by discovering failure modes (e.g., preference for 'the hell' insertions) in state-of-the-art 8B-parameter RMs. If the attribute-purity assumption is validated, this would give practitioners an inexpensive auditing tool for RLHF components. However, the paper's central empirical claim that the method yields high-quality explanations and that PFR rankings isolate attribute-level sensitivity rests on unvalidated assumptions about the LLM-generated perturbations; these are the subject of the major comments.
major comments (3)
- [Section 2.3 / Section 4.1] Attribute purity of the perturbations is unvalidated. The causal attribution of CF/SF labels to individual attributes assumes that a Step-2 perturbation generated for attribute a changes only that attribute and in the intended direction. The paper checks locality only through aggregate syntactic/semantic distances (Table 2) and never checks whether a perturbed response actually altered the intended attribute while holding the others fixed. If, for example, a 'harmlessness' perturbation also reduces helpfulness or changes verbosity, the preference flip is not attributable to harmlessness-specific sensitivity of the RM. This confounds the PFR rankings in Figure 3, the Kendall-tau correlations in Table 3, and the representative-example interpretations in Section 4.2. Section 5's limitation paragraph acknowledges that counterfactuals are not guaranteed but does not acknowledge this attribute-conflation issue. A concrete remedy is to add a human or LLM-based annotation study that labels which attributes changed in a sample of perturbed responses, or at least to report the rate at which an independent judge identifies only the intended attribute as changed. Without such validation, the central empirical claim of the paper is not established.
- [Section 4.1, Figure 3] Cross-attribute PFR comparisons are not normalized by perturbation magnitude. Attributes for which GPT-4o happens to make larger changes (larger reward deltas or larger edit distances) will mechanically yield higher flip rates, even for an RM that weights all 15 attributes equally. The paper reports distances only in aggregate over all attributes (Table 2) and does not provide per-attribute perturbation strength. The global sensitivity ranking may therefore reflect generator strength rather than RM sensitivity. The authors should report, per attribute, the distribution of reward deltas (or a normalized semantic distance) and show that PFR differences persist when conditioning on similar perturbation magnitude; alternatively, they should explicitly frame the PFR as an interaction between the generator and the RM rather than a pure RM property.
- [Section 2.2 / Section 3] The SF categorization does not verify that a perturbation moved in the intended opposing direction. In Eq. (2), any perturbed response that does not flip the preference is called a semifactual; this includes perturbations that accidentally improved y+ or worsened y-. The desiderata in Section 2.3 explicitly aim to make y+ worse and y- better, but the evaluation never measures whether the RM reward actually moved in that direction. Since an off-target SF can result from an unchanged response or from a change affecting a different attribute, the CF/SF coverage numbers in Table 1 can overstate how well the method characterizes the local decision boundary. The authors should report the fraction of generated perturbations that move the reward in the intended direction, perhaps stratified by attribute, as a basic sanity check on the generation step.
minor comments (4)
- [Tables 1 and 2] Polyjuice results are missing for the hs2 dataset, despite the paper stating that evaluation is performed against two baselines on three datasets. Please either include the PJ results for hs2 or explain why they are omitted.
- [Section 2.2] There is a typo in the list of categorized perturbation sets: 'YC+, YC-, YS- and YS-' should be 'YC+, YC-, YS+ and YS-' (or the fourth set should be YS+).
- [Section 4.2] Representative examples are selected by maximizing Kendall's tau between local and global rankings; if the same 500 comparisons are used for both the global sensitivity computation and the selection, the paper should state this explicitly and note that the selected example may overfit to the global pattern by construction.
- [Appendix A] The list of 15 attributes is induced from the test sets used in the experiments via GPT-4o prompting. The paper should briefly discuss the risk that the attribute list is dataset-specific and that perturbations might change attributes not on the list, which would reinforce the attribute-purity concern raised in the major comments.
Circularity Check
No significant circularity: CF/SF labels and global sensitivity metrics are computed from RM predictions on generated perturbations, and the attribute-conditioned perturbation is a treatment, not a fitted input.
full rationale
The paper's derivation chain is self-contained rather than circular. Section 2.2 defines counterfactuals and semifactuals purely by comparing RM rewards on original versus perturbed responses; the perturbed responses are generated independently in Section 2.3 via attribute-conditioned LLM prompting. No parameter is fitted to the quantity being explained, and no 'prediction' is read back from the same perturbation labels that define it. The global sensitivity measure in Section 4.1, the preference flip rate, is an empirical proportion computed from the RM's actual scalar rewards on generated perturbations; it is not an equation-level consequence of the attribute definitions. The interpretive step that CF attributes are 'strongly relevant' is an explicitly stated assumption grounded in prior contrastive-explanation literature, not a derivation from the paper's own inputs. The paper also provides an external validation: v2, trained on the harmless dataset, is most sensitive to harmlessness-related attributes, which is independent of the perturbation-generation prompts. The reader's concern that LLM perturbations may change multiple attributes at once is a correctness and validity risk about attribute purity, not a circularity in the derivation; it does not make any equation or fitted parameter equivalent to the claimed result. The representative-example workflow in Section 4.2 selects examples whose local ranking best matches the global ranking, so the later observation that these examples match global sensitivity is partly by selection; however, the paper presents this as an illustrative workflow rather than as an independent prediction, and the detailed reward patterns and model comparisons in the selected examples are not themselves constructed by that match. There are no load-bearing self-citations, no imported uniqueness theorems, and no renaming of known results. Overall, the central empirical claims rest on RM outputs and external validation, so no significant circularity is present.
Assumptions & free parameters
free parameters (2)
- Number of perturbed responses per original response =
15
- Evaluation attribute list (15 attributes) =
avoid-to-answer, appropriateness, assertiveness, clarity, coherence, complexity, correctness, engagement…
assumptions (5)
- standard math Bradley-Terry preference model: RM scalar rewards define binary preferences via comparison.
- domain assumption LLM-generated perturbations are valid, grammatical, local, and modify only the target attribute.
- ad hoc to paper The 15 hand-picked attributes span the meaningful space of response variations.
- domain assumption CF/SF distinction maps to causal relevance of attributes.
- domain assumption Text distance metrics (Levenshtein, Sentence-BERT cosine) capture locality and diversity.
invented entities (1)
-
High-level evaluation attribute space (15 attributes)
Cite this review
Pith. "Pith review of Interpreting Language Reward Models via Contrastive Explanations." pith.science (2026). https://pith.science/paper/CLYJY6HJ
@misc{pith2026241116502,
author = {Pith},
title = {Pith review of: Interpreting Language Reward Models via Contrastive Explanations},
year = {2026},
howpublished = {\url{https://pith.science/paper/CLYJY6HJ}},
note = {Machine review of arXiv:2411.16502}
}
read the original abstract
Reward models (RMs) are a crucial component in the alignment of large language models' (LLMs) outputs with human values. RMs approximate human preferences over possible LLM responses to the same prompt by predicting and comparing reward scores. However, as they are typically modified versions of LLMs with scalar output heads, RMs are large black boxes whose predictions are not explainable. More transparent RMs would enable improved trust in the alignment of LLMs. In this work, we propose to use contrastive explanations to explain any binary response comparison made by an RM. Specifically, we generate a diverse set of new comparisons similar to the original one to characterise the RM's local behaviour. The perturbed responses forming the new comparisons are generated to explicitly modify manually specified high-level evaluation attributes, on which analyses of RM behaviour are grounded. In quantitative experiments, we validate the effectiveness of our method for finding high-quality contrastive explanations. We then showcase the qualitative usefulness of our method for investigating global sensitivity of RMs to each evaluation attribute, and demonstrate how representative examples can be automatically extracted to explain and compare behaviours of different RMs. We see our method as a flexible framework for RM explanation, providing a basis for more interpretable and trustworthy LLM alignment.
Figures
Figures from the paper (3 more)
Forward citations
Cited by 1 Pith paper
-
Multi-Domain Explainability of Preferences
A concept-discovery plus hierarchical regression pipeline explains human, LLM-judge, and reward-model preferences at local and global levels across eight domains.
Reference graph
Works this paper leans on
-
[1]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,
-
[5]
Interactive analysis of llms using meaningful counterfactuals
Furui Cheng, Vil´em Zouhar, Robin Shing Moon Chan, Daniel F ¨urst, Hendrik Strobelt, and Menna- tallah El-Assady. Interactive analysis of llms using meaningful counterfactuals. arXiv preprint arXiv:2405.00708,
-
[6]
Llm-assisted content analysis: Using large language models to support deductive coding
Robert Chew, John Bollenbacher, Michael Wenger, Jessica Speer, and Annice Kim. Llm-assisted content analysis: Using large language models to support deductive coding. arXiv preprint arXiv:2306.14924,
-
[7]
Ultrafeedback: Boosting language models with high-quality feedback
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377,
-
[8]
Research agenda for sociotechnical approaches to ai safety
11 Published as a conference paper at ICLR 2025 Samuel Curtis, Ravi Iyer, Cameron Domenico Kirk-Giannini, Victoria Krakovna, David Krueger, Nathan Lambert, Bruno Marnette, Colleen McKenzie, Julian Michael, Evan Miyazono, et al. Research agenda for sociotechnical approaches to ai safety. online,
work page 2025
-
[9]
Rewardbench: Evaluating reward models for language modeling
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403.13787,
-
[10]
Skywork-reward: Bag of tricks for reward modeling in llms.arXiv preprint arXiv:2410.18451,
Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork-reward: Bag of tricks for reward modeling in llms.arXiv preprint arXiv:2410.18451,
-
[12]
12 Published as a conference paper at ICLR 2025 Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training langu...
work page 2025
Show all 24 references
-
[14]
A baseline analysis of reward mod- els’ ability to accurately analyze foundation models under distribution shift
Ben Pikus, Will LeVine, Tony Chen, and Sean Hendryx. A baseline analysis of reward mod- els’ ability to accurately analyze foundation models under distribution shift. arXiv preprint arXiv:2311.14743,
-
[17]
A long way to go: Investigating length correlations in rlhf
Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716,
-
[19]
Helpsteer2: Open-source dataset for training top-performing reward models
Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models. arXiv preprint arXiv:2406.08673, 2024b. Tongshuang Wu, Mar...
-
[21]
14 Published as a conference paper at ICLR 2025 A T HE HIGH -LEVEL EVALUATION ATTRIBUTES We adopted a semi-automated workflow to obtain the list of attributes presented in Section 2.3. Us- ing the same test sets of the three datasets (described in Section 3), we prompt GPT-4o ...
2024
-
[22]
As the former is not the focus of the datasets we experiment with, we only additionally include honesty, relabelled to avoid-to-answer for better relevance
apart from instruction-following and honesty. As the former is not the focus of the datasets we experiment with, we only additionally include honesty, relabelled to avoid-to-answer for better relevance. The list of attributes deviates from the list in the OASST2 dataset (K¨opf...
2023
-
[23]
To avoid LLM input and output being too long thus potentially hurting performance in each query, Step 2 prompt is repeated for each evaluation attribute. Step 1 prompt: In the task of response quality scoring, a trained deep learning model assigns real-valued scores for respon...
2025
-
[24]
We observe that only v2 and pythia have weak correlations in some cases, and the rest of the results are generally uncorrelated. harmless helpful hs2 v1 0.18 0.03 0.12 v2 0.49 0.30 0.28 pythia 0.28 0.31 0.07 Table 6: Ranking similarity of global sensitivity between the perturb...
2024
-
[2001]
Offsetbias: Lever- aging debiased data for tuning evaluators
Junsoo Park, Seungyeon Jwa, Meiying Ren, Daeyoung Kim, and Sanghyuk Choi. Offsetbias: Lever- aging debiased data for tuning evaluators. arXiv preprint arXiv:2407.06551,
-
[2016]
Indira Sen, Dennis Assenmacher, Mattia Samory, Isabelle Augenstein, Wil M. P. van der Aalst, and Claudia Wagner. People make better edits: Measuring the efficacy of llm-generated counterfac- tually augmented data for harmful language detection. In Proceedings of the 2023 Confe...
2023
-
[2017]
Interpretable preferences via multi-objective reward modeling and mixture-of-experts
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. arXiv preprint arXiv:2406.12845, 2024a. Zhilin Wang, Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel E...
-
[2019]
Llms for generating and evaluating counterfactuals: A comprehensive study
Van Bach Nguyen, Paul Youssef, J ¨org Schl ¨otterer, and Christin Seifert. Llms for generating and evaluating counterfactuals: A comprehensive study. arXiv preprint arXiv:2405.00722,
-
[2020]
Sentence-bert: Sentence embeddings using siamese bert- networks
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP- IJCNLP,...
2019
-
[2021]
Vera Liao, Rania Abdelghani, and Pierre-Yves Oudeyer
13 Published as a conference paper at ICLR 2025 Ziang Xiao, Xingdi Yuan, Q. Vera Liao, Rania Abdelghani, and Pierre-Yves Oudeyer. Supporting qualitative analysis with large language models: Combining codebook with GPT-3 for deductive coding. In Companion Proceedings of the 28t...
2025
-
[2022]
Zero-shot llm-guided coun- terfactual generation for text
Amrita Bhattacharjee, Raha Moraffah, Joshua Garland, and Huan Liu. Zero-shot llm-guided coun- terfactual generation for text. arXiv preprint arXiv:2405.04793,
-
[2023]
Training a helpful and harmless assistant with reinforcement learning from human feedback
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862,
-
[2024]
Rlhf deciphered: A critical analysis of reinforcement learning from human feedback for llms
Shreyas Chaudhari, Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik Narasimhan, Ameet Deshpande, and Bruno Castro da Silva. Rlhf deciphered: A critical analysis of reinforcement learning from human feedback for llms. arXiv preprint arXiv:2404.08555,
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.