{"id":"d37ea938-7736-4800-b536-8389580825dc","arxiv_id":"2411.16502","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Reward model preferences can be explained by generating counterfactual and semifactual answer variations along 15 hand-picked evaluation attributes and measuring which attribute changes flip the model's preference.","lead":"This paper presents a method to explain why a language reward model prefers one answer to another, by generating slightly modified answers and checking which changes flip or preserve the preference. The method gives a way to audit reward models, find their sensitivities to qualities like harmlessness and helpfulness, and surface hidden failure modes, which matters for safer LLM alignment.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Global sensitivity analysis depends on unvalidated attribute purity of LLM perturbations; without it, CF/SF labels cannot be attributed to single attributes.","rationale":"The reader's conditional verdict is appropriate: the method is clearly presented and potentially useful, but the central empirical claim that explanations are 'high-quality' and that global sensitivity rankings characterize RM behavior depends on an unvalidated assumption about attribute purity. My stress-test identifies this as the most load-bearing concern because the causal interpretation of CF/SF labels is the bridge from raw preference flips to attribute-level conclusions. If GPT-4o's perturbations routinely change multiple attributes, then every downstream analysis, global sensitivity, representative examples, and cross-model comparisons, is compromised. The proposed concrete test directly checks this assumption by re-running the analysis on purity-filtered perturbations. The related perturbation-magnitude confound is noted as a secondary issue that would survive even perfect purity, but the attribute-purity check is the most decisive single test. I agree with the reader's assessment and therefore recommend no change to the conditional verdict.","tokens_in":17787,"tokens_out":4245,"duration_ms":43322,"concrete_test":"Select a stratified sample of perturbed responses (e.g., 50 test comparisons per dataset, all 15 attributes), and have two independent human annotators (or an LLM judge calibrated against human agreement) label, for each perturbation, which attributes changed relative to the original and in which direction. Compute a per-attribute purity matrix: the fraction of perturbations that changed only the intended attribute. Then recompute Figure 3's PFRs using only perturbations with confirmed single-attribute purity and compare Kendall-tau rankings to the original PFR rankings; if the correlation drops below about 0.7, the global sensitivity conclusions are confounded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim that the method yields high-quality explanations and that the global PFR rankings of Section 4.1 reveal sensitivity to individual attributes rests on an unstated assumption: the Step-2 perturbation of Appendix B changes only the target attribute, and in the intended direction. The paper validates locality only through aggregate Levenshtein and SBERT distances (Table 2) and never checks whether a perturbed response altered only the intended attribute. If, for example, a 'harmlessness' perturbation also reduces helpfulness or changes verbosity, a preference flip cannot be attributed to harmlessness-specific sensitivity. This is not merely a missing human-evaluation nicety: the entire global sensitivity analysis (Figure 3, Table 3) and the representative-example case studies of Section 4.2 interpret CF/SF membership causally. The limitation section admits no guarantee of finding counterfactuals but does not acknowledge this attribute-conflation confound. A related and independent confound is perturbation magnitude: PFRs are compared across attributes without controlling for how strongly each attribute was changed; attributes that GPT-4o happens to perturb more aggressively will appear more 'sensitive' even for an RM that weights all attributes equally. Both issues are addressable by validation, so the work remains potentially useful, but as it stands the headline empirical claim is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a post hoc, model-agnostic framework for explaining binary preference comparisons made by reward models (RMs). For a given prompt x and a chosen/rejected response pair (y+, y-), the method uses a two-step LLM prompting strategy to generate 15 perturbed responses per original response, each targeting a specified high-level attribute (e.g., harmlessness, helpfulness) and aiming to make the chosen response worse or the rejected response better. The perturbed responses are scored by the RM and classified as counterfactuals (CF; preference flips) or semifactuals (SF; preference unchanged). The local explanations are aggregated into global attribute-level sensitivity via preference flip rates (PFR), and the paper presents a workflow for finding representative example comparisons. Quantitative evaluation on three datasets and three open-source RMs compares CF/SF coverage, syntactic/semantic distances, and diversity against Polyjuice and random LLM perturbation baselines. Qualitative case studies analyze OpenAssistant models and two 8B-parameter RMs, exposing behaviors such as assigning higher rewards to responses containing disrespectful language.","tokens_in":18065,"tokens_out":8091,"duration_ms":75390,"significance":"The framework is novel, simple, and practical: it is model-agnostic, requires only black-box access to the RM, and the appendices provide all prompts, making the method easy to reproduce. The global sensitivity analysis is a sensible way to aggregate local explanations, and the finding that v2 (trained on the harmless branch) is most sensitive to harmlessness on that dataset is a useful sanity check. The paper also demonstrates a valuable application by discovering failure modes (e.g., preference for 'the hell' insertions) in state-of-the-art 8B-parameter RMs. If the attribute-purity assumption is validated, this would give practitioners an inexpensive auditing tool for RLHF components. However, the paper's central empirical claim that the method yields high-quality explanations and that PFR rankings isolate attribute-level sensitivity rests on unvalidated assumptions about the LLM-generated perturbations; these are the subject of the major comments.","major_comments":[{"comment":"Attribute purity of the perturbations is unvalidated. The causal attribution of CF/SF labels to individual attributes assumes that a Step-2 perturbation generated for attribute a changes only that attribute and in the intended direction. The paper checks locality only through aggregate syntactic/semantic distances (Table 2) and never checks whether a perturbed response actually altered the intended attribute while holding the others fixed. If, for example, a 'harmlessness' perturbation also reduces helpfulness or changes verbosity, the preference flip is not attributable to harmlessness-specific sensitivity of the RM. This confounds the PFR rankings in Figure 3, the Kendall-tau correlations in Table 3, and the representative-example interpretations in Section 4.2. Section 5's limitation paragraph acknowledges that counterfactuals are not guaranteed but does not acknowledge this attribute-conflation issue. A concrete remedy is to add a human or LLM-based annotation study that labels which attributes changed in a sample of perturbed responses, or at least to report the rate at which an independent judge identifies only the intended attribute as changed. Without such validation, the central empirical claim of the paper is not established.","section":"Section 2.3 / Section 4.1"},{"comment":"Cross-attribute PFR comparisons are not normalized by perturbation magnitude. Attributes for which GPT-4o happens to make larger changes (larger reward deltas or larger edit distances) will mechanically yield higher flip rates, even for an RM that weights all 15 attributes equally. The paper reports distances only in aggregate over all attributes (Table 2) and does not provide per-attribute perturbation strength. The global sensitivity ranking may therefore reflect generator strength rather than RM sensitivity. The authors should report, per attribute, the distribution of reward deltas (or a normalized semantic distance) and show that PFR differences persist when conditioning on similar perturbation magnitude; alternatively, they should explicitly frame the PFR as an interaction between the generator and the RM rather than a pure RM property.","section":"Section 4.1, Figure 3"},{"comment":"The SF categorization does not verify that a perturbation moved in the intended opposing direction. In Eq. (2), any perturbed response that does not flip the preference is called a semifactual; this includes perturbations that accidentally improved y+ or worsened y-. The desiderata in Section 2.3 explicitly aim to make y+ worse and y- better, but the evaluation never measures whether the RM reward actually moved in that direction. Since an off-target SF can result from an unchanged response or from a change affecting a different attribute, the CF/SF coverage numbers in Table 1 can overstate how well the method characterizes the local decision boundary. The authors should report the fraction of generated perturbations that move the reward in the intended direction, perhaps stratified by attribute, as a basic sanity check on the generation step.","section":"Section 2.2 / Section 3"}],"minor_comments":[{"comment":"Polyjuice results are missing for the hs2 dataset, despite the paper stating that evaluation is performed against two baselines on three datasets. Please either include the PJ results for hs2 or explain why they are omitted.","section":"Tables 1 and 2"},{"comment":"There is a typo in the list of categorized perturbation sets: 'YC+, YC-, YS- and YS-' should be 'YC+, YC-, YS+ and YS-' (or the fourth set should be YS+).","section":"Section 2.2"},{"comment":"Representative examples are selected by maximizing Kendall's tau between local and global rankings; if the same 500 comparisons are used for both the global sensitivity computation and the selection, the paper should state this explicitly and note that the selected example may overfit to the global pattern by construction.","section":"Section 4.2"},{"comment":"The list of 15 attributes is induced from the test sets used in the experiments via GPT-4o prompting. The paper should briefly discuss the risk that the attribute list is dataset-specific and that perturbations might change attributes not on the list, which would reinforce the attribute-purity concern raised in the major comments.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"This is a well-written paper with a promising framework and useful prompt-level reproducibility. My main concern is that the paper's central empirical claims go beyond what the current evaluation supports because attribute purity and perturbation magnitude are not validated. I would be willing to review a revision that adds such validation or appropriately weakens the claims. I also note that the missing Polyjuice baseline on hs2 and the 'first to propose' claim should be checked carefully before publication in a journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper adapts counterfactual and semifactual explanation to reward models' binary comparisons, using an attribute-conditioned LLM perturbation method. That specific framing is new, as far as I can tell, and the paper does it carefully. The two-step prompting (identify relevant words, then perturb around them) is sensible and the desiderata (locality, diversity, opposing-direction) are well chosen. The quantitative section compares against Polyjuice and random perturbation, and the ablation in Appendix D shows the centered-prompt choice is doing real work. Credit where due: the global sensitivity analysis has a nice internal validation—v2, trained on harmless data, is most sensitive to harmlessness-related attributes—and the qualitative finding that several RMs, including 8B ones, prefer disrespectful language is the kind of thing people will cite.\n\nNow the soft spots. The central claim of 'high-quality explanations' is assessed only with coverage, distance, and diversity metrics. None of these check whether the GPT-4o perturbation changed only the intended attribute. If a 'harmlessness' perturbation also shifts helpfulness or verbosity, then a preference flip can't be cleanly attributed to harmlessness. The paper never acknowledges this confound in the limitations section, and no code or data is released to let a reader check. A second, related issue: PFRs are compared across attributes without controlling for how strongly each attribute was perturbed; an attribute that GPT-4o happens to edit more aggressively will look more 'sensitive'. Both issues are fixable with human or LLM annotation of attribute purity and some magnitude normalization. They don't sink the method, but they mean the headline empirical claim is not established as cleanly as the paper suggests.\n\nWho is this for? Anyone building or auditing reward models. The method is immediately usable and the qualitative case studies are genuinely informative. It deserves a serious referee—the flaws are empirical validation gaps, not conceptual ones.\n\nMy recommendation: engage with it, ask the authors to run an attribute-purity check (even a small one) and release code/data. If they do, this becomes a solid toolkit paper.","headline":"A practical, clearly presented adaptation of contrastive explanations to reward models, with real qualitative payoff; the headline empirical claim is under-validated because attribute purity of perturbations is never checked.","tokens_in":18531,"tokens_out":1732,"would_cite":true,"duration_ms":15542,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes a post hoc, model-agnostic method to explain any binary reward-model comparison by generating attribute-conditioned LLM perturbations, classifying them as counterfactuals or semifactuals, and aggregating them into…","keywords":["reward models","contrastive explanations","counterfactuals","semifactuals","LLM perturbation","attribute sensitivity","RLHF alignment"],"falsifier":"Take a sample of the 15-attribute perturbation sets generated by the method, and have independent human annotators (or a separate judge model) label, for each perturbed response, which attributes actually changed relative to the original. If a large share of responses labelled harmlessness or helpfulness also change other attributes, the preference-flip rates and global sensitivity rankings would not cleanly attribute causes, and the central claim would need qualification.","tokens_in":99,"feed_emoji":"🔍","tokens_out":6163,"duration_ms":105992,"temperature":0.7,"pith_summary":"Reward models decide which of two responses a language model should prefer, but they are black boxes: a scalar score carries no reason for the choice. This paper proposes a post hoc method to explain any binary comparison by generating a diverse set of perturbed responses that deliberately change one of fifteen high-level evaluation attributes, such as harmlessness, helpfulness, clarity, or sensitivity. Each perturbed response is then classified as a counterfactual if the reward model's preference flips, or a semifactual if the preference stands, giving a local map of what matters to the model near the original comparison. Aggregating these labels across many comparisons yields a global sensitivity profile, showing how often each attribute flips the model's choice, and lets the authors pull out representative examples and compare different reward models.","feed_headline":"Controlled rewrites reveal why reward models prefer one answer","feed_subtitle":"Perturbs responses along 15 evaluation attributes, then sorts each into counterfactual or semifactual to map reward-model sensitivity.","key_machinery":"The load-bearing mechanism is the two-step attribute-conditioned perturbation prompt. In Step 1, the LLM is asked to identify, in each original response, the words relevant to each of fifteen high-level evaluation attributes (for example harmlessness, verbosity, relevance). In Step 2, the same LLM rewrites the chosen response to be worse (or the rejected response to be better) along one attribute, restricting edits to the words identified in Step 1 to keep perturbations local. The resulting perturbed responses are scored by the reward model and sorted into counterfactuals and semifactuals by whether the comparison preference flips; the preference-flip rate per attribute, aggregated over many comparisons, is the statistic that carries the global sensitivity analysis.","core_discovery":"The paper's central claim is that contrastive explanations of a reward model's binary comparison $(x, y_+, y_-)$ can be generated purely from black-box access to the model, using an external LLM to produce controlled perturbations. For a comparison with $r(x, y_+) > r(x, y_-)$, a perturbation $y'_+$ of the chosen response is a counterfactual when $r(x, y'_+) < r(x, y_-)$; a perturbation $y'_-$ of the rejected response is a counterfactual when $r(x, y'_-) > r(x, y_+)$. Perturbations that do not flip the preference are semifactuals. The authors show that generating perturbations via a two-step, attribute-conditioned prompting strategy yields both counterfactuals and semifactuals with high coverage on the HelpSteer2 and HH-RLHF datasets across three open reward models, and that aggregating preference-flip rates by attribute produces a ranking of attribute sensitivity that distinguishes reward models, including the finding that one model trained on the harmless dataset is markedly more sensitive to harmlessness, sensitivity, and neutrality than the others.","pith_inferences":["The method's usefulness depends on attribute purity: if the LLM's edits change more than the named attribute, the preference-flip rates conflate distinct causes; the paper validates locality with edit and semantic distances but does not directly measure attribute purity, so this is the natural next check.","If attribute purity holds, the same pipeline could generate attribute-labelled preference data automatically, providing a lower-cost route to fine-grained reward-model training sets in the direction the authors sketch in their conclusion.","The counterfactual/semifactual split could be refined further: semifactuals that sit between the two original rewards carry different information than those that overshoot, a distinction the paper notes as a limitation and future work."],"forward_implications":["Any black-box reward model can be audited locally without access to weights, gradients, or training data, because the method only queries the reward model for scores.","Attribute-level preference-flip rates give a global sensitivity ranking, enabling direct comparisons between reward models trained on different data and the discovery of failure modes such as preferring disrespectful rewrites.","Representative comparisons can be selected automatically by matching each comparison's local attribute ranking to the global sensitivity pattern, producing human-readable exemplars of model behaviour.","The same perturbation sets can be reused across multiple reward models, so behaviour differences can be attributed to the models themselves rather than to different inputs."],"supporting_citations":[{"why":"Defines the Bradley-Terry pairwise preference model that makes reward-model comparisons a binary classification task.","marker":"[6]"},{"why":"Supplies the HH-RLHF helpful and harmless datasets used to evaluate the generated explanations.","marker":"[4]"},{"why":"Provides the HelpSteer2 dataset and its multi-attribute labels that inform the high-level attribute list.","marker":"[42]"},{"why":"Describes Polyjuice, one of the two baseline perturbation methods against which the proposed approach is compared.","marker":"[43]"},{"why":"Documents the GPT-4 model family used as the external LLM for generating attribute-conditioned perturbations.","marker":"[1]"},{"why":"Provides the formal notion of semifactual explanations that the paper adapts to reward-model comparisons.","marker":"[22]"},{"why":"Provides the formal notion of counterfactual explanations that the paper adapts to reward-model comparisons.","marker":"[39]"},{"why":"Supplies the Sentence-BERT embeddings used to compute semantic distance and diversity metrics.","marker":"[35]"}],"fun_headline_variants":["Contrastive rewrites expose why reward models pick a response","Counterfactuals reveal reward model preferences","Reward model decisions explained via contrastive examples","Perturb answers to map reward-model sensitivity","Black-box reward models get contrastive explanations"],"cache_read_input_tokens":20736,"weakest_assumption_plain":"The whole analysis presumes that each LLM-generated perturbation changes only the intended evaluation attribute, while leaving the response's topic, style, and other qualities fixed, so that an observed preference flip can be credited to that attribute.","fun_headline_variants_meta":{"raw":{"variants":["Contrastive rewrites expose why reward models pick a response","Counterfactuals reveal reward model preferences","Reward model decisions explained via contrastive examples","Perturb answers to map reward-model sensitivity","Black-box reward models get contrastive explanations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000189,"raw_usage":{"total_tokens":1363,"prompt_tokens":1002,"completion_tokens":361,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":618,"completion_tokens_details":{"reasoning_tokens":290}},"tokens_in":618,"tokens_out":361,"duration_ms":4414,"temperature":1.0,"reasoning_tokens":290,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:01:28.113369+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a sample of the 15-attribute perturbation sets generated by the method, and have independent human annotators (or a separate judge model) label, for each perturbed response, which attributes actually changed relative to the original. If a large share of responses labelled harmlessness or helpfulness also change other attributes, the preference-flip rates and global sensitivity rankings would not cleanly attribute causes, and the central claim would need qualification.","supporting_citations":[{"cited_title":"As the former is not the focus of the datasets we experiment with, we only additionally include honesty, relabelled to avoid-to-answer for better relevance","cited_arxiv_id":null,"evidence_quote":"Provides the formal notion of semifactual explanations that the paper adapts to reward-model comparisons."}],"review_version":1}