Pith. sign in

REVIEW 4 major objections 7 minor 20 references

Discriminatively trained reward models memorize easy pairs, dataset shortcuts, and length/compliance heuristics instead of learning context-dependent quality.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 13:35 UTC pith:5VIYF53X

load-bearing objection First counterfactual-memorization maps of RMs; three clear patterns, but TG noise and a TM ceiling make the overgeneralization claim softer than the abstract sells. the 4 major comments →

arxiv 2607.24484 v1 pith:5VIYF53X submitted 2026-07-27 cs.LG cs.CL

What do Reward Models Memorize?

Ivo Verhoeven , Pushkar Mishra , Ekaterina Shutova This is my paper
classification cs.LG cs.CL
keywords reward modelscounterfactual memorizationhuman preference dataRLHFshortcut learningBradley-Terryreward hackingpreference alignment
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Reward models sit at the center of preference alignment: they score which model responses humans would prefer, and those scores steer later training. This paper asks what those models actually memorize when trained the usual way on human preference pairs. By measuring how much each pair’s classification probability changes when it is included versus held out of training, the authors map memorization across two large preference datasets and tie the map to surface features of the responses and the data collection process. They find three recurring patterns: memorization is spent on easy, high-margin pairs rather than hard ones; models lock onto dataset artifacts such as which base model wrote the answer or which recruitment wave a user came from; and simple correlates like verbosity and compliance are over-applied to unseen pairs. The practical claim is that this style of training yields biased proxies that still cannot judge response quality when context matters.

Core claim

Discriminative Bradley-Terry training of reward models on human preference data systematically produces three memorization patterns—misallocation of memorization to easy high-margin pairs, memorization of non-causal dataset shortcuts (model identity, user collection wave), and overgeneralization of simple heuristics such as length and compliance—so the resulting models are biased and not yet able to judge response quality in context-dependent settings.

What carries the argument

Counterfactual memorization (CM): the gap between expected preference probability when a pair is in the training set (training memorization) and when it is held out (testing generalization), estimated over many user-level data splits and visualized as a memorization map that is then explained by feature attributions and sparse autoencoder latents.

Load-bearing premise

That the difference in preference probability from only a few dozen user-level hold-out splits on a small adapter-tuned model is a faithful measure of what the reward model has memorized about each pair, and that surface-feature regressions can stand in for that memorization.

What would settle it

Retrain the same reward-model pipeline with an objective or curriculum that forces memorization onto low-margin, controversial pairs and strips model-identity and collection-wave cues; if the three CM patterns and the associated reward-hacking behaviors disappear on held-out user groups and models, the central claim is undermined.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • RM objectives should be redesigned to encourage complementary memorization of low-margin and context-sensitive pairs rather than maximizing margin on easy ones.
  • Evaluation benchmarks must hold out user groups and response models so that dataset-shortcut memorization cannot inflate scores.
  • Curriculum learning and causally informed training become natural remedies once misallocated memorization and heuristic overgeneralization are treated as the default outcome of plain Bradley-Terry training.
  • Surface heuristics such as length and compliance will keep enabling reward hacking on out-of-distribution pairs until dependence on them is actively mitigated.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If high-margin pairs dominate memorization capacity, simply scaling preference data without changing the objective may deepen the bias rather than fix it.
  • The same counterfactual-memorization map could be used as a diagnostic filter when building new preference datasets, discarding or reweighting pairs that land in the high-CM shortcut regions.
  • Implicit reward models inside direct preference methods may inherit the same three patterns unless their training signal is similarly audited.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The manuscript applies counterfactual memorization (Feldman 2020; Zhang et al. 2023; Dankers et al. 2023) to discriminatively trained Bradley–Terry reward models, a setting in which instance-level memorization has not previously been studied. Using 25 user-level splits of two human preference datasets (PRISM and COMMUNITY), the authors compute per-pair training memorization (TM), testing generalization (TG), and their difference (CM, Eq. 2), producing memorization maps (Fig. 2). They then attribute variation in these quantities to 84 hypothesis-driven features via an MLP regressor with SHAP, and to data-driven features via a conditional SAE. Three patterns are reported: (1) memorization is concentrated on high-margin, easy pairs rather than difficult ones; (2) dataset artifacts (response model identity in PRISM; user collection wave in COMMUNITY) are memorized; (3) simple heuristics (length, compliance) are overgeneralized, with violations of the heuristic requiring memorization. The authors conclude that discriminatively trained RMs are biased and not yet capable of context-dependent quality judgments.

Significance. If the results hold, this is a useful and timely contribution: it gives the RLHF community a concrete, instance-level account of *what* RMs memorize, connecting the memorization literature to well-documented reward-hacking behaviors (verbosity, sycophancy, model-identity bias). The work has real methodological strengths worth crediting: the user-level hold-out design with explicit minimization of user–user co-occurrence (App. B) is more careful than standard CM pipelines; the analysis is replicated across two datasets with quite different construction; the SAE feature descriptions are validated against independent LLM annotations with Bonferroni correction (App. G); and code is publicly released. The three patterns are stated crisply enough to be falsifiable by follow-up work (e.g., margin-aware objectives, model-held-out evaluation). That said, the paper's descriptive claims currently rest on noisy CM estimates whose variance is never propagated, and one of the three patterns is vulnerable to a scale artifact in the SHAP comparison, so the strength of the conclusions is not yet fully supported.

major comments (4)
  1. [§3, Eq. (2); §4.3; §8] TG (and hence CM) is estimated from only 5 held-out runs per pair, as the authors acknowledge in §8, but no uncertainty quantification is ever reported. Every load-bearing contrast in the paper — margin vs. CM (§4.3), the model-identity SHAP differences (Fig. 4), the 'Complete Secondary' education-group contrast (§4.3, Table 9), the block-level MAV rankings (Tables 4–5) — is presented as a point estimate. Given that the CM regressions only reach R²=0.39/0.33 (§4.3), estimation noise could account for a substantial fraction of the 'explained' variance, and the feature rankings could be unstable across split draws. This is fixable within the existing pipeline: please report per-pair standard errors on CM (bootstrap over the 5 hold-out runs), confidence intervals on the group contrasts that carry each pattern, and a robustness check showing that the SHAP feature rankings in Tables 4–5 are s
  2. [§4.3 'Heuristics'; Tables 4–5; Fig. 2] Pattern 3 (overgeneralization) rests on the observation that most feature blocks have larger SHAP MAV for TG than for TM. But Fig. 2 shows TM is near saturation, with most pairs clustered near (1,1) and little residual variance; SHAP magnitudes are expressed in units of the target and scale with its residual variance. Mechanically, then, almost any feature will show smaller MAV for TM than for TG, and CM attribution will be inflated for the few features (margin, model identity) that explain the residual TM variance. The Complexity block gap (TM 0.15 → TG 0.36, Table 4) is consistent with this compression rather than with RM behavior. The interpretation 'the RM attenuates these features for seen pairs' needs a variance-aware control: e.g., compare SHAP values normalized by target standard deviation, repeat the analysis on the subset of pairs with TM away from ceiling, or report the sign/o
  3. [§4.3 'User Preference Margin'; App. D] For PRISM, the (binned) user margin is an explicit term in the training loss (App. D). The finding that margin dominates TM and CM attribution for PRISM is therefore at least partly built into the objective rather than an emergent memorization pattern — the model is directly optimized to separate high-margin pairs. The authors are transparent about this ('not altogether surprising', §4.3) and lean on COMMUNITY, where the margin is a constructed proxy from cross-user prompt repetitions (App. F.2). But that proxy is itself noisy (only 27% of turns have entropy annotations, Table 9 'Turn Entropy' prevalence) and is derived from user–user disagreement, which is exactly the population the user-level split holds out. Please quantify how much of pattern 1 survives when (a) PRISM RMs are trained without the margin term, or (b) the analysis is restricted to COMMUNITY pairs with well-estimated pro
  4. [§3; App. C; §6] All RMs are LoRA fine-tunes (r=32, ~2% of parameters) of Llama-3.2-1B. The cited PEFT literature (App. C) indicates PEFT substantially reduces memorization relative to full fine-tuning, and a 1B backbone is far smaller than production RMs. The generalization claim in the abstract and §6 ('discriminative training of RMs ... results in biased RMs') is therefore made for a regime the paper itself argues is a lower bound on memorization. The direction of the bias may well be conservative, but the *patterns* (which features get memorized) could also change qualitatively with capacity. At minimum, the abstract and §6 should scope the claims to this regime; ideally, a small-scale full fine-tuning or larger-backbone replication of at least the margin and model-identity results would substantially strengthen the paper.
minor comments (7)
  1. [§3, Fig. 2] Fig. 2 and §3: the 'few negative CM preference pairs ... attributed to noise' claim would be more convincing with the per-pair noise floor from Major Comment 1; as stated it is an assumption.
  2. [App. D] App. D contains a likely typo with sign-flipped meaning: 'When training RMs on COMMUNITY we use the same methodology, except that we do include a user margin' — from context (and the preceding sentence saying COMMUNITY lacks scalar preferences) this should presumably read 'do not include'. Please clarify.
  3. [App. G] The caption and text refer to 'Figure 1' for the CM regions in App. G ('The regions are displayed in Figure 1, and Figure 11'), but Fig. 1 is the conceptual overview; presumably Fig. 2 or 11 is meant.
  4. [§5; App. G, Table 7] The SAE results (§5, Table 7) are noticeably weaker than the SHAP analysis — the authors themselves report difficulty replicating Movva et al.'s features and substantial multi-collinearity (App. G). The COMMUNITY feature list is dominated by near-duplicate formatting descriptions ('narrative paragraph format', 'written as narrative paragraphs', 'uses structured bullet-point lists'). Consider trimming or explicitly flagging redundancy.
  5. [References; Table 3] Several references are incomplete or malformed (e.g., 'Shu et al.' lacks a year and venue; 'Leo, 2025' and 'Int, 2026' in Table 3 are placeholder-style citations; OpenAI gpt-5/gpt-5-mini/gpt-4.1-nano are used without version or date stamps, which matters for reproducibility of the annotations).
  6. [§4.3, footnote 2] Footnote 2 (§4.3) explaining the reduced R² for CM is helpful but easy to miss; given that R²=0.39/0.33 is one of the more important numbers for interpreting the attribution strength, consider promoting this discussion to the main text.
  7. [Fig. 5] Fig. 5's secondary HMR axis values (printed numbers) are hard to connect to the bars; a short caption sentence explaining how to read them jointly with the SHAP difference would help.

Circularity Check

0 steps flagged

No significant circularity: CM is measured independently of the feature analyses; the three memorization patterns are post-hoc empirical associations, not predictions forced by construction.

full rationale

The load-bearing quantity is counterfactual memorization CM = TM − TG (Eq. 2), estimated from Bradley-Terry classification probabilities under user-level inclusion/exclusion splits. That definition does not refer to margin, model identity, length, compliance, or any of the later annotated features. Feature blocks are either dataset metadata or external LLM/tool annotations; SHAP on an MLP regressor and the conditional SAE are explicitly post-hoc explanations of an already-computed (TM, TG, CM) map. There is no uniqueness theorem, no self-citation that forbids alternatives, and no first-principles “prediction” that reduces to a fitted parameter. The one near-miss—PRISM’s binned user margin appearing inside the training loss (App. D)—is acknowledged by the authors and is not required for the claim, which they also report on COMMUNITY where no margin term is used; even on PRISM, high CM further requires low TG, which the margin term does not force. The SAE’s auxiliary region-classification head makes association weights supervised by construction, but that is ordinary supervised interpretation of latents, not a circular derivation of the memorization patterns themselves. The paper is descriptive measurement plus correlational attribution; it does not smuggle its conclusions in via definition or self-citation.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central claims rest on the operational definition of counterfactual memorization, the Bradley-Terry probabilistic model, the validity of automated feature annotations, and the assumption that LoRA fine-tunes on a 1B model exhibit the same qualitative memorization patterns as full-scale RMs. No new physical entities are postulated; free parameters are ordinary training and analysis hyperparameters.

free parameters (5)
  • number of random splits / hold-out multiplicity = 25 splits, 5 hold-outs
    25 splits with each user held out in 5; directly controls variance of TG and therefore CM estimates.
  • LoRA rank and alpha = r=32, α=32
    r=32, α=32; PEFT choice known to modulate memorization magnitude.
  • RM training epochs = 3
    3 epochs chosen explicitly to go past validation convergence in order to study memorization.
  • user-margin binning quantiles (PRISM) = quartile bins → {0,1/3,2/3,1}
    Margins mapped to {0,1/3,2/3,1} via 25/50/75/100 quantiles and added to the BT loss.
  • SAE latent size and active features = 32 / 4
    32 latents, average 4 active; Matryoshka Batch Top-k plus auxiliary region classifier.
axioms (5)
  • domain assumption Bradley-Terry model correctly represents human pairwise preference probabilities (Eq. 1).
    Standard in RM literature; used both for training and for the probability terms inside CM.
  • domain assumption Counterfactual memorization CM = TM − TG (Eq. 2) is a valid graded measure of instance-level memorization.
    Imported from Feldman / Zhang / Dankers; the entire analysis is built on this definition.
  • ad hoc to paper User-level splitting eliminates preference leakage while still permitting pair-level CM interpretation.
    Necessary computational compromise; authors argue within-user CM variance dominates, but this is unproven.
  • domain assumption Automated LLM and classifier annotations (compliance, sycophancy, emotions, dialog acts, etc.) are sufficiently accurate proxies for the true linguistic features.
    Feature blocks are generated by gpt-4.1-nano, midas, go-emotions, polite-guard, etc.; SHAP attributions inherit any annotation error.
  • domain assumption LoRA PEFT memorization patterns are qualitatively representative of full-parameter RM training.
    Authors cite literature that PEFT memorizes less; they claim directionality still holds (Appendix D).

pith-pipeline@v1.2.0-grok45-kimik3 · 48149 in / 3047 out tokens · 49864 ms · 2026-07-31T13:35:47.400066+00:00 · methodology

0 comments
read the original abstract

This paper studies what discriminatively trained reward models (RMs) memorize by measuring counterfactual memorization on two human preference datasets. We show that RMs 1) misallocate memorization to easy, high margin preference pairs, 2) memorize dataset-specific shortcuts (e.g., model identity, user sampling strategy), and 3) overgeneralize simple heuristic correlates of human preference (e.g., length, compliance) when confronted with unseen preference pairs. Overall, our findings indicate that discriminative training of RMs from human preference data results in biased RMs not yet capable of judging response quality in context-dependent scenarios.

Figures

Figures reproduced from arXiv: 2607.24484 by Ekaterina Shutova, Ivo Verhoeven, Pushkar Mishra.

Figure 1
Figure 1. Figure 1: Conceptual overview: We measure counter￾actual memorization by taking the difference in proba￾bility of preference for an RM when a preference pair is or is not in the training data. We use this to construct a ‘memorization map’, and try to determine what features are associated with different regions in this map. been explored. Counterfactual Memorization & RMs Counter￾factual memorization, as introduced … view at source ↗
Figure 2
Figure 2. Figure 2: Memorization maps: PRISM in the left pane and COMMUNITY in the right. The TM score is plotted along the horizontal axis, TG along the vertical axis. The joint histogram is plotted centrally. For each bin, color denotes the CM bin (see legend), and darkness the density of data present. The top and side panels give a KDE estimate of the marginal distributions of the preference pairs, for each CM bin independ… view at source ↗
Figure 3
Figure 3. Figure 3: Block-level feature importance: block-level net SHAP values, summarized by Mean Absolute Value (MAV) and Harmonic Mean Rank (HMR). Arrows point in direction of greater partial effect of the feature block on RM performance. The effect on TM , TG and CM are denoted by a red circle, a blue square, and a green triangle, respectively. The more extreme a value, the more important that block as a whole is to RM p… view at source ↗
Figure 4
Figure 4. Figure 4: Model identity partial effect: mean SHAP value induced whenever a particular model is chosen (orange), rejected (tan) or is used for both chosen and rejected responses (black). Each pane gives the SHAP value for a different endogenous variable. Models are sorted by average RM score, with left-most being the ‘worst’ performing model and right-most the ‘best’. nant is model identity. We plot the SHAP values … view at source ↗
Figure 5
Figure 5. Figure 5: Rule-based rewards & sycophancy partial effect: the change in expected SHAP when there is a rule￾based rewards or sycophancy feature difference across preference pairs: E[SHAPi,j |featj (y + i ) = 1, featj (y − i ) = 0] − E[SHAPi,j |featj (y + i ) = 0, featj (y − i ) = 1]. We interpret this as the effect of flipping a feature’s presence in the chosen and rejected responses. The secondary axis provides the … view at source ↗
Figure 6
Figure 6. Figure 6: SAE features for different CM regions: the feature descriptions of the conditional SAE latent space with the highest association weights with different regions in the counterfactual memorization map. These regions correspond to low/low (purple); medium/medium (blue); high/low (green); high/high (red) TM / TG , respectively. The darker the color, the greater the association between the SAE feature and that … view at source ↗
Figure 7
Figure 7. Figure 7: Dataset structure: Left is PRISM, right is COMMUNITY. Both datasets provide user-level metadata, and have the user interacting with the LLM also rate each response. In PRISM, these responses come from 1 of 21 models in the first turn, and the user rates each response on an ordinal scale, with the LLM providing the best response being used for subsequent turns. In COMMUNITY, the user selects the best respon… view at source ↗
Figure 8
Figure 8. Figure 8: Loss and gradient norm curves for all 25 model iterations when training on [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Loss and gradient norm curves for all 25 model iterations when training on [PITH_FULL_IMAGE:figures/full_fig_p017_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Loss and gradient norm curves for all 25 model iterations when training on [PITH_FULL_IMAGE:figures/full_fig_p017_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: SAE discriminative regions: we divide the memorization map into 4 distinct, similar triangles. The colors and numbers match those used in [PITH_FULL_IMAGE:figures/full_fig_p022_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Correlation between the mean user score of a chosen and rejected response and the mean RM score of a [PITH_FULL_IMAGE:figures/full_fig_p023_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Complexity partial effect: block-level net SHAP values induced by complexity differences between responses. Shapes indicate mean per bin, whereas solid lines give linear trend lines. We trim the data bins between the 0.1st and 99.9th percentiles. 2.5 0.0 2.5 0.04 0.02 0.00 0.02 0.04 Net SHAP Positive 2.5 0.0 2.5 Negative 2.5 0.0 2.5 Neutral (a) PRISM 2.5 0.0 2.5 0.04 0.02 0.00 0.02 0.04 Net SHAP Positive … view at source ↗
Figure 14
Figure 14. Figure 14: Emotion partial effect: block-level net SHAP values induced by emotion differences between responses. We aggregate SHAP scores based on sentiment categories, as described in Demszky et al. (2020)’s [PITH_FULL_IMAGE:figures/full_fig_p024_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 7 linked inside Pith

  1. [1]

    The greater this value, the more extreme a features impact on average

    MAV: the arithmetic mean of the absolute value of the Shapley values, across all sam- ples. The greater this value, the more extreme a features impact on average

  2. [2]

    Correlation( r): the Pearson correlation co- efficient between the SHAP values and the exogenous feature. This captures the direc- tion of the relationship between a feature’s value and the expected SHAP value 3.Beta(β): defined as β=correlation(ϕ j,ˆy)std(ˆy) std(ϕj) this captures the expected increase in Shap- ley values for a 1 unit increase in the exo...

  3. [4]

    InProceed- ings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 18647–18664, Vienna, Austria

    The AI Gap: How Socioeconomic Status Af- fects Language Technology Interactions. InProceed- ings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 18647–18664, Vienna, Austria. Associa- tion for Computational Linguistics. Reza Bayat, Mohammad Pezeshki, Elvis Dohmatob, David Lopez-Paz, and Pascal V...

    2024

  4. [5]

    InICML 2025 Workshop on Collaborative and Federated Agentic Workflows

    Mitigating Unintended Memorization with LoRA in Federated Learning for LLMs. InICML 2025 Workshop on Collaborative and Federated Agentic Workflows. Ralph Allan Bradley and Milton E. Terry. 1952. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons.Biometrika, 39(3/4):324. Bart Bussmann, Patrick Leask, and Neel Nanda

    2025

  5. [7]

    InThe Thirty-ninth 9 Annual Conference on Neural Information Process- ing Systems

    BLEUBERI: BLEU is a surprisingly effective reward for instruction following. InThe Thirty-ninth 9 Annual Conference on Neural Information Process- ing Systems. Myra Cheng, Kristina Gligoric, Tiziano Piccardi, and Dan Jurafsky. 2024. AnthroScore: A Computational Linguistic Measure of Anthropomorphism. InPro- ceedings of the 18th Conference of the European ...

    2024

  6. [10]

    Edward J

    Impact of Fine-Tuning Methods on Mem- orization in Large Language Models.Preprint, arXiv:2507.00258. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA: Low-rank adaptation of large language models. InInternational Conference on Learning Representations. Zeyu Huang, Zihan Qiu, Zili Wan...

    Pith/arXiv arXiv 2021

  7. [12]

    InFindings of the Associ- ation for Computational Linguistics: EMNLP 2024, pages 12891–12907, Miami, Florida, USA

    Enhancing Alignment using Curriculum Learn- ing & Ranked Preferences. InFindings of the Associ- ation for Computational Linguistics: EMNLP 2024, pages 12891–12907, Miami, Florida, USA. Associa- tion for Computational Linguistics. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn

    2024

  8. [13]

    InProceedings of the 37th International Conference on Neural Infor- mation Processing Systems, NIPS ’23, pages 53728– 53741, Red Hook, NY , USA

    Direct preference optimization: Your language model is secretly a reward model. InProceedings of the 37th International Conference on Neural Infor- mation Processing Systems, NIPS ’23, pages 53728– 53741, Red Hook, NY , USA. Curran Associates Inc. Manon Revel, Matteo Cargnelutti, Tyna Eloundou, and Greg Leppert. 2025. SEAL: Systematic error anal- ysis for...

    Pith/arXiv arXiv 2025

  9. [15]

    the margin term can indeed help the reward model perform better on more separable comparison pairs ... [it] also regresses performance on similar samples

    Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment.Preprint, arXiv:2501.09620. Fei Wang and Baochun Li. 2025. Leaner Train- ing, Lower Leakage: Revisiting Memorization in LLM Fine-Tuning with LoRA.Preprint, arXiv:2506.20856. Hao Wang, Licheng Pan, Zhichao Chen, Chunyuan Zheng, Zhixuan Chu, Xiaoxi Li, Yuan Lu, Xing- gao Liu, Haoxuan L...

    Pith/arXiv arXiv 2025

  10. [17]

    This model was specifically trained on human-LLM interactions Rule-based Rewards {0,1} 16 We annotate responses for possessing one of the identified LLM behaviors in Mu et al

    emotions over the entire response using AnasAlokla/multilingual_go_emotions PolitenessR4 We get logits for politeness levels over the entire response using Intel/polite-guard (Int, 2026). This model was specifically trained on human-LLM interactions Rule-based Rewards {0,1} 16 We annotate responses for possessing one of the identified LLM behaviors in Mu ...

    2026

  11. [20]

    Unlike the previous metrics, which capture the impact of a feature absolutely, this metric is relative to all other features active in a preference pair

    Harmonic Mean Rank: the harmonic mean of the row-wise rank of the absolute SHAP values 1 N NX i=1 (rank(ϕi)j)−1 !−1 We only compute this metric whenever a fea- ture is active (i.e., there is a difference be- tween responses in a preference pair). Unlike the previous metrics, which capture the impact of a feature absolutely, this metric is relative to all ...

    2025

  12. [1400]

    16 0 1 2 3 0.0 0.5 1.0 1.5 Loss 0 1 2 3 0 10 20 30 40 Grad Norm Figure 8: Loss and gradient norm curves for all 25 model iterations when training onPRISM

    We use the same learning-schedule as SFT, and optimise using AdamW (Kingma and Ba, 2015; Loshchilov and Hutter, 2019). 16 0 1 2 3 0.0 0.5 1.0 1.5 Loss 0 1 2 3 0 10 20 30 40 Grad Norm Figure 8: Loss and gradient norm curves for all 25 model iterations when training onPRISM. Dotted black lines denote epoch boundaries. 0 1 2 3 0.0 0.5 1.0 1.5 Loss 0 1 2 3 0 ...

    2015

  13. [2020]

    InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4040–4054, Online

    GoEmotions: A dataset of fine-grained emo- tions. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4040–4054, Online. Association for Computational Linguistics. Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ah- mad Beirami, Alex D’Amour, D. J. Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Deepak ...

    Pith/arXiv arXiv 2024

  14. [2021]

    Elisa Bassignana, Amanda Cercas Curry, and Dirk Hovy

    A general language assistant as a laboratory for alignment.Preprint, arXiv:2112.00861. Elisa Bassignana, Amanda Cercas Curry, and Dirk Hovy

    Pith/arXiv arXiv

  15. [2022]

    InPro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 1816– 1826, Abu Dhabi, United Arab Emirates

    An Empirical Analysis of Memorization in Fine-tuned Autoregressive Language Models. InPro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 1816– 1826, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. Rajiv Movva, Smitha Milli, Sewon Min, and Emma Pierson. 2025. What’s In My Human Feed...

    2022

  16. [2023]

    InProceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, pages 8323–8343, Singapore

    Memorisation Cartography: Mapping out the Memorisation-Generalisation Continuum in Neural Machine Translation. InProceedings of the 2023 Conference on Empirical Methods in Natural Lan- guage Processing, pages 8323–8343, Singapore. As- sociation for Computational Linguistics. Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade,...

    2023

  17. [2024]

    Bart Bussmann, Noa Nabeshima, Adam Karvonen, and Neel Nanda

    BatchTopK Sparse Autoencoders.Preprint, arXiv:2412.06410. Bart Bussmann, Noa Nabeshima, Adam Karvonen, and Neel Nanda. 2025. Learning Multi-Level Features with Matryoshka Sparse Autoencoders.Preprint, arXiv:2503.17547. Maarten Buyl, Hadi Khalaf, Claudio Mayrink Verdun, Lucas Monteiro Paes, Caio Cesar Vieira Machado, and Flavio Du Pin Calmon. 2025. AI Alig...

    Pith/arXiv arXiv 2025

  18. [2025]

    Leolani/cltl-dialogueclassification. Leolani

  19. [2026]

    Intel® Corporation

    Intel/polite-guard. Intel® Corporation. Lisa Alazraki, Yi-Chern Tan, Jon Ander Campos, Max- imilian Mozes, Marek Rei, and Max Bartolo. 2025. Reverse Engineering Human Preferences with Re- inforcement Learning. InThe Thirty-ninth Annual Conference on Neural Information Processing Sys- tems. Sotiris Anagnostidis, Gregor Bachmann, Lorenzo Noci, and Thomas Ho...

    2025

  20. [9471]

    Ruike Song, Zeen Song, Huijie Guo, and Wenwen Qiang

    Curran Associates, Inc. Ruike Song, Zeen Song, Huijie Guo, and Wenwen Qiang. 2026. Causal reward adjustment: Mitigating reward hacking in external reasoning via backdoor correction.Proceedings of the AAAI Conference on Artificial Intelligence, 40(39):33019–33027. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nik...

    Pith/arXiv arXiv 2026