Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling

Bo Chen; Dandan Guo; Guowei Rong; Mingyuan Zhou; Zhibin Duan; Zhuo Li

REVIEW 5 major objections 6 minor 1 cited by

Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling

T0 review · 5 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By rewriting an LLM reward head as a sparse non-negative sum of latent factors, BNRM claims to suppress reward hacking driven by length and formatting biases while yielding interpretable, uncertainty-aware reward scores.

desk verdict Promising idea, suggestive results, but the debiasing mechanism is not actually pinned down: the Bayesian prior is nearly inert, the algorithm description is internally inconsistent, and the learning-rate confound is unaddressed. read the letter →

arxiv 2602.10623 v2 pith:66SCQFUP submitted 2026-02-11 cs.LG cs.AI

Zhibin Duan , Guowei Rong , Zhuo Li , Bo Chen , Mingyuan Zhou , Dandan Guo This is my paper

classification cs.LGcs.AI

keywords rewardhackingRLHFnon-negativefactoranalysisBradley-Terrymodelvariationalinferencesparsityinterpretabilityover-optimization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that reward hacking in RLHF is driven by the dense, deterministic, uninterpretable reward representation. It proposes BNRM, which writes the reward as a sparse non-negative weighted sum of latent factors: per-response activations and global factor weights, trained end-to-end with a Bradley-Terry objective. Local sparsity is claimed to disentangle semantic reward axes; global sparsity is claimed to implicitly down-weight spurious correlations like length and formatting. Empirically, BNRM raises preference accuracy on in- and out-of-distribution benchmarks, cuts reward–length correlation from 0.488 to 0.123, withstands label noise, and aligns better with a gold reward under Best-of-N selection. If correct, this offers a lightweight single-model alternative to ensembles and explicit debiasing supervision.

What carries the argument

The load-bearing object is the non-negative reward decomposition r(x,y) = theta^T Phi. The instance-specific sparse vector theta encodes which reward factors are active for a given response; the global sparse dictionary Phi assigns a shared non-negative weight to each factor. Gamma priors enforce sparsity and non-negativity, and a Bradley-Terry likelihood closes the generative model. Scalability comes from an amortized variational inference network that maps the LLM's dense features to Weibull parameters for theta and Phi, enabling end-to-end backpropagation with reparameterized samples. Local sparsity is meant to disentangle factors; global sparsity is meant to debias.

What would settle it

Construct a preference dataset with a known spurious feature (e.g., response length) that is perfectly predictive during training but irrelevant or reversed on a held-out set. If BNRM's learned global weight for that feature stays high, or if the reward–length Pearson correlation does not drop on the held-out set, the implicit-debiasing claim is refuted. A controlled synthetic version would directly measure the Phi weight attached to the injected feature instead of relying on aggregate benchmark correlations.

Watch

Extended reading notes

Core claim

The central claim is that imposing a non-negative factor-analysis structure on the reward head—instead of a dense linear head—mitigates reward over-optimization. BNRM models the reward as the inner product of a sparse per-response non-negative loading vector and a sparse global non-negative dictionary, with both treated as random variables under Gamma priors. Inference is amortized through the LLM backbone using a reparameterizable Weibull posterior, trained on the Bradley-Terry preference likelihood. The authors argue that local sparsity induces disentangled, interpretable preference factors while global sparsity suppresses population-level non-causal biases, so no explicit debiasing superv

Load-bearing premise

The debiasing is an emergent hope: sparsity in the global factor weights is assumed to isolate causal reward axes and zero out non-causal ones, but the paper offers no formal guarantee—Section 4.4 is an intuition and Section 5.4 is a post hoc factor inspection, so a spuriously correlated factor could keep a high global weight if it happens to be useful across the training set.

Editorial extensions

If this is right

If the central claim holds, replacing a dense reward head with a sparse non-negative factor layer—without ensembling or explicit bias labels—should reduce reward over-optimization in standard RLHF pipelines.
Under scarce preference data, the model should be more data-efficient: with 1K training examples it roughly matches a standard Bradley-Terry model trained on 20K on at least one held-out benchmark, with wider gains as label noise increases.
Length and formatting biases should be encoded in low-weight factors, so Best-of-N selection with BNRM as the proxy tracks a gold reward as KL grows, whereas standard BT diverges.
The factor decomposition provides a mechanism for inspecting why a response received its score: per-response activations and global weights can be read as interpretable preference axes rather than a black-box scalar.
As a plug-and-play reward head, BNRM can be layered onto different base preference objectives and backbones, extending its benefits beyond a single training recipe.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If sparsity rather than the specific Gamma/Weibull parameterization is the active ingredient, then other sparse Bayesian priors (such as spike-and-slab or horseshoe priors over factors) should produce similar debiasing; a prior-ablation study would reveal the mechanism.
The paper's error-rectification analysis is post hoc, so a sharper test would train on a dataset with an artificially injected spurious feature and probe whether the learned global weight on that feature collapses—separating implicit regularization from representation learning.
Because the reward is a non-negative combination, factors that are useful in some contexts and harmful in others must either be split or globally down-weighted; this expressivity limit suggests a natural extension with signed or hierarchical factors.
The framework's success predicts that BNRM-style sparse factor heads could reduce the inference cost of uncertainty-aware RLHF, since a single model provides both calibrated uncertainty and debiasing rather than requiring an ensemble.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the paper proposes something worth looking at—replace the dense linear head in a Bradley-Terry reward model with sparse, non-negative latent factors, both per-sample and global, and let sparsity suppress spurious correlations like length bias. The empirical results are consistent with that story: consistent OOD gains on RewardBench, HHH, MT-Bench, and a real reduction in length correlation (Pearson 0.488 to 0.123). That is a concrete, useful signal, and the interpretability analysis via GPT-5 is a nice touch.

What's actually new: integrating non-negative factor analysis into a BT reward model for RLHF, with two-level sparsity and amortized Weibull variational inference, is not something I've seen in the cited literature. The method is lightweight relative to ensembles and the paper reports broad evaluation, including BoN and PPO. Credit where due: the core idea has legs.

Now the soft spots, and they are not minor. First, Algorithm 1 is internally inconsistent with the equations: the KL divergence is written as KL(Gamma || Weibull), but Eq. 9 and the generative process require KL(Weibull || Gamma). The algorithm also computes per-token rewards (shape R^{2B x L x 1}) without a pooling step, so it's not clear how the scalar preference score is formed. Second, the Bayesian prior is nearly inert—η=1e-5 makes the KL term negligible, so the sparsity is realistically coming from the ReLU activation on the variational scale, not from the Gamma prior. That undermines the 'Bayesian non-negative' mechanism. Third, BNRM is trained with 5x to 10x higher learning rates than the baselines (Table 6), so some of the gains could be a learning-rate effect rather than the model structure. There are no error bars, no multiple seeds, and no code. The post hoc factor analysis (Fig. 6, Table 8) is the only evidence for the debiasing mechanism, and it's equally consistent with any trained linear head down-weighting unhelpful features.

The stress-test note is right: the implicit debiasing claim is not causally tied to the sparsity/non-negativity. That said, it's a fair hypothesis, and it's not disproven either.

Who is this for: researchers working on reward modeling and alignment, especially those interested in sparse Bayesian structure as an inductive bias. It deserves a serious referee, but the referee should demand code, fixed algorithm details, multiple seeds, and an ablation that controls learning rate and isolates the sparsity mechanism. I would not cite it in its current form, but I'd read a revised version.

Referee Report

5 major / 6 minor

Summary. The paper proposes BNRM, a Bayesian non-negative factor model for reward modeling in RLHF. The scalar reward head is replaced by latent non-negative local factors theta and a global non-negative dictionary Phi, trained with a Bradley-Terry likelihood and amortized Weibull variational inference on top of an LLM backbone. The authors claim that sparsity and non-negativity at both local and global levels implicitly disentangle reward factors and suppress spurious correlations, thereby mitigating reward hacking, improving OOD generalization, and providing interpretable reward decompositions. Empirical sections report gains on Unified-Feedback, RM-Bench, RewardBench, HHH, MT-Bench, PPO fine-tuning, and Best-of-N tests.

Significance. If the central claims hold, BNRM would be a lightweight alternative to reward-model ensembles and information-bottleneck methods: it adds only a small number of parameters, yields uncertainty estimates, and produces factor-level interpretability. The paper evaluates on several external benchmarks, which is a genuine strength, and the BoN experimental design is a reasonable attempt to measure reward over-optimization directly. The low-resource and label-noise experiments address practically important settings. However, the evidence as currently presented is not yet load-bearing: the training algorithm as written is ill-defined, the claimed debiasing mechanism lacks a controlled ablation, and the reported gains are confounded by substantially different learning rates. These issues must be resolved before the empirical claims can be accepted.

major comments (5)

[A.1, Algorithm 1 (lines 8-23)] As written, the algorithm is ill-defined. The code constructs theta and r as 3D tensors of shape 2B x L x 1024 (lines 9-13, 22), but the BT loss on line 23 uses scalar quantities r^1_i - r^2_i. No pooling or reduction over the sequence dimension L is specified. This makes the central training objective non-reproducible. In addition, lines 14 and 21 compute KL(Gamma(1,1) || Weibull(...)), which is the reverse direction of the KL(q || p) terms in Eq. (9). If the actual implementation follows Algorithm 1, the stated ELBO is not what is optimized. Please specify the pooling operation and correct the KL direction, or explicitly justify the reversed direction.
[Table 6 / Sec. 5.1] BNRM is trained with learning rates 5e-5 vs. 1e-5 for the LoRA experiments and 2e-5 vs. 2e-6 for full fine-tuning, i.e. a 5-10x difference from the baselines. This alone could explain a large part of the observed accuracy gains. The central empirical claim that BNRM 'consistently boosts' BT and GRM baselines is therefore confounded by the optimization schedule. The comparison should be repeated at matched learning rates, ideally with a small LR sweep for both the baseline and BNRM.
[Sec. 4.4 / Sec. 5.4 / Appendix C.1] The paper attributes reward debiasing to sparsity in the local and global latent variables, but no controlled ablation isolates this mechanism. With eta=1e-5, the KL terms in Eq. (9) are numerically negligible, and the only explicit sparsity mechanism in Algorithm 1 is the ReLU activation on lambda (lines 9-11) and on the global weights (line 16). The post hoc factor analysis in Figure 6 and Table 8 is equally consistent with any trained linear head that down-weights unhelpful features. Please provide ablations that remove non-negativity, disable the KL term, or replace the sparse non-negative head with a dense non-negative linear head, and report sparsity levels actually achieved.
[Tables 1-3 / Figure 4] All headline results are reported as single runs without error bars, seeds, or significance tests. This is especially concerning in Table 3, where the BNRM policy improves TriviaQA by +39.35 on Llama3.1 but drops HellaSwag by -11.83 on OpenRLHF, suggesting high variance. Without repeated runs, the claim that BNRM 'substantially mitigates reward over-optimization' is not statistically supported. At minimum, standard deviations over three or more seeds are needed for the main comparisons.
[Sec. 5.3 / Appendix D] The BoN test uses reward-model-Mistral-7B-instruct-Unified-Feedback as the gold reward model. This gold model is trained on the same Unified-Feedback data family as the proxy models, so the BoN comparison is not fully independent evidence that BNRM tracks true human preferences. The external benchmarks (RewardBench, RM-Bench, HHH, MT-Bench) are stronger evidence. Please either add human evaluation on a small sample or explicitly discuss this limitation when drawing the reward-hacking conclusion.

minor comments (6)

[Sec. 4.4] The section title contains 'BNRA' instead of 'BNRM'.
[Eq. (9)] The optimization variables are listed as W_llm, W_vi, Phi, but q(Phi) is parameterized by its own variational parameters W_Phi. Please make the notation consistent.
[Algorithm 1 / Eq. (8)] The main text uses K latent factors, while Algorithm 1 fixes the dimension to 1024. Please state K=1024 explicitly or use a generic K throughout.
[Table 1] Several entries are malformed, e.g. '42.1↑‘0.0' and '95.6↓0.2', and the table alignment is erratic. Please proofread the tables.
[Appendix A.3 / Figure 7] The figure caption and text use lambda for what is elsewhere called eta. Please unify the symbol for the KL coefficient.
[Table 8] The text says factor selection uses top-k with k=20, but only three factors are shown. Please provide the complete set or state the selection criterion.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: central claims are evaluated on external benchmarks and independent gold reward models; self-citations are not load-bearing.

full rationale

The paper's derivation chain is not circular. BNRM is trained by maximizing an ELBO (Eq. 9) on preference data, and its central claims—robustness to over-optimization, debiasing, OOD generalization—are evaluated on external benchmarks (RM-Bench, RewardBench, HHH, MT-Bench) and on PPO fine-tuning benchmarks, not on quantities reconstructed from the training objective. The Best-of-N test uses a separately trained reward-model-Mistral-7B-instruct-Unified-Feedback as a gold approximator rather than BNRM's own scores, so the 'remains aligned with gold reward' result is not forced by construction. I found no equation or fitted parameter that is renamed as a prediction. The paper does contain self-citations (e.g., Hu et al. 2025, Zhou et al. 2012, Zhang et al. 2020), but these support standard NFA/Weibull inference machinery and are not load-bearing for the debiasing claim. Section 5.4's 'Error Rectification' analysis is a post-hoc decomposition of the fitted model, and the causal attribution of the gains to sparsity/non-negativity is under-identified (the BNRM runs use higher learning rates than baselines in Table 6, and η=1e-5 makes the KL term nearly inert); however, these are validity/identification concerns, not circularity of the type where an output is equivalent to an input by construction.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the assumption that a sparse, non-negative factor representation isolates causal preference signals without explicit debiasing supervision. The free parameters are mostly standard hyperparameters, but the KL weight and learning-rate choices play a large role in the reported gains. No new physical or algorithmic entities beyond the latent factor variables are introduced.

free parameters (3)

K (number of latent factors) = 1024
Set manually and used throughout Algorithm 1; no sensitivity analysis is provided.
eta (KL divergence weight) = 1e-5
Tuned on validation performance (Figure 7), and affects the balance between preference fit and sparsity.
Learning rate for BNRM = 5e-5
Chosen relative to baseline lr of 1e-5 (Table 6); the higher lr is a confound in comparisons.

assumptions (4)

domain assumption Bradley-Terry preference likelihood is the correct generative model for human preferences
Assumed in Section 3.1 and used to derive the BNRM objective; standard but unverified.
domain assumption Sparsity and non-negativity of latent factors implicitly debias reward models
Central to the method's claimed mechanism; argued intuitively in Section 4.4 with no formal proof or causal test.
standard math Amortized variational inference with Weibull posteriors is a sufficient approximation
A standard approximation technique (Zhang et al. 2020), but the quality of the approximation is not assessed.
domain assumption OOD benchmarks (RewardBench, HHH, MT-Bench) are faithful proxies for true human preference alignment
Used for evaluation; the paper does not perform human studies, and references a single gold reward model for Best-of-N.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling." pith.science (2026). https://pith.science/paper/66SCQFUP

@misc{pith2026260210623,
  author       = {Pith},
  title        = {Pith review of: Mitigating Reward Hacking in RLHF via Bayesian Non-negative Reward Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/66SCQFUP}},
  note         = {Machine review of arXiv:2602.10623}
}

read the original abstract

Reward models learned from human preferences are central to aligning large language models (LLMs) via reinforcement learning from human feedback, yet they are often vulnerable to reward hacking due to noisy annotations and systematic biases such as response length or style. We propose Bayesian Non-Negative Reward Model (BNRM), a principled reward modeling framework that integrates non-negative factor analysis into Bradley-Terry (BT) preference model. BNRM represents rewards through a sparse, non-negative latent factor generative process that operates at two complementary levels: instance-specific latent variables induce disentangled reward representations, while sparsity over global latent factors acts as an implicit debiasing mechanism that suppresses spurious correlations. Together, this disentanglement-then-debiasing structure enables robust uncertainty-aware reward learning. To scale BNRM to modern LLMs, we develop an amortized variational inference network conditioned on deep model representations, allowing efficient end-to-end training. Extensive empirical results demonstrate that BNRM substantially mitigates reward over-optimization, improves robustness under distribution shifts, and yields more interpretable reward decompositions than strong baselines.

Figures

Figures reproduced from arXiv: 2602.10623 by the authors.

**Figure 2.** Graphical model representations. (a) The standard BT model; (b) our proposed BNRM. Here, x denotes the prompt, y1, y2 are candidate responses, and nodes represent the predictive process for the preference y1 ≻ y2. 3.2. Non-negative Factor Analysis (NFA) Non-negative factor analysis methods, such as Poisson factor analysis (PFA) (Zhou et al., 2012), are widely used as topic models (Blei & Lafferty, 2009). They impo… view at source ↗

**Figure 3.** Variational Inferencer for BNRM. the corresponding local latent variable θi . Following prior work on scalable inference for non-negative latent variable models (Zhang et al., 2020), we adopt a Weibull distribution due to its convenient reparameterization and its ability to model sparse, positive random variables: q(θ | x, y) = Weibull(k,λ), (k,λ) = Activation(zWvi), (8) where Wvi ∈ R dmodel×2K, k,λ ∈ R K + denote … view at source ↗

**Figure 4.** ID and OOD evaluation results for BT-BNRM. (a): performance when training on a varying number of samples. (b): performance under different label-noise ratios. Solid bars denote the BT baseline, hatched bars denote our BT-BNRM. on 1K to 20K UF samples; (2) Label noise: training on 40K samples with noise rates from 0.1 to 0.4. Figure 4a shows that BNRM consistently outperforms BT with the performance gap widening as … view at source ↗

Figures from the paper (6 more)

**Figure 5.** Figure 5: Pearson correlation and mean reward score between response length and reward score on the RM-Bench Hard subset. The top plot shows the correlation between response length and reward score. The x-axis is log-scaled for better visual clarity. The bottom plot reports the …

**Figure 6.** Figure 6: Partial θ and Φ factor activations for the chosen and rejected responses on RewardBench. The left y-axis denotes the activation strength of θ, and the right y-axis denotes the activation strength of Φ, which represents the global weight of each latent factor in θ. Bars…

**Figure 7.** Figure 7: Influence of different values of λ on in-distribution (ID) and out-of-distribution (OOD) performance, where λ controls the strength of the KL-divergence regularization term relative to the BT preference loss. over the BT baseline by 5.8% and 3.4% on the two models, an…

**Figure 8.** Figure 8: Validation accuracy on Unified-Feedback with 40K training examples, fine-tuning Gemma-2B-it with LoRA. Our BNRM consistently achieves higher validation accuracy throughout training and at convergence compared with BT, GRM-DPO, GRM-DPO w/o ref, GRM-SFT, Label-smooth, a…

**Figure 9.** Figure 9: Pearson correlation and mean reward score between response length and reward score on the RM-Bench Hard subset. The top plot shows how the correlation between response length and reward score. The x-axis is log-scaled for better visual clarity. The bottom plot reports …

**Figure 10.** Figure 10: Best-of-N (BoN) performance for (a, b) Gemma-2B-it and (c, d) Gemma-2-2B-it. All rewards are normalized to start at 0. Compared to the BT baseline, BNRM remains better aligned with the gold reward, effectively mitigating reward hacking as KL divergence increases. 17 …

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Multimodal Reward Hacking in Reinforcement Learning
cs.AI 2026-07 conditional novelty 6.5 of 10

Imperfect multimodal RL rewards systematically create new failures (NRFR > RHR); scaling and answer-aware rewards help but do not eliminate hacking, and unreliable visual verifiers actively increase it.

Reference graph

Works this paper leans on

14 extracted references · 2 linked inside Pith · cited by 1 Pith paper

[1]

BT-Base (Bradley & Terry, 1952), a classical ranking- based preference objective

1952
[2]

BT-Margin (Touvron et al., 2023b; Wang et al., 2024) that optimizes a margin-based loss on score differences between chosen and rejected responses

2024
[3]

BT-Frozen that keeps the backbone frozen and only trains a lightweight reward head with the BT objective
[4]

BT-Ensemble (Coste et al., 2024) that trains three BT- Based reward models with an L2-regularized loss under different random seeds and averages their values as the final rewards

2024
[5]

BT-Label Smoothing (Wang et al., 2024) that penalizes overly sharp preference probabilities in the BT loss to reduce overfitting

2024
[6]

3https://github.com/vllm-project/vllm

GRM (Yang et al., 2024b) that jointly optimizes the lan- guage model head and the reward head to enhance gen- eralization under distribution shifts. 3https://github.com/vllm-project/vllm
[7]

good” or “bad

InfoRM (Miao et al., 2024) that designs to mitigate re- ward hacking from the perspective of mutual informa- tion. D. Best-of-N (BoN) Test Figure 10 presents the BoN results on the Gemma-2B and Gemma-2-2B Instruct models, where we adopted reward- model-Mistral-7B-instruct-Unified-Feedback (Yang et al., 2024b) as our gold reward model to approximate true h...

2024
[10]

Core concept: what human-understandable concept does this latent factor primarily capture (for example, code correctness, refusal to answer, toxic or aggressive language, politeness, step-by-step reasoning, verbosity, etc.)?

Show all 14 references

[11]

Justify your judgement using the high-activation examples above

Functional role / polarity: given that reward= P k θkΦk withθ k ≥0andΦ k ≥0, infer whether higher values of this factor are more characteristic of (i) high-quality / desirable behaviour, (ii) low-quality / undesirable behaviour, or (iii) a mixed or ambiguous pattern. Justify y...
[12]

Behavioural description: in 3–5 sentences, describe what kinds of behaviours, contents, or styles this factor is most sensitive to, and what makes the high-activation responses similar from the perspective of this factor
[13]

FactorName

Pattern summary: list 3–6 bullet points summarizing recurring patterns across the samples (tone, safety behaviour, helpfulness, reasoning style, level of detail, formatting, etc.). Required output format The model must respond inJSONwith the following fields: "FactorName"(2–5 ...
[14]

Sorry, but I can’t assist with that

Compared to the BT baseline, BNRM remains better aligned with the gold reward, effectively mitigating reward hacking as KL divergence increases. 17 BNRM: Non-negative Bayesian Reward Modeling Table 8.Case study of three latent factors with representative high-activation sample...
[281]

emnlp-main.281/

URLhttps://aclanthology.org/2025. emnlp-main.281/. Lin, Y ., Tan, L., Hao, Y ., Wong, H., Dong, H., Zhang, W., Yang, Y ., and Zhang, T. Spurious feature diversifica- tion improves out-of-distribution generalization.arXiv preprint arXiv:2309.17230, 2023. Liu, C. Y ., Zeng, L., ...

2025 arXiv
[2017]

URLhttps://arxiv.org/abs/1707. 06347. Singhal, P., Goyal, T., Xu, J., and Durrett, G. A long way to go: Investigating length correlations in rlhf.arXiv preprint arXiv:2310.03716, 2023. Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., V oss, C., Radford, A., Amodei, D.,...

2023 arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [1]

BT-Base (Bradley & Terry, 1952), a classical ranking- based preference objective

1952

[2] [2]

BT-Margin (Touvron et al., 2023b; Wang et al., 2024) that optimizes a margin-based loss on score differences between chosen and rejected responses

2024

[3] [3]

BT-Frozen that keeps the backbone frozen and only trains a lightweight reward head with the BT objective

[4] [4]

BT-Ensemble (Coste et al., 2024) that trains three BT- Based reward models with an L2-regularized loss under different random seeds and averages their values as the final rewards

2024

[5] [5]

BT-Label Smoothing (Wang et al., 2024) that penalizes overly sharp preference probabilities in the BT loss to reduce overfitting

2024

[6] [6]

3https://github.com/vllm-project/vllm

GRM (Yang et al., 2024b) that jointly optimizes the lan- guage model head and the reward head to enhance gen- eralization under distribution shifts. 3https://github.com/vllm-project/vllm

[7] [7]

good” or “bad

InfoRM (Miao et al., 2024) that designs to mitigate re- ward hacking from the perspective of mutual informa- tion. D. Best-of-N (BoN) Test Figure 10 presents the BoN results on the Gemma-2B and Gemma-2-2B Instruct models, where we adopted reward- model-Mistral-7B-instruct-Unified-Feedback (Yang et al., 2024b) as our gold reward model to approximate true h...

2024

[8] [10]

Core concept: what human-understandable concept does this latent factor primarily capture (for example, code correctness, refusal to answer, toxic or aggressive language, politeness, step-by-step reasoning, verbosity, etc.)?