Counterfactual Reward Model Training for Bias Mitigation in Multimodal Reinforcement Learning

N Harshit; Sheryl Mathew

REVIEW 5 major objections 4 minor 14 references

Counterfactual Reward Model Training for Bias Mitigation in Multimodal Reinforcement Learning

T0 review · 5 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A counterfactual trust score steers RLHF reward models away from biased signals, the paper argues, cutting spurious correlations while keeping accuracy high at 89.12% on a fake-news task.

desk verdict Plausible composite-trust-score idea undermined by circular scoring, undefined counterfactuals, and missing evaluation details. read the letter →

arxiv 2508.19567 v1 pith:ZBWPNYID submitted 2025-08-27 cs.LG

Sheryl Mathew , N Harshit This is my paper

classification cs.LG

keywords CounterfactualrewardmodelingFairness-awareRLHFMultimodalbiasmitigationCausalinferenceTrustScorePolicyrobustnessSpuriouscorrelationreductionFakenewsdetection

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Fair-RLHF, a reward-model training pipeline that adds counterfactual reasoning to reinforcement learning from human feedback. Its central object is the Counterfactual Trust Score, a per-batch number that penalizes drift, prediction uncertainty, fairness-rule violations, classification error, and reward swings when protected attributes are perturbed. The authors argue that training against this score rather than a plain reward signal makes the model less likely to amplify latent biases in multimodal data, and they report 89.12% accuracy on a fake-vs-true news task with reduced spurious correlations and unfair reward assignments. If this holds, fairness in RLHF can be monitored and enforced online without supervised bias labels.

What carries the argument

The Counterfactual Trust Score (CTS) is the load-bearing mechanism, defined as Trust_t = 1 − (αD_t + βU_t + γR_t + δE_t + ζC_t), where D_t aggregates drift metrics (PSI, JSD, autoencoder reconstruction error), U_t is softmax-margin uncertainty, R_t is a fairness-rule violation rate, E_t is classification error, and C_t is the average absolute reward change when protected attributes are counterfactually perturbed. Around it sit a CatBoost classifier as the reward model, a transformer-autoencoder for drift detection, and synthetic bias injection into later batches to test robustness.

What would settle it

Take the trained reward model and apply the same counterfactual perturbation procedure to a deliberately non-protected attribute, such as headline length. If C_t remains high, the fairness penalty is not specific to protected attributes and the bias-mitigation claim fails. Alternatively, on a held-out set with known biased labels, if the CTS-trained model's fairness violation rate is not lower than a plain CatBoost baseline at matched accuracy, the central claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that reward models for RLHF can be made bias-resilient by training them against a composite Counterfactual Trust Score instead of relying on passive constraints. The score combines four signals—counterfactual shifts that separate political framing from topical content, reconstruction uncertainty from a transformer autoencoder, explicit fairness-rule violations for protected attributes, and temporal reward changes—plus a counterfactual consistency penalty that punishes the reward function whenever flipping a protected attribute changes its output. On a multimodal fake-news dataset with injected bias, the authors report 89.12% accuracy and, more importantly for the

Load-bearing premise

The load-bearing premise is that perturbing protected attributes produces a counterfactual that differs only in that attribute; the paper does not define protected attributes or prove that the perturbation removes confounders, so the consistency penalty could end up measuring ordinary prediction noise rather than bias.

Editorial extensions

If this is right

If the CTS is a valid bias signal, RLHF reward models can be monitored per batch with a single scalar that signals when retraining or intervention is needed.
Because the method does not require bias labels, the same pipeline can be applied to any multimodal stream where protected attributes can be defined.
The tunable weights (α, β, γ, δ, ζ) give practitioners a direct way to control how aggressively to trade accuracy for fairness.
Embedding counterfactual consistency into the reward model should reduce preference-collapse failures where the policy overfits spurious cues.
Decomposing political framing from topical bias could be reused to separate content-based from style-based biases in other domains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The consistency penalty assumes counterfactuals that isolate protected attributes; if that assumption fails, the penalty would shrink whenever the model is generally uncertain, not specifically unfair. A natural test is to compare C_t against randomly perturbed non-protected features.
The CTS could be turned into an online intervention rule: when Trust_t drops below a threshold, switch to a conservative policy or trigger retraining. The authors describe the thresholds as tunable but do not develop the controller.
The 89.12% accuracy is reported on one fake-news dataset; the more durable contribution would be the fusion of drift, fairness, and uncertainty into one reward signal, which the authors themselves name as a direction for dialogue and recommendation systems.
Synthetic bias injected only into batches 6–10 tests sudden shifts; a gradual drift accumulating across all batches would stress the temporal component differently and is not examined here.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First, the punchline: the empirical claims are unsupported, and the central trust score is partly circular. The paper frames a real problem — reward models in RLHF can amplify latent bias, and monitoring should be prospective — and combining drift, uncertainty, fairness violations, and counterfactual consistency into a single score is a reasonable design goal. The synthetic bias injection setup is a sensible idea for stress testing. I want to credit that.

The soft spots, though, are load-bearing. Eq. 12 defines Trust_t as 1 minus a weighted sum that includes E_t, the classification error. So any reported relationship between trust and error is partly mechanical; it is not independent evidence that the score tracks bias. The counterfactual term C_t (Eq. 13) is not causally grounded. Algorithm 1 says 'perturbing protected attributes' but never says which attributes are protected, how they are identified, or what the perturbation process is. Without a causal graph or an explicit intervention, |f(x) − f(x_cf)| is just sensitivity to an arbitrary edit; it does not isolate fairness. This undermines the bias-mitigation claim.

There is also an internal inconsistency: Eq. 6 gives Trust_t without the ζC_t term, while Eq. 12 includes it. That is not a minor flourish; it is the central formula of the paper, and the reader cannot tell which version was used. The evaluation does not help: the dataset is unnamed, no baselines are provided, there are no error bars, and the 89.12% accuracy is reported without comparison. The references contain placeholder DOIs (e.g., 10.5555/12345678 and arXiv:2501.12345), which further damages credibility.

Who is this for? A reader interested in composite trust metrics might find the taxonomy useful, but as evidence of a working method, it does not hold up. I would not cite it in the next 12 months. If this lands on my desk, I would desk reject it. The right path for the authors is to rewrite with a named dataset, real baselines, a causally defined counterfactual, and a version of the trust score that does not contain its own outcome. Until then, it is not a research contribution in its current form.

Referee Report

5 major / 4 minor

Summary. The manuscript proposes a fairness-aware reward-modeling pipeline for multimodal RLHF, centered on a Counterfactual Trust Score (CTS) that combines drift statistics, prediction uncertainty, fairness-rule violations, classification error, and a counterfactual consistency penalty. The reward model is a CatBoost classifier trained on a multimodal fake/true news dataset with injected synthetic bias; the authors report 89.12% accuracy and claim reduced spurious correlations and unfair reinforcement signals. The paper includes an algorithm sketch, qualitative comparison tables, and discussion, but no quantitative experimental evaluation with baselines, error bars, or external fairness metrics.

Significance. The problem—reward models in RLHF amplifying latent biases—is important, and the idea of composing drift, uncertainty, and counterfactual consistency into a single trust score is a plausible design direction. However, the manuscript does not deliver the evidence needed to evaluate the proposal: the counterfactual consistency term, which is the claimed fairness mechanism, is not causally defined; the trust-score equations are internally inconsistent; and the evaluation section contains no quantitative results. If the causal grounding and empirical validation were supplied, the high-level monitoring framework could be useful, but in its current form the central claims are not established.

major comments (5)

[Section 3.5.1, Eq. (13), Algorithm 1 line 10] The counterfactual consistency penalty C_t is not causally grounded. Algorithm 1 line 10 only says 'generate counterfactuals x_cf by perturbing protected attributes [3]' but never identifies which attributes are protected in the fake-news dataset, what causal graph or identification assumptions justify the perturbation, or how the perturbation holds all other content fixed. Without this, C_t = E[|f(x)-f(x_cf)|] measures sensitivity to an arbitrary edit rather than bias. Since C_t is the only fairness-specific term in the trust score, the central bias-mitigation claim is unsupported. The injected synthetic biases (subject distribution, framing, temporal drift) are also not linked to the protected-attribute perturbation, so there is no evidence that C_t responds to the biases the paper claims to mitigate.
[Section 3.5.1, Eqs. (6), (7), (10), (12), Algorithm 1 lines 22/24] The trust score is defined inconsistently. Eq. (6) defines Trust_t = 1 - (αD_t + βU_t + γR_t + δE_t) with α+β+γ+δ=1, while Eq. (12) appends +ζC_t with no renormalization, so the weights no longer sum to 1 and the score is not uniquely specified. In addition, Eq. (7) defines U_t as the margin p_max - p_second-max, but Eq. (10) and Algorithm 1 line 22 define uncertainty as the complement 1 - (p_max - p_second-max), changing the sign of the uncertainty penalty. These are not merely cosmetic issues; they alter the behavior of the proposed method.
[Section 3.5.1, Eq. (12), Section 5, Fig. 1d] The paper claims to demonstrate a relationship between trust proxies and error rate, but classification error E_t is itself a component of Trust_t in Eq. (12). Any reported correlation between Trust_t (or its components) and E_t is therefore partly self-fulfilling. The visualizations in Fig. 1 are described qualitatively and do not provide an out-of-sample or ablative evaluation that would separate the predictive contribution of C_t from the mechanical inclusion of E_t.
[Section 3.3 and Algorithm 1] There is no reinforcement learning component in the described pipeline. The 'reward model' is a CatBoost classifier trained on fake/true labels; there is no policy, no preference data, no human-feedback loop, and no RL update. The 89.12% accuracy is a supervised classification result, and the paper provides no evidence that the proposed method improves reward modeling in an actual RLHF setting. Claims about 'unfair reinforcement signals' and 'bias mitigation in RLHF' are therefore outside what the experiments can support.
[Section 6, Tables 1-2, Abstract] The central empirical claim—89.12% accuracy, outperforming baseline reward models—is not backed by any quantitative comparison. No baseline accuracy, standard deviation, dataset size, class-imbalance ratio, batch size, or hyperparameter settings (α, β, γ, δ, ζ, η, λ, temperature) are reported. Tables 1 and 2 are qualitative self-assessments ('Very High', 'Full mitigation') rather than measurements. External fairness metrics (e.g., demographic parity, equalized odds) are never computed, and the synthetic bias injections are not tied to protected attributes. This is insufficient evidence for either the accuracy claim or the fairness/mitigation claim.

minor comments (4)

[Throughout] Numerous typos and spacing artifacts: 'Counterfactual T rust Score', 'F raming Disturbance', 'T emporal Drift', 'Bi-subject Distribution', and 'components that are aware of the law' (likely 'fairness-aware'). The text needs careful proofreading.
[Section 6] The section titled 'Evaluation Metrics' contains no evaluation metrics or experimental measurements; it is a qualitative comparison table. Either provide actual metrics or rename the section.
[Fig. 1] The figure captions do not state axis labels, units, or the exact quantity plotted. If figures are included, they need to be self-contained so readers can assess what is being shown.
[References] Reference [11] uses a placeholder DOI (10.5555/12345678) and reference [14] uses a placeholder arXiv identifier (2501.12345). These need to be verified or removed.

Circularity Check

1 steps flagged · score 6.0 of 10

Trust score includes classification error by construction, making reported trust–error correlations tautological; counterfactual consistency penalty is under-specified but that is an assumption gap, not circularity.

self definitional [Eq. 6 (Sec. 3.4.2), Eq. 12 (Sec. 3.5.1), Fig. 1d caption (Sec. 5)]
"Trustt = 1 − [αDt + βUt + γRt + δEt] (6) ... Trustt = 1 − (α · Dt + β · ¯ut + γ · Rt + δ · Et + ζ · Ct) (12) ... (d) Correlation Heatmap of key metrics, showing the relationships between our trust proxies and the final error rate."

Since Et (classification error) is one of the additive terms subtracted in the trust score, Trust_t is, by construction, a decreasing function of error. Any correlation or 'relationship' between trust and error in Fig. 1d is therefore not an empirical finding but an algebraic identity. The caption calls these 'trust proxies' and relates them to 'final error rate,' but the trust score itself contains the error rate. Thus the paper's use of this correlation as evidence of the trust score's validity is self-definitional.

full rationale

The paper's central derivation is not circular with respect to its external accuracy claim: the 89.12% fake-news classification accuracy is an independent benchmark, and the drift-detection components (PSI, JSD, transformer-autoencoder loss) are grounded in external references. However, one key evaluation step is circular: the Counterfactual Trust Score is defined to include classification error Et (Eq. 6 and Eq. 12), so any reported inverse relationship between trust and error is forced by the formula rather than discovered. The figure caption explicitly frames the correlation heatmap as showing relationships between trust proxies and the final error rate, which is tautological. The counterfactual consistency penalty Ct, while central to the fairness claim, is not causally grounded because the paper never specifies protected attributes or the perturbation model; this is a serious correctness gap but not a circular reduction. There is also an internal inconsistency: Eq. 6 omits ζCt while Eq. 12 includes it, meaning the trust score is not uniquely defined. One self-citation ([12], a co-author's prior work) is present for the transformer-autoencoder, but it is not load-bearing because the same loss is also supported by external references [9,10]. Overall, the circularity is partial: it affects the validation of the trust score's relationship to error, while the broader framework retains independent content.

Assumptions & free parameters 5 free parameters · 5 assumptions · 2 invented entities

The central claim rests on a composite score whose weights and several hyperparameters are not specified, and on assumptions that the unnamed dataset, the drift proxies, and the counterfactual perturbation mechanism are valid. These are all domain assumptions without independent evidence in the paper.

free parameters (5)

alpha, beta, gamma, delta, zeta = unspecified
Weights in the trust score (Eq. 12). Called tuneable, no values or search procedure given.
eta = unspecified
Variance penalty coefficient in the TAE loss (Eq. 9).
lambda = unspecified
EMA smoothing factor in Algorithm 1, line 27.
temperature scaling parameter = unspecified
Applied to CatBoost outputs (Algorithm 1, line 14), no value reported.
bias injection rates = unspecified
Rates for subject distribution increase, framing swaps, and label drift in the last batches (Section 3.2).

assumptions (5)

domain assumption The unnamed multimodal fake vs true news dataset exhibits framing bias, class imbalance, and temporal drift.
Section 3.1 states this but the dataset is not identified, so the claim cannot be checked.
domain assumption PSI, JSD, and autoencoder reconstruction error are valid unsupervised proxies for drift and bias.
Equations 4, 5, and 2 are used as D_t without independent validation in this paper; citations [1,2] are invoked.
domain assumption Perturbing protected attributes yields causally valid counterfactuals.
Eq. 13 and Algorithm 1 line 10 assume this, but no causal graph or identification strategy is provided.
domain assumption The softmax margin (Eq. 7) is a reliable uncertainty measure for this classifier.
Standard in deep learning but not calibrated or validated here.
standard math Standard definitions of PSI, JSD, attention, and autoencoder loss are accepted.
Equations 3-9 use textbook definitions; these are not in question.

invented entities (2)

Counterfactual Trust Score (CTS)
purpose: Aggregated reward-model reliability signal for RLHF.
Proposed in Eq. 6 and Eq. 12; no external benchmark or falsifiable prediction outside this paper.
Counterfactual consistency penalty C_t
purpose: Penalize reward changes when protected attributes are flipped.
Defined in Eq. 13; no evidence that reducing this quantity improves fairness.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Counterfactual Reward Model Training for Bias Mitigation in Multimodal Reinforcement Learning." pith.science (2026). https://pith.science/paper/ZBWPNYID

@misc{pith2026250819567,
  author       = {Pith},
  title        = {Pith review of: Counterfactual Reward Model Training for Bias Mitigation in Multimodal Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZBWPNYID}},
  note         = {Machine review of arXiv:2508.19567}
}

read the original abstract

In reinforcement learning with human feedback (RLHF), reward models can efficiently learn and amplify latent biases within multimodal datasets, which can lead to imperfect policy optimization through flawed reward signals and decreased fairness. Bias mitigation studies have often applied passive constraints, which can fail under causal confounding. Here, we present a counterfactual reward model that introduces causal inference with multimodal representation learning to provide an unsupervised, bias-resilient reward signal. The heart of our contribution is the Counterfactual Trust Score, an aggregated score consisting of four components: (1) counterfactual shifts that decompose political framing bias from topical bias; (2) reconstruction uncertainty during counterfactual perturbations; (3) demonstrable violations of fairness rules for each protected attribute; and (4) temporal reward shifts aligned with dynamic trust measures. We evaluated the framework on a multimodal fake versus true news dataset, which exhibits framing bias, class imbalance, and distributional drift. Following methodologies similar to unsupervised drift detection from representation-based distances [1] and temporal robustness benchmarking in language models [2], we also inject synthetic bias across sequential batches to test robustness. The resulting system achieved an accuracy of 89.12% in fake news detection, outperforming the baseline reward models. More importantly, it reduced spurious correlations and unfair reinforcement signals. This pipeline outlines a robust and interpretable approach to fairness-aware RLHF, offering tunable bias reduction thresholds and increasing reliability in dynamic real-time policy making.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

14 extracted references · 7 canonical work pages

[3]

Pinpoint Counterfactuals: Reducing social bias in foundation models via localized counterfactual generation

Sirotkin, K., Escudero-Vi˜ nolo, M., Carballeira, P., Maniparambil, M., Barata, C., O’Connor, N. E.: Pinpoint Counterfactuals: Reducing Social Bias in Foundation Models via Localized Counterfactual Generation . arXiv preprint, https://doi.org/10.48550/arXiv.2412.09160 (2024). 10

work page Pith review arXiv doi:10.48550/arxiv.2412.09160 2024
[1]

Unsupervised Concept Drift Detection from Deep Learning Representations in Real-time

Greco, S., Vacchetti, B., Apiletti, D., Cerquitelli, T.: Unsupervised Concept Drift Detection from Deep Learning Representations in Real Time. arXiv:2406.17813 (2024)

work page Pith review arXiv 2024
[2]

A., Benajiba, Y., Ballesteros, M.: Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Views

Margatina, K., Wang, S., Vyas, Y., John, N. A., Benajiba, Y., Ballesteros, M.: Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Views . Proceedings of the 17th Conference of the Euro- pean Chapter of the Association for Computational Linguistics, pages 2881–2898, https://doi.org/10.48550/arXiv.2303.11309 (2023)

arXiv doi:10.48550/arxiv.2303.11309 2023
[4]

J.: On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization

Xiao, J., Li, Z., Xie, X., Getzen, E., Fang, C., Su, W. J.: On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization. arXiv preprint, https://doi.org/10.48550/arXiv.2405.16455 (2024)

arXiv doi:10.48550/arxiv.2405.16455 2024
[5]

ACL Long Paper (2025)

Ouyang, S., Hu, Y., Chen, G., Li, Q., Zhang, F., Liu, Y.: Towards Reward Fair- ness in RLHF: From a Resource Allocation Perspective. ACL Long Paper (2025). https://aclanthology.org/2025.acl-long.163/

work page 2025
[6]

arXiv preprint (2024)

Banerjee, D., Gopalan, A.: Towards Reliable Alignment: Uncertainty-aware RLHF. arXiv preprint (2024). https://doi.org/10.48550/arXiv.2405.14758

arXiv doi:10.48550/arxiv.2405.14758 2024
[7]

arXiv preprint (2024)

Wang, J., Pyatkin, V., Morrison, J., et al.: Rewardbench: Evalu- ating Reward Models for Language Modeling . arXiv preprint (2024). https://doi.org/10.48550/arXiv.2403.13787

arXiv doi:10.48550/arxiv.2403.13787 2024
[8]

arXiv preprint (2023)

Dai, J., et al.: Safe Reinforcement Learning from Human Feedback. arXiv preprint (2023). https://doi.org/10.48550/arXiv.2310.12773

arXiv doi:10.48550/arxiv.2310.12773 2023

Show all 14 references

[9]

IJCAI (2024)

Lee, J., Nguyen, T., Roberts, K.: Explainable Drift Analysis in Multimodal Fake News Detection. IJCAI (2024). https://doi.org/10.24963/ijcai.2024/250

2024 doi
[10]

F AccT (2024)

Zhou, Y., Patel, S., Thompson, M.: Temporal Drift Detection in Multimodal Systems via Representation Learning . F AccT (2024). https://doi.org/10.1145/3689764.3693125

2024
[11]

NeurIPS Workshops (2024)

Kumar, A., Li, H., Gonzalez, J.: Evolving Bias in RLHF: Detection and Mitigation with Transformer-Based Representations. NeurIPS Workshops (2024). https://doi.org/10.5555/12345678

2024 doi
[12]

arXiv preprint arXiv:2508.07085 (2025)

Harshit, N., Mounvik, K.: Improving Real-Time Concept Drift Detection using a Hybrid Transformer-Autoencoder Framework. arXiv preprint arXiv:2508.07085 (2025). https://arxiv.org/abs/2508.07085

2025 arXiv
[13]

arXiv preprint (2023)

Amazon Science: Dynamic Benchmarking of Masked Lan- guage Models on Temporal Concept Drift . arXiv preprint (2023). https://www.amazon.science/publications/dynamic-benchmarking-of-masked- language-models-on-temporal-concept-drift

2023
[14]

arXiv preprint (2025)

TRiSM Team: TRiSM for Agentic AI: Trust and Risk Assessment Framework. arXiv preprint (2025). https://arxiv.org/abs/2501.12345 11

2025 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [3]

Pinpoint Counterfactuals: Reducing social bias in foundation models via localized counterfactual generation

Sirotkin, K., Escudero-Vi˜ nolo, M., Carballeira, P., Maniparambil, M., Barata, C., O’Connor, N. E.: Pinpoint Counterfactuals: Reducing Social Bias in Foundation Models via Localized Counterfactual Generation . arXiv preprint, https://doi.org/10.48550/arXiv.2412.09160 (2024). 10

work page Pith review arXiv doi:10.48550/arxiv.2412.09160 2024

[2] [1]

Unsupervised Concept Drift Detection from Deep Learning Representations in Real-time

Greco, S., Vacchetti, B., Apiletti, D., Cerquitelli, T.: Unsupervised Concept Drift Detection from Deep Learning Representations in Real Time. arXiv:2406.17813 (2024)

work page Pith review arXiv 2024

[3] [2]

A., Benajiba, Y., Ballesteros, M.: Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Views

Margatina, K., Wang, S., Vyas, Y., John, N. A., Benajiba, Y., Ballesteros, M.: Dynamic Benchmarking of Masked Language Models on Temporal Concept Drift with Multiple Views . Proceedings of the 17th Conference of the Euro- pean Chapter of the Association for Computational Linguistics, pages 2881–2898, https://doi.org/10.48550/arXiv.2303.11309 (2023)

arXiv doi:10.48550/arxiv.2303.11309 2023

[4] [4]

J.: On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization

Xiao, J., Li, Z., Xie, X., Getzen, E., Fang, C., Su, W. J.: On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization. arXiv preprint, https://doi.org/10.48550/arXiv.2405.16455 (2024)

arXiv doi:10.48550/arxiv.2405.16455 2024

[5] [5]

ACL Long Paper (2025)

Ouyang, S., Hu, Y., Chen, G., Li, Q., Zhang, F., Liu, Y.: Towards Reward Fair- ness in RLHF: From a Resource Allocation Perspective. ACL Long Paper (2025). https://aclanthology.org/2025.acl-long.163/

work page 2025

[6] [6]

arXiv preprint (2024)

Banerjee, D., Gopalan, A.: Towards Reliable Alignment: Uncertainty-aware RLHF. arXiv preprint (2024). https://doi.org/10.48550/arXiv.2405.14758

arXiv doi:10.48550/arxiv.2405.14758 2024

[7] [7]

arXiv preprint (2024)

Wang, J., Pyatkin, V., Morrison, J., et al.: Rewardbench: Evalu- ating Reward Models for Language Modeling . arXiv preprint (2024). https://doi.org/10.48550/arXiv.2403.13787

arXiv doi:10.48550/arxiv.2403.13787 2024

[8] [8]

arXiv preprint (2023)

Dai, J., et al.: Safe Reinforcement Learning from Human Feedback. arXiv preprint (2023). https://doi.org/10.48550/arXiv.2310.12773

arXiv doi:10.48550/arxiv.2310.12773 2023