RM-Distiller: Exploiting Generative LLM for Reward Model Distillation

Chenglong Wang; Fuhai Song; Hailong Cao; Hongli Zhou; Hui Huang; Lvyuan Han; Muyun Yang; Tiejun Zhao; Wei Liu; Wenhao Jiang

REVIEW 3 major objections 5 minor 1 cited by

A reward model distilled from an LLM teacher improves markedly when the teacher's refinement, scoring, and generation abilities all provide supervision, instead of using it as a binary annotator.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 09:18 UTC pith:A7SPZLVF

load-bearing objection A useful distillation recipe with a genuinely new scoring trick, but the paper's central contrastive-refinement mechanism is confounded with data volume and unverified. the 3 major comments →

arxiv 2601.14032 v2 pith:A7SPZLVF submitted 2026-01-20 cs.CL

RM-Distiller: Exploiting Generative LLM for Reward Model Distillation

Hongli Zhou , Hui Huang , Wei Liu , Chenglong Wang , Xingyuan Bu , Lvyuan Han , Fuhai Song , Muyun Yang

show 3 more authors

Wenhao Jiang Hailong Cao Tiejun Zhao

This is my paper

classification cs.CL

keywords reward model distillationLLM-as-a-judgecontrastive refinementmargin-aware regressiongenerative regularizationRLHFBradley-Terry modelpreference learning

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that the usual way of distilling a reward model from a generative LLM—asking the teacher to pick the better of two responses and training with a Bradley-Terry loss—throws away most of what the teacher knows. RM-Distiller instead adds three supervision signals: the teacher minimally edits rejected responses to create tightly matched contrastive pairs, it assigns continuous quality scores that are turned into a margin-aware regression target, and it regularizes the student by having it imitate the teacher's token distribution. The authors report that each signal improves benchmark accuracy on RewardBench and RM-Bench, and that policies trained with PPO, GRPO, and DAPO on the resulting reward model show better alignment and instruction-following. If the claims hold, better reward models can be produced without human labels, with no extra inference cost.

Core claim

On its own terms, the paper's central claim is that a 3B-parameter student reward model can be distilled from a larger LLM teacher more effectively by exploiting three teacher capabilities—refinement, scoring, and generation—than by treating the teacher as a binary annotator. The proposed loss combines a margin-aware regression term that aligns predicted reward differences with teacher score differences, and a generative regularization term that keeps the student's language modeling head close to the teacher's distribution. Across unlabeled and labeled settings, two teachers, and four evaluation benchmarks, the method outperforms existing distillation baselines, and the gain is attributed to

What carries the argument

The work is carried by three supervision mechanisms and a weighted loss. Contrastive Refinement prompts the teacher to diagnose why a rejected response is inferior and then minimally edit it into a chosen response, producing pairs that differ only in the preference-relevant attribute. Margin-Aware Regression replaces the binary ranking target with a squared-error loss between the predicted reward difference and the teacher's score difference, with scores self-calibrated against the rejected response's score. Generative Regularization adds NLL and KL terms that keep the student's LM head aligned with the teacher's generation distribution. The total objective is L_total = L_margin + L_reg with

Load-bearing premise

The whole mechanism for fine-grained contrastive learning rests on the assumption that the teacher's minimal edit produces rejected–chosen pairs that differ only in the attribute that determines preference, an assumption enforced only by prompt wording and a filter that removes unchanged responses, with no measured check on edit distance or semantic divergence.

What would settle it

Take a sample of refined pairs, compute token-level edit distance and semantic similarity (e.g., embedding cosine), and have humans judge whether the only difference is the preference-relevant attribute. Alternatively, replace the teacher's refinement with randomly generated minimal edits of comparable edit distance; if RM performance does not decline, the contrastive-signal explanation is not the source of the gain.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

Reward models distilled this way beat several established distillation methods and some larger open-source RMs on RewardBench and RM-Bench, from only 10k instructions.
Downstream RLHF (PPO, GRPO, DAPO) produces higher AlpacaEval, FollowBench, and CFBench scores when the policy is trained with RM-Distiller rewards than with binary-annotation rewards.
The method remains usable when only 1k instructions are available, and it adapts quickly to a new language (Arabic) with a small sampling of instructions.
Because the student is a small model and the extra supervision is applied during training only, the resulting RM has no additional inference cost over a standard BT-trained model.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The contrastive-refinement mechanism hinges on the teacher actually making minimal edits; since the paper does not measure edit distance or semantic similarity of the refined pairs, some of the reported gains could in principle come from increased training-data volume rather than fine-grained contrastivity. This could be tested by comparing against random minimal edits of the same edit distance.
The margin-aware regression component is independent of the refinement component and could be added to any existing RM training pipeline; its benefit might be separable from the other two modules.
If the mechanism is as effective as claimed, the same three-signal recipe might transfer to other preference-learning objectives beyond RM training, such as direct preference optimization (DPO).
The generative regularization serves as a guard against catastrophic forgetting, which suggests it could be a general recipe for any discriminative fine-tuning of generative backbones, not just reward modeling.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A useful distillation recipe with a genuinely new scoring trick, but the paper's central contrastive-refinement mechanism is confounded with data volume and unverified.

read the letter

Short version: this is a useful empirical recipe for RM distillation, with a genuinely new three-part objective, but the paper oversells its central mechanism. The ablation that supposedly proves Contrastive Refinement matters is confounded by training data volume, and the minimal-edit premise is asserted rather than verified. Still worth a careful referee, not a desk reject.

What's new: the combination of (i) using the teacher to rewrite rejected responses into near-minimal-edit chosen responses, (ii) regressing the student's reward margin onto teacher-assigned score differences with the rejected score as an anchor, and (iii) adding NLL/KL regularization from the teacher's generative distribution is not in prior work. The self-calibrated scoring protocol is the most interesting piece: anchoring the refined-response score to the rejected score and conditioning on the refinement history yields more discriminative signal than zero-shot pairwise scoring (Table 10). The experiment suite is broad: two teachers, three RL algorithms, domain adaptation with 1K instructions, bias benchmarks. Code is released. That is real work.

Soft spots: the stress-test note is correct. Section A.1 says D_final = D_sample ∪ D_refine, so the '+ Contrastive Refinement' row in Table 3 adds both new data and the margin supervision; the gain could be a data-volume effect. To support the claim that minimal-edit contrastive pairs are the active ingredient, the authors need a data-volume-matched baseline and at least some edit-distance or semantic-similarity statistics. The only evidence of minimal editing is Prompt D.2 and one toy example (Figure 3). Second, no table reports variance or seeds, so 'significantly outperforms' is statistically unsupported. Third, hyperparameters alpha, beta, and tau_s appear to be tuned on the evaluation benchmarks themselves (Table 11 and Section B.3), which raises a selection-on-evaluation concern. None of this kills the paper—the main gains are large and consistent—but it does mean the mechanism explanation is not yet established.

Who this is for: practitioners building reward models from API teachers, and RLHF teams who want cheap domain adaptation. It deserves a serious referee and could be publishable after revisions that add variance, a volume-controlled ablation, and some edit-distance verification.

Referee Report

3 major / 5 minor

Summary. The paper proposes RM-Distiller, a framework for distilling a reward model from a generative LLM teacher using three auxiliary supervision signals: (1) contrastive refinement, where the teacher is prompted to minimally edit a rejected response into a preferred response to create highly correlated preference pairs; (2) margin-aware regression, which fits the student's reward difference to the teacher's scalar score margin; and (3) generative regularization, which supervises the student's LM head with NLL/KL divergence to the teacher's output distribution. The student is a 3B Qwen2.5-Instruct with an added reward head. Experiments on RewardBench, RM-Bench, RLHF (PPO/GRPO/DAPO), domain adaptation (Arabic), bias/generalization benchmarks, and data efficiency report consistent improvements over several distillation baselines for both GPT-4o and Qwen3-14B teachers. The paper claims this is the first systematic study of RM distillation from generative LLMs.

Significance. If the results hold, RM-Distiller offers a practical, inference-cost-free way to improve reward model quality by extracting richer supervision from a generative teacher. The experimental design is broad: it covers two teacher types (closed/open), labeled and unlabeled settings, six baselines, three RL algorithms, and auxiliary benchmarks for bias, adaptation, and data efficiency. The three loss terms are clearly specified and the code is promised. However, the evidence for the central mechanistic claim is currently incomplete: the component ablation is confounded with data volume, the 'minimal edit' property is asserted but not measured, and no statistical significance is established. These are load-bearing issues for the abstract's claim that exploiting multifaceted teacher capabilities is 'critical.'

major comments (3)

[§4.3 / Appendix A.1] The incremental ablation does not isolate the contribution of Contrastive Refinement. Appendix A.1 states D_final = D_sample ∪ D_refine, so the '+ Contrastive Refinement' row in Table 3 adds D_refine to the training set. The observed gains on RewardBench/RM-Bench could therefore come from the increased data volume or from the extra score-margin supervision, rather than from the minimal-edit contrastive property. Please add a data-volume-matched baseline (e.g., training on an additional sample of D_sample or on teacher-generated non-refined pairs of equal size) and report the results. Without this control, the claim that refinement capability is 'critical' is not supported.
[§3.2, Eq. (3), Prompt D.2, Appendix B.3] The premise that refined chosen responses y_w* and rejected responses y_l 'differ only in the attributes determining preference' is asserted but not verified. The only enforcement is a prompt instruction and a filter τ_e > 0, yet Appendix B.3 sets τ_e = 0, so the filter only removes unchanged responses. No edit-distance distribution, semantic-similarity metric, or human/LLM inspection of refined pairs is reported. Please quantify edit distances of accepted pairs and, if possible, manually/automatically verify that refinements are minimal edits rather than wholesale rewrites. Otherwise the mechanism explanation (fine-grained contrast vs. data augmentation) remains speculative.
[§4.2–§4.4, Tables 1–3] The paper uses 'significantly outperforms' throughout, but no table reports standard deviations, confidence intervals, or significance tests. Some differences are small (e.g., Table 1, Qwen3-14B labeled scene: RM-Bench avg 66.4 vs. 66.0 for Margin BT). Since the results appear to be from single runs, the central comparative claim is not statistically verified. Please report at least 3 seeds, bootstrap confidence intervals, or significance tests for the main tables and the RLHF results.

minor comments (5)

[Tables 1 and 3] Several values are run together with missing separators (e.g., '70.391.9', '91.962.3'). Please fix the formatting so each cell is clearly delimited.
[Appendix B.3 vs §3.3] Appendix B.3 states τ_e = 0, which conflicts with the §3.3 claim that the edit-distance filter 'ensures that the refinement has introduced meaningful changes.' Clarify what threshold is used and what exactly τ_e > 0 vs. τ_e = 0 implies.
[Figure 3] The numeric example is confusing: the rejected response says '9.11 is bigger than 9.9', while the refined chosen response says '9.11 is smaller than 9.9'. This appears to be an intentional correction of a factual error, but the presentation makes it look like a formatting inconsistency. Consider clarifying the example.
[§4.5] The phrase 'filtered by GPT-5' in the Arabic evaluation setup needs more detail: which model version, what prompt, and how was filtering validated? Alternatively, replace with a less vague description.
[Abstract / §1] The claim 'first systematic research on RM distillation from generative LLMs' is strong given the existing literature (SynRM, CLoud, RMBoost, SteerLM). Consider softening or explicitly differentiating the contribution from these prior works.

Circularity Check

0 steps flagged

No significant circularity: the student is trained on teacher-generated signals and evaluated on external benchmarks, so the central claim is not equivalent to its inputs.

full rationale

RM-Distiller's derivation chain is a supervised distillation pipeline rather than a first-principles prediction. Eq. (2) directly regresses the student's reward difference onto the teacher's score difference, but this is the declared training objective, not a hidden 'prediction' of an already-fitted quantity; the teacher scores are training targets, and the paper's claims of improved RM quality are tested on external benchmarks (RewardBench, RM-Bench, EvalBiasBench, IFBench, and downstream RLHF evaluations) whose labels are independent of the teacher's distillation labels. The Contrastive Refinement step constructs training pairs from teacher refinements, and the paper's minimal-edit claim is enforced only by prompt and a tau_e > 0 filter; if that mechanism is unverified it weakens the causal explanation, but it does not make the evaluation circular. Self-citations (e.g., [Huang et al., 2025], [Zhou et al., 2024]) are used only as background motivation and are not load-bearing for the central result. No equation reduces to its own input or renames a fitted value as a prediction; the principal limitation is a potential confound between added data volume and the contrastive mechanism, which is an empirical validity concern, not circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The method assumes teacher LLM judgments are a proxy for human preference, that prompted minimal editing produces attribute-isolated pairs, and that token-level regularization preserves linguistic competence; none of these are verified with independent measurements. No new entities are introduced.

free parameters (3)

alpha (NLL regularization weight) = 0.2
Chosen by ablation on RewardBench/RM-Bench (Table 11); no held-out validation split is described.
beta (KL regularization weight) = 0.2
Chosen by ablation on RewardBench/RM-Bench (Table 11); no held-out validation split is described.
tau_s (minimum teacher score margin) = 3
Selected by sweeping values 0-4 on RewardBench (Table 12) to maximize average accuracy.

axioms (3)

domain assumption Teacher LLM judgments and scalar scores are a valid proxy for human preference and preference strength.
The entire distillation approach in Section 2.2 and 3.3 relies on teacher scores being meaningful enough to train a reward model that performs well on human-labeled benchmarks.
domain assumption The contrastive-refinement prompt yields minimally edited response pairs that differ only in quality-relevant attributes.
Section 3.2 asserts this as the guarantee of 'highly correlated' pairs, but the only enforcement is a prompt instruction and a filter tau_e>0.
domain assumption NLL/KL regularization on teacher-generated tokens preserves the student's foundational linguistic knowledge and prevents catastrophic forgetting.
Section 3.4 motivates Generative Regularization with this claim; no direct measurement of linguistic competence or forgetting is provided.

pith-pipeline@v1.3.0-alltime-deepseek · 19962 in / 14006 out tokens · 140567 ms · 2026-08-03T09:18:00.842213+00:00 · methodology

0 comments

read the original abstract

Reward models (RMs) play a pivotal role in aligning large language models (LLMs) with human preferences. Due to the difficulty of obtaining high-quality human preference annotations, distilling preferences from generative LLMs has emerged as a standard practice. However, existing approaches predominantly treat teacher models as simple binary annotators, failing to fully exploit the rich knowledge and capabilities for RM distillation. To address this, we propose RM-Distiller, a framework designed to systematically exploit the multifaceted capabilities of teacher LLMs: (1) Refinement capability, which synthesizes highly correlated response pairs to create fine-grained and contrastive signals. (2) Scoring capability, which guides the RM in capturing precise preference strength via a margin-aware optimization objective. (3) Generation capability, which incorporates the teacher's generative distribution to regularize the RM to preserve its fundamental linguistic knowledge. Extensive experiments demonstrate that RM-Distiller significantly outperforms traditional distillation methods both on RM benchmarks and reinforcement learning-based alignment, proving that exploiting multifaceted teacher capabilities is critical for effective reward modeling. To the best of our knowledge, this is the first systematic research on RM distillation from generative LLMs.

Figures

Figures reproduced from arXiv: 2601.14032 by Chenglong Wang, Fuhai Song, Hailong Cao, Hongli Zhou, Hui Huang, Lvyuan Han, Muyun Yang, Tiejun Zhao, Wei Liu, Wenhao Jiang, Xingyuan Bu.

**Figure 2.** Figure 2: The illustration of RM-Distiller. We first initialize preference pairs from diverse candidate models, and then synthesize highly cor [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Figure 3: A concrete example of Contrastive Refinement. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

**Figure 4.** Figure 4: The variation of statistical metrics on ShareGPT using the BT Classifier across different RL algorithms. [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗

**Figure 5.** Figure 5: The variation of statistical metrics on ShareGPT using the RM-Distiller across different RL algorithms. [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization
cs.CL 2026-03 conditional novelty 6.0

JudgeBiasBench measures 12 LLM-judge bias types under a 4-dimension taxonomy, and bias-aware GRPO/InfoNCE training cuts BSR substantially while preserving general evaluation performance.

Reference graph

Works this paper leans on

2 extracted references · cited by 1 Pith paper

[1]

NLL weightα: With GPT-4o as the teacher model, we setβ= 0and evaluate the student’s performance across different values ofα
[2]

KL weightβ: With Qwen3-14B as the teacher model, we set the optimalαfrom the last stage and evaluate the student’s performance across different values ofβ. As shown in Table 11, incorporating Generative Regular- ization helps prevent the model from overfitting to superficial preference patterns and maintains its generalization across di- verse tasks. More...

2025

[1] [1]

NLL weightα: With GPT-4o as the teacher model, we setβ= 0and evaluate the student’s performance across different values ofα

[2] [2]

KL weightβ: With Qwen3-14B as the teacher model, we set the optimalαfrom the last stage and evaluate the student’s performance across different values ofβ. As shown in Table 11, incorporating Generative Regular- ization helps prevent the model from overfitting to superficial preference patterns and maintains its generalization across di- verse tasks. More...

2025