REVIEW 4 major objections 4 minor 27 references
Policy-labeled Preference Learning: Is Preference Enough for RLHF?
T0 review · 4 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Pairwise preferences alone mislabel suboptimal behavior as optimal in stochastic environments; labeling each trajectory with its behavior policy, via regret, fixes the mismatch and improves offline and online RLHF.
desk verdict Solid theoretical extension of CPL to stochastic MDPs, but the experiments validate the model on labels made by the same regret estimator, so the headline claim about human preferences is not yet earned. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is negative regret used as the preference score, $-\mathrm{Reg}^\pi_{\pi^*}(s,a) := -(V^{\pi^*}(s) - Q^\pi(s,a))$, which unlike the optimal advantage $Q^{\pi^*}(s,a)-V^{\pi^*}(s)$ retains the behavior policy $\pi$. Its decomposition into the log-likelihood of the optimal policy minus the sequential forward KL divergence $\bar{D}_{\mathrm{KL}}(\pi\|\pi^*;s,a)$ is what turns regret into a directly optimizable policy objective. The Policy Deviation Theorem (Theorem 3.4) supplies that decomposition, and the resulting contrastive KL regularization, implemented as an $L$-horizon, undiscounted lookahead over rollouts, is the mechanism that aligns the learned policy with preferred trajectories while contrasting it against dispreferred ones. Corollary 3.5, which shows regret is invariant under state-dependent reward transformations, is what keeps the score stable without extra variance-reduction machinery.
What would settle it
Train PPL on an offline dataset from a known stochastic behavior policy in a stochastic MetaWorld task, once with true policy labels and once with deterministic pseudo-labels as in Algorithm 1. The mechanism predicts the true-label version clearly outperforms the pseudo-label version in a heterogeneous or stochastic setting; if the two perform the same, behavior-policy information is not what drives the gains. A second check: setting the lookahead to $L=0$ collapses the objective to CPL, so the claim predicts a performance drop; matching results would show the sequential-KL term is decorative.
Extended reading notes
Core claim
The paper's central claim is that the "optimal advantage" score used by prior work (for instance CPL) silently assumes every trajectory was generated by the optimal policy, and that this causes a likelihood mismatch in stochastic MDPs: differences in outcome quality that actually come from behavior-policy suboptimality are misattributed to environmental randomness. The fix is to score segments by negative regret, $-\mathrm{Reg}^\pi_{\pi^*}(s,a) = \alpha\left(\log\pi^*(a|s) - \bar{D}_{\mathrm{KL}}(\pi\|\pi^*;s,a)\right)$, which folds the behavior policy $\pi$ into the score. With the learned policy substituted for $\pi^*$, this yields a policy-labeled loss whose contrastive KL term pulls the learned policy toward preferred segments' policies and pushes it away from dispreferred ones. The paper further proves (Theorem 3.4) that any policy's soft Q-function differs from the optimal soft Q-function exactly by $\alpha$ times a sequential forward KL divergence, and shows that when all preferred segments come from the optimal policy the objective reduces to CPL, and when the segment policies are the reference policy it reduces to DPO. Empirically, on six MetaWorld tasks across homogeneous and heterogeneous offline datasets, PPL matches or beats CPL and the reward-based P-IQL, and its online variant matches PEBBLE while using about 8.8% of PEBBLE's parameter count.
Load-bearing premise
The load-bearing premise is that the current learned policy can stand in for the true optimal policy inside the regret formula when the training loss is written; the paper asserts this substitution rather than proving it, so the loss is no longer a regret against the true optimum once the learned policy drifts far from optimal.
Editorial extensions
If this is right
- PPL strictly generalizes CPL: when every preferred segment is assumed to come from the optimal policy, the contrastive KL term vanishes and the loss reduces to CPL's objective.
- PPL also contains DPO: if the behavior policies of both segments equal the reference policy, the framework recovers the original DPO formulation, so the regret view unifies the two prior methods.
- Because regret eliminates the state-dependent shaping term by construction, the method needs no explicit advantage-estimation or variance-reduction scheme to stay stable (Corollary 3.5).
- In offline settings the policy label matters most when data is heterogeneous: deterministic pseudo-labels cost little on homogeneous data but produce a large performance gap on mixed-policy datasets, so collecting true policy labels is what buys the improvement.
- Online, where policy labels are free by construction, PPL learns from scratch without a reward model or critic and matches PEBBLE's success rates with roughly 8.8% of PEBBLE's parameters.
Reading between the lines
- The likelihood-mismatch diagnosis suggests a dataset-design rule the paper leaves implicit: preference datasets should record behavior-policy identity, and the contrastive KL term will be most informative when queried segment pairs come from policies of known, differing quality rather than from a single average policy.
- A testable extension is to replace the rollout-based lookahead with a model-based or replay-based estimate of the sequential KL term, which would let PPL apply where the environment is not resettable and only fixed offline segments exist.
- The regret decomposition implies a reweighting interpretation: segments whose behavior policy already matches the learned policy contribute little gradient, which suggests an active-query strategy that preferentially queries segments from policies currently far from the learner.
- If the mechanism is right, a sharp prediction follows: in a stochastic environment with genuinely stochastic behavior policies, PPL trained with true policy labels should strictly dominate its deterministic-pseudo-label variant, and the gap should grow with the stochasticity of the behavior policy.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes policy-labeled preference learning (PPL), a DPO-style method that augments pairwise preference segments with the behavior policy that generated them. The motivating claim is that modeling preferences with negative regret, rather than with the optimal advantage used in CPL, resolves a 'likelihood mismatch' when trajectories come from suboptimal stochastic policies. The theoretical section defines an (α,π*)-equivalence class of rewards, proves a policy deviation theorem (Theorem 3.4) expressing Q^{π*}_* − Q^π_* as a discounted sequential forward KL divergence, and derives a contrastive KL regularizer. Empirically, PPL is evaluated on MetaWorld in offline homogeneous/heterogeneous datasets and in an online setting, reporting large margins over CPL on newly generated datasets, parity on the original CPL datasets, and online performance comparable to PEBBLE with fewer parameters.
Significance. The theoretical core is solid and potentially useful: Lemma 3.2's bijective characterization, Lemma 3.3's contraction argument, and Theorem 3.4's closed-form decomposition are clean and go beyond existing bandit/token-level DPO analyses. If the substitution of the current policy for the optimal one inside the regret model could be justified, PPL would be a principled way to add behavior-policy information to preference learning. However, the reported empirical advantages are conditional on the current experimental design: labels are generated by the same regret score that PPL optimizes, and on the original CPL preference datasets PPL only matches CPL. The significance of the paper therefore rests on an unresolved equivalence between the theoretical regret objective and the practical loss, and on non-circular experiments.
major comments (4)
- [Section 3.1, Eq. (3)–LPPL; Appendix B.1] The central derivation substitutes the current policy πψ for the true α-optimal policy π* without proof. Equation (3) and Eq. (5) state a decomposition of negative regret relative to the true π*; immediately after Eq. (3) the objective LPPL is written with Reg^{π+}_{πψ}(s_t^+, a_t^+) and Reg^{π−}_{πψ}(s_t^-, a_t^-), i.e., the learned policy has silently become the reference policy. Corollary 3.6 only shows that, if πψ is treated as the optimal policy, the MaxEnt objective with negative regret equals a forward-KL minimization; it does not prove that this iterative reparameterization minimizes regret with respect to the true optimal policy or converges to it. This is a load-bearing gap because the entire likelihood-mismatch argument depends on the meaning of 'regret' being preserved during training. Please either prove a fixed-point or stationarity result for the πψ-reparameterized loss, or explicitly reframe PPL as an approximation with an error analysis.
- [Section 4.1 and Table 6 (Appendix E.3)] The empirical validation is partially self-referential. Preference labels are generated by a pretrained SAC critic using 'estimated regret' (Section 4.1), and PPL's loss (Section 3.1) is built from exactly the same regret-based score with behavior-policy labels; CPL is evaluated with advantage-based scores and without policy labels. On the original CPL datasets (Table 6) PPL matches CPL, while the large improvements in Table 2 appear only on the newly generated regret-labeled datasets. As written, the experiments demonstrate that if preferences are generated by the regret model plus known behavior policies, PPL exploits that structure; they do not establish the paper's advertised conclusion that policy labels improve learning from general or human preferences. Add a benchmark with human labels or with labels from a different preference model family (e.g., return-based or advantage-based), keeping PPL's algorithm fixed.
- [Section 3.3, Algorithm 1 lines 4–5; Section 4.3] The pseudo-label rule π(·|s_t) ← δ_{a_t} is correct only for deterministic behavior policies. The offline datasets are generated by stochastic SAC policies, and Appendix E.4 adds Gaussian noise during rollouts, so the pseudo-labels are inconsistent with the true generative process; Section 4.3 indeed shows a substantial drop in heterogeneous settings. This limits the paper's claim that policy labeling is inexpensive in the common unknown-policy offline setting. The authors should either fit a stochastic behavior-policy estimator from the data, with an analysis of the resulting bias, or clearly restrict the offline contribution to settings where true or reliable policy labels are available.
- [Section 3.3, contrastive KL approximation] The implementation replaces the infinite discounted sequential forward KL from Theorem 3.4 with an L-horizon undiscounted sum evaluated by reusing the observed segment as a rollout of π+ and π−. The paper acknowledges this approximation but gives no analysis of the bias introduced when segments are truncated, when the continuation is off-policy for the stated behavior policy, or when L is small. Because this regularizer is the main algorithmic novelty, the comparison between the theoretical objective and the implemented loss needs at least an explicit derivation of the approximation error and, ideally, an offline ablation over L with a discussion of the truncation effects.
minor comments (4)
- [Appendix B.2, Eq. (6)] In the second bracket of Eq. (6), the state-dependent term should be β(s_t^-) and the transition expectation should be over P(·|s_t^-, a_t^-); as printed, the shaped DPO objective is not well-formed.
- [Section 3.2, after Eq. (4)] The sentence 'Theorem 3.6 implies that...' should read 'Corollary 3.6 implies...' because the result is labeled Corollary 3.6.
- [Section 3.3] The text says 'We normalize the contrastive KL regularization to balance their scale,' but the equations show only a 1/L factor; please state the exact normalization used, including whether it also divides by segment length, entropy scale, or another quantity.
- [Section 4.1 and Appendix E.4] For reproducibility, please state the exact regret formula used by the SAC critic at query time, including whether the temperature α in the label generator equals the α used in the PPL loss and whether the critic is the same network used for rollouts or a separately trained one.
Circularity Check
PPL's headline gains come from datasets whose preference labels are generated by the very regret score PPL assumes; on the original CPL datasets PPL only matches CPL.
-
self definitional
[Section 4.1 (Experimental Setup) and Section 3.1, Eq. (3); cf. Appendix E.3, Table 6]
"To generate preference queries without human supervision, we pretrain an SAC model as an oracle that achieves a 100% success rate. Using this pretrained model as a critic, we uniformly sampled segments of length 64 and assigned labels based on estimated regret."
The preference labels are computed from 'estimated regret,' which is exactly the negative-regret score that Eq. (3) defines and LPPL optimizes. Thus the ground-truth preference signal in the main experiments already encodes PPL's central modeling assumption that preferences are regret; the benchmark can show that PPL can fit regret-generated labels, but it cannot independently test the claim that regret plus policy labels models human preferences. The self-referential nature is exposed by Appendix E.3 (Table 6): on the original CPL datasets, PPL is comparable to CPL, while the large margins in Table 2 appear only on the newly generated regret-labeled datasets. The headline empirical claim is therefore conditional on the paper's own regret label generator.
full rationale
The theoretical chain (Lemmas 3.2-3.3, Theorem 3.4, Eq. (2)-(3)) is self-contained: it derives the regret decomposition from MaxEnt optimality and the soft Bellman operator without importing the target result. The practical loss, however, substitutes the learned policy πψ for the true α-optimal policy π* when writing LPPL and Corollary 3.6 (Section 3.1, Appendix B.1-B.3); this is an unproven reparameterization rather than a formal derivation step, and it is a correctness and robustness risk rather than a circular reduction. The main circularity concern is empirical: Section 4.1 generates all preference labels with an SAC critic using 'estimated regret,' the same regret score PPL is built on, so the comparisons in Table 2 are partly self-referential. On the external CPL-provided datasets (Appendix E.3), PPL matches CPL rather than outperforming it, confirming that the advertised gains depend on the regret-based label generator. There is no load-bearing self-citation chain or imported uniqueness theorem. Overall, the mathematical derivation is mostly independent, but the headline experimental support partially reduces to testing PPL on data generated by PPL's own regret model.
Assumptions & free parameters
free parameters (3)
- temperature α =
0.1
- asymmetric regularizer λ =
0.5 offline, 1.0 online
- rollout horizon L for contrastive KL =
not specified; online ablation over {5,10,20}
assumptions (5)
- standard math MaxEnt optimality: the target policy is α-optimal for an implicit reward, and soft value functions satisfy V(s)=E[Q(s,a)-α log π(a|s)].
- domain assumption Human/AI preferences follow a Bradley-Terry model whose score is a sum of negative regrets per timestep.
- domain assumption Each trajectory segment is generated by an identifiable behavior policy π+ or π-, and this label is available or can be pseudo-labeled.
- ad hoc to paper The learned policy πψ can be substituted for the true optimal policy π* in the regret decomposition when constructing LPPL.
- ad hoc to paper The sequential forward KL can be approximated by reusing the observed segments as rollouts and truncating to L steps.
Cite this review
Pith. "Pith review of Policy-labeled Preference Learning: Is Preference Enough for RLHF?." pith.science (2026). https://pith.science/paper/H4326EWV
@misc{pith2026250506273,
author = {Pith},
title = {Pith review of: Policy-labeled Preference Learning: Is Preference Enough for RLHF?},
year = {2026},
howpublished = {\url{https://pith.science/paper/H4326EWV}},
note = {Machine review of arXiv:2505.06273}
}
read the original abstract
To design rewards that align with human goals, Reinforcement Learning from Human Feedback (RLHF) has emerged as a prominent technique for learning reward functions from human preferences and optimizing policies via reinforcement learning algorithms. However, existing RLHF methods often misinterpret trajectories as being generated by an optimal policy, causing inaccurate likelihood estimation and suboptimal learning. Inspired by Direct Preference Optimization framework which directly learns optimal policy without explicit reward, we propose policy-labeled preference learning (PPL), to resolve likelihood mismatch issues by modeling human preferences with regret, which reflects behavior policy information. We also provide a contrastive KL regularization, derived from regret-based principles, to enhance RLHF in sequential decision making. Experiments in high-dimensional continuous control tasks demonstrate PPL's significant improvements in offline RLHF performance and its effectiveness in online settings.
Figures
Figures from the paper (16 more)
Reference graph
Works this paper leans on
-
[1]
L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774,
-
[5]
B., Hatgis-Kessell, S., Booth, S., Niekum, S., Stone, P., and Allievi, A
Knox, W. B., Hatgis-Kessell, S., Booth, S., Niekum, S., Stone, P., and Allievi, A. Models of human prefer- ence for learning reward functions.arXiv preprint arXiv:2206.02231,
-
[6]
Offline reinforce- ment learning with implicit q-learning.arXiv preprint arXiv:2110.06169,
Kostrikov, I., Nair, A., and Levine, S. Offline reinforce- ment learning with implicit q-learning.arXiv preprint arXiv:2110.06169,
-
[8]
Liu, Z., Lu, M., Zhang, S., Liu, B., Guo, H., Yang, Y ., Blanchet, J., and Wang, Z. Provably mitigating overopti- mization in rlhf: Your sft loss is implicitly an adversarial regularizer.arXiv preprint arXiv:2405.16436,
-
[9]
Simpo: Simple preference optimization with a reference-free reward.arXiv preprint arXiv:2405.14734,
Meng, Y ., Xia, M., and Chen, D. Simpo: Simple preference optimization with a reference-free reward.arXiv preprint arXiv:2405.14734,
-
[11]
Asynchronous rlhf: Faster and more efficient off-policy rl for language models
Noukhovitch, M., Huang, S., Xhonneux, S., Hosseini, A., Agarwal, R., and Courville, A. Asynchronous rlhf: Faster and more efficient off-policy rl for language models. arXiv preprint arXiv:2410.18252,
-
[12]
Park, J., Seo, Y ., Shin, J., Lee, H., Abbeel, P., and Lee, K. Surf: Semi-supervised reward learning with data augmen- tation for feedback-efficient preference-based reinforce- ment learning.arXiv preprint arXiv:2203.10050,
-
[15]
Proximal policy optimization algorithms
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,
Show all 27 references
-
[16]
Show, don’t tell: Aligning language models with demonstrated feedback.arXiv preprint arXiv:2406.00888,
Shaikh, O., Lam, M., Hejna, J., Shao, Y ., Bernstein, M., and Yang, D. Show, don’t tell: Aligning language models with demonstrated feedback.arXiv preprint arXiv:2406.00888,
-
[17]
Forward kl regularized preference optimization for aligning diffusion policies.arXiv preprint arXiv:2409.05622,
Shan, Z., Fan, C., Qiu, S., Shi, J., and Bai, C. Forward kl regularized preference optimization for aligning diffusion policies.arXiv preprint arXiv:2409.05622,
-
[19]
Be- yond reverse kl: Generalizing direct preference optimiza- tion with diverse divergence constraints.arXiv preprint arXiv:2309.16240,
Wang, C., Jiang, Y ., Yang, C., Liu, H., and Chen, Y . Be- yond reverse kl: Generalizing direct preference optimiza- tion with diverse divergence constraints.arXiv preprint arXiv:2309.16240,
-
[20]
Xiao, T., Yuan, Y ., Zhu, H., Li, M., and Honavar, V . G. Cal-dpo: Calibrated direct preference optimiza- tion for language model alignment.arXiv preprint arXiv:2412.14516,
-
[21]
Dichotomy of control: Separating what you can control from what you cannot.arXiv preprint arXiv:2210.13435,
Yang, M., Schuurmans, D., Abbeel, P., and Nachum, O. Dichotomy of control: Separating what you can control from what you cannot.arXiv preprint arXiv:2210.13435,
-
[22]
Token-level direct preference optimization.arXiv preprint arXiv:2404.11999,
Zeng, Y ., Liu, G., Ma, W., Yang, N., Zhang, H., and Wang, J. Token-level direct preference optimization.arXiv preprint arXiv:2404.11999,
-
[23]
Dpo meets ppo: Reinforced token optimization for rlhf.arXiv preprint arXiv:2404.18922,
Zhong, H., Feng, G., Xiong, W., Cheng, X., Zhao, L., He, D., Bian, J., and Wang, L. Dpo meets ppo: Reinforced token optimization for rlhf.arXiv preprint arXiv:2404.18922,
-
[25]
= (s,a). Proof. Let ˜Qπ ∗(s,a) =Q π∗ ∗ (s,a)−α P t>0γtEτ∼Pπ h DKL(π(·|st)||π∗(·|st)) s0 =s,a 0 =a i for all (s,a)∈S×A . Then Tπ ∗ ˜Qπ ∗(s,a) =Q π∗ ∗ (s,a)−γE P h α Hπ∗ (·|s′)−H π(·|s′) +Eπ∗[Qπ∗ ∗ (s′,a′)]−E π[ ˜Qπ ∗(s′,a′)] i =Q π∗ ∗ (s,a)−γE P α Hπ∗ (·|s′)−H π(·|s′) +Eπ∗[αlog...
1999
-
[26]
is our primary baseline, where the optimal advantage is defined as the score function: SCPL(πψ;ζ +)−S CPL(πψ;ζ−) = X t≥0 logπψ(a+ t|s+ t ) πψ(a− t|s− t ). The objective is to minimize the following loss function: LCPL(πψ;D) =−E (ζ+,ζ−)∼D h logσ SCPL(πψ;ζ +)−S CPL(πψ;ζ−) i A ke...
2023
-
[1999]
Reward model learning vs
Nika, A., Mandal, D., Kamalaruban, P., Tzannetos, G., Radanovi´c, G., and Singla, A. Reward model learning vs. direct policy optimization: A comparative analysis of learning from human preferences.arXiv preprint arXiv:2403.01857,
-
[2010]
M., Stiennon, N., Wu, J., Brown, T
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences.arXiv preprint arXiv:1909.08593,
1909 arXiv
-
[2015]
High-dimensional continuous control using generalized advantage estimation.arXiv preprint arXiv:1506.02438,
Schulman, J., Moritz, P., Levine, S., Jordan, M., and Abbeel, P. High-dimensional continuous control using generalized advantage estimation.arXiv preprint arXiv:1506.02438,
-
[2017]
Quantifying differences in reward functions.arXiv preprint arXiv:2006.13900,
Gleave, A., Dennis, M., Legg, S., Russell, S., and Leike, J. Quantifying differences in reward functions.arXiv preprint arXiv:2006.13900,
2006 arXiv
-
[2018]
While following this data generation procedure, we found a step in the reference code where transitions following a success signal were explicitly truncated
checkpoints, which achieved an approximate 50% success rate. While following this data generation procedure, we found a step in the reference code where transitions following a success signal were explicitly truncated. This truncation was intended to prevent segments from bein...
2023
-
[2020]
Llama 2: Open foundation and fine- tuned chat models.arXiv preprint arXiv:2307.09288,
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y ., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models.arXiv preprint arXiv:2307.09288,
-
[2021]
Pebble: Feedback- efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training.arXiv preprint arXiv:2106.05091,
Lee, K., Smith, L., and Abbeel, P. Pebble: Feedback- efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training.arXiv preprint arXiv:2106.05091,
-
[2022]
From r to q∗: Your language model is secretly a q-function.arXiv preprint arXiv:2404.12358, 2024a
Rafailov, R., Hejna, J., Park, R., and Finn, C. From r to q∗: Your language model is secretly a q-function.arXiv preprint arXiv:2404.12358, 2024a. Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Er- mon, S., and Finn, C. Direct preference optimization: Your language mo...
-
[2023]
Training a helpful and harmless assistant with rein- forcement learning from human feedback.arXiv preprint arXiv:2204.05862,
Bai, Y ., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with rein- forcement learning from human feedback.arXiv preprint arXiv:2204.05862,
-
[2024]
B., and Sadigh, D
Hejna, J., Rafailov, R., Sikchi, H., Finn, C., Niekum, S., Knox, W. B., and Sadigh, D. Contrastive prefence learn- ing: Learning from human feedback without rl.arXiv preprint arXiv:2310.13639,
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.