{"id":"1ae0fbbb-b842-41b7-9bf7-61ee420bf537","arxiv_id":"2505.06273","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Policy-labeled Preference Learning models preferences with regret and behavior-policy labels, adds contrastive KL regularization, and reports improved RLHF performance on MetaWorld offline and online control tasks.","lead":"This paper proposes policy-labeled preference learning (PPL), an RLHF method that uses the behavior policy behind each trajectory plus a regret-based score to avoid mistaking every trajectory for the optimal policy. A smart generalist should read it because it addresses a known blind spot in Direct Preference Optimization for non-deterministic environments, with theory and control experiments showing gains over prior DPO variants.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PPL's headline gains come from datasets whose preference labels were generated by the same regret estimator used in PPL; on the original CPL datasets PPL only matches CPL, so the claim that policy labels help for general human preferences is not established.","rationale":"The reader's weakest_assumption focuses on the unproven substitution of the trainable policy for the true optimal policy in the regret decomposition. That is a legitimate gap in exposition: Eq. (3) is stated for the true alpha-optimal policy, and the loss is written with the current policy in its place. However, this can be read as a parametric family of regret-based preference models indexed by the target policy, analogous to DPO's reward parameterization, so the substitution may be salvageable with a consistency and identifiability argument. The more direct weakness is the alignment between the label generator and the proposed loss. The paper's own Table 6 shows near-parity with CPL on the original CPL datasets, while the strong gains in Table 2 come from datasets whose labels were generated by the regret estimate used in PPL's score. This pattern suggests the improvement may reflect label/objective alignment rather than a general benefit of including behavior-policy information. A non-regret label condition is therefore the decisive check. Since the reader already assigned CONDITIONAL and noted synthetic labels in the rationale, the verdict does not need to change, but the stress-test highlights a different load-bearing mechanism than the one named in the weakest_assumption field.","tokens_in":21491,"tokens_out":27862,"duration_ms":279273,"concrete_test":"Take the heterogeneous offline datasets from Section 4.2 and regenerate preference labels using the original CPL/PEBBLE protocol, which scores segments by partial sums of the SAC reward rather than by regret, while retaining the same behavior-policy labels and hyperparameters. Train PPL and CPL on this shared data. If PPL's heterogeneous-dataset margin over CPL disappears or drops to the level seen in Table 6, the conclusion that policy-labeled regret modeling resolves likelihood mismatch for preferences in general is not supported by the current experiments.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing concern is that the experimental support for the central claim is generated by the same model the paper proposes. In Section 4.1, preference queries are labeled by a pretrained SAC critic using 'estimated regret', and PPL's loss (Section 3.1, Eq. 3) is exactly a regret-based score with behavior-policy labels. PPL is therefore evaluated on data whose generative process matches its own likelihood; CPL is not given the policy labels and uses an advantage-based score. Consistent with this, on the original CPL datasets (Table 6, Appendix E.3) PPL matches CPL rather than clearly beating it, while the large margins in Table 2 appear only on the newly generated regret-labeled datasets. Since the abstract and introduction frame the contribution as 'modeling human preferences with regret', the reported gains do not yet establish that policy labels help for general or human preferences; they establish only that if preferences are exactly regret, then using the regret model plus the true behavior policy helps. The theoretical analysis in Section 3.2 is coherent as a regret-based preference family, but without a human or at least non-regret preference benchmark, the central claim remains conditional.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes policy-labeled preference learning (PPL), a DPO-style method that augments pairwise preference segments with the behavior policy that generated them. The motivating claim is that modeling preferences with negative regret, rather than with the optimal advantage used in CPL, resolves a 'likelihood mismatch' when trajectories come from suboptimal stochastic policies. The theoretical section defines an (α,π*)-equivalence class of rewards, proves a policy deviation theorem (Theorem 3.4) expressing Q^{π*}_* − Q^π_* as a discounted sequential forward KL divergence, and derives a contrastive KL regularizer. Empirically, PPL is evaluated on MetaWorld in offline homogeneous/heterogeneous datasets and in an online setting, reporting large margins over CPL on newly generated datasets, parity on the original CPL datasets, and online performance comparable to PEBBLE with fewer parameters.","tokens_in":21806,"tokens_out":7904,"duration_ms":78307,"significance":"The theoretical core is solid and potentially useful: Lemma 3.2's bijective characterization, Lemma 3.3's contraction argument, and Theorem 3.4's closed-form decomposition are clean and go beyond existing bandit/token-level DPO analyses. If the substitution of the current policy for the optimal one inside the regret model could be justified, PPL would be a principled way to add behavior-policy information to preference learning. However, the reported empirical advantages are conditional on the current experimental design: labels are generated by the same regret score that PPL optimizes, and on the original CPL preference datasets PPL only matches CPL. The significance of the paper therefore rests on an unresolved equivalence between the theoretical regret objective and the practical loss, and on non-circular experiments.","major_comments":[{"comment":"The central derivation substitutes the current policy πψ for the true α-optimal policy π* without proof. Equation (3) and Eq. (5) state a decomposition of negative regret relative to the true π*; immediately after Eq. (3) the objective LPPL is written with Reg^{π+}_{πψ}(s_t^+, a_t^+) and Reg^{π−}_{πψ}(s_t^-, a_t^-), i.e., the learned policy has silently become the reference policy. Corollary 3.6 only shows that, if πψ is treated as the optimal policy, the MaxEnt objective with negative regret equals a forward-KL minimization; it does not prove that this iterative reparameterization minimizes regret with respect to the true optimal policy or converges to it. This is a load-bearing gap because the entire likelihood-mismatch argument depends on the meaning of 'regret' being preserved during training. Please either prove a fixed-point or stationarity result for the πψ-reparameterized loss, or explicitly reframe PPL as an approximation with an error analysis.","section":"Section 3.1, Eq. (3)–LPPL; Appendix B.1"},{"comment":"The empirical validation is partially self-referential. Preference labels are generated by a pretrained SAC critic using 'estimated regret' (Section 4.1), and PPL's loss (Section 3.1) is built from exactly the same regret-based score with behavior-policy labels; CPL is evaluated with advantage-based scores and without policy labels. On the original CPL datasets (Table 6) PPL matches CPL, while the large improvements in Table 2 appear only on the newly generated regret-labeled datasets. As written, the experiments demonstrate that if preferences are generated by the regret model plus known behavior policies, PPL exploits that structure; they do not establish the paper's advertised conclusion that policy labels improve learning from general or human preferences. Add a benchmark with human labels or with labels from a different preference model family (e.g., return-based or advantage-based), keeping PPL's algorithm fixed.","section":"Section 4.1 and Table 6 (Appendix E.3)"},{"comment":"The pseudo-label rule π(·|s_t) ← δ_{a_t} is correct only for deterministic behavior policies. The offline datasets are generated by stochastic SAC policies, and Appendix E.4 adds Gaussian noise during rollouts, so the pseudo-labels are inconsistent with the true generative process; Section 4.3 indeed shows a substantial drop in heterogeneous settings. This limits the paper's claim that policy labeling is inexpensive in the common unknown-policy offline setting. The authors should either fit a stochastic behavior-policy estimator from the data, with an analysis of the resulting bias, or clearly restrict the offline contribution to settings where true or reliable policy labels are available.","section":"Section 3.3, Algorithm 1 lines 4–5; Section 4.3"},{"comment":"The implementation replaces the infinite discounted sequential forward KL from Theorem 3.4 with an L-horizon undiscounted sum evaluated by reusing the observed segment as a rollout of π+ and π−. The paper acknowledges this approximation but gives no analysis of the bias introduced when segments are truncated, when the continuation is off-policy for the stated behavior policy, or when L is small. Because this regularizer is the main algorithmic novelty, the comparison between the theoretical objective and the implemented loss needs at least an explicit derivation of the approximation error and, ideally, an offline ablation over L with a discussion of the truncation effects.","section":"Section 3.3, contrastive KL approximation"}],"minor_comments":[{"comment":"In the second bracket of Eq. (6), the state-dependent term should be β(s_t^-) and the transition expectation should be over P(·|s_t^-, a_t^-); as printed, the shaped DPO objective is not well-formed.","section":"Appendix B.2, Eq. (6)"},{"comment":"The sentence 'Theorem 3.6 implies that...' should read 'Corollary 3.6 implies...' because the result is labeled Corollary 3.6.","section":"Section 3.2, after Eq. (4)"},{"comment":"The text says 'We normalize the contrastive KL regularization to balance their scale,' but the equations show only a 1/L factor; please state the exact normalization used, including whether it also divides by segment length, entropy scale, or another quantity.","section":"Section 3.3"},{"comment":"For reproducibility, please state the exact regret formula used by the SAC critic at query time, including whether the temperature α in the label generator equals the α used in the PPL loss and whether the critic is the same network used for rollouts or a separately trained one.","section":"Section 4.1 and Appendix E.4"}],"recommendation":"major_revision","confidential_remarks":"The main gap is between the clean theoretical statements and the actual training objective: the πψ-in-place-of-π* substitution needs a proof or a careful approximation analysis, and the headline empirical gains currently come from datasets labeled by the same regret model that PPL optimizes. If the authors can add a human-preference or non-regret label experiment and resolve the substitution issue, the paper would be strong. I see no indication of misconduct; the paper is transparent about pseudo-labels, dataset construction, and the reproducibility check against CPL."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The theory in this paper is worth a look. Theorem 3.4 genuinely extends the Q-difference identity to stochastic MDPs without an explicit reference policy, and the regret decomposition that yields the contrastive KL term is a real extension of CPL. The authors also do an honest reproducibility check on the original CPL datasets and openly report the pseudo-label degradation. So there is substance here.\n\nThe soft spot is the empirical core. Section 4.1 says preference labels were generated by a pretrained SAC critic using estimated regret, and PPL's loss is exactly a regret-based score with behavior-policy labels. On the original CPL datasets (Table 6), PPL matches CPL rather than beating it. The large margins in Table 2 appear only on the newly created regret-labeled datasets. That does not establish that policy labels help for general or human preferences; it shows that if preferences are exactly regret, then using the regret model plus the true behavior policy helps. The paper frames the contribution as modeling human preferences, but the experiments never use human or non-regret labels.\n\nThere is also a derivation gap. Equation (3) is an identity about the true optimal policy π*, but then the learned policy πψ is substituted into the regret without proof. If that substitution is not valid during training, the loss is no longer minimizing regret relative to π*. The authors note this in Appendix B.1 but do not resolve it. The deterministic pseudo-label is another acknowledged weakness: it is wrong for stochastic behavior policies and the paper shows it hurts heterogeneous performance. These are real issues but they are addressable rather than fatal.\n\nThe theory section is coherent and the central idea—that behavior-policy information matters when trajectories come from multiple policies—is plausible. But as it stands, the empirical validation is self-referential in a way that undercuts the main claim. The paper deserves a serious referee because the question is important and the theoretical contribution is nontrivial, but a revision needs to (1) test on human labels or at least on a non-regret preference source, and (2) either prove or explicitly justify the πψ substitution in the PPL loss.\n\nI'd send it to peer review, but with the expectation that the experiments be redone before acceptance.","headline":"Solid theoretical extension of CPL to stochastic MDPs, but the experiments validate the model on labels made by the same regret estimator, so the headline claim about human preferences is not yet earned.","tokens_in":22267,"tokens_out":2017,"would_cite":false,"duration_ms":20117,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pairwise preferences alone mislabel suboptimal behavior as optimal in stochastic environments; labeling each trajectory with its behavior policy, via regret, fixes the mismatch and improves offline and online RLHF.","keywords":["reinforcement learning from human feedback","preference-based reinforcement learning","regret-based preference model","direct preference optimization","likelihood mismatch","offline reinforcement learning","contrastive KL regularization","maximum entropy reinforcement learning"],"falsifier":"Train PPL on an offline dataset from a known stochastic behavior policy in a stochastic MetaWorld task, once with true policy labels and once with deterministic pseudo-labels as in Algorithm 1. The mechanism predicts the true-label version clearly outperforms the pseudo-label version in a heterogeneous or stochastic setting; if the two perform the same, behavior-policy information is not what drives the gains. A second check: setting the lookahead to $L=0$ collapses the objective to CPL, so the claim predicts a performance drop; matching results would show the sequential-KL term is decorative.","tokens_in":21337,"feed_emoji":"🤖","tokens_out":12175,"duration_ms":99262,"temperature":0.7,"pith_summary":"This paper asks whether human preference comparisons alone suffice for reinforcement learning from human feedback in stochastic environments, and answers no: the behavior policy that generated each segment must also be known. It proposes Policy-labeled Preference Learning (PPL), which scores segments by negative regret relative to the optimal policy, a quantity that decomposes into a term raising the likelihood of preferred actions and a term reducing the sequential forward KL divergence between behavior and learned policies. The paper proves a policy-deviation theorem relating any policy's soft Q-function to the optimal one, and shows the resulting contrastive KL regularization aligns the learned policy with preferred segments and away from dispreferred ones. If the claim holds, preference data plus a cheap policy label trains policies directly, beating previous direct and reward-based methods on offline robotic control and matching an order-of-magnitude larger online baseline.","feed_headline":"Label trajectories to fix RLHF's likelihood mismatch","feed_subtitle":"The loss separates behavior-policy effects from environmental noise and beats prior methods with fewer parameters.","key_machinery":"The load-bearing object is negative regret used as the preference score, $-\\mathrm{Reg}^\\pi_{\\pi^*}(s,a) := -(V^{\\pi^*}(s) - Q^\\pi(s,a))$, which unlike the optimal advantage $Q^{\\pi^*}(s,a)-V^{\\pi^*}(s)$ retains the behavior policy $\\pi$. Its decomposition into the log-likelihood of the optimal policy minus the sequential forward KL divergence $\\bar{D}_{\\mathrm{KL}}(\\pi\\|\\pi^*;s,a)$ is what turns regret into a directly optimizable policy objective. The Policy Deviation Theorem (Theorem 3.4) supplies that decomposition, and the resulting contrastive KL regularization, implemented as an $L$-horizon, undiscounted lookahead over rollouts, is the mechanism that aligns the learned policy with preferred trajectories while contrasting it against dispreferred ones. Corollary 3.5, which shows regret is invariant under state-dependent reward transformations, is what keeps the score stable without extra variance-reduction machinery.","core_discovery":"The paper's central claim is that the \"optimal advantage\" score used by prior work (for instance CPL) silently assumes every trajectory was generated by the optimal policy, and that this causes a likelihood mismatch in stochastic MDPs: differences in outcome quality that actually come from behavior-policy suboptimality are misattributed to environmental randomness. The fix is to score segments by negative regret, $-\\mathrm{Reg}^\\pi_{\\pi^*}(s,a) = \\alpha\\left(\\log\\pi^*(a|s) - \\bar{D}_{\\mathrm{KL}}(\\pi\\|\\pi^*;s,a)\\right)$, which folds the behavior policy $\\pi$ into the score. With the learned policy substituted for $\\pi^*$, this yields a policy-labeled loss whose contrastive KL term pulls the learned policy toward preferred segments' policies and pushes it away from dispreferred ones. The paper further proves (Theorem 3.4) that any policy's soft Q-function differs from the optimal soft Q-function exactly by $\\alpha$ times a sequential forward KL divergence, and shows that when all preferred segments come from the optimal policy the objective reduces to CPL, and when the segment policies are the reference policy it reduces to DPO. Empirically, on six MetaWorld tasks across homogeneous and heterogeneous offline datasets, PPL matches or beats CPL and the reward-based P-IQL, and its online variant matches PEBBLE while using about 8.8% of PEBBLE's parameter count.","pith_inferences":["The likelihood-mismatch diagnosis suggests a dataset-design rule the paper leaves implicit: preference datasets should record behavior-policy identity, and the contrastive KL term will be most informative when queried segment pairs come from policies of known, differing quality rather than from a single average policy.","A testable extension is to replace the rollout-based lookahead with a model-based or replay-based estimate of the sequential KL term, which would let PPL apply where the environment is not resettable and only fixed offline segments exist.","The regret decomposition implies a reweighting interpretation: segments whose behavior policy already matches the learned policy contribute little gradient, which suggests an active-query strategy that preferentially queries segments from policies currently far from the learner.","If the mechanism is right, a sharp prediction follows: in a stochastic environment with genuinely stochastic behavior policies, PPL trained with true policy labels should strictly dominate its deterministic-pseudo-label variant, and the gap should grow with the stochasticity of the behavior policy."],"forward_implications":["PPL strictly generalizes CPL: when every preferred segment is assumed to come from the optimal policy, the contrastive KL term vanishes and the loss reduces to CPL's objective.","PPL also contains DPO: if the behavior policies of both segments equal the reference policy, the framework recovers the original DPO formulation, so the regret view unifies the two prior methods.","Because regret eliminates the state-dependent shaping term by construction, the method needs no explicit advantage-estimation or variance-reduction scheme to stay stable (Corollary 3.5).","In offline settings the policy label matters most when data is heterogeneous: deterministic pseudo-labels cost little on homogeneous data but produce a large performance gap on mixed-policy datasets, so collecting true policy labels is what buys the improvement.","Online, where policy labels are free by construction, PPL learns from scratch without a reward model or critic and matches PEBBLE's success rates with roughly 8.8% of PEBBLE's parameters."],"supporting_citations":[{"why":"Supplies CPL, the primary baseline whose optimal-advantage score PPL replaces, along with the rollout and preference data used for evaluation.","marker":"(Hejna et al., 2023)"},{"why":"Provides the regret-based preference model and the critique of partial-sum-of-rewards scoring that motivates regret as a segment score.","marker":"(Knox et al., 2022)"},{"why":"Gives the maximum-entropy framework, including the link between optimal policies and soft Q-functions used throughout the proofs.","marker":"(Ziebart, 2010)"},{"why":"Establishes DPO, the direct-preference-optimization paradigm PPL extends, and the surjection lemma that Lemma 3.2 sharpens into a bijection.","marker":"(Rafailov et al., 2024b)"},{"why":"Provides PEBBLE, the online baseline whose success rates and parameter count the online PPL experiments are compared against.","marker":"(Lee et al., 2021)"},{"why":"Supplies P-IQL, the reward-based offline RLHF baseline that PPL is compared with and outperforms on most tasks.","marker":"(Hejna & Sadigh, 2024)"},{"why":"States policy invariance under reward shaping, the classical result that Lemma 3.2 refines by specifying the action-dependent term preserving $\\alpha$-optimality.","marker":"(Ng et al., 1999)"}],"fun_headline_variants":["Policy-labeled preferences fix RLHF's likelihood mismatch","Regret-based scoring for RLHF reduces likelihood errors","Labeling policies in preferences improves RLHF offline and online","PPL: A small-parameter RLHF that beats prior methods","Unifying CPL and DPO with regret-based policy scoring"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the current learned policy can stand in for the true optimal policy inside the regret formula when the training loss is written; the paper asserts this substitution rather than proving it, so the loss is no longer a regret against the true optimum once the learned policy drifts far from optimal.","fun_headline_variants_meta":{"raw":{"variants":["Policy-labeled preferences fix RLHF's likelihood mismatch","Regret-based scoring for RLHF reduces likelihood errors","Labeling policies in preferences improves RLHF offline and online","PPL: A small-parameter RLHF that beats prior methods","Unifying CPL and DPO with regret-based policy scoring"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000779,"raw_usage":{"total_tokens":3454,"prompt_tokens":964,"completion_tokens":2490,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":2409}},"tokens_in":580,"tokens_out":2490,"duration_ms":17220,"temperature":1.0,"reasoning_tokens":2409,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:48:03.716503+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train PPL on an offline dataset from a known stochastic behavior policy in a stochastic MetaWorld task, once with true policy labels and once with deterministic pseudo-labels as in Algorithm 1. The mechanism predicts the true-label version clearly outperforms the pseudo-label version in a heterogeneous or stochastic setting; if the two perform the same, behavior-policy information is not what drives the gains. A second check: setting the lookahead to $L=0$ collapses the objective to CPL, so the claim predicts a performance drop; matching results would show the sequential-KL term is decorative.","supporting_citations":[],"review_version":1}