Pith. sign in

REVIEW 5 major objections 5 minor 16 references

Multi-Task Reward Learning from Human Ratings

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Human ratings carry both category and magnitude; reward learning should use both.

desk verdict Plausible extension of rating-based reward learning, but synthetic-only ratings and an overclaimed PPO comparison keep it at workshop level. read the letter →

arxiv 2506.09183 v2 pith:PJQUY7FF submitted 2025-06-10 cs.LG cs.AI

Mingkang Wu , Devin White , Evelyn Rose , Vernon Lawhern , Nicholas R Waytowich , Yongcan Cao This is my paper
classification cs.LGcs.AI
keywords reinforcementlearningfromhumanfeedbackrating-basedmulti-taskrewardinferenceuncertainty-weightedlossclassificationandregressionsyntheticratingsDeepMindControlSuite
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Reinforcement learning from human feedback typically reduces ratings to a classification task: predict which rating bucket a behavior falls into. This paper argues that humans also judge magnitude—how far a behavior is from a quality threshold—so reward learning should be a multi-task problem. It proposes training the reward predictor with both a classification loss and a regression loss, where discrete ratings are first mapped through a log-scaled formula to continuous reward targets, and the two losses are combined with learnable uncertainty weights. Experiments on six continuous-control environments with synthetic human ratings show the combined objective consistently beats the classification-only rating baseline and, for suitable numbers of rating classes, can exceed PPO trained on the true environment reward. If this holds with real raters, it would make rating feedback a more informative and cheaper signal for aligning agents in reward-free settings.

What carries the argument

The load-bearing object is the multi-task loss of Eq. (5), built from a rating-induced reward assignment $R_i(\sigma) = \log(1 + \alpha \sum_t y_i)$ and the classification probability $Q_\sigma(i)$ from the base rating method. The log-scaled assignment turns discrete rating classes into continuous, smoothly varying regression targets while keeping large ratings from producing unstable gradients, which matters because early random-policy data are dominated by low ratings. The loss combines the classifier's cross-entropy with mean-squared error against those targets and weights each term by an inverse learned variance, so the model adaptively emphasizes whichever view of the ratings—category or magnitude—is currently more certain. The $\log \lambda$ terms prevent the weights from collapsing to zero, which would otherwise let the model ignore both losses.

What would settle it

Collect real human ratings on the same six DeepMind Control environments under the same 2 to 6 class rubrics, fix the same hyperparameters, and compare mean returns. If the multi-task method no longer consistently beats classification-only rating-based reinforcement learning, or if PPO trained on the true reward wins across the board, the synthetic rating proxy—not the combined loss—was the source of the reported improvement.

Watch

Extended reading notes

Core claim

The central claim is that the ordinal and continuous information in human ratings is thrown away when rewards are learned only through classification. The paper proposes a rating-induced reward assignment $R_i(\sigma) = \log(1 + \alpha \sum_{t} y_i)$ that converts discrete rating classes into smoothed continuous reward labels, and a multi-task loss $L = \frac{1}{2\lambda_{\mathrm{cls}}^2} L_{\mathrm{CE}} + \log \lambda_{\mathrm{cls}} + \frac{1}{2\lambda_{\mathrm{reg}}^2} L_{\mathrm{reg}} + \log \lambda_{\mathrm{reg}}$ that trains the reward predictor to both classify the rating class and regress the continuous value, with $\lambda_{\mathrm{cls}}$ and $\lambda_{\mathrm{reg}}$ learned from data. The learnable weights let the model lean on classification when that signal is confident and on regression when the continuous value is more informative. Across six DeepMind Control environments and 2 to 6 rating classes, this joint objective consistently outperforms the classification-only rating-based reinforcement learning baseline, and in several configurations it matches or beats PPO, which is trained on the environment's handcrafted reward.

Load-bearing premise

All ratings are synthetic, generated by manually defined reward boundaries per environment; the claim that the method mimics human decision-making and captures human uncertainty assumes those boundaries faithfully model how real people assign rating categories and values.

Editorial extensions

If this is right

  • Adding a regression term to the rating classifier raises mean returns relative to classification-only rating-based reinforcement learning in all six environments tested.
  • With a suitable number of rating classes, the reward-free method can match or exceed PPO, which receives the true environment reward, across every environment in the study.
  • Learnable uncertainty weights outperform fixed equal weighting and either objective alone, indicating that the balance between classification and regression should change during training.
  • The log-scaled mapping from ratings to rewards makes training stable enough to learn from scratch on data dominated by low-rating segments, since the agent starts from a random policy.
  • Across the tested 2 to 6 rating-class configurations, the improvement is consistent rather than limited to one rating granularity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If real human rating data confirm the synthetic results, the method implies that raters' numerical scores carry usable magnitude information beyond category labels, which would justify collecting scalar ratings instead of only pairwise preferences.
  • A natural test is to rerun the six environments with genuine human raters; if the margin over the classification-only baseline shrinks or disappears, the manually defined reward boundaries used to generate synthetic ratings were the driver, not the multi-task loss.
  • Because the uncertainty weights are trained per task, the same framework could be extended to switch emphasis within a trajectory or to weight ratings by annotator confidence, though the paper itself tests only per-environment weighting.
  • The paper's own impact statement notes that biased human ratings could be reinforced; a concrete follow-up would be to audit the learned reward for rating-bias amplification before deployment in sensitive settings.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes a multi-task reward learning framework for reinforcement learning from human ratings. The method trains a reward predictor by jointly optimizing a classification loss (predicting discrete rating classes) and a regression loss (predicting a smoothed continuous reward target derived from ratings), with learnable uncertainty-based weights that balance the two objectives. The authors evaluate the method on six DeepMind Control environments with synthetic ratings, comparing against standard PPO and rating-based RL (RbRL), and report that the proposed method consistently outperforms RbRL and can sometimes surpass PPO. An ablation study compares the full method against fixed-equal-weight, classification-only, and regression-only variants.

Significance. If the central empirical claim held, the paper would provide a useful incremental extension of rating-based reinforcement learning: adding an uncertainty-weighted regression term to a rating classifier is a natural idea, and the adaptive weighting mechanism is well motivated by the multi-task learning literature. The paper includes ablations across several environments, multiple rating-class counts, and 10 independent runs, which are strengths. However, the empirical validation rests entirely on synthetic ratings generated as a deterministic, noiseless quantization of true returns using manually chosen boundaries. This does not test the paper's stated goal of capturing the inherent uncertainty in human decision-making, and it leaves open the possibility that the reported gains are artifacts of the synthetic protocol. The contribution is therefore promising but not yet convincingly established.

major comments (5)
  1. [Section 5] The rating data are generated synthetically by 'manually defining the reward boundaries for each environment,' which yields a deterministic, noiseless quantization of the true normalized return with no human noise or inconsistency. The abstract and introduction claim the method 'captures the inherent uncertainty in human decision-making,' but the experiments never expose the reward model to such uncertainty. Consequently, Figures 2 and 3 demonstrate only that the multi-task loss can recover a reward from a known quantized oracle signal, not that it mimics human rating behavior. Please add experiments with real human raters or, at minimum, synthetic ratings with calibrated noise and a sensitivity analysis over the manual boundary choices, and temper the claims about mimicking human decision-making.
  2. [Section 4.1, Eq. (3)] Eq. (3) defines Ri(σ) = log(1 + α Σ_{t=0}^k yi), but yi is not indexed by t, so the summation is ambiguous: if the rating is assigned to the whole segment, the sum should not appear; if ratings are assigned per timestep, the notation should be y_t or similar. Moreover, the target Ri(σ) is not normalized to [0,1], while the predicted return R̂(σ) is normalized, so the MSE in Eq. (4) compares quantities on different scales. Please define the target clearly and specify the normalization used before computing Lreg.
  3. [Section 5] The results section contains a self-contradictory pair of sentences: it first says the method 'exceeds PPO performance given a suitable number of rating classes' and then claims that 'even in reward-free environments, our method surpasses PPO across all environments by adapting to different numbers of rating classes.' These statements are mutually inconsistent about whether PPO is surpassed in all environments or only for some rating-class counts. Please correct the text and state precisely what Figures 2 and 3 show, including which configurations fail to surpass PPO.
  4. [Section 5, Eq. (2)] The synthetic ratings are generated using manually defined reward boundaries, while Eq. (2) defines the classifier's rating boundaries R̄i as derived from the human label distribution. It is unclear whether the same manually defined boundaries used for generating labels are also used as R̄i when training the classifier. If they are, the classification subtask is given the ground-truth thresholds and may be artificially easy, which could inflate the reported advantage over RbRL. Please clarify how R̄i is set in the experiments and, if the boundaries are shared, analyze sensitivity to the specific boundary choices.
  5. [Section 5 and Appendix A] The experimental description is missing essential details needed to reproduce the results: the architecture of the reward predictor, the number of reward-model updates per policy step, how and when the 2,000 synthetic ratings are collected relative to policy training, and the exact procedure for normalizing predicted returns. Please provide a complete specification of the training pipeline, including all reward-model hyperparameters, so that the empirical claims can be independently checked.
minor comments (5)
  1. [Section 4.1] The sentence 'Where yi ∈ n denotes the human-provided rating class' should be written as 'yi ∈ {0, 1, ..., n−1}' or similar; the notation 'n' alone is not a set.
  2. [Section 4, after Eq. (5)] The phrase 'A similar ideal applies' should read 'A similar idea applies.'
  3. [Acknowledgment] The heading 'Acknowledgment' is misspelled; it should be 'Acknowledgments' or 'Acknowledgement' depending on the journal style.
  4. [Appendix A] The appendix provides only PPO hyperparameters; please also state the reward-model architecture, optimizer, learning-rate schedule, and number of training iterations, as these are needed for reproducibility.
  5. [General] Several equations and inline math have spacing artifacts (e.g., 'M DP\R', 'λ cls', 'logλ cls'); please format them consistently.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the reward model is trained on human ratings, and the evaluation metric is the environment's true return, so the central comparison does not fit its own evaluation target.

full rationale

The paper's derivation chain is self-contained rather than circular. The classification component (Eqs. 1-2) is adopted from the authors' prior RbRL work (White et al., 2024) and is treated as a baseline; this is a normal self-citation, and the baseline result is independently published, not invoked to preclude alternatives. The proposed regression target (Eq. 3) is a hand-defined deterministic transform of the provided rating class, not a fitted parameter, and no fitted quantity is later relabeled as a prediction of environment return. The learnable uncertainty weights (Eq. 5) are trained on the same rating labels, which is standard multi-task learning, and the policy evaluation compares against the environment's true cumulative reward, so the evaluation target is not used to fit the model. The main limitation is external validity rather than circularity: Section 5 generates ratings synthetically by manually defining reward boundaries, so the claimed mimicry of human decision-making is not validated with real human raters; additionally, Eq. (3) is not well defined as written because y_i does not depend on the summation index t. These issues undermine the empirical claim's generality but do not make the derivation reduce to its own inputs, so the circularity score is minimal.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new entities. It relies on hand-set constants (alpha, k), a synthetic rating generator with undisclosed boundaries, and the domain assumption that humans combine categorical and continuous judgment. These choices are load-bearing because they determine the regression target and the entire empirical evaluation.

free parameters (3)
  • alpha (rating scaling factor) = 0.5
    Hand-set in Eq. (3); controls the nonlinearity of the rating-to-reward mapping. No sensitivity analysis is provided, yet it directly defines the regression target.
  • k (temperature in classification softmax) = 30
    Set in Eq. (2), inherited from White et al. (2024); affects the shape of predicted rating probabilities in both the baseline and the proposed method.
  • Manual rating boundaries per environment = not specified
    The synthetic rating generator 'manually defines the reward boundaries for each environment' (Section 5), so the human labels are constructed by hand and the exact boundaries are not disclosed.
assumptions (4)
  • domain assumption Human ratings encode both categorical quality and continuous, ordinal magnitude information.
    The entire motivation for combining classification and regression rests on this premise (Sections 1 and 4.1); the paper provides no human-subject data to test it.
  • standard math The uncertainty-weighting derivation of Kendall et al. (2018) transfers to this RLHF reward-learning setting.
    The loss (5) is adopted from Kendall et al. without re-deriving its assumptions for the scaled-softmax classification term used here.
  • ad hoc to paper Synthetic ratings generated from manually defined boundaries are a faithful proxy for real human ratings.
    Section 5 generates all ratings synthetically; the paper's claims about mimicking human decision-making depend on this proxy.
  • ad hoc to paper The regression target log(1 + alpha * sum ratings) is a meaningful reward signal.
    Eq. (3) introduces this mapping without derivation; alpha is hand-set and the target's scale is not evaluated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Task Reward Learning from Human Ratings." pith.science (2026). https://pith.science/paper/PJQUY7FF

@misc{pith2026250609183,
  author       = {Pith},
  title        = {Pith review of: Multi-Task Reward Learning from Human Ratings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PJQUY7FF}},
  note         = {Machine review of arXiv:2506.09183}
}
read the original abstract

Reinforcement learning from human feedback (RLHF) has become a key factor in aligning model behavior with users' goals. However, while humans integrate multiple strategies when making decisions, current RLHF approaches often simplify this process by modeling human reasoning through isolated tasks such as classification or regression. In this paper, we propose a novel reinforcement learning (RL) method that mimics human decision-making by jointly considering multiple tasks. Specifically, we leverage human ratings in reward-free environments to infer a reward function, introducing learnable weights that balance the contributions of both classification and regression models. This design captures the inherent uncertainty in human decision-making and allows the model to adaptively emphasize different strategies. We conduct several experiments using synthetic human ratings to validate the effectiveness of the proposed approach. Results show that our method consistently outperforms existing rating-based RL methods, and in some cases, even surpasses traditional RL approaches.

Figures

Figures reproduced from arXiv: 2506.09183 by the authors.

Figure 1
Figure 1. A schematic illustration of the proposed reward learning framework with human ratings. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Empirical return comparison among different algorithms across six environments. The plots show the mean with [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Ablation study comparing empirical returns across different configurations in six environments. The plots show [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

16 extracted references · 9 canonical work pages

  1. [1]

    F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D

    Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  2. [2]

    Safe rlhf: Safe reinforcement learning from human feedback

    Dai, J., Pan, X., Sun, R., Ji, J., Xu, X., Liu, M., Wang, Y., and Yang, Y. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310.12773, 2023

    arXiv 2023

  3. [3]

    Multi-task learning using uncertainty to weigh losses for scene geometry and semantics

    Kendall, A., Gal, Y., and Cipolla, R. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 7482--7491, 2018

    work page 2018

  4. [4]

    Playing atari with deep reinforcement learning

    Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013

    arXiv 2013

  5. [5]

    Performance Optimization of Ratings-Based Reinforcement Learning

    Rose, E., White, D., Wu, M., Lawhern, V., Waytowich, N. R., and Cao, Y. Performance optimization of ratings-based reinforcement learning. arXiv preprint arXiv:2501.07755, 2025

    work page Pith review arXiv 2025

  6. [6]

    Schluchter, M. D. Mean square error. Encyclopedia of Biostatistics, 5, 2005

    work page 2005

  7. [7]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017

    arXiv 2017

  8. [8]

    Sutton, R. S. and Barto, A. G. Reinforcement Learning: A n Introduction . The MIT Press, Cambridge, MA, 1998

    work page 1998

Show all 16 references
  1. [9]

    Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction. Robotica, 17 0 (2): 0 229--235, 1999

    1999

  2. [11]

    Deepmind control suite, 2018 b

    Tassa, Y., Doron, Y., Muldal, A., Erez, T., Li, Y., de Las Casas, D., Budden, D., Abdolmaleki, A., Merel, J., Lefrancq, A., Lillicrap, T., and Riedmiller, M. Deepmind control suite, 2018 b . URL https://arxiv.org/abs/1801.00690

    2018 arXiv

  3. [12]

    Mujoco: A physics engine for model-based control

    Todorov, E., Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp.\ 5026--5033. IEEE, 2012

    2012

  4. [13]

    J., Waytowich, N., and Cao, Y

    White, D., Wu, M., Novoseller, E., Lawhern, V. J., Waytowich, N., and Cao, Y. Rating-based reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp.\ 10207--10215, 2024

    2024

  5. [14]

    Value of potential field in reward specification for robotic control via deep reinforcement learning

    Wu, M., Tao, F., and Cao, Y. Value of potential field in reward specification for robotic control via deep reinforcement learning. In AIAA SCITECH 2023 Forum, pp.\ 0505, 2023

    2023

  6. [15]

    Offline reinforcement learning with failure under sparse reward environments

    Wu, M., Siddique, U., Sinha, A., and Cao, Y. Offline reinforcement learning with failure under sparse reward environments. In 2024 IEEE 3rd International Conference on Computing and Machine Intelligence (ICMI), pp.\ 1--5. IEEE, 2024

    2024

  7. [16]

    R., and Cao, Y

    Wu, M., White, D., Lawhern, V., Waytowich, N. R., and Cao, Y. Rbrl2. 0: Integrated reward and policy learning for rating-based reinforcement learning. arXiv preprint arXiv:2501.07502, 2025

    2025

  8. [17]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.