Pith. sign in

REVIEW 4 major objections 5 minor 19 references

Training People to Reward Robots

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Machine-teaching guidance sharpens novice reward demonstrations for reinforcement-learning robots and transfers to skills the teachers never trained on.

desk verdict Useful extension with solid within-task training results, but the headline transfer claim to unseen skills is not supported by its own statistics. read the letter →

arxiv 2505.10151 v1 pith:MQNGCGPX submitted 2025-05-15 cs.RO

Endong Sun , Yuqing Zhu , Matthew Howard This is my paper · ORCID
classification cs.RO
keywords reinforcementlearningfromdemonstrationmachineteachingscaffoldingtrainingrewarddesigntransferofskillleast-squarespolicyiterationhuman-robotinteraction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether machine teaching—the practice of choosing the smallest set of examples that lets a learner reach a target—can train novice humans to give better reward demonstrations to robots that learn by reinforcement learning from demonstration (RLfD). The authors build a scaffolding interface that shows trainees the ideal reward for each state-action pair while they adjust a reward slider, using only eight demonstrations for a simple reaching skill. Their central claim is that this guidance improves the robot's learning on the training skill and transfers: trainees also produce better demonstrations for a second skill they never practiced. On the training skill, the target group's reward error fell 83% and the average root-mean-square trajectory error fell 89%; on the unseen skill, reward error fell 64%, while trajectory-level gains were large but not statistically significant, which the paper attributes to reward ambiguity in that task.

What carries the argument

The machinery is an interactive scaffolding training interface built on a machine-teaching formulation for RLfD. For a linear action-value model $Q^{\pi}(x,u)=\omega^{\top}\psi(x,u)$ learned by least-squares policy iteration, the teaching risk is $\rho=\|\omega-\bar{\omega}\|_2$, and with teaching dimension $T$ the teaching budget is exactly $N=T=8$ state-action-reward tuples. The interface shows the current state and action, lets the user assign a reward with a slider, and displays the ideal reward as a reference bar. The training curriculum P3-P7 progressively fixes state, action direction, and action magnitude so novices internalise the reward structure. The argument that this improves robot learning passes through Eq. (20), $\|\omega-\bar{\omega}\|_2\propto\|j-\bar{\jmath}\|_2$, which connects the measurable reward error to the value-parameter error that determines policy quality.

What would settle it

Compute the ratio $\|\omega-\bar{\omega}\|_2 / \|j-\bar{\jmath}\|_2$ across many random state-action samples in the least-squares policy iteration setup; if it varies by orders of magnitude, the proportionality in Eq. (20) fails and reward-error gains would not reliably become robot-learning gains. A companion check: rerun the transfer experiment with an unseen skill that has a unique optimal policy; if trajectory-level gains still fail to reach significance, the reward-ambiguity explanation is not sufficient.

Watch

Extended reading notes

Core claim

The central claim is that a teacher's absolute reward error, $\|j-\bar{\jmath}\|$, can serve as a training signal for novice humans: with a least-squares policy iteration learner and linear value features, the value-parameter error $\|\omega-\bar{\omega}\|$ is taken to be proportional to that reward error, so guiding people toward ideal rewards should improve what the robot learns. The experiment supports this on the trained skill: eight machine-teaching-guided demonstrations cut the target group's reward error by 83% and the learned policy's trajectory error by 89%, with no significant change in the control group. The transfer claim is that this teaching skill generalises: on a line-reaching skill not seen in training, reward error fell by 64%, a statistically significant change, while the larger trajectory improvements (70% lower ARMSE, 91% lower ATR) did not reach significance, which the paper explains by noting that multiple policies give the same cumulative reward for this task.

Load-bearing premise

The load-bearing premise is that bringing a novice's reward judgments closer to the ideal will bring the robot's learned value function closer to the target by a proportional amount; the paper derives this from a matrix-norm inequality, but the inequality only guarantees a bound, so the strength of that link can vary from task to task.

Editorial extensions

If this is right

  • Eight guided demonstrations on one reaching skill suffice to produce significant gains in a novice's reward-teaching accuracy on that skill.
  • The teaching improvement transfers to a never-practiced skill at the demonstration level: reward error fell 64% on the line-reaching task.
  • On the training skill, the robot's trajectory error fell 89% and its true-reward shortfall by 98%, so better reward labels translate into better control.
  • On the unseen skill, trajectory-level gains (70% lower trajectory error, 91% lower total-reward shortfall) were large but not statistically significant; the paper attributes this to reward ambiguity.
  • Trained teachers who teach via reinforcement-learning demonstrations produce policies with better long-horizon stability than teachers trained for supervised learning-from-demonstration, which matters when rollouts are long.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Taken as an inference from the mathematical fact the paper cites, Eq. (20) is an inequality in disguise: the constant linking reward error to value-parameter error depends on the conditioning of the sampled state-action matrix, so in poorly conditioned samples the training benefit may not propagate to the robot's policy.
  • The reward-ambiguity explanation for the non-significant S2 trajectory gains is directly testable: repeat the transfer test with a skill whose optimal policy is unique; if trajectory metrics become significant, ambiguity is confirmed, and if not, the transfer claim needs a different mechanism.
  • The scaffolding curriculum separates state distance, action direction, and action magnitude, so the same sequence could be applied to other learners with linear value features, though the machine-teaching target would no longer be closed-form if the learner changes.
  • A practical consequence the authors leave implicit is that short, targeted reward-teaching drills could substitute for extensive demonstration training in workplaces, since the measured benefit appears in demonstration quality and transfers across reward functions.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a machine-teaching-based scaffolding framework for training novice humans to provide reward demonstrations for reinforcement learning from demonstration (RLfD). In a between-subjects experiment, participants teach a two-link robot arm reaching skills S1 (point reaching) and S2 (line reaching); guided participants receive visual feedback based on the ideal reward values in a five-phase curriculum. The paper reports large within-task improvements in demonstration error and robot learning metrics on S1 and claims transfer to S2 with a 64% reduction in demonstration error and a 70% improvement in robot learning. The abstract and introduction present the 70% transfer improvement on robot learning as a headline result.

Significance. If the within-task results hold, the paper demonstrates that a short scaffolded curriculum with visual reward feedback can substantially improve novices' reward giving in RLfD, and the comparison with supervised LfD is a useful additional data point. The experimental design has real strengths: random assignment, a control group, a power analysis, multiple evaluation metrics, and a separate transfer task. The paper does not provide code or machine-checked proofs, so all claims rest on the reported statistics, which makes the mismatch between the significant E_ADE transfer result and the non-significant robot-learning metrics on S2 particularly damaging for the abstract's central claim. The transfer claim as written is not supported by the reported tests.

major comments (4)
  1. [§IV-C (Transfer of Teaching Ability); Abstract] The central transfer claim is not supported by the reported inferential statistics. For unseen skill S2, comparing P2 and P8 gives a significant 64% reduction in E_ADE (p=4.73e-9), but the direct measures of robot learning, E_ARMSE (p=0.27) and E_ATR (p=0.16), are not statistically significant. The abstract's statement that MT-guidance 'causes a 70% improvement in robot learning performance on skills not seen by subjects during training' therefore overstates the evidence. The post-hoc attribution of this discrepancy to 'reward ambiguity' is not tested with any quantitative measure and cannot convert non-significant learning-outcome differences into evidence of transfer. Please either report the transfer claim only for demonstration quality (E_ADE), or supply an analysis that directly supports the robot-learning transfer claim.
  2. [§III-B, Eq. (20)] Equation (20) asserts ∥ω−ω̄∥2 ∝ ∥j−j̄∥2 and attributes the proof to the sub-multiplicative property of matrix norms, citing prior work [4]. Sub-multiplicativity yields at most an upper bound of the form ∥ω−ω̄∥ ≤ ∥(Ψ⊤(Ψ−γΨ′))−1∥ ∥Ψ⊤∥ ∥j−j̄∥, with a condition-number-dependent constant; it does not establish proportionality, and no lower bound is given. In addition, the reported metric E_ADE is an ℓ1-style sum over individual reward differences, whereas Eq. (20) is stated for an ℓ2 norm. This relation is the theoretical bridge from the significant E_ADE result to claims about learned policies, so it cannot be left as a self-cited 'it can be shown' step; the paper should either prove a precise two-sided bound or weaken the interpretation of E_ADE as a proxy for robot learning.
  3. [§IV-B/C (between-subjects analysis)] The h1 analysis compares within-group changes only: P1 versus P9 for the target group and P1 versus P9 for the control group. A significant within-group improvement in one group and a non-significant change in the other does not by itself establish a treatment effect. The paper should report a between-group comparison of change scores or a group-by-time interaction test for E_ADE, E_ARMSE, and E_ATR to support the causal claim that guidance, rather than practice or time, produced the improvement.
  4. [§IV-C (Transfer of Teaching Ability)] The control-group results are used to infer 'the absence of training effects' from p>0.1 (S1) and p>0.05 (S2). Failure to reject the null in a study with ten participants per group is not evidence of absence. Please report effect sizes and confidence intervals for the control-group comparisons and avoid framing non-significant p-values as demonstrating that no effect exists.
minor comments (5)
  1. [§IV-A, Eq. (21)-(22)] The parameter definitions are confusing: Eq. (21) sets R=βI, while Eq. (22) also carries a β factor, and the text says Q=R for S1 yet Eq. (22) defines a different Q for S2. Please spell out the exact matrices used for each skill and report the sensitivity of the results to the chosen β and ε.
  2. [§II-B / §IV-B] P1 and P9 use the same skill S1 with newly sampled random probes, so h1 is a within-task improvement rather than evidence of generalization to new reward functions; h2 is the true generalization test. The phrase 'generalise this to previously unseen ones' in the abstract should be aligned with this distinction.
  3. [§IV-C, Fig. 6] The supervised-LfD versus RLfD comparison appears to be descriptive only; no statistical tests or error bars are reported for the threshold crossing times, so the crossover points t=276 and t=69 should be labeled as exploratory.
  4. [§IV-A (metric m1)] The definition of E_ADE in metric m1 starts the sum at n=0; since demonstrations are indexed n=1,...,N=8, the sum should run from n=1 to N unless a separate term is intended.
  5. [General] The paper does not state whether ethics approval or informed consent was obtained for the human-subject experiment; this should be reported in the experimental protocol.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central robot-learning claims are measured by external trajectory metrics, not by the displayed guidance.

full rationale

The paper's derivation chain is largely self-contained. The MT guidance is computed from the known optimal reward for the training skill S1 and displayed to subjects as a blue slider, while the metric E_ADE is defined as the distance from the subject's rewards to that same optimal reward vector. Thus the substantial improvement in E_ADE on S1 is partly a manipulation check: subjects are shown the target values and then measured on how closely they reproduce them. However, the paper's central claims about robot learning performance rest on E_ARMSE and E_ATR, which are computed from simulated robot trajectories and are not defined in terms of the displayed guidance. These metrics improved significantly for S1, and the transfer skill S2 was never shown during training, so the S2 E_ADE reduction is not a direct rehearsal of the training content. The control group provides an independent baseline, and the paper explicitly reports that the S2 robot-learning metrics E_ARMSE (p = 0.27) and E_ATR (p = 0.16) were not statistically significant, which is a statistical-support issue rather than a circularity issue. The only self-citation is [4] for the inequality in Eq. (20), but that theoretical bound is not used to generate the empirical results; robot learning is measured directly. Accordingly, no load-bearing circularity is present, though a minor self-citation and the E_ADE/guidance alignment justify a score slightly above zero.

Assumptions & free parameters 2 free parameters · 6 assumptions · 0 invented entities

The framework relies on standard linear algebra and LSPI assumptions, plus several domain assumptions about the task family and human learning that are not independently established. The most fragile is the Eq. (20) link between demonstration error and value-function error, which is self-cited and only an upper bound.

free parameters (2)
  • Action cost weight β = 0.01
    Chosen by hand to define the cost function in Eq. (21); determines the ideal rewards shown as guidance, so the training and evaluation depend on this value.
  • Regularization ε = 1e-15
    Chosen to avoid numerical instability in the S2 cost matrix (Eq. 22); not derived from data.
assumptions (6)
  • standard math Submultiplicative property of matrix norms and invertibility of Ψ^T(Ψ−γΨ')
    Used in Section III-B to derive the teaching-risk bound; the invertibility of the LSPI matrix is assumed without conditions.
  • domain assumption Target skills are quadratic optimal control problems with known cost matrices (Eqs. 6-9, 21-22)
    The experiment tasks S1 and S2 are LQR problems; the ideal rewards used as guidance are computed from these known costs, which limits the scope to this task family.
  • domain assumption LSPI converges to the optimal value function from demonstrations (Eqs. 12-13)
    Convergence is assumed from cited algorithm properties [10], [11]; the paper does not verify convergence conditions for the specific data sampled.
  • ad hoc to paper Novice teachers can internalize reward-assignment rules from an 8-example scaffolding curriculum and transfer them to new state-action pairs and a new skill
    This is the empirical premise of hypothesis h2; the curriculum P3-P7 is hand-designed and not derived from the MT formalization, so the transfer claim rests on this assumption.
  • domain assumption Teaching dimension of the linear learner equals the feature dimension T, so N=8 demonstrations suffice (Eq. 15-18)
    Taken from [12]; the feature vector ψ has 8 components, and the paper sets the effort budget to T, assuming 8 demonstrations are sufficient for LSPI to identify the value function.
  • ad hoc to paper ∥ω−ω̄∥∝∥j−j̄∥ (Eq. 20) is treated as an equality justifying E_ADE as a teaching-risk proxy
    The derivation only supports an upper bound with a condition-number dependent constant, yet the paper proceeds as if proportionality holds, citing self-prior work [4].

how reviews work

0 comments
Cite this review

Pith. "Pith review of Training People to Reward Robots." pith.science (2026). https://pith.science/paper/MQNGCGPX

@misc{pith2026250510151,
  author       = {Pith},
  title        = {Pith review of: Training People to Reward Robots},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MQNGCGPX}},
  note         = {Machine review of arXiv:2505.10151}
}
read the original abstract

Learning from demonstration (LfD) is a technique that allows expert teachers to teach task-oriented skills to robotic systems. However, the most effective way of guiding novice teachers to approach expert-level demonstrations quantitatively for specific teaching tasks remains an open question. To this end, this paper investigates the use of machine teaching (MT) to guide novice teachers to improve their teaching skills based on reinforcement learning from demonstration (RLfD). The paper reports an experiment in which novices receive MT-derived guidance to train their ability to teach a given motor skill with only 8 demonstrations and generalise this to previously unseen ones. Results indicate that the MT-guidance not only enhances robot learning performance by 89% on the training skill but also causes a 70% improvement in robot learning performance on skills not seen by subjects during training. These findings highlight the effectiveness of MT-guidance in upskilling human teaching behaviours, ultimately improving demonstration quality in RLfD.

Figures

Figures reproduced from arXiv: 2505.10151 by the authors.

Figure 1
Figure 1. Overview of training workflow. The novice user provides low-quality demonstrations, causing the robot to fail at [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Visual training interface setup for example training [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Scaffolding training (P3–P7) for S1 (see §IV). The ideal reward for the illustrated state (black dots)-action (red [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Comparison of subjects’ performance in teaching S1: [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Comparison of subjects’ performance in teaching S2: [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Comparison of learning outcomes for teachers trained [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 17 canonical work pages

  1. [4]

    Using machine teaching to boost novices’ robot teaching skill,

    Y . Zhu, E. Sun, and M. Howard, “Using machine teaching to boost novices’ robot teaching skill,” arXiv, 2024

    work page 2024

  2. [1]

    A survey of demonstration learn- ing,

    A. Correia and L. A. Alexandre, “A survey of demonstration learn- ing,” arXiv, 2023

    work page 2023

  3. [2]

    An algorithmic perspective on imitation learning,

    T. Osa, J. Pajarinen, G. Neumann, J. Bagnell, P. Abbeel, and J. Peters, “An algorithmic perspective on imitation learning,” Foundations and Trends in Robotics, 2018

    work page 2018

  4. [3]

    How can everyday users efficiently teach robots by demonstrations?

    M. Sakr, Z. Zhang, B. Li, et al., “How can everyday users efficiently teach robots by demonstrations?” arXiv, 2023

    work page 2023

  5. [5]

    When should we prefer offline reinforcement learning over behavioral cloning?

    A. Kumar, J. Hong, A. Singh, and S. Levine, “When should we prefer offline reinforcement learning over behavioral cloning?” arXiv, 2022

    work page 2022

  6. [6]

    Where machines could re- place humans-and where they can’t (yet),

    M. Chui, J. Manyika, and M. Miremadi, “Where machines could re- place humans-and where they can’t (yet),” The McKinsey Quarterly, pp. 1–12, 2016

    work page 2016

  7. [7]

    Artificial intelligence and work: A critical review of recent research from the social sciences,

    J.-P. Deranty and T. Corbin, “Artificial intelligence and work: A critical review of recent research from the social sciences,” AI & SOCIETY, pp. 1–17, 2022

    work page 2022

  8. [8]

    Ai, robotics, and the future of jobs,

    A. Smith and J. Anderson, “Ai, robotics, and the future of jobs,” Pew Research Center, vol. 6, 2014

    work page 2014

Show all 19 references
  1. [9]

    An overview of machine teaching,

    X. Zhu, A. Singla, S. Zilles, and A. N. Rafferty, “An overview of machine teaching,” CoRR, vol. abs/1801.05927, 2018

    2018 arXiv

  2. [10]

    Least-squares policy iteration algorithms for robotics: Online, continuous, and automatic,

    S. R. Friedrich, M. Schreibauer, and M. Buss, “Least-squares policy iteration algorithms for robotics: Online, continuous, and automatic,” Engineering Applications of Artificial Intelligence , vol. 83, pp. 72– 84, 2019

    2019

  3. [11]

    Locally weighted least squares policy iteration for model-free learning in uncertain environments,

    M. Howard and Y . Nakamura, “Locally weighted least squares policy iteration for model-free learning in uncertain environments,” inIROS, IEEE, 2013

    2013

  4. [12]

    The teaching dimension of linear learners,

    J. Liu and X. Zhu, “The teaching dimension of linear learners,” JMLR, vol. 17, no. 162, pp. 1–25, 2016

    2016

  5. [13]

    R. A. Horn and C. R. Johnson, Matrix analysis. Cambridge university press, 2012

    2012

  6. [14]

    N. R. Council, B. on Behavioral, S. Sciences, C. on Developments in the Science of Learning with additional material from the Committee on Learning Research, and E. Practice, How people learn: Brain, mind, experience, and school: Expanded edition. National Academies Press, 200...

    2000

  7. [15]

    D Norman, Design of everyday things , 1988

    1988

  8. [16]

    A taxonomy of mixed reality visual displays,

    P. Milgram and F. Kishino, “A taxonomy of mixed reality visual displays,” IEICE Trans. Inf.& Syst 1994 ,

    1994

  9. [17]

    How do humans teach: On curriculum learning and teaching dimension,

    F. Khan, B. Mutlu, and J. Zhu, “How do humans teach: On curriculum learning and teaching dimension,” NIPS, vol. 24, 2011

    2011

  10. [18]

    Curriculum learning,

    Y . Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in ICML 2009

    2009

  11. [19]

    LQR-trees: Feedback motion planning on sparse ran- domized trees,

    R. Tedrake, “LQR-trees: Feedback motion planning on sparse ran- domized trees,” in Robotics: Science & Systems , 2009

    2009

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.