REVIEW 3 major objections 6 minor 26 references
CueLearner: Bootstrapping and local policy adaptation from relative feedback
T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Relative feedback—short directional hints like 'more to the left'—can be learned as an action-refinement model, cutting label needs 5-10x versus scalar feedback on sparse-reward tasks and enabling fast local policy adaptation.
desk verdict A practical, honestly-bounded method for learning from relative feedback in deep RL, with a generalization claim that is broader than the direct evidence supports. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is a learned feedback model $\phi_\theta$: a deep network that takes the current state and action and outputs an action-space correction vector, trained by L2 regression toward $\epsilon\cdot h$, where $h$ is the trainer's normalized improvement direction. At inference it is applied iteratively to the base policy's action until refinements converge, and clipping keeps the refined action within a distance $\tau_{\max}$ of the base action when the user has prior knowledge that the optimal action is local. To prevent compounding errors from drifting off the training distribution, the model is retrained with a Dataset Aggregation loop that re-queries feedback on refined actions. This refinement module is separate from the policy, which lets the same tool bootstrap exploration—by refining actions from a uniform random base policy during off-policy training—and later adapt a deployed policy without touching its weights.
What would settle it
Collect relative-feedback labels in an adaptation scenario where the true required correction is consistently larger than the clipping bound $\tau_{\max}$, then check whether refined actions still improve success; the paper's Table II already provides such a case (navigation needing $\pm180^\circ$ corrections), where the method worsens performance, so a clean test is whether any variant with these labels can recover the oracle's gains.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that relative feedback can be fully absorbed into a learned feedback model $\phi_\theta(s,a)$ trained to match a normalized improvement direction $h$ in action space, and this model can then replace the human trainer. The model is applied through the refinement recurrence $a_{k+1}=a_k+\phi_\theta(s,a_k)$, stopped by a convergence threshold or iteration cap, with optional clipping of refinements to a ball of radius $\tau_{\max}$ around the base action. Guided by this module, a double deep Q-learning agent learns sparse-reward billiards and quadruped navigation with 350-1000 relative labels, roughly 5-10 times fewer labels than scalar feedback for the same speedup. In post-hoc adaptation, the same module restores performance after sensor miscalibration, embodiment bias, or gravity changes about an order of magnitude faster than RL finetuning, matching a residual-learning baseline while requiring less precise corrections. The paper also reports that the method is suited to localized adjustments and fails when a task demands global action changes beyond its bound.
Load-bearing premise
The whole method leans on the premise that the correct action lies inside a small box around the base policy's proposed action, because refinements are clipped to that region and the model is only trained on corrections of that size.
Editorial extensions
If this is right
- On sparse-reward tasks, a few hundred relative-feedback labels can replace thousands of scalar-feedback labels, making human-guided RL practical for nonexpert trainers.
- Because the feedback model generalizes to novel states, the human trainer can stop after a short labeling session; the learned corrections then cover the state space.
- For local adaptation, the method restores much of a policy's lost performance (e.g., from 72% to 82% success after lidar misalignment) an order of magnitude faster than RL finetuning.
- The same 500-label procedure that bootstraps exploration in simulation transfers to a real ANYmal, suggesting deployment-ready policies can be produced without sim-to-real retraining.
- When action corrections need to exceed the preset bound, e.g., more than $\pm60^\circ$ in navigation, the method at best preserves and at worst degrades performance.
Reading between the lines
- Because the refinement module is policy-agnostic, the same trained feedback model could be attached to any base controller—including classical planners—not just DDQN agents, turning relative feedback into a portable correction layer.
- The paper's own Table II suggests a practical diagnostic: if the human's corrections consistently exceed the assumed $\tau_{\max}$, the task is outside the method's regime and full retraining or a new action space is needed; this could be measured during label collection.
- The 5-10x label-efficiency gap over scalar feedback hints that richer-but-still-cheap input channels (e.g., ordinal or coarse directional commands) may be broadly underused in human-in-the-loop RL.
- The real-world ANYmal result is qualitative evidence that offline-trained feedback models keep working under sensor noise and geometry mismatch; a quantitative extension would measure collision rate against baseline on multiple courses.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces CueLearner, a method that learns a deep-network feedback model phi_theta from relative feedback labels (normalized improvement directions in action space), and then uses iterative action refinement to guide exploration in off-policy RL or to adapt a trained policy. The method is evaluated on two sparse-reward tasks: a simplified billiards game and quadruped navigation in the Orbit simulator, plus a real-world deployment on an ANYmal. The authors report improved sample efficiency relative to scalar feedback, comparable or better label efficiency than demonstrations in billiards, faster adaptation than RL finetuning, and label efficiency comparable to residual learning from optimal corrections. The paper also reports an honest failure case when the required action changes are globally large rather than local.
Significance. If the central claim holds, CueLearner would be a practically useful interface: relative feedback is easier to provide than demonstrations while being more informative than scalar feedback, and learning a feedback model that replaces the human trainer could substantially reduce annotation cost. The paper is transparent about its limitations, especially the local-action assumption in Eq. (6) and the failure in the substantially changed navigation environment (Table II), and it includes a real-world deployment with actual human feedback. However, the headline claim that the learned feedback model 'can generalize feedback to novel states' is not directly tested, and the main empirical comparisons are reported without error bars or significance tests. These gaps currently limit the strength of the conclusions.
major comments (3)
- [Section IV-B, Eqs. (7)-(8); Abstract; Section II-C] The central claim that the learned feedback model 'can generalize feedback to novel states without further querying the human trainer' is not directly supported. In the exploration use case, labels are collected under a uniformly random base policy (Eq. 7) and the model is applied to actions proposed by the RL policy (Eq. 8) without DAgger retraining during RL. Correct refinement at a state-action pair outside the label-collection support requires true generalization, but no held-out state evaluation, coverage analysis, or plot of prediction error versus distance from the training distribution is provided. The end-to-end success on billiards and navigation is also consistent with the alternative that the random base policy already covers the state regions visited during guided exploration. This is load-bearing because it is the property that distinguishes the method from prior relative-feedback work and justifies the label-efficiency advantage. Please add a direct evaluation of phi_theta's generalization, for example by validating on state-action pairs drawn from the RL policy's visitation distribution or by analyzing prediction error as a function of distance from the label-collection support.
- [Figures 3-5; Tables I-II] The main quantitative claims are reported as means across five seeds without error bars, confidence intervals, or significance tests. This applies to the 'five times less labels' and 'almost ten times less labels' comparisons in Section VI-A, the 'slightly outperforms demonstrations' result in Figure 3, and the 'on par with residual learning' claim in Section VI-B. Given the high variance typical of DDQN training with sparse rewards, these differences may fall within seed noise. Please report per-seed results or confidence intervals, and where possible paired tests across the shared seeds.
- [Abstract; Section VI-C; Eq. (6); Table II] The adaptation contribution is advertised as adapting to 'changes in the environment or the user's preferences' without the locality qualifier, but the method relies on the assumption that the optimal action lies within tau_max of the base action (Eq. 6). Table II shows that when the environment change requires corrections beyond approximately ±60 degrees, the method fails while a PRM oracle with a ±180-degree bound succeeds. The paper honestly states this limitation in Section VI-C, but the abstract and the list of contributions should be reworded to make explicit that the adaptation is local in action space; otherwise the scope of the central claim is broader than the evidence. Alternatively, add an analysis characterizing when the locality assumption holds.
minor comments (6)
- [Section VI-D] The real-world experiment uses a discrete three-option interface (optimal, counterclockwise, clockwise), while the method in Eq. (3) trains on continuous normalized gradients h as defined in Eq. (2). The paper does not specify how the discrete labels are converted to the continuous h used by the feedback model, nor how the strength parameter epsilon is applied to a discrete choice. This missing detail affects reproducibility of the real-world result.
- [Section V-C] For the billiards task, the oracle is described as 'an agent trained using our baseline (double deep Q-learning)', but the paper does not explain how this oracle computes the improvement direction h, which is defined as the normalized gradient of Q in Eq. (2). Please specify the exact procedure used to generate the oracle labels.
- [Appendix] Several task-dependent hyperparameters are mentioned in the main text but not listed in the Appendix: the guidance frequency ('every second episode'), the episode cutoff for guidance, the iterative refinement stopping threshold tau, and Kmax. Please include these values to make the experiments fully reproducible.
- [General] There are numerous typographical errors, including 'reiforcement', 'comparible', 'In constrast', 'substitues', 'obtacles', 'perfoms', and 'slighly'. A careful proofreading pass is recommended.
- [Section VI-A] The paper does not include an empirical comparison with Celemin et al. [8] even in a simple domain where their linear-policy method could be applied. While the authors state that the prior method is not compatible with deep RL, a small comparison in a suitable setting would strengthen the claim that the new method is more general.
- [Section VI-D] The real-world deployment is described only qualitatively ('handles sensor noise well', 'minimal collisions', 'minor scrapes'). Reporting quantitative metrics, such as success rate, collision rate, or time to reach goal across the obstacle courses, would make the deployment result more informative.
Circularity Check
No significant circularity: the paper's claims are empirical and are not reduced to fitted inputs, definitions, or self-citations.
full rationale
CueLearner is an empirical method paper: its central claims—improved sample efficiency in sparse-reward RL and faster post-hoc adaptation—are supported by measured learning curves (Figures 3-5, Tables I-II) against independent baselines, not by a derivation that folds its own inputs back into its outputs. Equation (3) trains the feedback model φ_θ to regress the epsilon-scaled normalized action gradients h defined in Equation (2), and Equation (4) applies that model as an iterative action-refinement operator; these equations define the algorithm rather than producing a predicted quantity that was already used as a fit target. The label-efficiency comparisons are external empirical measurements, and the paper makes no load-bearing self-citations: references to prior relative-feedback work (Celemin et al.) are contextual and not used to justify correctness. The main unverified assertion—that φ_θ generalizes to novel states without retraining—is an untested empirical claim and a possible coverage or overfitting risk, but not circularity, because no fitted parameter is relabeled as a prediction and no equation reduces the claimed result to its own inputs. The acknowledged limitation that the method only supports local action adjustments (Table II and surrounding text) concerns scope and assumptions, not circularity. Oracle-generated labels and hand-tuned hyperparameters introduce dependence on benchmark choices, but they do not make the reported empirical results circular.
Assumptions & free parameters
free parameters (6)
- epsilon (feedback strength) =
5 deg (billiards exploration), 0.1 deg (billiards adaptation), 20 deg (nav exploration), 2 deg (nav adaptation)
- tau_max (action clipping bound) =
+/-1 deg (embodiment), +/-0.3 deg (billiards other), +/-60 deg (nav adaptation), +/-180 deg (extreme)
- Scalar feedback threshold =
5 deg (billiards), 20 deg (navigation)
- Guidance frequency and cutoff =
every second episode, predetermined number of episodes (not specified)
- Iterative refinement stopping threshold tau and Kmax =
not reported
- DAgger retraining interval =
every 10 new feedback labels
assumptions (6)
- standard math The MDP formulation with continuous state and action spaces and a scalar reward function.
- domain assumption Human feedback h is the normalized gradient of the action-value function Q with respect to action (Eq. 2).
- domain assumption Iterative refinement (Eq. 4) converges to an improved action.
- ad hoc to paper The optimal action lies within a small L2 ball of radius tau_max around the base policy's action (Eq. 6).
- domain assumption A uniformly random base policy provides sufficient coverage of the state-action space for training the feedback model (Eq. 7).
- domain assumption Off-policy RL with a replay buffer can effectively use guided transitions.
invented entities (1)
-
None
Cite this review
Pith. "Pith review of CueLearner: Bootstrapping and local policy adaptation from relative feedback." pith.science (2026). https://pith.science/paper/EKOOWMWD
@misc{pith2026250704730,
author = {Pith},
title = {Pith review of: CueLearner: Bootstrapping and local policy adaptation from relative feedback},
year = {2026},
howpublished = {\url{https://pith.science/paper/EKOOWMWD}},
note = {Machine review of arXiv:2507.04730}
}
read the original abstract
Human guidance has emerged as a powerful tool for enhancing reinforcement learning (RL). However, conventional forms of guidance such as demonstrations or binary scalar feedback can be challenging to collect or have low information content, motivating the exploration of other forms of human input. Among these, relative feedback (i.e., feedback on how to improve an action, such as "more to the left") offers a good balance between usability and information richness. Previous research has shown that relative feedback can be used to enhance policy search methods. However, these efforts have been limited to specific policy classes and use feedback inefficiently. In this work, we introduce a novel method to learn from relative feedback and combine it with off-policy reinforcement learning. Through evaluations on two sparse-reward tasks, we demonstrate our method can be used to improve the sample efficiency of reinforcement learning by guiding its exploration process. Additionally, we show it can adapt a policy to changes in the environment or the user's preferences. Finally, we demonstrate real-world applicability by employing our approach to learn a navigation policy in a sparse reward setting.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Learning Complex Dexterous Manip- ulation with Deep Reinforcement Learning and Demonstrations,
A. Rajeswaran, V . Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine, “Learning Complex Dexterous Manip- ulation with Deep Reinforcement Learning and Demonstrations,” in Proceedings of Robotics: Science and Systems (RSS) , 2018
work page 2018
-
[2]
Deep q-learning from demonstrations,
T. Hester, M. Vecerik, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, I. Osband, G. Dulac-Arnold, J. Agapiou, J. Z. Leibo, and A. Gruslys, “Deep q-learning from demonstrations,” in Proceedings of the Thirty-Second AAAI Con- ference on Artificial Intelligence and Thirtieth Innovative Applica- tions of Artificial Intelligenc...
work page 2018
-
[3]
Action advising with advice imitation in deep reinforcement learning,
E. Ilhan, J. Gow, and D. P. Liebana, “Action advising with advice imitation in deep reinforcement learning,” in Adaptive Agents and Multi-Agent Systems , 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:233296282
work page 2021
-
[4]
Dqn-tamer: Human-in-the-loop reinforcement learning with intractable feedback,
R. Arakawa, S. Kobayashi, Y . Unno, Y . Tsuboi, and S. ichi Maeda, “Dqn-tamer: Human-in-the-loop reinforcement learning with intractable feedback,” 2018
work page 2018
-
[5]
Combining manual feedback with sub- sequent mdp reward signals for reinforcement learning
W. B. Knox and P. Stone, “Combining manual feedback with sub- sequent mdp reward signals for reinforcement learning.” in AAMAS, vol. 10, 2010, pp. 5–12
work page 2010
-
[6]
K. Lee, L. M. Smith, and P. Abbeel, “PEBBLE: feedback- efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training,” CoRR, vol. abs/2106.05091, 2021. [Online]. Available: https://arxiv.org/abs/2106.05091
arXiv 2021
-
[7]
Interactive learning with corrective feedback for policies based on deep neural networks,
R. P ´erez-Dattari, C. Celemin, J. R. del Solar, and J. Kober, “Interactive learning with corrective feedback for policies based on deep neural networks,” in International Symposium on Experimental Robotics , 2018. [Online]. Available: https://api.semanticscholar.org/CorpusID:52902162
work page 2018
-
[8]
Reinforcement learning of motor skills using policy search and human corrective advice,
C. Celemin, G. J. Maeda, J. R. del Solar, J. Peters, and J. Kober, “Reinforcement learning of motor skills using policy search and human corrective advice,” The International Journal of Robotics Research, vol. 38, pp. 1560 – 1580, 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID:199530701
work page 2019
Show all 26 references
-
[9]
No, to the right: Online language corrections for robotic manipulation via shared autonomy,
Y . Cui, S. Karamcheti, R. Palleti, N. Shivakumar, P. Liang, and D. Sadigh, “No, to the right: Online language corrections for robotic manipulation via shared autonomy,” Proceedings of the 2023 ACM/IEEE International Conference on Human-Robot Interaction , 2023. [Online]. Avai...
2023
-
[10]
Yell at your robot: Improving on-the-fly from language corrections,
L. X. Shi, Z. Hu, T. Z. Zhao, A. Sharma, K. Pertsch, J. Luo, S. Levine, and C. Finn, “Yell at your robot: Improving on-the-fly from language corrections,” arXiv preprint arXiv: 2403.12910 , 2024
2024 arXiv
-
[11]
An interactive framework for learning continuous actions policies based on corrective feedback,
C. Celemin and J. R. del Solar, “An interactive framework for learning continuous actions policies based on corrective feedback,” Journal of Intelligent & Robotic Systems , vol. 95, no. 1, pp. 77–97, Jul. 2019. [Online]. Available: https://doi.org/10.1007/s10846-018-0839-z
2019 doi
-
[12]
Algorithms for inverse reinforcement learning,
A. Y . Ng and S. J. Russell, “Algorithms for inverse reinforcement learning,” in Proceedings of the Seventeenth International Conference on Machine Learning , ser. ICML ’00. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 2000, p. 663–670
2000
-
[13]
Interactively shaping agents via human reinforcement: The TAMER framework,
W. B. Knox and P. Stone, “Interactively shaping agents via human reinforcement: The TAMER framework,” in The Fifth International Conference on Knowledge Capture , September 2009
2009
-
[14]
Interactive learning from policy-dependent human feedback,
J. MacGlashan, M. K. Ho, R. Loftin, B. Peng, G. Wang, D. L. Roberts, M. E. Taylor, and M. L. Littman, “Interactive learning from policy-dependent human feedback,” in Proceedings of the 34th International Conference on Machine Learning , ser. Proceedings of Machine Learning Res...
2017
-
[15]
Deep reinforcement learning from human preferences,
P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei, “Deep reinforcement learning from human preferences,” in Advances in Neural Information Processing Systems , I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Ed...
2017
-
[16]
Few-shot preference learning for human- in-the-loop rl,
J. Hejna and D. Sadigh, “Few-shot preference learning for human- in-the-loop rl,” in Conference on Robot Learning , 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:253546930
2022
-
[17]
Integrating behavior cloning and reinforcement learning for improved performance in sparse reward environments,
V . G. Goecks, G. M. Gremillion, V . J. Lawhern, J. Valasek, and N. R. Waytowich, “Integrating behavior cloning and reinforcement learning for improved performance in sparse reward environments,” in Adaptive Agents and Multi-Agent Systems , 2019. [Online]. Available: https://a...
2019
-
[18]
Agent- advising approaches in an interactive reinforcement learning scenario,
F. Cruz, P. W ¨uppen, S. Magg, A. Fazrie, and S. Wermter, “Agent- advising approaches in an interactive reinforcement learning scenario,” in 2017 Joint IEEE International Conference on Development and Learning and Epigenetic Robotics (ICDL-EpiRob), 2017, pp. 209–214
2017
-
[19]
Modem: Accelerating visual model-based reinforcement learning with demonstrations,
N. Hansen, Y . Lin, H. Su, X. Wang, V . Kumar, and A. Rajeswaran, “Modem: Accelerating visual model-based reinforcement learning with demonstrations,” arXiv preprint, 2022
2022
-
[20]
Interactive robot learning from verbal correction,
H. Liu, A. Chen, Y . Zhu, A. Swaminathan, A. Kolobov, and C.-A. Cheng, “Interactive robot learning from verbal correction,” 2023
2023
-
[21]
A reduction of imitation learning and structured prediction to no-regret online learning,
S. Ross, G. Gordon, and D. Bagnell, “A reduction of imitation learning and structured prediction to no-regret online learning,” in Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics , ser. Proceedings of Machine Learning Research, ...
2011
-
[22]
Orbit: A unified simulation framework for interactive robot learning environments,
M. Mittal, C. Yu, Q. Yu, J. Liu, N. Rudin, D. Hoeller, J. L. Yuan, R. Singh, Y . Guo, H. Mazhar, A. Mandlekar, B. Babich, G. State, M. Hutter, and A. Garg, “Orbit: A unified simulation framework for interactive robot learning environments,” IEEE Robotics and Automa- tion Lette...
2023
-
[23]
Anymal - a highly mobile and dynamic quadrupedal robot,
M. Hutter, C. Gehring, D. Jud, A. Lauber, C. D. Bellicoso, V . Tsounis, J. Hwangbo, K. Bodie, P. Fankhauser, M. Bloesch, R. Diethelm, S. Bachmann, A. Melzer, and M. Hoepflinger, “Anymal - a highly mobile and dynamic quadrupedal robot,” in 2016 IEEE/RSJ Interna- tional Conferen...
2016
-
[24]
Deep reinforcement learning with double q-learning,
H. Van Hasselt, A. Guez, and D. Silver, “Deep reinforcement learning with double q-learning,” in Proceedings of the AAAI conference on artificial intelligence, vol. 30, no. 1, 2016
2016
-
[25]
Probabilistic roadmaps for path planning in high-dimensional configuration spaces,
L. Kavraki, P. Svestka, J.-C. Latombe, and M. Overmars, “Probabilistic roadmaps for path planning in high-dimensional configuration spaces,” IEEE Transactions on Robotics and Automation , vol. 12, no. 4, pp. 566–580, 1996
1996
-
[26]
Reinforcement learning: An introduction,
R. Sutton and A. Barto, “Reinforcement learning: An introduction,” IEEE Transactions on Neural Networks , vol. 9, no. 5, pp. 1054–1054, 1998. APPENDIX a) Billiards Task: We train the DDQN policy using an MLP with layers [400, 400, 400, 300, 300, 300], a 1:1 gradient-to-environ...
1998
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.