{"id":"24b10b78-ba21-4ad9-af01-06f413f92150","arxiv_id":"2507.04730","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"CueLearner learns a relative-feedback model from a small number of human directional corrections and uses it to guide off-policy RL exploration or refine a deployed policy.","lead":"This paper introduces CueLearner, a method that learns a model of directional human corrections (like 'more to the left') and uses it to speed up reinforcement learning or adapt a trained robot policy. It is tested on a billiards game, a simulated quadruped navigation task, and a real robot.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's claim that the feedback model generalizes to novel states is not directly tested; the reported gains could stem from the RL policy staying within the label-collection distribution, not from true generalization.","rationale":"The reader's weakest_assumption focuses on the tau_max bound in Eq. 6 and Table II, which the authors already present as a stated limitation: the title says 'local policy adaptation' and Table II explicitly shows failure when global action changes are required. That assumption is scoped and honest, so it is not the most dangerous point of failure. The more consequential, untested assumption is that phi_theta, trained on a finite set of labels from one policy, produces reliable corrections at state-action pairs outside that labeling distribution. Both use cases depend on this: exploration applies phi_theta to the RL policy's actions (Eq. 8), and adaptation applies it iteratively at refined actions. The paper's DAgger retraining in adaptation (Section IV-A) mitigates distribution shift for that use case, but the exploration use case explicitly does not retrain, relying on the model's generalization. Without a direct test, the 5-10x label-efficiency advantage over scalar feedback could be an artifact of the simple tasks' state coverage rather than a property of the feedback model. A holdout-state experiment would settle this. I therefore recommend keeping the verdict conditional, with the added condition that the authors report held-out generalization accuracy for the feedback model and qualify the abstract if the claim is not supported.","tokens_in":11167,"tokens_out":15037,"duration_ms":173675,"concrete_test":"Use the navigation domain and its PRM oracle. Collect the 350-500 relative feedback labels exactly as in the paper with a uniformly random base policy, then hold out all labels whose states lie in a contiguous region (e.g., all states with the goal to the robot's right within 5 meters). Train phi_theta on the remaining labels and measure the angular error of its predicted correction on the held-out states against the oracle. Then run the full guided-RL pipeline using only the held-out labels for feedback-model training and compare the success-rate curve to the full-label run. If held-out angular error is high and the success-rate curve degrades substantially, the model does not generalize to novel states; if both are preserved, the generalization claim is supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's key novelty over prior relative-feedback work (Celemin et al.) is that the learned feedback model phi_theta fully replaces the human and 'can generalize feedback to novel states without further querying the human trainer' (Abstract, Section II-C). In the exploration use case (Section IV-B), labels are collected with a uniformly random base policy (Eq. 7), and the model is then applied to refine actions proposed by the RL policy (Eq. 8) with no DAgger retraining during exploration. Correct refinement at a state-action pair therefore requires phi_theta to interpolate or extrapolate beyond the support of the random-policy rollout used for label collection. The paper provides no direct measurement of this generalization: no held-out state evaluation, no coverage analysis, and no plot of prediction error versus distance from the training distribution. The end-to-end success on the two tasks is also consistent with the alternative hypothesis that the random base policy already covers the state regions visited during guided RL in these simple, low-dimensional environments (e.g., billiards states span the table). If phi_theta fails on genuinely novel states, the abstract's central generalization claim is unsupported, and the method degenerates into a fixed action-bias module that would need new labels in each region of state space. This is load-bearing because it is precisely the property that justifies the label-efficiency advantage and the comparison to Celemin et al.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CueLearner, a method that learns a deep-network feedback model phi_theta from relative feedback labels (normalized improvement directions in action space), and then uses iterative action refinement to guide exploration in off-policy RL or to adapt a trained policy. The method is evaluated on two sparse-reward tasks: a simplified billiards game and quadruped navigation in the Orbit simulator, plus a real-world deployment on an ANYmal. The authors report improved sample efficiency relative to scalar feedback, comparable or better label efficiency than demonstrations in billiards, faster adaptation than RL finetuning, and label efficiency comparable to residual learning from optimal corrections. The paper also reports an honest failure case when the required action changes are globally large rather than local.","tokens_in":11482,"tokens_out":5884,"duration_ms":68867,"significance":"If the central claim holds, CueLearner would be a practically useful interface: relative feedback is easier to provide than demonstrations while being more informative than scalar feedback, and learning a feedback model that replaces the human trainer could substantially reduce annotation cost. The paper is transparent about its limitations, especially the local-action assumption in Eq. (6) and the failure in the substantially changed navigation environment (Table II), and it includes a real-world deployment with actual human feedback. However, the headline claim that the learned feedback model 'can generalize feedback to novel states' is not directly tested, and the main empirical comparisons are reported without error bars or significance tests. These gaps currently limit the strength of the conclusions.","major_comments":[{"comment":"The central claim that the learned feedback model 'can generalize feedback to novel states without further querying the human trainer' is not directly supported. In the exploration use case, labels are collected under a uniformly random base policy (Eq. 7) and the model is applied to actions proposed by the RL policy (Eq. 8) without DAgger retraining during RL. Correct refinement at a state-action pair outside the label-collection support requires true generalization, but no held-out state evaluation, coverage analysis, or plot of prediction error versus distance from the training distribution is provided. The end-to-end success on billiards and navigation is also consistent with the alternative that the random base policy already covers the state regions visited during guided exploration. This is load-bearing because it is the property that distinguishes the method from prior relative-feedback work and justifies the label-efficiency advantage. Please add a direct evaluation of phi_theta's generalization, for example by validating on state-action pairs drawn from the RL policy's visitation distribution or by analyzing prediction error as a function of distance from the label-collection support.","section":"Section IV-B, Eqs. (7)-(8); Abstract; Section II-C"},{"comment":"The main quantitative claims are reported as means across five seeds without error bars, confidence intervals, or significance tests. This applies to the 'five times less labels' and 'almost ten times less labels' comparisons in Section VI-A, the 'slightly outperforms demonstrations' result in Figure 3, and the 'on par with residual learning' claim in Section VI-B. Given the high variance typical of DDQN training with sparse rewards, these differences may fall within seed noise. Please report per-seed results or confidence intervals, and where possible paired tests across the shared seeds.","section":"Figures 3-5; Tables I-II"},{"comment":"The adaptation contribution is advertised as adapting to 'changes in the environment or the user's preferences' without the locality qualifier, but the method relies on the assumption that the optimal action lies within tau_max of the base action (Eq. 6). Table II shows that when the environment change requires corrections beyond approximately ±60 degrees, the method fails while a PRM oracle with a ±180-degree bound succeeds. The paper honestly states this limitation in Section VI-C, but the abstract and the list of contributions should be reworded to make explicit that the adaptation is local in action space; otherwise the scope of the central claim is broader than the evidence. Alternatively, add an analysis characterizing when the locality assumption holds.","section":"Abstract; Section VI-C; Eq. (6); Table II"}],"minor_comments":[{"comment":"The real-world experiment uses a discrete three-option interface (optimal, counterclockwise, clockwise), while the method in Eq. (3) trains on continuous normalized gradients h as defined in Eq. (2). The paper does not specify how the discrete labels are converted to the continuous h used by the feedback model, nor how the strength parameter epsilon is applied to a discrete choice. This missing detail affects reproducibility of the real-world result.","section":"Section VI-D"},{"comment":"For the billiards task, the oracle is described as 'an agent trained using our baseline (double deep Q-learning)', but the paper does not explain how this oracle computes the improvement direction h, which is defined as the normalized gradient of Q in Eq. (2). Please specify the exact procedure used to generate the oracle labels.","section":"Section V-C"},{"comment":"Several task-dependent hyperparameters are mentioned in the main text but not listed in the Appendix: the guidance frequency ('every second episode'), the episode cutoff for guidance, the iterative refinement stopping threshold tau, and Kmax. Please include these values to make the experiments fully reproducible.","section":"Appendix"},{"comment":"There are numerous typographical errors, including 'reiforcement', 'comparible', 'In constrast', 'substitues', 'obtacles', 'perfoms', and 'slighly'. A careful proofreading pass is recommended.","section":"General"},{"comment":"The paper does not include an empirical comparison with Celemin et al. [8] even in a simple domain where their linear-policy method could be applied. While the authors state that the prior method is not compatible with deep RL, a small comparison in a suitable setting would strengthen the claim that the new method is more general.","section":"Section VI-A"},{"comment":"The real-world deployment is described only qualitatively ('handles sensor noise well', 'minimal collisions', 'minor scrapes'). Reporting quantitative metrics, such as success rate, collision rate, or time to reach goal across the obstacle courses, would make the deployment result more informative.","section":"Section VI-D"}],"recommendation":"major_revision","confidential_remarks":"I am sympathetic to this paper: the method is simple, the failure case in Table II is reported honestly, and the real-world deployment is a useful addition. The main concern is that the abstract's generalization claim is not directly tested, and the empirical comparisons lack error bars. Both issues are fixable with additional experiments and analysis, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"David,\n\nQuick take: this is a genuine, if modest, contribution to human-in-the-loop RL. What's new is learning a deep feedback model from relative labels and plugging it into off-policy deep RL—both for guiding exploration and for post-hoc adaptation. That's a real extension of Celemin et al., which was restricted to policy search. The paper does a decent job showing the method beats scalar feedback in label efficiency, does okay against demonstrations, and transfers to a real robot with 500 human labels.\n\nThe experiments are solid in structure: two sparse-reward tasks, multiple seeds, and an honest failure case in Table II where the method can't handle global action changes. That honesty is a strength, not a weakness; they know their method is for local tweaks.\n\nWhere the paper is soft is the generalization claim. The abstract says the feedback model 'can generalize feedback to novel states without further querying the human trainer.' But there's no direct test of that—no held-out states, no coverage analysis, no prediction error as a function of distribution shift. In the exploration use case, labels are collected under a uniform random base policy, then applied to the RL policy's actions. That works fine if the random policy covers the states the RL policy visits, which is likely in these simple sims. But the claim is broader than the evidence. The stress-test note you passed along lands on this correctly.\n\nThe tau_max clipping is the load-bearing assumption. The authors state it outright and show what happens when it breaks. That's an acknowledged limitation, not a hidden one.\n\nMinor complaints: no error bars in the figures, no released code, and a handful of hand-tuned hyperparameters. None of these are deal-breakers, but code would help reproducibility.\n\nOverall, this is a worthy paper for a serious referee. The method is practical, the comparisons are reasonable, and the limitations are clearly stated. I'd recommend accept-with-revisions if the authors can either directly test out-of-distribution generalization or soften the claim.","headline":"A practical, honestly-bounded method for learning from relative feedback in deep RL, with a generalization claim that is broader than the direct evidence supports.","tokens_in":11964,"tokens_out":2741,"would_cite":true,"duration_ms":28693,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Relative feedback—short directional hints like 'more to the left'—can be learned as an action-refinement model, cutting label needs 5-10x versus scalar feedback on sparse-reward tasks and enabling fast local policy adaptation.","keywords":["relative feedback","reinforcement learning","human-in-the-loop","action refinement","sparse rewards","policy adaptation","off-policy RL","quadruped navigation"],"falsifier":"Collect relative-feedback labels in an adaptation scenario where the true required correction is consistently larger than the clipping bound $\\tau_{\\max}$, then check whether refined actions still improve success; the paper's Table II already provides such a case (navigation needing $\\pm180^\\circ$ corrections), where the method worsens performance, so a clean test is whether any variant with these labels can recover the oracle's gains.","tokens_in":10973,"feed_emoji":"🎯","tokens_out":5767,"duration_ms":60175,"temperature":0.7,"pith_summary":"This paper introduces a way to use relative feedback—short directional hints such as 'more to the left'—in deep reinforcement learning. The authors train a deep network to predict the improvement direction for a given state-action pair, then apply that model iteratively to refine actions proposed by the agent. They show this guidance can steer exploration in sparse-reward domains, making off-policy RL several times more label-efficient than scalar feedback and competitive with demonstrations on one task. The same refinement module can adapt an already-trained policy to small environment changes or preference shifts, and 500 human labels collected in 40 minutes sufficed to bootstrap a real-world quadruped navigation policy. The central thesis is that relative feedback offers a practical middle ground: richer than scalar scores, cheaper than demonstrations, and learnable enough to generalize to novel states.","feed_headline":"Relative feedback cuts RL label needs 5-10x vs scalar scores","feed_subtitle":"A learned model turns 'more to the left' hints into action corrections that guide exploration and adapt trained policies.","key_machinery":"The machinery is a learned feedback model $\\phi_\\theta$: a deep network that takes the current state and action and outputs an action-space correction vector, trained by L2 regression toward $\\epsilon\\cdot h$, where $h$ is the trainer's normalized improvement direction. At inference it is applied iteratively to the base policy's action until refinements converge, and clipping keeps the refined action within a distance $\\tau_{\\max}$ of the base action when the user has prior knowledge that the optimal action is local. To prevent compounding errors from drifting off the training distribution, the model is retrained with a Dataset Aggregation loop that re-queries feedback on refined actions. This refinement module is separate from the policy, which lets the same tool bootstrap exploration—by refining actions from a uniform random base policy during off-policy training—and later adapt a deployed policy without touching its weights.","core_discovery":"On the paper's own terms, the central discovery is that relative feedback can be fully absorbed into a learned feedback model $\\phi_\\theta(s,a)$ trained to match a normalized improvement direction $h$ in action space, and this model can then replace the human trainer. The model is applied through the refinement recurrence $a_{k+1}=a_k+\\phi_\\theta(s,a_k)$, stopped by a convergence threshold or iteration cap, with optional clipping of refinements to a ball of radius $\\tau_{\\max}$ around the base action. Guided by this module, a double deep Q-learning agent learns sparse-reward billiards and quadruped navigation with 350-1000 relative labels, roughly 5-10 times fewer labels than scalar feedback for the same speedup. In post-hoc adaptation, the same module restores performance after sensor miscalibration, embodiment bias, or gravity changes about an order of magnitude faster than RL finetuning, matching a residual-learning baseline while requiring less precise corrections. The paper also reports that the method is suited to localized adjustments and fails when a task demands global action changes beyond its bound.","pith_inferences":["Because the refinement module is policy-agnostic, the same trained feedback model could be attached to any base controller—including classical planners—not just DDQN agents, turning relative feedback into a portable correction layer.","The paper's own Table II suggests a practical diagnostic: if the human's corrections consistently exceed the assumed $\\tau_{\\max}$, the task is outside the method's regime and full retraining or a new action space is needed; this could be measured during label collection.","The 5-10x label-efficiency gap over scalar feedback hints that richer-but-still-cheap input channels (e.g., ordinal or coarse directional commands) may be broadly underused in human-in-the-loop RL.","The real-world ANYmal result is qualitative evidence that offline-trained feedback models keep working under sensor noise and geometry mismatch; a quantitative extension would measure collision rate against baseline on multiple courses."],"forward_implications":["On sparse-reward tasks, a few hundred relative-feedback labels can replace thousands of scalar-feedback labels, making human-guided RL practical for nonexpert trainers.","Because the feedback model generalizes to novel states, the human trainer can stop after a short labeling session; the learned corrections then cover the state space.","For local adaptation, the method restores much of a policy's lost performance (e.g., from 72% to 82% success after lidar misalignment) an order of magnitude faster than RL finetuning.","The same 500-label procedure that bootstraps exploration in simulation transfers to a real ANYmal, suggesting deployment-ready policies can be produced without sim-to-real retraining.","When action corrections need to exceed the preset bound, e.g., more than $\\pm60^\\circ$ in navigation, the method at best preserves and at worst degrades performance."],"supporting_citations":[{"why":"Prior work on learning continuous-action policies from corrective feedback; the closest method that this paper extends to deep RL.","marker":"[11]"},{"why":"Combines relative feedback with policy search; its incompatibility with deep RL motivates the new method and the comparison setup.","marker":"[8]"},{"why":"Dataset Aggregation is used to retrain the feedback model on refined actions, preventing compounding errors.","marker":"[21]"},{"why":"Double deep Q-learning is the base off-policy RL algorithm used in all experiments.","marker":"[24]"},{"why":"Probabilistic roadmaps provide the navigation oracle that generates feedback labels and serves as an upper bound in Table II.","marker":"[25]"},{"why":"Manual feedback combined with MDP rewards is the scalar-feedback baseline that the label-efficiency comparison is built against.","marker":"[5]"},{"why":"Action advising with advice imitation is a demonstration-style guidance baseline used to frame the relative-feedback comparison.","marker":"[3]"},{"why":"The Orbit simulator provides the quadruped navigation environment for training and adaptation experiments.","marker":"[22]"},{"why":"The ANYmal platform is the real robot used in the real-world deployment experiment.","marker":"[23]"}],"fun_headline_variants":["Relative feedback: 5-10x fewer RL labels, faster adaptation","CueLearner turns 'more left' hints into 5-10x label savings","Relative feedback guides RL exploration: 5-10x label cut","Learn RL from relative hints: 5-10x fewer labels, quick adaptation","Human relative feedback slashes RL training labels by 5-10x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole method leans on the premise that the correct action lies inside a small box around the base policy's proposed action, because refinements are clipped to that region and the model is only trained on corrections of that size.","fun_headline_variants_meta":{"raw":{"variants":["Relative feedback: 5-10x fewer RL labels, faster adaptation","CueLearner turns 'more left' hints into 5-10x label savings","Relative feedback guides RL exploration: 5-10x label cut","Learn RL from relative hints: 5-10x fewer labels, quick adaptation","Human relative feedback slashes RL training labels by 5-10x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000639,"raw_usage":{"total_tokens":2947,"prompt_tokens":956,"completion_tokens":1991,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":1890}},"tokens_in":572,"tokens_out":1991,"duration_ms":17893,"temperature":1.0,"reasoning_tokens":1890,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:40:16.050951+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect relative-feedback labels in an adaptation scenario where the true required correction is consistently larger than the clipping bound $\\tau_{\\max}$, then check whether refined actions still improve success; the paper's Table II already provides such a case (navigation needing $\\pm180^\\circ$ corrections), where the method worsens performance, so a clean test is whether any variant with these labels can recover the oracle's gains.","supporting_citations":[{"cited_title":"An interactive framework for learning continuous actions policies based on corrective feedback,","cited_arxiv_id":null,"evidence_quote":"Prior work on learning continuous-action policies from corrective feedback; the closest method that this paper extends to deep RL."},{"cited_title":"Reinforcement learning of motor skills using policy search and human corrective advice,","cited_arxiv_id":null,"evidence_quote":"Combines relative feedback with policy search; its incompatibility with deep RL motivates the new method and the comparison setup."},{"cited_title":"A reduction of imitation learning and structured prediction to no-regret online learning,","cited_arxiv_id":null,"evidence_quote":"Dataset Aggregation is used to retrain the feedback model on refined actions, preventing compounding errors."},{"cited_title":"Combining manual feedback with sub- sequent mdp reward signals for reinforcement learning","cited_arxiv_id":null,"evidence_quote":"Manual feedback combined with MDP rewards is the scalar-feedback baseline that the label-efficiency comparison is built against."},{"cited_title":"Action advising with advice imitation in deep reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Action advising with advice imitation is a demonstration-style guidance baseline used to frame the relative-feedback comparison."},{"cited_title":"Anymal - a highly mobile and dynamic quadrupedal robot,","cited_arxiv_id":null,"evidence_quote":"The ANYmal platform is the real robot used in the real-world deployment experiment."}],"review_version":1}