{"id":"8595e6e4-a211-4360-b089-d64e5c8dc241","arxiv_id":"2412.13630","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A small, bounded residual policy, trained online with SAC and introduced gradually, refines frozen Behavior Transformer and Diffusion Policy models to near-perfect success on ManiSkill and Adroit.","lead":"This paper wraps a frozen robot policy with a small online-trained 'decorator' policy that adds bounded corrections during interaction. On eight simulated manipulation tasks it lifts two standard imitation learning policies to near-perfect success while keeping their smooth motions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The unquantified boundary of Assumption 3 is the load-bearing gap: if base-policy failures occur far from the demonstration manifold, an α-bounded residual cannot recover, so the 'near-perfect on all tasks' claim is stronger than the evidence.","rationale":"The Reader's conditional verdict is appropriate. The main experimental evidence is broad, the ablations are honest, and the analysis of why fine-tuning baselines fail (App. F, including the Cal-QL setup interpolation) shows care rather than cherry-picking. The single place where the central claim is stronger than the evidence is the unquantified boundary of Assumption 3. This is not an internal inconsistency; the paper explicitly states the assumption. However, because the headline is an unconditional 'near-perfect on all tasks' claim, the missing quantification of where base-policy failures fall relative to the demonstration manifold is load-bearing. The proposed weak-base test would either confirm that the assumption is robust in practice or force the authors to qualify the scope of their central claim. Missing code and per-task tuning remain secondary reproducibility concerns but do not change the conditional verdict.","tokens_in":28456,"tokens_out":8848,"duration_ms":92820,"concrete_test":"Train a base Behavior Transformer on 100 randomly chosen StackCube demonstrations instead of 1000, producing a policy whose failures occur earlier and in states farther from the demonstration support; then run Policy Decorator with the same hyperparameters as Table 8 (α=0.03, H=1M) and ten seeds. If final success remains close to the weak base success instead of rising to near-100%, Assumption 3 is a substantive scope condition and the headline claim should be qualified; if PD still recovers near-perfect success, the assumption is robust in practice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Assumption 3 (Sec. 3) is the load-bearing premise of the headline claim: the method can only improve a base policy if its failures are locally recoverable by a bounded action-space correction. The paper motivates this with Fig. 3 ('small adjustments can bring deviated trajectories back on track') and reports one low-performing-checkpoint experiment (App. D.2), but it never measures where base-policy failures actually occur relative to the demonstration manifold. If compounding error or object variation (TurnFaucet, PushChair) drives the base policy into states far from demonstrated states, an α-bounded residual cannot return the trajectory to a successful region, and with sparse rewards no learning signal exists. The Sec. 6 assertion that 'only a few critical states need adjustment' is a post-hoc interpretation, not a measurement. Consequently, 'near-perfect on all tasks' is a claim about an unmeasured subclass of base policies; the empirical successes support the mechanism but do not delineate its boundary. This does not invalidate the method within its stated scope, but it means the central claim's outer boundary is unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Policy Decorator, a method for improving a frozen, offline-trained imitation-learning policy through online reinforcement learning of a small residual policy. The residual action is added to the base-policy action, bounded by a hyperparameter alpha, and introduced through a progressive schedule that interpolates between the base policy alone and the combined policy. The method is evaluated on eight ManiSkill and Adroit tasks with Behavior Transformer and Diffusion Policy base policies, against fine-tuning and non-fine-tuning baselines, with ten seeds and extensive ablations. The headline claim is that Policy Decorator improves both base policies to near-perfect success on all evaluated tasks while preserving the smooth motion of the imitation policy.","tokens_in":28695,"tokens_out":6665,"duration_ms":61226,"significance":"If the empirical results hold, the paper makes a practically useful contribution: large, non-differentiable imitation policies can be improved online without backpropagating through the large model, which sidesteps both architectural incompatibility and the computational cost of fine-tuning. The evaluation is unusually thorough for this area: eight tasks, two base policy families, ten seeds, several strong baselines, a comparison with DPPO, visual-observation experiments, and detailed appendices analyzing why fine-tuning and non-fine-tuning baselines fail. The paper also provides concrete hyperparameter tables and a careful validation of the Cal-QL baseline implementation in Appendix F.3. The main caveats are the per-task tuning of the two key hyperparameters and the unquantified boundary of the assumption on base-policy quality; these limit the strength of the general claim but do not undermine the core demonstration.","major_comments":[{"comment":"The headline results in Figs. 6-8 are produced with per-task values of the two key hyperparameters, the residual bound alpha and the schedule length H. Table 8 lists alpha from 0.03 to 0.8 and H from 30K to 8M across tasks, so there is no cross-task fixed setting. The ablations in Figs. 10-11 show robustness only on StackCube and PushChair, and even there the robust ranges differ between the two tasks. The abstract and Sec. 5.3 state that Policy Decorator 'improves the performance of both offline-trained policies to a near-perfect level on all tasks,' which is stronger than the evidence: the results demonstrate per-task tuned success, not general success without task-specific tuning. Please add an experiment with a fixed (alpha, H) across all tasks, or explicitly qualify the claim to per-task tuned settings.","section":"Sec. 5.4.2, Table 8"},{"comment":"The method's load-bearing premise is that base-policy failures are correctable by a bounded, per-step action residual. Assumption 3 formalizes this only as the base policy exhibiting 'reasonable initial performance,' and the paper never measures where base-policy failures actually occur relative to the demonstration-covered state manifold. Appendix D.2 tests one low-performing BeT checkpoint on TurnFaucet, which is a useful start, but it does not systematically vary base-policy quality or quantify the state deviation that an alpha-bounded residual can recover. The claim 'near-perfect on all tasks' is therefore demonstrated only for the particular base policies and tasks selected, and its boundary remains uncharacterized. Please add a systematic study of base-policy quality (e.g., checkpoints of varying success rates, with analysis of failure-state distances and required correction magnitudes), and soften the general claim accordingly.","section":"Sec. 3, Assumption 3"},{"comment":"The paper advertises the preservation of smooth, natural motion as a key advantage over pure RL policies, but the evidence is qualitative: videos and the multi-modality analysis in Appendix J. No quantitative smoothness metric is reported across tasks or seeds. Since this property is part of the paper's central contribution, please add a measurable proxy such as action jerk, action total variation, or a power-spectral measure, and compare the refined policy with both the base policy and a pure-RL baseline.","section":"Sec. 5.5"}],"minor_comments":[{"comment":"The listed parameters for the illustrative bimodal distribution, mu1 = mu2 = 0.5 and sigma1 = sigma2 = 1, make the two Gaussian components identical, so the 'bimodal' base distribution is actually unimodal. Please correct the parameter values or the description so that the example matches the claim.","section":"Appendix J.1"},{"comment":"The label 'Resiudual RL' contains a typo and should read 'Residual RL.'","section":"Fig. 17"},{"comment":"The row 'Adroit: Door (Diffusion Policy, visual)' has no corresponding curve in Fig. 8. Please clarify whether this experiment was run and, if so, where it is reported.","section":"Appendix B.2, Table 8"},{"comment":"The phrase 'both offline-trained policies to a near-perfect level on all tasks' is too broad given that for Diffusion Policy, StackCube and Door were not tested because the base policy already achieves near-optimal performance. Please state explicitly that the claim concerns the tasks on which improvement was evaluated, or distinguish 'improved' from 'maintained near-perfect.'","section":"Abstract and Sec. 5.3"},{"comment":"The bullet list references Appendix D.1, D.2, and D.3 without their titles; including the titles in the bullets would improve readability.","section":"Sec. 5.4.3"}],"recommendation":"major_revision","confidential_remarks":"The empirical work is substantial and the baseline analysis is careful; I found no indication of misconduct. The main risk is overclaiming: the per-task hyperparameter tuning and the unquantified Assumption 3 boundary should be addressed either by additional experiments or by sharp qualifications of the headline claims. If the authors add a fixed-hyperparameter experiment or reformulate the claims, the paper would be a strong candidate for acceptance. The Cal-QL setup validation in Appendix F.3 is exemplary and should be kept."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Policy Decorator is a solid empirical systems paper: it wraps a frozen imitation policy with a small SAC-trained residual policy, bounds the residual action by alpha, and ramps up its influence via a progressive schedule H. The combination is not conceptually new—residual RL has been around—but the systematic demonstration on Behavior Transformer and Diffusion Policy across eight ManiSkill/Adroit tasks, with ten seeds and a long list of baselines, is real evidence. The paper is honestly written, the ablations are informative, and the failure analysis of fine-tuning baselines (random critic, long horizon) actually strengthens the case for the method.\n\nThe central claim—that a bounded residual can refine a base policy to near-perfect success while preserving smooth motion—is supported by the data. The stress-test note worries that Assumption 3 (base policy has reasonable initial performance) is unquantified: if failures occur far from the demonstration manifold, an alpha-bounded residual can't recover. That is a fair boundary condition, but the paper states it clearly, includes a low-performing checkpoint experiment, and does not oversell universality. It does however say 'near-perfect on all tasks' when two Diffusion Policy tasks were skipped because the base policy was already near-perfect; that is the claim being slightly stronger than the evidence, though not by much.\n\nSoft spots: the two key hyperparameters alpha and H are tuned per task in Table 8. The paper shows robustness within ranges, but the per-task values suggest tuning is needed for each new environment. The RLPD baseline is softened by omitting the Q-ensemble and high UTD ratio, with justification, but it weakens the comparison. No code is released, which makes reproduction harder.\n\nWho it's for: people working on improving large imitation policies, residual RL, or fine-tuning. It deserves a serious referee; the experiments are extensive and the method is likely to be useful. I'd recommend accepting it with revisions that address the hyperparameter sensitivity discussion and release code.","headline":"Solid empirical paper on residual RL for large imitation policies; the per-task hyperparameters and softened RLPD baseline are the main soft spots.","tokens_in":29184,"tokens_out":2002,"would_cite":true,"duration_ms":18261,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A small residual policy trained online with sparse rewards can refine frozen imitation policies to near-perfect performance on eight manipulation tasks while preserving smooth motion.","keywords":["residual policy","online reinforcement learning","imitation learning","behavior cloning","model-agnostic refinement","controlled exploration","sparse rewards","robot manipulation"],"falsifier":"Take a base policy whose rollouts on a target task land mostly in states far from the demonstration state distribution, measurable by the density of demonstrations around the states the policy visits. Run Policy Decorator with the paper's settings: if success does not rise substantially, or if the learned residual saturates at the $\\alpha$ bound on most steps while reward stays near zero, the central 'small adjustments bring deviated trajectories back on track' premise is falsified for that policy.","tokens_in":28264,"feed_emoji":"🤖","tokens_out":11403,"duration_ms":90298,"temperature":0.7,"pith_summary":"Policy Decorator claims that a large, offline-trained imitation policy can be improved through online interaction without changing any of its weights. The paper's proposal is to learn a small residual policy—a per-step correction added to the base policy's action—using reinforcement learning with only the task's sparse success signal. Its argument rests on the observation that imitation policies fail mainly through compounding error: small deviations that a bounded correction can pull back onto the demonstration-covered path. With residual actions squashed to $(-\\alpha,\\alpha)$ and introduced through a schedule that starts from the base policy alone, the method reaches near-perfect success on the tasks it evaluates with low-dimensional state observations, for both Behavior Transformer and Diffusion Policy bases, while preserving the smooth, multi-modal motion of the imitation policy.","feed_headline":"Learned patch lifts frozen robot policies to near-perfect success","feed_subtitle":"A small online residual policy fixes imitation-learned robots without retraining the large model, keeping motion smooth.","key_machinery":"The load-bearing object is the bounded residual policy combined with the progressive exploration schedule. The residual is a small Gaussian network (MLP for state observations, CNN for images) trained by SAC; after a $\\tanh$ squashing function and scaling by $\\alpha$, its output lies in $(-\\alpha,\\alpha)$, so the correction is small by construction. The behavior policy executes $\\pi_{\\text{base}}(s)$ with probability $1-\\epsilon$ and $\\pi_{\\text{base}}(s)+\\pi_{\\text{res}}(s)$ otherwise, with $\\epsilon$ increasing linearly from 0 to 1 over $H$ environment steps. This design keeps the agent inside states where the sparse success signal is still reachable, which is what makes online refinement sample-efficient; because the base policy is only queried and never differentiated, the method works for any policy architecture that maps observations to actions.","core_discovery":"The central claim is that a frozen imitation policy $\\pi_{\\text{base}}$ can be wrapped, decorator-style, by a small learnable residual policy $\\pi_{\\text{res}}$ trained with Soft Actor-Critic on the environment's sparse success reward; the action executed in the environment is $\\pi_{\\text{base}}(s)+\\pi_{\\text{res}}(s)$. Two controls make this stable: the residual output is $\\tanh$-squashed and scaled by $\\alpha$ so that the combined trajectory stays near the base trajectory, and a progressive schedule mixes the residual in gradually, using the base policy alone at the start and increasing the probability of adding the residual over time. The paper reports near-perfect success on every task it evaluates with low-dimensional state observations (for Diffusion Policy, StackCube and Adroit Door were skipped because the base policy was already near-optimal), strong results with image observations, and consistent superiority over fine-tuning baselines (LoRA-based SAC, DIPO, Cal-QL, RLPD, ROT) and non-fine-tuning baselines (JSRL, Residual RL, FISH). The refined policy also keeps the base policy's smooth motion and, in the Behavior Transformer case, its multi-modal action distribution.","pith_inferences":["The paper's assumptions imply a cheap rejection test: measure how far a base policy's rollouts wander from the demonstration manifold before investing in online refinement; success is only promised for policies whose failures are small local deviations.","A natural extension is state-dependent residual bounds: replacing the global $\\alpha$ with per-state or per-action-dimension caps could improve sample efficiency on high-precision tasks without changing the method.","Because the base policy is never modified, the decorator framing suggests compositional refinement—several residuals trained for different objectives or constraints could be stacked or combined without retraining the base.","The preservation of multi-modality should degrade as $\\alpha$ grows large relative to the mode separation of the base policy; the paper's $\\alpha$ ablations are consistent with this prediction, and a direct test would compute the combined action distribution's modality for increasing $\\alpha$."],"forward_implications":["Any frozen policy with reasonable initial performance—Behavior Transformer, Diffusion Policy, MLP, RNN, or CNN—can be improved online using only sparse success signals, without gradient access to the base model.","Fine-tuning large imitation policies, which the paper finds prone to early unlearning and long-horizon reward starvation, can be replaced by forward-only refinement that costs a fraction of the wall-clock time (7h23m vs 33h52m on StackCube with Behavior Transformer).","Refinement transfers to novel objects: on TurnFaucet and PushChair, base policies trained on source objects improved online on target objects not present in the demonstrations.","The refined policy keeps the smooth, multi-modal motion of imitation learning rather than the jerky motion of pure RL policies, so large improvements in success need not sacrifice motion quality.","The recipe is not tied to one RL algorithm or observation modality: it also works with PPO and with image observations, and on low-performing base checkpoints."],"supporting_citations":[{"why":"Supplies the compounding-error diagnosis that motivates why small corrections can refine a base policy.","marker":"(Ross et al., 2011)"},{"why":"Establishes residual reinforcement learning on top of a base policy, the foundation this work extends with controlled exploration.","marker":"(Johannink et al., 2019)"},{"why":"Provides the Soft Actor-Critic algorithm used to train the residual policy.","marker":"(Haarnoja et al., 2018)"},{"why":"Inspires the epsilon-greedy progressive schedule that gradually introduces residual actions.","marker":"(Mnih et al., 2015)"},{"why":"Supplies the Behavior Transformer base policy, whose non-differentiable action clustering makes RL fine-tuning non-trivial.","marker":"(Shafiullah et al., 2022)"},{"why":"Supplies the Diffusion Policy base with receding-horizon control, the second model class the method refines.","marker":"(Chi et al., 2023)"},{"why":"Provides the Adroit benchmark and the 25 human-teleoperation demonstrations used for its four tasks.","marker":"(Rajeswaran et al., 2017)"},{"why":"Provides the ManiSkill2 benchmark, tasks, and demonstrations, including the novel-object splits used online.","marker":"(Gu et al., 2023)"}],"fun_headline_variants":["Small residual policy perfects frozen robot policies","Tiny patch lifts frozen robot policies to near-perfect","Decorator method refines large robot policies online","Model-agnostic residual policy boosts imitation learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The base policy must already be reasonably competent, since the method only learns small per-step corrections: if the policy has drifted into states far outside its demonstration distribution, a bounded action-level patch cannot recover and no sparse success signal will be available to train the residual.","fun_headline_variants_meta":{"raw":{"variants":["Small residual policy perfects frozen robot policies","Tiny patch lifts frozen robot policies to near-perfect","Decorator method refines large robot policies online","Model-agnostic residual policy boosts imitation learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000256,"raw_usage":{"total_tokens":1573,"prompt_tokens":941,"completion_tokens":632,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":572}},"tokens_in":557,"tokens_out":632,"duration_ms":6067,"temperature":1.0,"reasoning_tokens":572,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T12:56:31.096474+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a base policy whose rollouts on a target task land mostly in states far from the demonstration state distribution, measurable by the density of demonstrations around the states the policy visits. Run Policy Decorator with the paper's settings: if success does not rise substantially, or if the learned residual saturates at the $\\alpha$ bound on most steps while reward stays near zero, the central 'small adjustments bring deviated trajectories back on track' premise is falsified for that policy.","supporting_citations":[],"review_version":1}