{"id":"2ce81f49-e761-4fa0-a2a0-57fbe9cc1c0c","arxiv_id":"1908.08659","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Task-space impedance control as an RL action space reaches success thresholds in fewer training samples than joint torque, PD, or inverse dynamics control across three simulated manipulation tasks.","lead":"This paper compares four ways to specify robot actions for reinforcement learning and finds that task-space impedance control learns manipulation skills fastest. The result is a practical guide for choosing action spaces in robot learning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table I rests on a single run per condition; without seed variance the uniform ordering of action spaces is not statistically established.","rationale":"I agree with the reader's identification of the single-run design as the load-bearing weakness. The paper's central claim is an empirical ranking across six conditions, and Table I provides exactly one number per cell. Because RL experiments are highly sensitive to seed, the absence of any variance information means the uniform ordering could be a statistical fluke. This is not an internal inconsistency or a disagreement with consensus; it is a genuine evidential gap. The paper has strengths that make the result plausible: a consistent qualitative story about task-space impedance reducing exploration difficulty, detailed hyperparameters, and publicly available code. The concern is concrete and testable with additional runs, so the appropriate outcome is to keep the reader's CONDITIONAL verdict rather than raise or lower it. The test I propose directly targets the weakest point: if the ordering survives multiple seeds with non-overlapping intervals, the claim is substantially strengthened; if not, it should be hedged to 'preliminary evidence' without the word 'significantly.'","tokens_in":9871,"tokens_out":4032,"duration_ms":44141,"concrete_test":"Using the provided code and hyperparameters, run each of the 24 conditions (3 tasks x 4 action spaces x 2 algorithms) with at least 5 different random seeds, recording the number of environment steps to reach the stated success criteria. Compute the mean and standard deviation (or bootstrap confidence intervals) for each condition, and test whether the impedance controller remains the fastest in all six task/algorithm cells and whether its margin over the next-best controller exceeds the pooled variance. If the ordering flips in any cell or the margins fall inside the noise, the central claim of uniform sample-efficiency gains is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that an impedance-controller action space 'significantly reduces' the samples needed to reach success criteria across all tasks and algorithms. The evidence is Table I, which reports a strict ordering (impedance fastest in all six task/algorithm cells). However, the paper reports no repeated seeds, no variance estimates, and no seed counts for any of the 24 conditions. In RL, sample-efficiency curves are notoriously sensitive to random initialization and environment stochasticity; a single run per condition cannot distinguish a robust ordering from one that arises by chance. The abstract uses 'significantly reduces' and Section VI states 'we found the impedance controller action space to have learned the fastest' without statistical qualification. The paper's own language ('preliminary evidence') tempers the claim, but the empirical comparison as presented does not support the strength of the conclusion. This is a methodological gap about the evidence, not a dispute about the plausibility of the result, and it is addressable because the code and hyperparameters are public.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares four action spaces (direct torque, joint-space PD, inverse dynamics, and task-space impedance) for reinforcement learning of three simulated manipulation tasks (peg insertion, hammering, pushing) with two algorithms (PPO and SAC). The central claim is that defining the action space as references to a task-space impedance controller substantially improves sample efficiency, with a consistent ranking across all six task/algorithm combinations. The evidence is Table I, which reports the number of steps to reach task-specific success thresholds, together with qualitative observations about learned behavior. The paper frames its contribution as preliminary evidence that incorporating model-based controllers, particularly impedance control, into the action space reduces sample complexity for dynamic manipulation.","tokens_in":10039,"tokens_out":2901,"duration_ms":30345,"significance":"If the claimed effect is real, the paper provides a practically useful and systematic comparison for a design choice that is often made ad hoc: how to parameterize the action space for model-free RL on manipulation tasks. Its scope is broad (four action spaces, three tasks, two RL algorithms), the code and hyperparameters are released, and the finding that impedance references outperform direct torque and joint-space PD is plausible and consistent with prior work on residual and model-based control. The main limitation is statistical: with a single run per condition and no variance estimates, the paper cannot yet support the strength of its abstract and conclusion claims. The manuscript is a reasonable pilot study, but the present evidence is too thin for the current wording.","major_comments":[{"comment":"The central claim rests entirely on Table I, but each of the 24 conditions appears to have been run exactly once, with no repeated seeds, no error bars, and no variance estimates. In deep RL, sample-efficiency curves are strongly seed-dependent, so a strict ordering observed in single runs cannot distinguish a robust effect from stochastic variation. The abstract's 'significantly reduces' and Section VI's 'we found the impedance controller action space to have learned the fastest' are therefore not supported by the evidence as presented. Please report multiple seeds (at least 5) with medians and interquartile ranges, or explicitly downgrade the language to 'in the runs we performed' throughout the abstract and conclusions.","section":"Section VI, Table I and Abstract"},{"comment":"The comparison is partly a comparison of the authors' tuning effort rather than of the action spaces alone. The PD proportional gains (Kp in Eq. (3)), the impedance stiffness K and damping B in Eq. (11), and the derivative gains are hand-chosen, but their numerical values are not reported anywhere. Section VI even states that increasing PD gains by an order of magnitude would harm compliance, which shows that the ranking is sensitive to these choices. Please report the exact gain values and the tuning procedure used, or state clearly that the conclusions are conditional on the specific gains selected.","section":"Section IV and Appendix"},{"comment":"The success thresholds (80% average nail depth, 80% insertion success rate, 80% of normalized pushing distance) are arbitrary, and the sample counts in Table I are measured against these thresholds. Different thresholds could easily change the relative ordering, especially where the values are close (e.g., SAC insertion: impedance 0.39 vs. ID 0.47, or PPO pushing: impedance 0.21 vs. PD 0.23). Please justify the threshold choices or provide a sensitivity analysis over a range of thresholds.","section":"Section VI, success criteria"}],"minor_comments":[{"comment":"The SAC pushing row reads '0.18 0.24 0.90 * 1'; there appear to be five entries for four columns, and the footnote says the torque action space never crossed the threshold. Please fix the table formatting so the PD and torque entries are unambiguous.","section":"Table I"},{"comment":"The text says 'The episode lengths were capped at 2 seconds for object pushing and peg insertion and 3 seconds for hamming.' The word 'hamming' should be 'hammering'.","section":"Appendix"},{"comment":"Equation (10) applies the pseudoinverse J^+ to a sum that includes (K(x_des - x(q)) + \\dot{x}_des - B J \\dot{q} - \\dot{J} \\dot{q}); the parentheses are unbalanced and the notation mixes Cartesian and joint quantities. The simplified law in Eq. (11) then drops the \\dot{J}\\dot{q} term. Please clarify the derivation or add a sentence explaining that Eq. (11) is the implemented approximation.","section":"Section IV-D, Eq. (10)"},{"comment":"Figure 2 is referenced as showing learning trends, but the figure itself contains only qualitative stills without learning curves or axes. Either replace it with actual learning curves or remove the reference in the text.","section":"Figure 2"},{"comment":"The phrase 'inject very little bias into the learning problem' is used for torque and PD control; it would be clearer to specify what kind of bias is meant, for example, the controller does not impose a particular dynamic response.","section":"Section II"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is likely a short workshop-style contribution; the single-seed evidence and unreported tuning parameters are the principal barriers to publication in a regular archival venue. The code release and the breadth of the comparison are strengths, and the conclusion is plausible, so I do not recommend rejection. The authors should be encouraged to run multiple seeds and be much more careful with the strength of their claims. I would also note that the paper may be a better fit for a workshop than a full journal if the experiments are not expanded."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nYou should know about this one: a clean, well-scoped empirical comparison of four action spaces (torque, PD, inverse dynamics, task-space impedance) for three simulated manipulation tasks with PPO and SAC. The headline finding — that impedance references learn fastest, often by 3–10x — is plausible and consistent across all six task/algorithm cells. It extends Peng and van de Panne's locomotion result to manipulation and is the first systematic comparison to include inverse dynamics and impedance controllers. The paper is transparent: code is public, hyperparameters are in the appendix, and the tasks cover precision, impulse, and contact-rich behaviors. The qualitative explanations (task-space actions align with reward geometry; ID compensates inertial coupling) are sensible and not over-sold.\n\nThe soft spots are real, though. The biggest is the evidence base for the ranking: every condition appears to be a single run. No seeds, no error bars, no variance estimates. In RL, sample-efficiency curves are noisy; a strict ordering of six cells could easily shuffle under reseeding. The paper's own language wavers between \"preliminary evidence\" (intro) and \"significantly reduces\" (abstract and conclusions), and the conclusions lean on the stronger phrasing. The stress-test note is accurate: the central claim is not statistically established, even if the effect sizes look large.\n\nSecond, the impedance controller gains (stiffness K, damping B) are hand-chosen and not reported. Same for the PD gains. The comparison therefore encodes the authors' tuning effort, and without those numbers a reader cannot judge whether the impedance advantage is due to the action-space framing or to particularly good gain selection. This is a free-parameter concern, not a fatal one — the point is exactly that the controller structure matters — but it should be stated transparently.\n\nThird, two tasks disable gravity to give torque control a fair shot, which limits generality. The pushing task uses gravity compensation. These are acknowledged, but they narrow the claim.\n\nThe paper deserves a serious referee. It is a useful, honest empirical contribution that practitioners will cite for action-space design. The right path is revision with repeated seeds (at least 5–10 per condition), interquartile ranges or standard errors, and reporting the controller gains. That is tractable, and the code is already out there.\n\nIf it were my call: conditional accept, with those revisions requested. Worth a slot at a reading group, and I would cite it in future work on manipulation RL.\n\nBest,\n[You]","headline":"Useful empirical comparison of action spaces for manipulation RL, but the central ordering rests on single runs with no uncertainty estimates.","tokens_in":10558,"tokens_out":2321,"would_cite":true,"duration_ms":23217,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Impedance actions beat torque in every manipulation task tested","keywords":["reinforcement learning","manipulation","action spaces","impedance control","inverse dynamics control","sample efficiency","proximal policy optimization","soft actor-critic"],"falsifier":"Re-run the same three tasks and two algorithms with, say, ten seeds per action space and report the distribution of steps to success; if the impedance-reference action space is not fastest in most cells after averaging, the claimed uniform ordering is not supported.","tokens_in":9663,"feed_emoji":"🤖","tokens_out":7176,"duration_ms":70978,"temperature":0.7,"pith_summary":"The paper asks whether the action space of a reinforcement learning policy—what the policy's output actually commands—matters as much as the learning algorithm. It argues that defining actions as reference inputs to a task-space impedance controller makes dynamic manipulation tasks dramatically more sample-efficient than outputting joint torques or joint-space PD references. Across three simulated tasks (peg insertion, hammering, and pushing) and two algorithms (PPO and SAC), the impedance action space reached success criteria fastest in all six conditions, often by a factor of several. If this holds, a large share of the sample cost of learning manipulation can be avoided by better wiring the policy output through existing model-based controllers, without changing rewards or algorithms.","feed_headline":"Impedance actions beat torque in every manipulation task tested","feed_subtitle":"The action-space choice, not the algorithm, drove sample efficiency: impedance references beat torques by up to 12x.","key_machinery":"The central object is the action space of the Markov decision process, specifically the choice to make actions references $(x_{\\mathrm{des}}, \\dot{x}_{\\mathrm{des}})$ for a task-space impedance controller. This is a low-level controller that drives the end effector to behave like a mass-spring-damper toward a desired pose and velocity while using an inverse-dynamics model to compensate for the arm's inertial, Coriolis, and gravitational terms; a null-space damping term handles redundant degrees of freedom. Its work in the paper is to replace the policy's burden of learning joint-level dynamics and inverse kinematics with a much simpler problem: nudging a compliant task-space target, so that exploration is spent on the task rather than on robot mechanics.","core_discovery":"On its own terms, the paper claims a consistent ordering of learning speed across tasks and algorithms: impedance references are fastest, inverse dynamics next in most cases, then joint PD, then direct torque. The authors write that in all experiments the impedance controller action space learned fastest, often followed by inverse dynamics, PD, and then torque. They attribute the advantage to two mechanisms: the impedance controller provides inherent mechanical compliance, which matters for contact-rich tasks such as pushing a block without tipping it, and because actions are specified in task space, the policy's output aligns directly with reward-relevant quantities, so it does not need to solve inverse kinematics during exploration. The margin can be large: in hammering with SAC, impedance reached the success criterion in 0.012 million steps versus 0.145 million for torque.","pith_inferences":["The task-space alignment mechanism suggests a testable extension: if the reward is re-expressed in joint coordinates, the impedance advantage should shrink, and if the reward is purely task-space, it should grow.","A natural next experiment is to make the impedance gains themselves learnable or state-dependent, since the paper keeps them fixed and does not explore variable impedance.","If the same ordering transfers to physical hardware, the impedance action space may also reduce the sim-to-real gap, because the learned policy commands compliant task-space goals rather than raw torques while the low-level controller absorbs dynamic effects.","The paper's framing implies that the largest remaining sample-efficiency gains in manipulation may come from choosing the right interface between the policy and the controller rather than from further advances in RL algorithms alone."],"forward_implications":["Practitioners can reduce the sample budget for learning manipulation tasks by choosing impedance-reference actions over torque or PD references, without changing the reward function or learning algorithm.","Exploiting known inertial and kinematic models of the arm helps even when contact dynamics are not modeled; the gains come from using available structure, not from modeling the environment.","Inverse dynamics control is a cheaper middle ground that outperforms PD control in most conditions, particularly in velocity-dominated tasks such as hammering.","The action-space choice changes qualitative behavior: policies trained in compliant action spaces produce slower, controlled motions, while torque policies tend to collide aggressively with the environment.","A fair comparison of torque and PD baselines requires careful engineering, including gravity compensation and mass scaling, so naive baselines may underestimate their sample cost."],"supporting_citations":[{"why":"Supplies the PPO algorithm used as one of the two learning methods in every comparison cell.","marker":"[14]"},{"why":"Supplies the SAC algorithm used as the off-policy comparison method.","marker":"[15]"},{"why":"Extends the SAC method with the improvements, including automatic temperature tuning and double Q-learning, used in the experiments.","marker":"[16]"},{"why":"Provides the earlier action-space comparison for locomotion that motivates testing PD control over torque and situates the manipulation-specific result.","marker":"[31]"},{"why":"Provides the MuJoCo physics simulator in which all three manipulation tasks are implemented.","marker":"[37]"},{"why":"Supports the value of learning residuals to model-based controllers, cited to justify exploiting available models.","marker":"[12]"},{"why":"Supports the practice of adding gravity compensation to torque-control learning, which the paper uses to make the torque baseline viable.","marker":"[11]"},{"why":"Another source for gravity-compensated torque control, used to strengthen the torque baseline in the comparison.","marker":"[5]"}],"fun_headline_variants":["Impedance actions beat torque for RL manipulation, up to 12x fewer samples","Action space choice, not algorithm, drives RL sample efficiency","Impedance control needs far fewer RL samples for delicate manipulation tasks","Task-space impedance speeds RL manipulation over torque and joint PD"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking in Table I is based on one untested run per condition, so the claim assumes those single runs are representative of each action-space and algorithm combination.","fun_headline_variants_meta":{"raw":{"variants":["Impedance actions beat torque for RL manipulation, up to 12x fewer samples","Action space choice, not algorithm, drives RL sample efficiency","Impedance control needs far fewer RL samples for delicate manipulation tasks","Task-space impedance speeds RL manipulation over torque and joint PD"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000223,"raw_usage":{"total_tokens":1427,"prompt_tokens":888,"completion_tokens":539,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":464}},"tokens_in":504,"tokens_out":539,"duration_ms":6257,"temperature":1.0,"reasoning_tokens":464,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:32:29.601982+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same three tasks and two algorithms with, say, ten seeds per action space and report the distribution of steps to success; if the impedance-reference action space is not fastest in most cells after averaging, the claimed uniform ordering is not supported.","supporting_citations":[{"cited_title":"Learning Locomotion Skills Using DeepRL: Does the Choice of Action Space Matter?,","cited_arxiv_id":null,"evidence_quote":"Provides the earlier action-space comparison for locomotion that motivates testing PD control over torque and situates the manipulation-specific result."},{"cited_title":"MuJoCo: A physics engine for model-based control,","cited_arxiv_id":null,"evidence_quote":"Provides the MuJoCo physics simulator in which all three manipulation tasks are implemented."},{"cited_title":"Learning Force Control Policies for Compliant Manipulation,","cited_arxiv_id":null,"evidence_quote":"Supports the practice of adding gravity compensation to torque-control learning, which the paper uses to make the torque baseline viable."}],"review_version":1}