{"id":"96167fdd-ed05-4417-a1a7-ef036fccdbb2","arxiv_id":"2510.12363","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Pretraining an inverse dynamics model on task-agnostic exploration data and using its weights to initialize PPO actor and critic improves sample efficiency by ~37% and final performance by ~7% across nine simulated locomotion tasks.","lead":"This paper tests a way to give robot control policies a head start: first train a model that predicts which motor commands produce a desired body movement, then use its weights to initialize reinforcement learning. Across nine simulated locomotion tasks on three robots, the warm start cut training iterations needed to reach a performance target by about 37% and raised final performance by about 7% on average.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 36.9% sample-efficiency claim rests on a threshold metric that can count an initial-performance advantage as 'convergence'; the reported improvement may be partly an artifact.","rationale":"The reader's weakest assumption is task-agnostic coverage of the pretraining data. That is a plausible concern, but the paper's own quantitative experiments include parkour tasks on terrains unseen in pretraining, and the pretrained method never performs worse than random init on those tasks, so the coverage worry is partially addressed empirically. A more load-bearing issue is the sample-efficiency metric itself. The central strongest claim pairs a +7.3% final-performance gain with a +36.9% sample-efficiency gain; the latter is computed against 90% of the baseline's final performance, a fixed threshold that can be reached at iteration 0 by a method with a better start. This makes the 36.9% number difficult to interpret as sample efficiency. A reanalysis with own-final-threshold or AUC metrics would settle whether the sample-efficiency claim is real or an artifact of the threshold. The final-performance comparison is more robust, so the overall verdict should remain CONDITIONAL rather than moving to REJECT: the method likely provides some benefit, but the headline quantitative claim needs a metric correction or justification. I also note a secondary reporting inconsistency: the abstract states 36.2%/4.3% while Section 5.4/Table 1 report 36.9%/7.3%; this should be reconciled, though it does not by itself overturn the central result.","tokens_in":23824,"tokens_out":12210,"duration_ms":110822,"concrete_test":"Recompute sample efficiency for all 9 tasks using two alternative definitions from the raw learning curves: (a) iterations to reach 90% of each method's own final performance (e.g., mean of the last 10% of training), and (b) area under the training curve up to a common iteration budget. If the average pretrained-vs-random-init advantage under (a) or (b) is substantially smaller than 36.9%, the headline sample-efficiency claim should be revised; if it remains close to 37%, the concern is resolved.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 5.4 defines 'Num. iters. to converge' as the iteration count needed to reach 90% of the final performance of the PIDM (Random Init) baseline, not 90% of the evaluated method's own final performance. This is a fixed low threshold: any method whose initial or early performance already exceeds 0.9 × baseline_final is classified as 'converged' at iteration 0, regardless of whether its own learning curve is still climbing. Because PIDM (Pretrained) has higher final performance than PIDM (Random Init) on most tasks in Table 1, the threshold is often below the pretrained method's early performance, so the reported 36.9% improvement conflates a better starting point with faster convergence. The training curves in Figure 6 show that pretrained runs can start with an advantage, but the aggregate metric does not separate this initial gain from genuine convergence-speed gains. Thus the strongest claim's sample-efficiency component is not yet established by the chosen metric; the final-performance component (+7.3%) is less affected because it compares end-of-training values, though it also inherits any issues from the random-init baseline's instability and excluded runs.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript proposes a task-agnostic pretraining method for actor-critic RL in legged locomotion. A Proprioceptive Inverse Dynamics Model (PIDM) is trained from transitions collected by an exploration policy, and its weights are used to initialize the actor and critic of PPO. Across 9 tasks and 3 embodiments, the authors report a 7.3% final-performance improvement and a 36.9% sample-efficiency improvement relative to random initialization of the same PIDM architecture, with ablations, coverage analysis, and a sim-to-real deployment.","tokens_in":24138,"tokens_out":6818,"duration_ms":57613,"significance":"The contribution is potentially useful: it offers a plug-in initialization that requires no reward-labeled offline data or expert demonstrations, and the main comparison (pretrained vs. random-init PIDM) isolates the effect of pretraining from architecture. Strengths include a clear three-stage pipeline, a reproducible experimental setup (code bundled), ablations for actor/critic and data source, and a sim-to-real demo. The idea is not fully novel—inverse-model pretraining and exploration-based data collection exist—but applying it to warm-start locomotion PPO across multiple embodiments is a reasonable empirical contribution. However, the headline sample-efficiency claim rests on a metric that can count an initial advantage as 'convergence,' and the aggregate statistics are weakened by excluded failed runs and missing error bars.","major_comments":[{"comment":"The 'Number of iterations to converge' is defined as iterations to reach 90% of the final performance of PIDM (Random Init), not 90% of the evaluated method's own final performance. If a pretrained run's initial or early performance already exceeds 0.9 × baseline_final, it is counted as converged at iteration 0. Since the pretrained runs often start with an advantage (Figure 6) and have higher final performance than the baseline, the reported '36.9% sample-efficiency improvement' is not a clean measure of convergence speed; it conflates an initial advantage with faster learning. I recommend redefining the threshold relative to each method's own final performance (or a common reference performance), and reporting area-under-curve or full time-to-threshold curves.","section":"§5.4 (metric definition)"},{"comment":"The headline numbers are aggregates over five seeds without error bars, and five random-init baseline runs are excluded (Jump: 3/5 failed; Walk and Crouch: 1/5 each). This makes the quantitative conclusions fragile. In particular, the Jump improvement of +5.9% is based on only two successful baseline runs. Please report per-seed results, confidence intervals, and a sensitivity analysis where failed runs are assigned a worst-case performance or convergence time. As it stands, the +7.3% final-performance figure could be partly an artifact of dropping the most unstable baseline runs.","section":"Table 1 / Table 8"},{"comment":"The task-agnostic transfer premise is validated only for two flat-terrain tasks (pedipulation and locomotion). Figure 5a shows coverage of initial exploration for these tasks, but the claim that the initial state distribution is similar across all downstream tasks is used to justify transfer to five perceptive parkour tasks on terrains unseen in pretraining. Either extend the coverage analysis to representative parkour tasks, or moderate the mechanistic claim. The empirical results are encouraging, but the stated mechanism is not fully evidenced.","section":"§4.2 / §5.3"}],"minor_comments":[{"comment":"The abstract reports inconsistent numbers: one version states 36.2% sample efficiency and 4.3% task performance, while the body and the other abstract version state 36.9% and 7.3%. Reconcile all occurrences.","section":"Abstract"},{"comment":"The claim that the randomly initialized Action Synthesizer preserves near-unit Gaussian action distribution at initialization is stated without quantitative support. A small experiment or reference to a figure would strengthen this load-bearing design explanation.","section":"§4.5"},{"comment":"The modifications to benchmark tasks (removed curricula, unified collision model, changed pedipulation action space) are documented, but there is no quantitative evidence that they leave the relative ranking of methods unchanged. A brief comparison of the vanilla MLP under original versus modified settings would improve comparability.","section":"Appendix A.6.2"},{"comment":"The derived 'improvement over vanilla MLP' values in the main text (5.0% final, 18.8% sample efficiency) do not exactly match the differences between rows in Table 1 (7.3 - 2.2 = 5.1; -36.9 - (-17.6) = -19.3). Please reconcile or clarify how these were computed.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is worth pursuing. The sample-efficiency metric issue is fixable with a re-analysis rather than a full re-run, and the final-performance claim is less affected. The main risk is that after re-analysis the 36.9% number shrinks considerably, but the method still shows consistent positive improvements across tasks. I would not reject this manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper before reading it. First, the core idea is genuinely practical and new enough: collect task-agnostic exploration transitions, train a proprioceptive inverse dynamics model (PIDM) on them, then use those weights to initialize both actor and critic in PPO. Second, the headline \"36.9% sample efficiency improvement\" is not established by the metric they use. The stress-test note is right: \"Num. iters. to converge\" is defined as iterations to reach 90% of the final performance of the PIDM (Random Init) baseline, not 90% of the method's own final performance. If the pretrained method starts above that threshold, it is counted as converged at iteration 0, even if its learning curve is still climbing. So the sample-efficiency claim conflates a better starting point with faster convergence. The final-performance gain (+7.3%) is a cleaner comparison, though it inherits some noise from the excluded runs and task modifications.\n\nWhat the paper does well: the experimental design is genuinely careful. Nine tasks across three embodiments, five seeds, a controlled comparison between pretrained and randomly initialized PIDM architectures, ablations on actor vs. critic initialization and on data source, a PIDM-error-vs-RL-performance study, and a sim-to-real demo. The authors are transparent about task modifications (removed curricula, unified collision model, added symmetry loss) and about failed runs in Table 8. The fact that pretrained initialization helps consistently across all nine tasks, even when the final performance gains are small, makes me believe the effect is real. The dynamics-probing experiment in Section 5.2 is a nice attempt to motivate why inverse dynamics is the right thing to pretrain, even if it is only correlational.\n\nThe soft spots are real but not fatal. The convergence metric is the biggest issue; it should be redefined or supplemented with something like time-to-return-threshold against the method's own final performance, or a fixed reward threshold. Table 1 lacks error bars and reports aggregate percentages, which hides the variance visible in the training curves. Five runs failed and were excluded, mostly for PIDM (Random Init), which makes the baseline easier to beat. The \"drop-in replacement\" claim is weakened by the modifications and by the fact that the PIDM architecture is about 4x larger than the vanilla MLP. There is no code artifact yet, only a promise to open-source after publication. The coverage assumption (Section 4.2) is validated only on two flat-terrain tasks; the parkour terrains were unseen in pretraining, though the results suggest the pretrained weights still help.\n\nWho is this for: anyone working on legged locomotion RL, especially in Isaac Lab, who wants to reduce training cost. It is an engineering contribution, not a paradigm shift. It deserves a serious referee; the metric problem is fixable and the qualitative result is likely to survive. I would engage with it.","headline":"A useful warm-starting trick for locomotion RL with a solid empirical sweep, though the headline sample-efficiency number is inflated by a threshold metric that counts initial advantage as convergence.","tokens_in":24586,"tokens_out":2334,"would_cite":true,"duration_ms":19576,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that warm-starting PPO with a pretrained inverse-dynamics model lifts locomotion performance by 7.3% and cuts training iterations by 36.9% across nine tasks and three robot bodies.","keywords":["pretraining-finetuning","actor-critic reinforcement learning","inverse dynamics model","warm-start initialization","reward-free exploration","robot locomotion","sample efficiency","legged robots"],"falsifier":"Repeat the paper's own UMAP overlap analysis (Figure 5a) between the flat-terrain pretraining buffer and the first 100 PPO iterations of a perceptive parkour task (e.g., Climb Up or Jump) started from random initialization. If the overlap is comparable to the flat-terrain case, the coverage rationale survives; if it is small yet warm-starting still helps, the stated mechanism is wrong even though the method works. A sharper mechanistic test: scramble the pretrained backbone's weights while preserving the input–output statistics of the initial policy, and check whether the gains persist; if the","tokens_in":23684,"feed_emoji":"🤖","tokens_out":22349,"duration_ms":178661,"temperature":0.7,"pith_summary":"Most robot locomotion skills are still learned from scratch, even when they run on the same robot body. This paper argues that what a policy absorbs in its earliest, clumsiest training steps — basic kinematics, contact behavior, how the joints react to commands — is knowledge about the body, not about any particular task, and so it should be captured once and reused. The recipe: let a robot explore with a curiosity-driven policy to gather reward-free transition data; train a Proprioceptive Inverse Dynamics Model (PIDM), a compact MLP that predicts which action moves the joints toward a desired next state; then load that learned backbone into both the actor and critic of PPO (Proximal Policy Optimization) before task-specific training starts. Across nine locomotion tasks on two quadrupeds and one humanoid, the pretrained start improves final performance in eight of nine tasks (tying the ninth) and speeds convergence in all nine, averaging a 7.3% performance gain and a 36.9% reduction in iterations to convergence compared with random initialization of the same architecture. If the recipe holds, the expensive stumbling phase of locomotion RL can be paid for once per robot body instead of once per skill.","feed_headline":"37% faster robot learning with a pretrained dynamics model","feed_subtitle":"Learn a robot body's dynamics once, reuse them to warm-start every new locomotion skill — no task data needed.","key_machinery":"The Proprioceptive Inverse Dynamics Model (PIDM) is the key object: a modular MLP mapping a history of proprioception and actions plus a desired next-state change (delta-state) to the action that produces it. An ensemble of five PIDMs drives data collection; the spread of their predictions is an intrinsic reward steering the robot toward uncertain states. The same objective is then trained by supervised regression on that buffer. At fine-tuning, the pretrained backbone becomes the shared trunk of PPO's actor and critic, with a randomly initialized intention-encoder replacing the delta-encoder and a randomly initialized synthesizer replacing the decoder. Modularity is the source of task-agnos","core_discovery":"The central claim: a motor policy separates 'which state do I want next' from 'which action gets me there', and that second mapping — the body's inverse dynamics — is nearly task-independent. The paper labels this Hypothesis 1, supported by a probing experiment: deeper layers of trained policies carry progressively less information about future joint state. The PIDM is built to be that late-stage function, trained by supervised L1 regression on reward-free exploration data, then spliced into the actor and critic as a shared trunk behind randomly initialized task-specific heads and fine-tuned by unmodified PPO. Reported result: +7.3% final performance and 36.9% fewer iterations to converge ve","pith_inferences":["The paper validates the early-training coverage assumption for two flat-terrain tasks only; a direct check of whether flat-terrain exploration data covers the early state distribution of the perceptive parkour tasks would either confirm the stated mechanism or force a revised explanation.","The weight-update measurements suggest the pretrained backbone changes how the random heads learn, not just what the initial actions are — a test the paper does not run (freezing the backbone for the first iterations) could separate 'good initial behavior' from 'better optimization geometry' as the causal channel.","The success of a single shared embodiment module hints at a wider design principle for legged robots — an embodiment trunk plus task-specific heads — that could compose with skill libraries, multi-task learning, or cross-task distillation beyond the single-task fine-tuning studied here.","Because gains scale with dynamics-model accuracy, the recipe is probably not at its ceiling; a larger or temporal backbone could widen the improvements if the RL update stays stable."],"forward_implications":["New skills on a pretrained robot converged to the same performance level in about a third fewer training iterations, so the pretrained start directly saves compute per downstream task.","Because the pretrained weights carry no reward or task information, one PIDM per embodiment can serve tasks with different commands, observations, rewards, curricula, and even terrains not present in pretraining.","The method is a drop-in swap: rewards, curricula, observations, and PPO hyperparameters are left untouched, so existing locomotion training pipelines can adopt it without retuning.","Better dynamics models translate into better RL: checkpoints with lower PIDM error produced larger downstream gains, suggesting headroom from improved pretraining.","A policy trained with this initialization transferred from simulation to a real quadruped, indicating that the warm-started network preserves deployability."],"fun_headline_variants":["Pretrained dynamics model speeds up robot RL by 36%","Warm-start robot RL with a pretrained inverse dynamics model","One dynamics model to fast-start every robot skill","Pretrain once, reuse for all: RL speedup for robots","Task-agnostic pretraining boosts PPO for locomotion by 36%"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that reward-free exploration data collected on flat and lightly rough ground covers the state distribution every downstream task visits during early training — including the five parkour tasks whose terrains never appear in pretraining — and this overlap is verified for only two flat-terrain tasks.","fun_headline_variants_meta":{"raw":{"variants":["Pretrained dynamics model speeds up robot RL by 36%","Warm-start robot RL with a pretrained inverse dynamics model","One dynamics model to fast-start every robot skill","Pretrain once, reuse for all: RL speedup for robots","Task-agnostic pretraining boosts PPO for locomotion by 36%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000174,"raw_usage":{"total_tokens":1127,"prompt_tokens":758,"completion_tokens":369,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":282}},"tokens_in":502,"tokens_out":369,"duration_ms":3389,"temperature":1.0,"reasoning_tokens":282,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T09:56:46.733220+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the paper's own UMAP overlap analysis (Figure 5a) between the flat-terrain pretraining buffer and the first 100 PPO iterations of a perceptive parkour task (e.g., Climb Up or Jump) started from random initialization. If the overlap is comparable to the flat-terrain case, the coverage rationale survives; if it is small yet warm-starting still helps, the stated mechanism is wrong even though the method works. A sharper mechanistic test: scramble the pretrained backbone's weights while preserving the input–output statistics of the initial policy, and check whether the gains persist; if the","supporting_citations":[],"review_version":1}