{"id":"27590a80-31cc-485a-bd24-9ffa0a8f395c","arxiv_id":"2509.00215","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"By computing gradients through a learned dynamics model while unrolling trajectories in the real simulator, DMO achieves SHAC-level sample efficiency with standard simulators and deploys on a real quadruped.","lead":"DMO is a reinforcement learning method that rolls out robot trajectories in a regular simulator but computes policy gradients through a small learned model of the simulator, avoiding compounding prediction errors. It matches the sample efficiency of differentiable-simulator methods like SHAC without requiring the simulator to be differentiable, and it transferred to a real Go2 robot walking and balancing on its front legs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The decisive assumption is that the learned model's Jacobians match the true simulator's along visited states; MLE training and the cosine-similarity evidence do not establish this, so the central claim is conditionally supported at best.","rationale":"The reader's weakest_assumption identifies the same load-bearing point: the learned model's Jacobians must be accurate along true trajectories for DMO's decoupled gradient to approximate a differentiable-simulator gradient. My reading agrees and sharpens it. The paper's own Limitations section acknowledges the MLP world model is 'ill-suited for complex inputs,' but the more immediate risk is not input modality: it is that the one-step MLE objective used to fit f_hat (Eq. 4) does not directly control the derivative ∂f_hat/∂(s,a). In contact-rich control, the true transition Jacobian is piecewise and discontinuous, while an MLP produces a smooth approximation; the mismatch can be large exactly where policy updates need reliable direction information. The cosine-similarity experiment (Figure 5 right) is the right kind of evidence, but it compares full policy gradients rather than Jacobian matrices and mixes in differences in the forward trajectories, so it cannot by itself settle the Jacobian-accuracy question. The concrete test I propose would separate the two factors: hold the forward states and actions fixed, swap only the Jacobian, and measure both Jacobian error and resulting gradient alignment. The paper has genuine strengths—the ablation isolating decoupling (Figure 3 right) is convincing, and the real-robot Go2 deployment is a meaningful proof of concept—so the verdict should remain conditional rather than being rejected. The reader's CONDITIONAL verdict is therefore appropriate, and no verdict change is needed; the condition should explicitly include a Jacobian-fidelity check rather than only reporting clarifications on normalization and variant selection.","tokens_in":16983,"tokens_out":10666,"duration_ms":132403,"concrete_test":"On each of the five DFlex environments, freeze a trained DMO policy and sample K=500 state-action pairs from the replay buffer at three training stages (early/mid/converged). Compute the true DFlex Jacobian J_f(s,a) and the learned J_hat(s,a) at the same points; report mean relative Frobenius error ||J_hat-J_f||_F/(||J_f||_F+ε). Then, with the forward trajectory fixed to the real simulator for both, recompute policy gradients using J_hat vs J_f at the same states/actions (only the Jacobian swapped) and report cosine similarity. If relative Jacobian error remains below ~0.1–0.3 at all stages, the assumption holds; if it is large yet the fixed-state gradient cosine is high, the correctness story needs to be revised to explain why biased Jacobians still suffice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation (3) defines the policy gradient through ∂f/∂s and ∂f/∂a; DMO substitutes ∂f_hat/∂s and ∂f_hat/∂a evaluated on true simulator states (Algorithm 1, Appendix A.1.3). For the central claim to hold, these learned Jacobians must be close to the true ones in the region the policy actually visits. The model is trained with the one-step MLE objective (Eq. 4), which controls the conditional distribution of s′ but gives no guarantee on the derivative of the conditional mean. In the DFlex/IsaacGym contact-rich tasks the true transition is non-smooth; an MLP with smooth activations will smooth the contact switch, and the error in ∂f_hat can be large precisely where the gradient direction changes. The only direct evidence (Figure 5 right) measures cosine similarity of full policy gradients, not Jacobian error, and the comparison includes model-rollout trajectories with different forward states, so it does not isolate Jacobian fidelity at the same (s,a). If this Jacobian error is large, DMO's gradient is a biased heuristic and the 'as accurate as a differentiable simulator' part of the claim is not established; the empirical gains could be coming from the critic/value bootstrapping or reward shaping rather than from the decoupled dynamics derivative.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DMO (Decoupled forward-backward Model-based policy Optimization), a first-order model-based RL method that unrolls trajectories with a high-fidelity (possibly non-differentiable) simulator while computing policy gradients by backpropagating through a learned differentiable dynamics model. The learned model is trained on replay data with a one-step maximum-likelihood objective, and its Jacobians are evaluated at the true simulator states during the backward pass. The authors instantiate DMO on top of BPTT, SHAC, and SAPO, evaluate it on five DFlex control tasks plus an AllegroHand task, and deploy policies on a real Unitree Go2 in both quadrupedal and bipedal modes. They report large sample-efficiency gains over PPO and SAC, competitive final performance with SHAC/SAPO, and a wall-clock advantage over model-free baselines. The core decoupling idea is supported by an ablation against a model-rollout variant and by a cosine-similarity analysis of gradient directions.","tokens_in":17194,"tokens_out":4913,"duration_ms":61897,"significance":"If the central claim is correct, DMO offers a practical way to obtain cheap first-order policy gradients without a differentiable simulator, avoiding the compounding-error problem of standard FoG-MBRL while retaining the sample-efficiency benefits of analytical gradients. The decoupled backpropagation formula in Appendix A.1.3 is correct and is implemented with a simple PyTorch trick; the paper honestly credits prior work (SVG(∞), PILCO) for the general idea and provides a systematic modern evaluation. The internal ablation against model-based forward rollouts is well designed and shows a real performance difference, which strengthens the paper's main mechanistic claim. The real-robot deployment is a valuable addition, though it is presented as a demonstration rather than a controlled study. The main weakness is that the paper does not directly establish that the learned model's Jacobians are accurate enough along the visited state-action distribution; this is the key assumption on which the 'as accurate as a differentiable simulator' claim rests.","major_comments":[{"comment":"The central claim that DMO yields policy gradients 'as accurate as those of a differentiable simulator' depends on the learned model's Jacobians ∂f̂/∂s and ∂f̂/∂a being close to the true simulator Jacobians along the states actually visited by the policy. The training objective in Eq. (4) is a one-step maximum-likelihood fit of the conditional next-state distribution; it controls the conditional mean, but gives no guarantee on the derivative of the conditional mean. In contact-rich tasks, the true transition is non-smooth while the MLP is smooth, so ∂f̂ can be substantially biased exactly where the policy gradient changes. The only direct evidence, Figure 5 (right), measures cosine similarity of full policy gradients, not Jacobian error, and the comparison against the model-forward trajectory uses different forward states, so it does not isolate Jacobian fidelity at the same (s,a). Pleas","section":"Section 3.2 and Eq. (4)"},{"comment":"There is an indexing error in the description of the backward pass. Algorithm 1 and the text in Section 3.2 state that DMO uses ∂f̂(s,a)/∂s evaluated at (s_{t+1}, a_{t+1}) and ∂f̂(s,a)/∂a at (s_{t+1}, a_{t+1}) to approximate the true dynamics Jacobians. However, the analytical policy gradient in Eq. (3) requires ∂f/∂s and ∂f/∂a evaluated at (s_t, a_t). Appendix A.1.3 correctly writes the decoupled formula with ∂f̂(s,a)/∂s at (s_t, a_t). As written, Algorithm 1 would backpropagate through the wrong time step. Please correct the subscripts in Algorithm 1 and in the Section 3.2 discussion to (s_t, a_t), and make the implementation consistent with Appendix A.1.3.","section":"Algorithm 1 and Section 3.2"},{"comment":"The gradient-similarity experiment is not sufficient to support the claim that decoupled gradients are 'more precise' than model-rollout gradients. The three trajectories are unrolled 'in parallel' and 'under the same conditions,' but it is not stated whether the actions and initial states are identical across the three graphs. If the model-rollout trajectory diverges in state space, then the cosine similarity between DFlex and model-forward gradients conflates state-distribution mismatch with Jacobian error. Even for the DMO graph, the cosine similarity between DFlex and DMO gradients is an aggregate over states and does not show where or why the gradients differ. Please clarify the experimental protocol and report a per-transition Jacobian comparison at identical (s,a) points, or at least a state-conditioned breakdown. This is necessary to attribute the observed performance gains to th","section":"Figure 5 (right) and Section 4.3"}],"minor_comments":[{"comment":"The objective written as L_f(ϕ) = E_{(s,a,s')∼B}[ p_ϕ(s' | s,a) ] is not a maximum-likelihood objective as stated; MLE would maximize the log-density, E[ log p_ϕ(s'|s,a) ]. Please correct the equation or clarify the notation.","section":"Eq. (4)"},{"comment":"In the third line of Eq. (3), the derivative ∂πθ(s)/∂θ is written with a subscript (θ̃, s_t), where θ̃ is not defined. This appears to be a typo; please replace with the proper evaluation notation.","section":"Section 3.1, Eq. (3)"},{"comment":"The claim that DMO 'achieves up to 20% improvement' in wall-clock time is not tied to a specific baseline in the main text. Figure 4 (right) compares only PPO and SAC, not MAAC or SHAC. Please state the comparison explicitly.","section":"Section 4.2"},{"comment":"The class name 'GradientSwapingFunction' contains a typo ('Swaping' -> 'Swapping'). Also, the code comment and explanation could be clearer about the fact that the backward path through f̂ receives the gradient with respect to the true next state by using a clone node.","section":"Appendix A.1.3"},{"comment":"The phrase 'generated under the exact same conditions as DMO-SHAC' is ambiguous: it should specify whether the same random seed, initial state, and action sequence are used for the learned-model rollouts. This matters for interpreting Figure 5 (right).","section":"Section 4.3"},{"comment":"SHAC/SAPO results are only shown as final performance; no learning curves or wall-clock times are given for these baselines. The paper's claim of being competitive with SHAC in 'efficiency and final convergence accuracy' would be strengthened by reporting SHAC's sample/time curves on the shared benchmarks.","section":"Appendix A.2.1, Figure 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid revisit of an idea from SVG(∞)/PILCO, modernized and systematically ablated, with a correct decoupled backpropagation formula. The main uncertainty is whether the learned model's Jacobians are accurate enough along visited states; this is testable within the paper's own framework by comparing against DFlex's analytic derivatives. The indexing error in Algorithm 1 is easily fixed. I do not see grounds for rejection, but the 'as accurate as a differentiable simulator' claim needs direct Jacobian-fidelity evidence before the paper can be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a solid paper, and the core idea is not as new as the framing suggests—the authors know that, since they cite SVG(infinity) and PILCO up front. What's actually new is the modern instantiation: decoupled forward/backward passes on GPU-accelerated non-differentiable simulators, three concrete variants, a neat PyTorch gradient-swapping trick that avoids graph surgery, and a clean ablation isolating the decoupling effect. The math in Appendix A.1.3 is correct, and the ablation versus model-based forward passes is a fair comparison that shows a real performance difference. The real-robot Go2 deployment, including bipedal front-leg balancing, is a useful proof of concept. The main soft spot is exactly what the stress-test note flags: the claim that decoupled gradients are 'as accurate as a differentiable simulator' rests on the learned model's Jacobians being close to the true dynamics along visited states. The paper's cosine-similarity evidence (Figure 5 right) is suggestive but not direct—it compares full policy gradients, not Jacobian error, and mixes in different forward states. That said, this is not a load-bearing flaw. The internal ablation shows DMO beats model-based forward with the same learned model, which demonstrates the decoupling helps even if the Jacobians are imperfect. So the central mechanism is empirically supported, just not with the precision the headline implies. Secondary issues: the aggregate normalized scores in Figures 3 and 4 do not specify the normalization procedure, DMO performance selects the best variant per environment, the wall-clock improvement percentage lacks a clear baseline definition, and the real-robot evaluation is qualitative. These should be clarified before the strongest claims are taken at face value. The stated limitations—differentiable reward functions and the MLP world model—are honest and do not undercut the contribution within its scope. Who is this for? Anyone working in model-based RL for robot learning who wants a practical way to get differentiable-simulator-style sample efficiency without building a differentiable simulator. It deserves a serious peer review; the requests should be for clarification, not for new experiments. I'd bring it to a reading group and likely cite it when discussing decoupled gradient tricks.","headline":"A clean, well-executed revival of the SVG(infinity) decoupling idea, with a fair ablation and a real-robot demo; the central mechanism holds up, but the strongest performance claims need a bit more care.","tokens_in":628,"tokens_out":733,"would_cite":true,"duration_ms":26501,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By unrolling trajectories in an ordinary simulator and backpropagating through a learned model, DMO obtains policy gradients nearly as accurate as a differentiable simulator's.","keywords":["Model-Based Reinforcement Learning","First-Order Policy Optimization","Decoupled Backpropagation","Learned Dynamics Models","Sample Efficiency","Quadruped Locomotion","Sim-to-Real Transfer"],"falsifier":"On a contact-rich task with an exact differentiable simulator available, compute DMO's policy gradient and the simulator's true policy gradient on identical rollouts, then deliberately corrupt the learned model's Jacobians, for example by dropping contact transitions from the replay buffer. If DMO still trains successfully despite low cosine similarity between the two gradients, the paper's accuracy-of-gradients mechanism is not what drives the gains.","tokens_in":16790,"feed_emoji":"🤖","tokens_out":6518,"duration_ms":71211,"temperature":0.7,"pith_summary":"This paper tries to show that reinforcement learning can get the sample efficiency of differentiable-simulator optimizers without ever needing simulator gradients. The proposed method, DMO, unrolls policy trajectories in an ordinary high-fidelity simulator, then computes policy gradients by backpropagating through a learned neural-network model of the simulator's dynamics, evaluated at the true simulator states rather than at the model's own predicted states. That separation is meant to prevent the compounding prediction errors that hurt standard model-based RL. The authors report that DMO converges with roughly tenfold fewer samples than PPO, matches the efficiency of SHAC-style differentiable-simulator training, and transfers to a real quadruped robot in both four-legged and two-legged walking.","feed_headline":"One gradient trick gives robot RL ten times the sample efficiency","feed_subtitle":"DMO unrolls trajectories in a normal simulator but backpropagates through a learned model, matching differentiable-simulator training.","key_machinery":"The central object is the decoupled gradient swap: during training the simulator produces the real next state, while a learned MLP dynamics model produces a predicted next state from the same state-action pair; an autodiff gradient-swapping function returns the real state for the forward pass but routes backpropagation through the predicted state, so gradients flow through the learned model's Jacobians evaluated at the accurate simulator state. This uses the learned model's Jacobian at the true state as a proxy for the simulator's Jacobian. A truncated-return objective with a learned value bootstrap for the SHAC and SAPO variants, or without one for the BPTT variant, closes the loop.","core_discovery":"On its own terms, the paper's finding is that the accuracy of first-order policy gradients is set by where the dynamics-model derivatives are evaluated, not by which function generates the trajectory. By replacing only the forward unrolling with a high-fidelity simulator and leaving the backward pass inside a learned differentiable model, DMO produces gradients whose cosine similarity to exact differentiable-simulator gradients is high, and it avoids the ill behavior seen in methods that unroll fully inside the learned model. The same decoupling also lets the value critic be fit to true simulator rollouts, which the paper argues is more accurate. Three instantiations, DMO-BPTT, DMO-SHAC, and","pith_inferences":["If the decoupling claim generalizes, the field's investment in differentiable simulators may be partly redundant: any simulator plus a locally accurate learned Jacobian could give the same first-order updates, at least where rewards are differentiable.","The same trick should extend to richer world models such as latent or vision-based ones, since the forward unrolling stays in the simulator; the paper's MLP limitation concerns the model class, not the decoupling idea itself.","A testable consequence is that DMO's policy-gradient quality should track the Jacobian accuracy of the learned model along the on-policy state distribution; measuring that correlation would directly check the mechanism.","The requirement of differentiable rewards is a design constraint rather than a detail: applying DMO to sparse-reward tasks would force the value bootstrap to carry more of the learning, so reward shaping becomes a first-class concern."],"forward_implications":["DMO can be dropped onto existing first-order gradient methods such as SHAC, SAPO, and BPTT with only a few lines of code, because decoupling is implemented as a gradient-swapping autodiff node.","It reaches asymptotic performance in under four million samples, about ten times fewer than PPO, while improving wall-clock time by up to 20% despite learning a model.","It works with non-differentiable simulators: the real-robot experiments were trained in a GPU simulator that provides no gradients.","Decoupling itself, not just model learning, drives the gains: the model-based-forward ablation nearly halves asymptotic performance.","The critic can be learned from true simulator rollouts, which the paper claims is more accurate, and the method remains stable even with a batch size of one on dense-reward tasks."],"supporting_citations":[{"why":"Provides the SHAC algorithm and the DFlex differentiable simulator that DMO-SHAC extends and that supplies the exact-gradient baseline.","marker":"[18]"},{"why":"Provides SAPO, whose exploration enhancements define DMO-SAPO and the AllegroHand benchmark.","marker":"[19]"},{"why":"MAAC, the modern FoG-MBRL baseline that unrolls trajectories in the learned model and is compared for efficiency.","marker":"[23]"},{"why":"SVG(∞), the prior decoupled-gradient formulation that DMO revisits and contrasts with full-trajectory optimization.","marker":"[52]"},{"why":"Documents compounding prediction errors in learned dynamics models, the failure mode decoupling is designed to remove.","marker":"[38]"},{"why":"Defines the BPTT/APG-style truncated-backprop configuration used by DMO-BPTT.","marker":"[17]"},{"why":"Isaac Gym, the non-differentiable GPU simulator used for the real-robot quadruped and bipedal experiments.","marker":"[14]"}],"fun_headline_variants":["Decoupled backprop makes model-based RL as fast as SHAC","Simulator rollouts with learned gradients beat PPO sample efficiency","Hybrid RL: real sim for trajectories, learned model for gradients","Gradients from learned model, rollouts from simulator: faster RL"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The learned dynamics model's derivatives have to be faithful enough to the real simulator's derivatives along the states the policy actually visits; otherwise the decoupled gradient is a wrong gradient computed on a correct trajectory.","fun_headline_variants_meta":{"raw":{"variants":["Decoupled backprop makes model-based RL as fast as SHAC","Simulator rollouts with learned gradients beat PPO sample efficiency","Hybrid RL: real sim for trajectories, learned model for gradients","Gradients from learned model, rollouts from simulator: faster RL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000632,"raw_usage":{"total_tokens":2749,"prompt_tokens":735,"completion_tokens":2014,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":1939}},"tokens_in":479,"tokens_out":2014,"duration_ms":17334,"temperature":1.0,"reasoning_tokens":1939,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T13:49:40.119363+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a contact-rich task with an exact differentiable simulator available, compute DMO's policy gradient and the simulator's true policy gradient on identical rollouts, then deliberately corrupt the learned model's Jacobians, for example by dropping contact transitions from the replay buffer. If DMO still trains successfully despite low cosine similarity between the two gradients, the paper's accuracy-of-gradients mechanism is not what drives the gains.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SHAC algorithm and the DFlex differentiable simulator that DMO-SHAC extends and that supplies the exact-gradient baseline."},{"cited_title":"Heess, G","cited_arxiv_id":null,"evidence_quote":"SVG(∞), the prior decoupled-gradient formulation that DMO revisits and contrasts with full-trajectory optimization."},{"cited_title":"Makoviychuk, L","cited_arxiv_id":null,"evidence_quote":"Isaac Gym, the non-differentiable GPU simulator used for the real-robot quadruped and bipedal experiments."}],"review_version":1}