Pith. sign in

REVIEW 3 major objections 6 minor 40 references

ABPT: Amended Backpropagation through Time with Partially Differentiable Rewards

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ABPT combines the N-step return with a 0-step value gradient so that non-differentiable reward components no longer bias backpropagation-through-time, yielding faster convergence and higher final rewards on quadrotor flight tasks.

desk verdict A simple and plausibly useful gradient-mixing trick for differentiable simulators, but the empirical claim is weakened by missing the two closest baselines and by the absence of released code. read the letter →

arxiv 2501.14513 v2 pith:Q5I5NJUI submitted 2025-01-24 cs.RO cs.AIcs.LG

Fanxing Li , Fangyu Sun , Tianbao Zhang , Danping Zou This is my paper · ORCID
classification cs.ROcs.AIcs.LG
keywords differentiablesimulationbackpropagationthroughtimepartiallyrewardsgradientbiasvaluequadrotorcontrolactor-criticentropyregularization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Backpropagation-through-time (BPTT) trains policies through a differentiable simulator, but quadrotor reward functions usually contain non-differentiable pieces such as success bonuses, and those pieces silently drop out of the gradient, biasing training. This paper proposes Amended Backpropagation-through-Time (ABPT), which optimizes an objective that averages the N-step return with a 0-step return: the learned value function's gradient is computed through the action-value function, so it is differentiable even when the reward is not. That value-gradient term compensates the missing reward gradients and keeps parameter updates heading in the right direction. Across hovering, tracking, landing, and racing, ABPT is reported to converge faster and reach higher final rewards than PPO, plain BPTT, and SHAC, with real-world flight validation. The paper also adds entropy regularization and state replay to improve exploration.

What carries the argument

The load-bearing object is the 0-step return $J_0^\theta = V_\phi(s_t)$, defined through the action-value function with an entropy bonus: $V_\phi(s) = \mathbb{E}_{a\sim\pi}[Q_\phi(s,a)] + \kappa H(\pi_\theta(\cdot|s))$. Unlike a state-only value $V_\phi(s)$, this form depends on $\theta$ through the policy, so its gradient with respect to $\theta$ is nonzero and can be backpropagated. The N-step return $J_N^\theta = \sum_{k=0}^{N-1}\gamma^k R(s_{t+k}) + (1-d)\gamma^N V_\phi(s_{t+N})$ supplies the direct first-order reward path. Averaging the two yields a gradient whose value-gradient half is immune to non-differentiable rewards; a target critic, TD-$\lambda$ return estimation, entropy regularization, and a state replay buffer stabilize the learned value and exploration.

What would settle it

Compare ABPT with plain BPTT on the racing task while scaling up the non-differentiable gate-passing bonus: if the bonus dominates and ABPT's converged gate-crossing rate or reward falls to BPTT's level, the claimed compensation is bounded. A sharper test is to replace the learned critic with a deliberately poor one at initialization and measure the angle between ABPT's parameter update and the exact analytical gradient of a fully differentiable surrogate reward; if the update direction is no closer than BPTT's, the value-gradient correction is not carrying the argument.

Watch

Extended reading notes

Core claim

The central claim is that the biased first-order gradient caused by partially differentiable rewards can be corrected by averaging in the value gradient. With the objective $J_\theta = \frac{1}{2|B|}\sum_i \left(J_N^\theta + J_0^\theta\right)$, where $J_0^\theta = V_\phi(s_t)$ is the state value computed from the learned action-value function $Q_\phi(s, \pi_\theta(s))$, backpropagation produces the average of the value gradient $\nabla^{[q]}_\theta G_{t:t+N}$ and the first-order gradient plus the terminal value term. Because $Q_\phi$ is a differentiable function of $\theta$ regardless of reward differentiability, the 0-step term supplies gradient information about the non-differentiable part of the reward (for example, gate-passing bonuses) that the direct reward path drops. The paper states that this combination 'remains effective in guiding the parameter updates toward the correct direction, when the first-order gradient is biased,' and supports it with a parameter-residual experiment in Appendix A.

Load-bearing premise

The method's correction depends on the learned value function being accurate enough in the states actually visited; if the critic is wrong early in training, the added value-gradient term can push the policy in the wrong direction, and the paper provides no guarantee that the equal-weight average stays unbiased.

Editorial extensions

If this is right

  • Where rewards are partially differentiable, ABPT's update should stay closer to the true policy gradient than plain BPTT because the 0-step value gradient carries information about non-differentiable reward components.
  • The equal-weight average, rather than a variance-based adaptive mixture, keeps the per-step overhead small, making the method practical on GPU-parallel differentiable simulators.
  • On fully differentiable tasks, ABPT should retain BPTT-level convergence speed while gaining exploration from entropy and state replay, as demonstrated for hovering and tracking.
  • When non-differentiable rewards dominate the total reward, performance may still degrade; the paper's own limitation statement says the bias may not be fully eliminated in that regime.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the bias-correction story holds generally, the same 0-step/value-gradient averaging should transfer to other differentiable simulators with contact-rich or hybrid rewards, such as legged locomotion with foot-contact bonuses, not just quadrotors.
  • The method depends on the critic's accuracy early in training; a testable extension would adaptively weight the value-gradient term by critic uncertainty instead of a fixed one-half, which the paper explicitly leaves open.
  • Because ABPT stores only states rather than transitions, its replay buffer could be augmented with goal-conditioned or adversarially sampled states to stress-test generalization to corner cases.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes ABPT (Amended Backpropagation through Time), an on-policy actor-critic method for training quadrotor policies when the reward function is only partially differentiable. The core idea is to combine a 0-step return (a value term computed through a learned Q-function) with the N-step return used by first-order gradient methods such as SHAC, and to update the actor by differentiating the averaged objective in Eq. (9). Entropy regularization and a state-only replay buffer are added to improve exploration and sample efficiency. The method is evaluated on four quadrotor tasks (hovering, tracking, landing, racing) in the VisFly simulator, with qualitative real-world demonstrations, and is compared against PPO, BPTT, and SHAC. The paper claims that ABPT converges faster and reaches higher final rewards than these baselines, especially when rewards contain non-differentiable components.

Significance. If the empirical claims hold, ABPT is a useful and simple modification of the SHAC/BPTT pipeline for differentiable-simulation policy learning: it is easy to implement, preserves first-order gradient efficiency, and appears to mitigate the bias caused by non-differentiable reward terms. The paper includes four tasks with varying reward differentiability, ablations of the three proposed components, reward-robustness and learning-rate robustness experiments, and real-world video demonstrations. These are genuine strengths. The significance is moderated, however, by the absence of comparisons with the closest prior gradient-mixing methods (AGPO and AOBG), by an algebraic inconsistency in the central derivation in Appendix A, and by the lack of released code or complete implementation details, which makes the empirical claims hard to verify independently.

major comments (3)
  1. [§6.2, Appendix E] The central empirical claim that ABPT converges significantly faster and achieves higher ultimate rewards than existing learning algorithms is not tested against AGPO [36] and AOBG [11], the two prior methods explicitly designed for biased or high-variance first-order gradients in differentiable simulation. The manuscript only asserts in §6.2 and Appendix E that these methods 'require extremely long training times due to the need for variance estimation at each step' without reporting any experiment or wall-clock measurement. Since ABPT's update is a fixed equal-weight average of a value gradient and a first-order gradient, and AGPO already mixes Q-gradients with first-order gradients using an adaptive ratio, the claimed superiority over existing gradient-mixing methods is currently unverified. The authors should either add these baselines or explicitly restrict the headline claim to the evaluated baselines.
  2. [Appendix A, Eqs. (16) and (19)] The bias-correction argument hinges on the unproven approximation G_{t:t+N} ≈ V_t - (1-d)γ^N V_{t+N+1} in Eq. (16). The paper gives no formal statement of the approximation error, no condition under which the critic is accurate enough for the value-gradient term to be reliable, and no bound on the residual bias after the equal-weight average. Furthermore, Eq. (19) is algebraically inconsistent: the first line writes ∇J as \bar∇G + (1-d)γ^N ∇V_{t+N+1} with \bar∇G = (∇[q]G + ∇G)/2, while the second line writes it as (∇[q]G + ∇G + (1-d)γ^N ∇V_{t+N+1})/2; these two expressions differ by a factor of two on the terminal-value term. This does not prevent direct implementation of Eq. (9), but it invalidates the derivation as written and should be corrected, together with a precise statement of what gradient is actually being averaged.
  3. [§6.2, Appendix F] The empirical evaluation is not reproducible from the manuscript as submitted. The abstract states that code will be released at a GitHub URL, while the main text refers to a placeholder 'Anonymous' link, and no code or supplementary material is provided. Network architectures, critic update schedules, the exact adaptive entropy coefficient computation, the TD-λ target details beyond Eq. (13), and the state-buffer sampling mechanism are not specified. For a paper whose contributions are primarily empirical, these details are necessary for verification; please provide complete implementation details or the actual code.
minor comments (6)
  1. [§1] The sentence 'Though reinforcement learning (RL) can tackle such limitation by self-exploration of agents, it relies on RL relies on zero-order gradient (ZOG) approximations' contains a duplicated phrase 'it relies on RL relies on' and should be reworded.
  2. [Eq. (4)] The notation '∇[0]θ Jθ' is nonstandard and the definition of the advantage A^{πθ}(s_t, a_t) is not given; please define it explicitly and use a consistent gradient notation throughout.
  3. [Eqs. (6) and (7)] The first-order gradient formulas omit the expectation over trajectories and the minibatch index; this makes it unclear whether the gradient is computed per trajectory or averaged over the batch.
  4. [Table 1] In the Racing row, the observation is described as 'state & next 2 gates' but the task text in Appendix C describes four gates; please clarify whether the policy observes only the next two gates and, if so, how this is reconciled with the task description.
  5. [§6.3 and Figure 1] The real-world validation is qualitative only; no quantitative metrics such as success rate, tracking error, or number of trials are reported. If real-world performance is part of the claimed contribution, please add quantitative results.
  6. [Abstract and §6.2] The abstract and main text are inconsistent about code availability: the abstract gives a GitHub URL, while the body says the code is 'to be released' at an anonymous placeholder, and the acknowledgments section contains boilerplate text that appears to be from a template; please clean this up.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the gradient-averaging identity is a direct differentiation of the paper's own objective, and the empirical superiority claim is validated against external baselines and real-world flight.

full rationale

The derivation in Appendix A is not circular. The objective (9) is explicitly defined as an average of the N-step return and the 0-step return, so the 'proof' that its gradient equals the average of the corresponding gradients is simply the chain rule applied to the definition; no conclusion is imported from the definition. The only nontrivial ingredient is the approximate Bellman consistency (16), which the paper states as an assumption ('Supposed the value function Qϕ is well trained...'), and which is a standard critic-bootstrap condition rather than a restatement of the paper's performance claim. The 0-step value gradient is obtained from a learned Q-function trained with TD-λ targets (Eqs. 12-14) and the reparameterized Gaussian policy (Eq. 11); it is a fitted critic, but it is not relabeled as a held-out prediction, and the claim that the combination reduces bias is tested empirically against PPO, BPTT, and SHAC, with real-world validation. The paper even states a limitation (Appendix B) that ABPT 'may still fail to fully eliminate extreme bias,' which confirms the claim is not true by construction. The main self-citation, VisFly [29], is used as the evaluation simulator, but it is a separate released environment, the training curves are only one part of the evidence, and the real-world trials make the comparison externally falsifiable; this does not make the central claim reduce to a self-citation. The absence of AGPO/AOBG from the headline comparisons is a benchmarking and completeness concern, not a circularity issue.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical or mathematical entities. Its free parameters are conventional RL hyperparameters, tuned per task, and its load-bearing assumption is that the learned value function supplies a reliable gradient around non-differentiable reward terms. The team's own VisFly simulator and prior differentiable-flight papers are used as tools, not as the target conclusion.

free parameters (4)
  • Per-task learning rate = 0.01 / 0.002 / 0.005 / 0.0002 depending on task and algorithm
    Tuned for each method and task to achieve optimal training curves; makes cross-method conclusions dependent on tuning choices.
  • Horizon length H = 96 for ABPT and SHAC, 256 or 512 for BPTT
    Chosen per algorithm and task; longer horizons give BPTT more gradient horizon but also more variance.
  • Entropy temperature κ = Adaptive, following SAC
    Controls the exploration term in Eq. (15); a design choice, not derived from first principles.
  • Replay buffer size = 1,000,000 for hovering, tracking, landing; 50,000 for racing
    Chosen per task for state initialization; affects sampling of corner cases.
assumptions (4)
  • domain assumption VisFly provides accurate differentiable quadrotor dynamics with correct gradients.
    All training and evaluation rely on VisFly; no verification of gradient accuracy is provided.
  • domain assumption The critic Qϕ is accurate enough for Eq. (16) to hold.
    Eq. (16) in Appendix A rewrites the accumulated reward using the value function, and the whole value-gradient correction depends on this approximation.
  • domain assumption The Gaussian policy with tanh normalization can represent the needed control distributions.
    The policy class is used for reparameterized gradients and exploration; no expressiveness analysis is given.
  • standard math Standard RL and TD learning results are accepted.
    Policy gradient and TD-λ are taken as background knowledge.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ABPT: Amended Backpropagation through Time with Partially Differentiable Rewards." pith.science (2026). https://pith.science/paper/Q5I5NJUI

@misc{pith2026250114513,
  author       = {Pith},
  title        = {Pith review of: ABPT: Amended Backpropagation through Time with Partially Differentiable Rewards},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q5I5NJUI}},
  note         = {Machine review of arXiv:2501.14513}
}
read the original abstract

Quadrotor control policies can be trained with high performance using the exact gradients of the rewards to directly optimize policy parameters via backpropagation-through-time (BPTT). However, designing a fully differentiable reward architecture is often challenging. Partially differentiable rewards will result in biased gradient propagation that degrades training performance. To overcome this limitation, we propose Amended Backpropagation-through-Time (ABPT), a novel approach that mitigates gradient bias while preserving the training efficiency of BPTT. ABPT combines 0-step and N-step returns, effectively reducing the bias by leveraging value gradients from the learned Q-value function. Additionally, it adopts entropy regularization and state initialization mechanisms to encourage exploration during training. We evaluate ABPT on four representative quadrotor flight tasks \li{in both real world and simulation}. Experimental results demonstrate that ABPT converges significantly faster and achieves higher ultimate rewards than existing learning algorithms, particularly in tasks involving partially differentiable rewards. The code will be released at http://github.com/Fanxing-LI/ABPT.

Figures

Figures reproduced from arXiv: 2501.14513 by the authors.

Figure 1
Figure 1. Our trained policies were validated in real world. Two rows are separately four trials of [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An illustration for explaining biased gradient. In a racing tasks for drone which we intro [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. A overview of ABPT. ABPT combines 0-step return and n-step returns together, to com [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: Training curves of PPO, SHAC, BPTT, and our ABPT in both time-step ( [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: The curve shows the the difference between the parameters trained with fully differen [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Quadrotor tasks (left to right): hovering, tracking, landing, and racing. We illustrate [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Ablation study: the key components of ABPT are sequentially removed in turn to evaluate [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Training curves with different rewards: position-based rewards ( [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Training curves with different learning rates [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 18 canonical work pages

  1. [36]

    F. Gao, L. Shi, S. Zhang, Z. Wang, and Y . Wu. Adaptive-Gradient Policy Optimization: En- hancing Policy Learning in Non-Smooth Differentiable Simulations. In Proceedings of the 41st International Conference on Machine Learning , pages 14844–14858. PMLR, July 2024. URL https://proceedings.mlr.press/v235/gao24m.html. ISSN: 2640-3498

    work page 2024

  2. [11]

    H. J. Suh, M. Simchowitz, K. Zhang, and R. Tedrake. Do Differentiable Simulators Give Better Policy Gradients? In Proceedings of the 39th International Conference on Machine Learning , pages 20668–20696. PMLR, June 2022. URL https://proceedings.mlr.press/v162/ suh22b.html. ISSN: 2640-3498

    work page 2022

  3. [1]

    Loquercio, E

    A. Loquercio, E. Kaufmann, R. Ranftl, M. M ¨uller, V . Koltun, and D. Scaramuzza. Learning high-speed flight in the wild. Science Robotics, 6(59):eabg5810, 2021. ISSN 2470-9476

    work page 2021

  4. [2]

    Loquercio, E

    A. Loquercio, E. Kaufmann, R. Ranftl, A. Dosovitskiy, V . Koltun, and D. Scaramuzza. Deep drone racing: From simulation to reality with domain randomization. IEEE Transactions on Robotics, 36(1):1–14, 2019. ISSN 1552-3098

    work page 2019

  5. [3]

    Kaufmann, A

    E. Kaufmann, A. Loquercio, R. Ranftl, A. Dosovitskiy, V . Koltun, and D. Scaramuzza. Deep drone racing: Learning agile flight in dynamic environments. pages 133–145. PMLR, 2018. ISBN 2640-3498

    work page 2018

  6. [4]

    R. S. Sutton and A. G. Barto. Reinforcement Learning, second edition: An Introduction . MIT Press, Nov. 2018. ISBN 978-0-262-35270-3. Google-Books-ID: uWV0DwAAQBAJ

    work page 2018

  7. [5]

    Zhang, Y

    Y . Zhang, Y . Hu, Y . Song, D. Zou, and W. Lin. Back to newton’s laws: Learning vision-based agile flight via differentiable physics. arXiv preprint arXiv:2407.10648, 2024

    arXiv 2024

  8. [6]

    Wiedemann, V

    N. Wiedemann, V . W¨uest, A. Loquercio, M. M¨uller, D. Floreano, and D. Scaramuzza. Training Efficient Controllers via Analytic Policy Gradient. pages 1349–1356, June 2023. doi:10.1109/ ICRA48891.2023.10160581

    arXiv 2023

Show all 40 references
  1. [7]

    J. Lv, Y . Feng, C. Zhang, S. Zhao, L. Shao, and C. Lu. Sam-rl: Sensing-aware model-based reinforcement learning via differentiable physics-based simulation and rendering. The Inter- national Journal of Robotics Research, page 02783649241284653, 2023

    2023

  2. [8]

    Y . Song, S. Kim, and D. Scaramuzza. Learning quadruped locomotion using differentiable simulation. arXiv preprint arXiv:2403.14864, 2024

    2024 arXiv

  3. [9]

    Y . Hu, Y . Zhang, Y . Song, Y . Deng, F. Yu, L. Zhang, W. Lin, D. Zou, and W. Yu. Seeing through pixel motion: Learning obstacle avoidance from optical flow with one camera. IEEE Robotics and Automation Letters, pages 1–8, 2025. doi:10.1109/LRA.2025.3560842

    2025

  4. [10]

    Zhang, W

    S. Zhang, W. Jin, and Z. Wang. Adaptive Barrier Smoothing for First-Order Policy Gradi- ent with Contact Dynamics. In Proceedings of the 40th International Conference on Ma- chine Learning, pages 41219–41243. PMLR, July 2023. URL https://proceedings.mlr. press/v202/zhang23s.ht...

    2023

  5. [12]

    V . Mnih. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013

    2013 arXiv

  6. [13]

    Lillicrap

    T. Lillicrap. Continuous control with deep reinforcement learning. arXiv preprint arXiv:1509.02971, 2015

    2015 arXiv

  7. [14]

    Fujimoto, H

    S. Fujimoto, H. Hoof, and D. Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning , pages 1587–1596. PMLR, 2018

    2018

  8. [15]

    Haarnoja, A

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861–1870. PMLR, 2018

    2018

  9. [16]

    Schulman

    J. Schulman. Trust region policy optimization. arXiv preprint arXiv:1502.05477, 2015

    2015 arXiv

  10. [17]

    Schulman, F

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 16

    2017 arXiv

  11. [18]

    T. M. Moerland, J. Broekens, A. Plaat, and C. M. Jonker. Model-based reinforcement learning: A survey. F oundations and Trends® in Machine Learning , 16(1):1–118, 2023. ISSN 1935- 8237

    2023

  12. [19]

    Deisenroth and C

    M. Deisenroth and C. E. Rasmussen. Pilco: A model-based and data-efficient approach to pol- icy search. In Proceedings of the 28th International Conference on machine learning (ICML- 11), pages 465–472, 2011

    2011

  13. [20]

    R. S. Sutton. Integrated architectures for learning, planning, and reacting based on approximat- ing dynamic programming. In Machine learning proceedings 1990 , pages 216–224. Elsevier, 1990

    1990

  14. [21]

    K. Chua, R. Calandra, R. McAllister, and S. Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018

    2018

  15. [22]

    Watter, J

    M. Watter, J. Springenberg, J. Boedecker, and M. Riedmiller. Embed to control: A locally linear latent dynamics model for control from raw images. Advances in neural information processing systems, 28, 2015

    2015

  16. [23]

    Hafner, T

    D. Hafner, T. Lillicrap, J. Ba, and M. Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv:1912.01603, 2019

    1912 arXiv

  17. [24]

    Y . Hu, L. Anderson, T.-M. Li, Q. Sun, N. Carr, J. Ragan-Kelley, and F. Durand. DiffTaichi: Differentiable Programming for Physical Simulation, Feb. 2020. URL http://arxiv.org/ abs/1910.00935. arXiv:1910.00935

    2020 arXiv

  18. [25]

    C. D. Freeman, E. Frey, A. Raichuk, S. Girgin, I. Mordatch, and O. Bachem. Brax – A Differentiable Physics Engine for Large Scale Rigid Body Simulation, June 2021. URL http://arxiv.org/abs/2106.13281. arXiv:2106.13281

    2021 arXiv

  19. [26]

    Todorov, T

    E. Todorov, T. Erez, and Y . Tassa. MuJoCo: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026– 5033, Oct. 2012. doi:10.1109/IROS.2012.6386109. URLhttps://ieeexplore.ieee.org/ abstract/document/638...

    2012

  20. [27]

    Heiden, D

    E. Heiden, D. Millard, E. Coumans, Y . Sheng, and G. S. Sukhatme. NeuralSim: Augmenting Differentiable Simulators with Neural Networks. In 2021 IEEE International Conference on Robotics and Automation (ICRA) , pages 9474–9481, May 2021. doi:10.1109/ICRA48506. 2021.9560935. URL...

    2021

  21. [28]

    T. A. Howell, S. L. Cleac’h, J. Br¨udigam, J. Z. Kolter, M. Schwager, and Z. Manchester. Dojo: A Differentiable Physics Engine for Robotics, Mar. 2023. URL http://arxiv.org/abs/ 2203.00806. arXiv:2203.00806

    2023 arXiv

  22. [29]

    F. Li, F. Sun, T. Zhang, and D. Zou. VisFly: An Efficient and Versatile Simulator for Training Vision-based Flight, Sept. 2024. URL http://arxiv.org/abs/2407.14783. arXiv:2407.14783

    2024 arXiv

  23. [30]

    Savva, A

    M. Savva, A. Kadian, O. Maksymets, Y . Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V . Koltun, J. Malik, et al. Habitat: A platform for embodied ai research. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9339–9347, 2019

    2019

  24. [31]

    Schoenholz and E

    S. Schoenholz and E. D. Cubuk. JAX MD: A Framework for Differentiable Physics. In Advances in Neural Information Processing Systems , volume 33, pages 11428–11441. Cur- ran Associates, Inc., 2020. URLhttps://proceedings.neurips.cc/paper/2020/hash/ 83d3d4b6c9579515e1679aca8cbc8...

    2020

  25. [32]

    Paszke, S

    A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation in pytorch. 2017

    2017

  26. [33]

    M. C. Mozer. A focused backpropagation algorithm for temporal pattern recognition. In Backpropagation, pages 137–169. Psychology Press, 2013

    2013

  27. [34]

    M. A. Z. Mora, M. Peychev, S. Ha, M. Vechev, and S. Coros. PODS: Policy Optimization via Differentiable Simulation. In Proceedings of the 38th International Conference on Machine Learning, pages 7805–7817. PMLR, July 2021. URL https://proceedings.mlr.press/ v139/mora21a.html. ...

    2021

  28. [35]

    J. Xu, V . Makoviychuk, Y . Narang, F. Ramos, W. Matusik, A. Garg, and M. Macklin. Ac- celerated Policy Learning with Parallel Differentiable Simulation, Apr. 2022. URL http: //arxiv.org/abs/2204.07137. arXiv:2204.07137

    2022 arXiv

  29. [37]

    D. P. Kingma. Auto-encoding variational bayes. 2013

    2013

  30. [38]

    V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. Human-level control through deep rein- forcement learning. nature, 518(7540):529–533, 2015

    2015

  31. [39]

    Haarnoja, A

    T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, 2018. URL https://arxiv.org/abs/ 1801.01290

    2018 arXiv

  32. [40]

    Raffin, A

    A. Raffin, A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann. Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research , 22 (268):1–8, 2021. 18

    2021

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.