Pith. sign in

REVIEW 3 major objections 4 minor 20 references

DiLQR: Differentiable Iterative Linear Quadratic Regulator via Implicit Differentiation

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper shows that the exact gradient of a converged iterative Linear Quadratic Regulator can be computed analytically through implicit differentiation, reducing backward-pass cost to O(1) in the number of solver iterations.

desk verdict DiLQR is a genuinely useful differentiable-iLQR recipe with a sign error in its central printed theorem that must be corrected before the paper can be relied on. read the letter →

arxiv 2506.17473 v1 pith:QZ647M3I submitted 2025-06-20 cs.RO

Shuyuan Wang , Philip D. Loewen , Michael Forbes , Bhushan Gopaluni , Wei Pan This is my paper · ORCID
classification cs.RO
keywords differentiablecontroliterativelinearquadraticregulatorimplicitdifferentiationtrajectoryoptimizationend-to-endlearningimitationmodel-basedcomputationalefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DiLQR treats the iterative Linear Quadratic Regulator as a trainable module whose parameter gradient is computed at the solver's fixed point rather than by unrolling every iteration. The paper's central claim is Proposition 4.1: the trajectory Jacobians obey a block linear system whose explicit block-elimination solution yields the exact gradient in O(1) time with respect to iteration count. In imitation-learning and system-identification experiments the method reports backward-pass speedups of 21x to 128x over automatic differentiation, imitation losses orders of magnitude below neural-network policies, and better parameter recovery than differentiable controllers that approximate the last solver layer as constant. If the claim is right, iLQR becomes a cheap, exact differentiable layer for long-horizon and high-dimensional end-to-end control.

What carries the argument

The load-bearing object is the fixed-point equation of a single iLQR iteration, written as $X=F(X,U,\theta)$, $U=G(X,U,\theta)$ for the state sequence $X$, control sequence $U$, and learnable parameters $\theta$. The argument differentiates this identity through the implicit function theorem, turns the result into a block linear system, and eliminates blocks to obtain an explicit solution in terms of $M=(I-F_X)^{-1}$ and $K=I-G_U$. Two further mechanisms carry the practical speed: a forward pass that reuses $\nabla_\theta x_t$ from one time step to the next, and a parallelized sparse treatment of tensor Jacobians such as $\partial D/\partial X$, whose off-diagonal blocks vanish.

What would settle it

Stop iLQR after a single iteration and compare the closed-form implicit gradient with the automatic-differentiation gradient of that one unrolled step: if they differ, the formula is not the gradient of the finite computation. A sharper test is to choose a parameter near a bifurcation where the fixed point is non-isolated and the matrix $K-G_X M F_U$ is nearly singular, and check whether the closed form diverges while finite differences of the converged fixed point remain bounded.

Watch

Extended reading notes

Core claim

At a converged trajectory, a single iLQR iteration leaves the state and control sequences unchanged, giving the fixed-point identity $X^\star=F(X^\star,U^\star,\theta)$, $U^\star=G(X^\star,U^\star,\theta)$. Differentiating this identity with respect to $\theta$ produces a linear system in the Jacobians $\nabla_\theta X^\star$ and $\nabla_\theta U^\star$, and Proposition 4.1 solves it in closed form: $$\nabla_\$\theta$ U^\star=(K-G_X M F_U)^{-1}(G_X M F_\$\theta$+G_\$\theta$),\qquad \nabla_\$\theta$ X^\star=M(F_\$\theta$+F_U\nabla_\$\theta$ U^\star),$$ with $M=(I-F_X)^{-1}$ and $K=I-G_U$. The formula is exact for the converged fixed point, unlike methods that differentiate only the last solver layer and treat its input as constant. The paper completes the recipe with a forward algorithm for parameter derivatives of the linearized dynamics that reuses information across time steps and exploits block-diagonal sparsity, so the remaining tensor computations scale linearly with horizon rather than quadratically.

Load-bearing premise

The method is exact only when iLQR has actually converged to an isolated fixed point for the current parameters and the matrix $K-G_X M F_U$ is invertible; with early stopping, or at a bifurcation where the fixed point is non-isolated, the implicit gradient differs from the gradient of the finite computation.

Editorial extensions

If this is right

  • Backward-pass cost and memory become independent of iLQR iteration count, so long-horizon trajectory optimization can be trained without storing the unrolled computation graph.
  • The exact analytic gradient replaces the common last-layer-only approximation that treats the solver input as fixed, which should improve parameter recovery in imitation and system-identification tasks.
  • Because the gradient is exact at the fixed point, iLQR can serve as a reliable differentiable module inside larger networks, including those with image encoders and decoders.
  • The same fixed-point differentiation recipe applies to other recursive optimal-control solvers whose iteration map depends on the trajectory itself.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to monitor the residual $\|(X,U)-(F,G)\|$ and the condition number of $K-G_X M F_U$ during training, using them as stopping criteria or as signals to re-run the solver to tighter convergence before trusting the gradient.
  • Because training no longer needs to store the unrolled solver, one can afford to re-solve iLQR from many different initial states at every optimization step, making bi-level loops such as model-based reinforcement learning practical at a scale that was previously prohibitive.
  • The derivation assumes first-order difference-equation dynamics; an obvious extension is to continuous-time or stochastic dynamics, where the fixed-point map changes but the block-elimination structure should survive as long as the relevant inverse exists.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces DiLQR, a framework for differentiating through the Iterative Linear Quadratic Regulator (iLQR) by applying implicit differentiation at a fixed point of the solver. The central claim is that the Jacobians of the converged iLQR trajectory with respect to learnable parameters can be computed analytically through a closed-form block-elimination solution (Proposition 4.1), giving O(1) backward-pass cost in the number of iLQR iterations. The authors derive the fixed-point equations, use known LQR derivative formulas to compute the necessary Jacobian blocks, propose a forward algorithm for parameter derivatives, and report experiments on cartpole and inverted pendulum imitation learning, model recovery, ablation studies, and a visual control task.

Significance. If the central claim holds, DiLQR would be a practically valuable contribution: it would make iLQR a scalable differentiable module with backward cost independent of the number of solver iterations, directly addressing the memory and compute bottlenecks of unrolled differentiation. The paper has clear strengths: the derivation is a first-principles application of implicit differentiation to a defined operator, it builds on independent prior results for LQR derivatives (Amos et al.), the speedup measurements are clearly reported, and a limitations section explicitly acknowledges the fixed-point assumption. The numerical experiments show consistent computational gains and the visual control demonstration adds an end-to-end dimension. The main technical obstacle is the sign inconsistency in the stated central theorem, which must be fixed before the claims can be accepted.

major comments (3)
  1. [Section 4.2, Eq. (11); Appendix A.1, Eqs. (18) and (23)] The formula for ∇θX⋆ in Proposition 4.1 contains a sign error: the main text and the appendix proposition statement both show −Gθ inside the parentheses, whereas the block-elimination proof in Appendix A.1 concludes with +Gθ in Eq. (23). Substituting the derived dU/dθ = (K−GXMFU)^{-1}(GXMFθ+Gθ) into dX/dθ = M(Fθ+FU dU/dθ) yields the plus sign. A reader implementing Eq. (11) as written would compute incorrect state-trajectory gradients even when the fixed-point assumptions hold, so this is a load-bearing error in the central theorem.
  2. [Section 4.2 and Section 6.4] The claim that the method gives an exact analytical gradient is only true at a fixed point; for early-stopped iLQR, the implicit gradient does not equal the gradient of the finite unrolled computation. The paper acknowledges the fixed-point assumption in Section 6.4 but does not qualify the contributions accordingly, nor does it report convergence behavior for the experiments in Section 5. The authors should state clearly that exactness is a fixed-point property and provide evidence that their experimental runs satisfy it.
  3. [Proposition 4.1 and Eq. (10)] The stated solution requires invertibility of K−GXMFU and differentiability of the fixed point map, but the paper gives no conditions under which these hold. The authors should state explicit hypotheses (for example, nonsingularity of the Schur complement, an isolated fixed point, and Lipschitz regularity of F and G) or discuss the degenerate cases where the formula breaks down.
minor comments (4)
  1. [Section 5.2 and Figure 3] The evaluation protocol of selecting the best validation loss during training and reporting the corresponding test loss is a form of peeking; the authors should report the test loss at the epoch selected by training loss alone, or justify the protocol as inherited from prior work.
  2. [Appendix A.5] The note about AI-assisted data summarization and truncated runs should be accompanied by the exact filtering pipeline used in the final experiments, so that the corrected numbers in Table 1 are reproducible and readers can assess the impact of the excluded runs.
  3. [Abstract and Section 5.2] Several numeric claims are typeset inconsistently, with '10^6' and '10^4' appearing as '106' and '104' in the text; these should be formatted uniformly.
  4. [Section 5.2, Comparison to Other Differentiable Control Methods] The comparison against SafePDP and IDOC would be more convincing if all alignment steps were fully described, including the number of trajectories, the initialization, and the exact loss curves, rather than a single aggregated paragraph.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: DiLQR's implicit-differentiation derivation is self-contained and its cited LQR gradient formulas are independent external results.

full rationale

The central derivation differentiates the fixed-point identity X*=F(X*,U*,θ), U*=G(X*,U*,θ) (eq. 8) and solves the resulting linear system (eq. 10) by block elimination. Proposition 4.1 is a closed-form solution to that linear system, not an input assumed into the system; it is not defined in terms of the gradients it purports to predict. The ingredients ∂H/∂D, ∂H/∂d, ∂H/∂C, ∂H/∂c are taken from Amos et al. (2018) and Amos & Kolter (2017), which are prior, independent works by other authors, and these are used only to assemble the Jacobians of the single iLQR iteration operator, not to presuppose the fixed-point sensitivity. No parameter is fitted and then renamed a prediction; the claims are checked against external baselines (autodiff, DiffMPC) and against ground-truth model parameters. The manuscript's acknowledged reliance on convergence to a fixed point (Section 6.4) is an applicability condition, not a circularity. The internal sign discrepancy between Proposition 4.1 (eq. 11) and the Appendix proof (eq. 23) is a correctness/inconsistency concern, and the A.5 note about AI-assisted data summarization is an experimental-reporting concern; neither is a reduction of the derivation to its own inputs. Accordingly, the circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central derivation introduces no free parameters: the learned quantities in the experiments (dynamics parameters, cost weights) are the targets of learning, not inputs to the derivation. It assumes standard differentiability and convergence conditions for iLQR and relies on the implicit function theorem.

assumptions (4)
  • domain assumption The iLQR iteration converges to a fixed point for each parameter, and the fixed point is isolated and differentiable.
    Section 4.2 defines the sequence of trajectories converging to a fixed point and Section 6.4 acknowledges reliance on convergence. If convergence fails or the fixed point is non-isolated, the implicit function theorem does not apply.
  • domain assumption The dynamics and costs are at least twice differentiable, so the LQR data D_t, d_t, C_t, c_t exist and the chain-rule terms in eq. (13) are defined.
    Used throughout Section 3.1 for linearization and quadratic approximation and Section 4.3 for the chain-rule decomposition.
  • domain assumption The LQR subproblem in eq. (4) has a unique, differentiable solution for the control update, requiring positive definiteness of the relevant matrix (with regularization or projected-Newton handling of box constraints) at the fixed point.
    The derivative formulas for the LQR solution from Amos et al. (2018) are invoked in Section 4.3 and assume a well-posed LQR solution.
  • standard math The Implicit Function Theorem is applicable to the fixed-point identity in eq. (8).
    Used implicitly in Section 4.2 to differentiate the identity and solve the linear system in eq. (10).

how reviews work

0 comments
Cite this review

Pith. "Pith review of DiLQR: Differentiable Iterative Linear Quadratic Regulator via Implicit Differentiation." pith.science (2026). https://pith.science/paper/QZ647M3I

@misc{pith2026250617473,
  author       = {Pith},
  title        = {Pith review of: DiLQR: Differentiable Iterative Linear Quadratic Regulator via Implicit Differentiation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QZ647M3I}},
  note         = {Machine review of arXiv:2506.17473}
}
abstract

While differentiable control has emerged as a powerful paradigm combining model-free flexibility with model-based efficiency, the iterative Linear Quadratic Regulator (iLQR) remains underexplored as a differentiable component. The scalability of differentiating through extended iterations and horizons poses significant challenges, hindering iLQR from being an effective differentiable controller. This paper introduces DiLQR, a framework that facilitates differentiation through iLQR, allowing it to serve as a trainable and differentiable module, either as or within a neural network. A novel aspect of this framework is the analytical solution that it provides for the gradient of an iLQR controller through implicit differentiation, which ensures a constant backward cost regardless of iteration, while producing an accurate gradient. We evaluate our framework on imitation tasks on famous control benchmarks. Our analytical method demonstrates superior computational performance, achieving up to 128x speedup and a minimum of 21x speedup compared to automatic differentiation. Our method also demonstrates superior learning performance ($10^6$x) compared to traditional neural network policies and better model loss with differentiable controllers that lack exact analytical gradients. Furthermore, we integrate our module into a larger network with visual inputs to demonstrate the capacity of our method for high-dimensional, fully end-to-end tasks. Codes can be found on the project homepage https://sites.google.com/view/dilqr/.

Figures

Figures reproduced from arXiv: 2506.17473 by the authors.

Figure 1
Figure 1. An overview of iLQR, and AutoDiff vs our proposed planner with implicit differentiation. As shown in the [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Backward computation time comparison between AutoDiff and our proposed method across different iLQR [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Learning results on the pendulum and cartpole. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (7 more)
Figure 5
Figure 5. Figure 5: Model loss curves for dynamics learning under [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Training loss comparison on the Cartpole task. [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 8
Figure 8. Figure 8: Diagram of the end-to-end control architecture. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 7
Figure 7. Figure 7: Computation time comparison across different [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 9
Figure 9. Figure 9: Imagined trajectory in the pendulum domain. [PITH_FULL_IMAGE:figures/full_fig_p009_9.png]
Figure 10
Figure 10. Figure 10: Comparison of image prediction error across [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]
Figure 11
Figure 11. Figure 11: Aggregated Cost Learning Curve (Cartpole, [PITH_FULL_IMAGE:figures/full_fig_p014_11.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 13 canonical work pages

  1. [8]

    Actor-critic model predictive control: Differentiable opti- mization meets reinforcement learning

    Romero, A., Aljalbout, E., Song, Y ., and Scaramuzza, D. Actor-critic model predictive control: Differentiable opti- mization meets reinforcement learning. Romero, A., Song, Y ., and Scaramuzza, D. Actor-critic model predictive control. In2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 14777–14784, 2024a. doi: 10.1109/ICRA57147.2...

    arXiv

  2. [9]

    Value Iteration Networks on Multiple Levels of Abstraction

    Schleich, D., Klamt, T., and Behnke, S. Value iteration networks on multiple levels of abstraction.arXiv preprint arXiv:1905.11068,

    work page Pith review arXiv 1905

  3. [10]

    Proximal policy optimization algorithms

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,

    arXiv

  4. [12]

    A., Brown, M., and Gerdes, J

    Spielberg, N. A., Brown, M., and Gerdes, J. C. Neural network model predictive motion control applied to auto- mated driving with unknown friction.IEEE Transactions on Control Systems Technology, 30(5):1934–1945,

    work page 1934

  5. [18]

    Zhao, L., Xu, H., and Wong, L. L. Scaling up and stabilizing differentiable planning with implicit differentiation.arXiv preprint arXiv:2210.13542,

    work page Pith review arXiv

  6. [20]

    Appendix A.1

    12 DiLQR: Differentiabl ILQR via Implicit Differentiation A. Appendix A.1. Proof of proposition 1 Proposition A.1.Define Fθ := ∂F ∂θ ,FU := ∂F ∂U ,FX := ∂F ∂X ,Gθ := ∂G ∂θ ,GU := ∂G ∂U ,GX := ∂G ∂X . DefineM:= (I−F X)−1, andK:=I−G U . The analytical form of the gradients dX dθ and dU dθ are given as follows: dX dθ =M(F θ +FU(K−G XMFU)−1(GXMFθ−Gθ)) dU dθ =...

    work page 2018

  7. [2004]

    Gpt- driver: Learning to drive with gpt.arXiv preprint arXiv:2310.01415,

    Mao, J., Qian, Y ., Ye, J., Zhao, H., and Wang, Y . Gpt- driver: Learning to drive with gpt.arXiv preprint arXiv:2310.01415,

    arXiv

  8. [2009]

    DiffTORI: Differentiable Trajectory Optimization for Deep Reinforcement and Imitation Learning

    Wan, W., Wang, Z., Wang, Y ., Erickson, Z., and Held, D. Difftori: Differentiable trajectory optimization for deep reinforcement and imitation learning.arXiv preprint arXiv:2402.05421,

    work page Pith review arXiv

Show all 20 references
  1. [2011]

    Differentiable optimal control via differential dy- namic programming.arXiv preprint arXiv:2209.01117,

    Dinev, T., Mastalli, C., Ivan, V ., Tonneau, S., and Vijayaku- mar, S. Differentiable optimal control via differential dy- namic programming.arXiv preprint arXiv:2209.01117,

    arXiv

  2. [2012]

    Deep reinforce- ment learning with double Q-learning

    van Hasselt, H., Guez, A., and Silver, D. Deep reinforce- ment learning with double Q-learning. InProceedings of the 30th AAAI conference on artificial intelligence, vol- ume 30, pp. 2094–2100,

  3. [2015]

    Learning model predictive controllers with real-time attention for real-world navigation.arXiv preprint arXiv:2209.10780,

    Xiao, X., Zhang, T., Choromanski, K., Lee, E., Francis, A., Varley, J., Tu, S., Singh, S., Xu, P., Xia, F., et al. Learning model predictive controllers with real-time attention for real-world navigation.arXiv preprint arXiv:2209.10780,

    arXiv

  4. [2016]

    doi: 10.1609/aaai.v30i1. 10295. Vrabie, D., Pastravanu, O., Abu-Khalaf, M., and Lewis, F. Adaptive optimal control for continuous-time linear systems based on policy iteration.Automatica, 45(2): 477–484,

    doi

  5. [2017]

    v31i1.10916

    doi: 10.1609/aaai. v31i1.10916. Bai, S., Kolter, J. Z., and Koltun, V . Deep equilibrium mod- els.Advances in neural information processing systems, 32,

    doi

  6. [2018]

    Path integral net- works: End-to-end differentiable optimal control.arXiv preprint arXiv:1706.09597,

    Okada, M., Rigazio, L., and Aoshima, T. Path integral net- works: End-to-end differentiable optimal control.arXiv preprint arXiv:1706.09597,

    arXiv

  7. [2019]

    D., An, G

    Pereira, M., Fan, D. D., An, G. N., and Theodorou, E. MPC- inspired neural network policies for sequential decision making.arXiv preprint arXiv:1802.05803,

    arXiv

  8. [2020]

    J., and Johnson, A

    Zhu, J., Payne, J. J., and Johnson, A. M. Convergent ilqr for safe trajectory planning and control of legged robots. arXiv preprint arXiv:2304.00346,

    arXiv

  9. [2021]

    Constrained iterative lqr for on-road autonomous driving motion planning

    Chen, J., Zhan, W., and Tomizuka, M. Constrained iterative lqr for on-road autonomous driving motion planning. In 2017 IEEE 20th International conference on intelligent transportation systems (ITSC), pp. 1–7. IEEE,

    2017

  10. [2022]

    Infinite-horizon differentiable model predictive control

    10 DiLQR: Differentiabl ILQR via Implicit Differentiation East, S., Gallieri, M., Masci, J., Koutnik, J., and Cannon, M. Infinite-horizon differentiable model predictive control. Proceedings of ICLR 2020,

    2020

  11. [2023]

    M., Wang, Y ., Brunton, S

    Soudbakhsh, D., Annaswamy, A. M., Wang, Y ., Brunton, S. L., Gaudio, J., Hussain, H., Vrabie, D., Drgona, J., and Filev, D. Data-driven control: Theory and applications. In2023 American Control Conference (ACC), pp. 1922–

    1922

  12. [2024]

    P., Loewen, P

    Wang, S., Duan, J., Lawrence, N. P., Loewen, P. D., Forbes, M. G., Gopaluni, R. B., and Zhang, L. Guiding reinforce- ment learning with incomplete system dynamics.arXiv preprint arXiv:2410.16821,

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.