{"id":"20bf3b52-4dd7-4eea-a9e4-9fec186f9913","arxiv_id":"2506.17473","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"DiLQR computes gradients of a converged iLQR controller with implicit differentiation, making the backward pass cost constant in the number of solver iterations.","lead":"The paper derives analytical gradients for an iterative LQR controller using implicit differentiation, giving a backward-pass cost that stays constant as the number of solver iterations grows. This could make model-based controllers practical as trainable layers in neural networks for control and imitation learning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 4.1 contains a sign error in the closed-form Jacobian of the state; the main-text formula contradicts its own derivation, so the central theorem is false as stated.","rationale":"The reader's verdict of CONDITIONAL is appropriate, and my read does not change it. The reader identified a sign error in the main text but treated the weakest assumption as convergence of iLQR and invertibility of the linear system in eq. (10). I agree that invertibility and convergence are real assumptions and are disclosed in Section 6.4. However, the sign inconsistency between Proposition 4.1 and its own proof in Appendix A.1 is the most load-bearing concern because the central contribution is explicitly that closed-form analytical solution. A formula that is wrong as printed cannot support the claim of exact analytical gradients, regardless of whether a corrected version appears nearby in the appendix. This is not an ad hominem or an appeal to consensus; it is a checkable internal inconsistency in the mathematical core. The concrete test is straightforward and would settle the issue. The paper's broader methodology—implicit differentiation at the iLQR fixed point with O(1) backward cost—remains plausible and valuable, and the speedup measurements are not invalidated by this sign error. Therefore the verdict stays CONDITIONAL: the authors must correct Proposition 4.1, align the main text with the appendix, and ideally add a numerical validation of the stated formula against unrolled autodiff before the paper can be accepted.","tokens_in":17545,"tokens_out":8466,"duration_ms":90125,"concrete_test":"On a small cartpole or pendulum problem with known parameters, run iLQR to a tight convergence tolerance. Then compute the Jacobians dX/dθ and dU/dθ by three routes: (1) full unrolled autodiff through the iLQR iterations (the ground truth for the finite computation), (2) Proposition 4.1 exactly as printed in the main text (with −Gθ), and (3) the corrected formula from Appendix eq. (23) (with +Gθ). If route (2) does not match the autodiff result while route (3) does, the main-text formula is demonstrably wrong and must be corrected. As a secondary check, repeat the comparison at 50 and 300 iterations with the same tolerance to verify whether the implicit gradient matches unrolled autodiff only in the converged limit, which would confirm the convergence assumption is also empirically load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of the paper, that Proposition 4.1 provides the exact analytical solution for the iLQR fixed-point Jacobians, is undermined by an internal inconsistency. In the main text, Proposition 4.1 (eq. 11) gives ∇θX⋆ = M(Fθ + FU(K−GX M FU)^{-1}(GX M Fθ − Gθ)), while the proof in Appendix A.1, specifically the substitution step leading to eq. (23), derives ∇θX⋆ = M(Fθ + FU(K−GX M FU)^{-1}(GX M Fθ + Gθ)). The sign on Gθ differs: the proposition statement has a minus, the proof has a plus. Substituting the derived dU/dθ = (K−GX M FU)^{-1}(GX M Fθ + Gθ) into dX/dθ = M(Fθ + FU dU/dθ) yields the plus sign, so the printed Proposition 4.1 is incorrect as written and the error is not merely typographical in a secondary term—it affects the gradient of the full state trajectory, which is at the core of the method's claimed exactness. A reader implementing the formula from the main text would compute incorrect gradients even when iLQR has fully converged and the implicit function theorem assumptions hold. While the surrounding derivation suggests the correct formula, the published central theorem does not match it. This is a load-bearing correctness issue distinct from the already-acknowledged convergence limitation: the convergence concern affects the applicability regime, whereas the sign error affects the stated mathematical result itself. I therefore treat this as the single most load-bearing concern.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DiLQR, a framework for differentiating through the Iterative Linear Quadratic Regulator (iLQR) by applying implicit differentiation at a fixed point of the solver. The central claim is that the Jacobians of the converged iLQR trajectory with respect to learnable parameters can be computed analytically through a closed-form block-elimination solution (Proposition 4.1), giving O(1) backward-pass cost in the number of iLQR iterations. The authors derive the fixed-point equations, use known LQR derivative formulas to compute the necessary Jacobian blocks, propose a forward algorithm for parameter derivatives, and report experiments on cartpole and inverted pendulum imitation learning, model recovery, ablation studies, and a visual control task.","tokens_in":17816,"tokens_out":6476,"duration_ms":65060,"significance":"If the central claim holds, DiLQR would be a practically valuable contribution: it would make iLQR a scalable differentiable module with backward cost independent of the number of solver iterations, directly addressing the memory and compute bottlenecks of unrolled differentiation. The paper has clear strengths: the derivation is a first-principles application of implicit differentiation to a defined operator, it builds on independent prior results for LQR derivatives (Amos et al.), the speedup measurements are clearly reported, and a limitations section explicitly acknowledges the fixed-point assumption. The numerical experiments show consistent computational gains and the visual control demonstration adds an end-to-end dimension. The main technical obstacle is the sign inconsistency in the stated central theorem, which must be fixed before the claims can be accepted.","major_comments":[{"comment":"The formula for ∇θX⋆ in Proposition 4.1 contains a sign error: the main text and the appendix proposition statement both show −Gθ inside the parentheses, whereas the block-elimination proof in Appendix A.1 concludes with +Gθ in Eq. (23). Substituting the derived dU/dθ = (K−GXMFU)^{-1}(GXMFθ+Gθ) into dX/dθ = M(Fθ+FU dU/dθ) yields the plus sign. A reader implementing Eq. (11) as written would compute incorrect state-trajectory gradients even when the fixed-point assumptions hold, so this is a load-bearing error in the central theorem.","section":"Section 4.2, Eq. (11); Appendix A.1, Eqs. (18) and (23)"},{"comment":"The claim that the method gives an exact analytical gradient is only true at a fixed point; for early-stopped iLQR, the implicit gradient does not equal the gradient of the finite unrolled computation. The paper acknowledges the fixed-point assumption in Section 6.4 but does not qualify the contributions accordingly, nor does it report convergence behavior for the experiments in Section 5. The authors should state clearly that exactness is a fixed-point property and provide evidence that their experimental runs satisfy it.","section":"Section 4.2 and Section 6.4"},{"comment":"The stated solution requires invertibility of K−GXMFU and differentiability of the fixed point map, but the paper gives no conditions under which these hold. The authors should state explicit hypotheses (for example, nonsingularity of the Schur complement, an isolated fixed point, and Lipschitz regularity of F and G) or discuss the degenerate cases where the formula breaks down.","section":"Proposition 4.1 and Eq. (10)"}],"minor_comments":[{"comment":"The evaluation protocol of selecting the best validation loss during training and reporting the corresponding test loss is a form of peeking; the authors should report the test loss at the epoch selected by training loss alone, or justify the protocol as inherited from prior work.","section":"Section 5.2 and Figure 3"},{"comment":"The note about AI-assisted data summarization and truncated runs should be accompanied by the exact filtering pipeline used in the final experiments, so that the corrected numbers in Table 1 are reproducible and readers can assess the impact of the excluded runs.","section":"Appendix A.5"},{"comment":"Several numeric claims are typeset inconsistently, with '10^6' and '10^4' appearing as '106' and '104' in the text; these should be formatted uniformly.","section":"Abstract and Section 5.2"},{"comment":"The comparison against SafePDP and IDOC would be more convincing if all alignment steps were fully described, including the number of trajectories, the initialization, and the exact loss curves, rather than a single aggregated paragraph.","section":"Section 5.2, Comparison to Other Differentiable Control Methods"}],"recommendation":"major_revision","confidential_remarks":"The sign inconsistency in the central proposition is concerning but appears to be a fixable error, since the proof itself contains the correct expression. The Appendix A.5 disclosure about AI-assisted data summarization is an open integrity concern; although the final numbers are presented as corrected, the paper would benefit from a clear statement of the exact reproduction protocol. The core methodology is defensible and the experiments show strong computational gains, so a careful revision addressing the theorem statement and its hypotheses should be sufficient."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know two things about DiLQR. First, the central idea is genuinely useful: differentiate the iLQR iteration at its fixed point via the implicit function theorem, get a backward pass that costs O(1) in the number of iterations, and add a forward recursion to propagate derivatives of the linearized dynamics. That is a real engineering contribution and the speedups over unrolled autodiff (21–128x) are plausible and cleanly measured. Second, the main-text statement of Proposition 4.1 has a sign error in the X-star formula: eq. (11) has (GX M Fθ − Gθ), while the appendix proof of the same proposition (eq. 23) derives (GX M Fθ + Gθ). The U-star formula matches the proof, so this is not a systematic inconsistency — but a reader implementing the printed theorem will get wrong gradients for the state trajectory. That is load-bearing, and it must be fixed before the paper is reliable as written.\n\nWhat the paper does well: the fixed-point formulation of iLQR is new in this exact form; prior differentiable LQR/MPC work either unrolls or differentiates the last layer treating the input as constant. The forward algorithm for ∇θDt exploiting temporal reuse is a useful practical contribution. The appendix block elimination is correct modulo the typo. The authors also disclose in A.5 that a rebuttal statistic was recomputed with a stricter pipeline — that kind of transparency is rare and should be credited.\n\nSoft spots, in proportion: the sign error is the big one. Next, the exactness claim is conditional on iLQR actually converging to a fixed point; the paper states this in 6.4 but the experiments don't consistently verify how close the trajectories were to fixed points. The comparison to SafePDP and IDOC changes the setup in their favor, which is fine for a scientific claim, but the headline '10^6x' improvement over NN is on toy benchmarks with small datasets and should not be overread. The visual control experiment is qualitative and not statistically decisive.\n\nWho this is for: anyone building differentiable model-based control pipelines, or doing RL/imitation with embedded trajectory optimizers. The paper deserves a serious referee, not a desk reject. I'd recommend major revision: fix the sign in Proposition 4.1, add convergence checks, and re-run the SafePDP/IDOC comparison under a fairness-neutral protocol.\n\nBring it to reading group? Yes. I wouldn't cite the current version until the theorem is corrected.","headline":"DiLQR is a genuinely useful differentiable-iLQR recipe with a sign error in its central printed theorem that must be corrected before the paper can be relied on.","tokens_in":18410,"tokens_out":2287,"would_cite":false,"duration_ms":21365,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that the exact gradient of a converged iterative Linear Quadratic Regulator can be computed analytically through implicit differentiation, reducing backward-pass cost to O(1) in the number of solver iterations.","keywords":["differentiable control","iterative linear quadratic regulator","implicit differentiation","trajectory optimization","end-to-end learning","imitation learning","model-based control","computational efficiency"],"falsifier":"Stop iLQR after a single iteration and compare the closed-form implicit gradient with the automatic-differentiation gradient of that one unrolled step: if they differ, the formula is not the gradient of the finite computation. A sharper test is to choose a parameter near a bifurcation where the fixed point is non-isolated and the matrix $K-G_X M F_U$ is nearly singular, and check whether the closed form diverges while finite differences of the converged fixed point remain bounded.","tokens_in":17310,"feed_emoji":"🤖","tokens_out":10692,"duration_ms":96636,"temperature":0.7,"pith_summary":"DiLQR treats the iterative Linear Quadratic Regulator as a trainable module whose parameter gradient is computed at the solver's fixed point rather than by unrolling every iteration. The paper's central claim is Proposition 4.1: the trajectory Jacobians obey a block linear system whose explicit block-elimination solution yields the exact gradient in O(1) time with respect to iteration count. In imitation-learning and system-identification experiments the method reports backward-pass speedups of 21x to 128x over automatic differentiation, imitation losses orders of magnitude below neural-network policies, and better parameter recovery than differentiable controllers that approximate the last solver layer as constant. If the claim is right, iLQR becomes a cheap, exact differentiable layer for long-horizon and high-dimensional end-to-end control.","feed_headline":"Exact iLQR gradients at 128x speed","feed_subtitle":"Implicit differentiation at the solver's fixed point removes the cost of unrolling hundreds of iterations.","key_machinery":"The load-bearing object is the fixed-point equation of a single iLQR iteration, written as $X=F(X,U,\\theta)$, $U=G(X,U,\\theta)$ for the state sequence $X$, control sequence $U$, and learnable parameters $\\theta$. The argument differentiates this identity through the implicit function theorem, turns the result into a block linear system, and eliminates blocks to obtain an explicit solution in terms of $M=(I-F_X)^{-1}$ and $K=I-G_U$. Two further mechanisms carry the practical speed: a forward pass that reuses $\\nabla_\\theta x_t$ from one time step to the next, and a parallelized sparse treatment of tensor Jacobians such as $\\partial D/\\partial X$, whose off-diagonal blocks vanish.","core_discovery":"At a converged trajectory, a single iLQR iteration leaves the state and control sequences unchanged, giving the fixed-point identity $X^\\star=F(X^\\star,U^\\star,\\theta)$, $U^\\star=G(X^\\star,U^\\star,\\theta)$. Differentiating this identity with respect to $\\theta$ produces a linear system in the Jacobians $\\nabla_\\theta X^\\star$ and $\\nabla_\\theta U^\\star$, and Proposition 4.1 solves it in closed form: $$\\nabla_\\$\\theta$ U^\\star=(K-G_X M F_U)^{-1}(G_X M F_\\$\\theta$+G_\\$\\theta$),\\qquad \\nabla_\\$\\theta$ X^\\star=M(F_\\$\\theta$+F_U\\nabla_\\$\\theta$ U^\\star),$$ with $M=(I-F_X)^{-1}$ and $K=I-G_U$. The formula is exact for the converged fixed point, unlike methods that differentiate only the last solver layer and treat its input as constant. The paper completes the recipe with a forward algorithm for parameter derivatives of the linearized dynamics that reuses information across time steps and exploits block-diagonal sparsity, so the remaining tensor computations scale linearly with horizon rather than quadratically.","pith_inferences":["A testable extension is to monitor the residual $\\|(X,U)-(F,G)\\|$ and the condition number of $K-G_X M F_U$ during training, using them as stopping criteria or as signals to re-run the solver to tighter convergence before trusting the gradient.","Because training no longer needs to store the unrolled solver, one can afford to re-solve iLQR from many different initial states at every optimization step, making bi-level loops such as model-based reinforcement learning practical at a scale that was previously prohibitive.","The derivation assumes first-order difference-equation dynamics; an obvious extension is to continuous-time or stochastic dynamics, where the fixed-point map changes but the block-elimination structure should survive as long as the relevant inverse exists."],"forward_implications":["Backward-pass cost and memory become independent of iLQR iteration count, so long-horizon trajectory optimization can be trained without storing the unrolled computation graph.","The exact analytic gradient replaces the common last-layer-only approximation that treats the solver input as fixed, which should improve parameter recovery in imitation and system-identification tasks.","Because the gradient is exact at the fixed point, iLQR can serve as a reliable differentiable module inside larger networks, including those with image encoders and decoders.","The same fixed-point differentiation recipe applies to other recursive optimal-control solvers whose iteration map depends on the trajectory itself."],"supporting_citations":[{"why":"It is the reference the paper cites for iLQR as the numerical controller being differentiated and for the benchmark task setup.","marker":"[Todorov et al., 2012]"},{"why":"It supplies the two-pass backward/forward iLQR update whose fixed point is the object being differentiated.","marker":"[Tassa et al., 2014]"},{"why":"It provides the Riccati-Mayne value-function recursion and projected-Newton action optimization used inside one iLQR iteration.","marker":"[Mayne et al., 2000]"},{"why":"It provides the O(T) sensitivity formulas for the LQR subproblem that the paper adopts for the Jacobian terms.","marker":"[Amos et al., 2018]"},{"why":"It supplies the differentiable-optimization machinery behind the efficient calculation of tensor sensitivities.","marker":"[Amos & Kolter, 2017]"},{"why":"It establishes the deep-equilibrium idea of differentiating through a fixed point rather than unrolling, which motivates the O(1) backward pass.","marker":"[Bai et al., 2019]"}],"fun_headline_variants":["Implicit differentiation gives exact iLQR gradients at 128x speed","Trainable iLQR: closed-form gradients, 128x speedup","Fixed-point trick yields exact iLQR gradients, no unrolling","Differentiable iLQR with constant backward cost, 128x faster","Exact gradients for iLQR without unrolling, up to 128x speed"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method is exact only when iLQR has actually converged to an isolated fixed point for the current parameters and the matrix $K-G_X M F_U$ is invertible; with early stopping, or at a bifurcation where the fixed point is non-isolated, the implicit gradient differs from the gradient of the finite computation.","fun_headline_variants_meta":{"raw":{"variants":["Implicit differentiation gives exact iLQR gradients at 128x speed","Trainable iLQR: closed-form gradients, 128x speedup","Fixed-point trick yields exact iLQR gradients, no unrolling","Differentiable iLQR with constant backward cost, 128x faster","Exact gradients for iLQR without unrolling, up to 128x speed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000243,"raw_usage":{"total_tokens":1566,"prompt_tokens":1020,"completion_tokens":546,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":636,"completion_tokens_details":{"reasoning_tokens":447}},"tokens_in":636,"tokens_out":546,"duration_ms":5374,"temperature":1.0,"reasoning_tokens":447,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:09:31.168767+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Stop iLQR after a single iteration and compare the closed-form implicit gradient with the automatic-differentiation gradient of that one unrolled step: if they differ, the formula is not the gradient of the finite computation. A sharper test is to choose a parameter near a bifurcation where the fixed point is non-isolated and the matrix $K-G_X M F_U$ is nearly singular, and check whether the closed form diverges while finite differences of the converged fixed point remain bounded.","supporting_citations":[],"review_version":2}