REVIEW 3 major objections 3 minor 15 references
Efficient Hypergradient Descent for Inverse Reinforcement Learning
T0 review · 3 major / 3 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read At an exactly solved inner optimum of bilevel IRL, the inner Hessian equals α times the trajectory Fisher information matrix, turning the implicit hypergradient into a damped inverse-Fisher solve computable from sampled trajectories with…
desk verdict The Hessian-Fisher identity is a real result, but the sketching algorithm has a scaling error that invalidates the reported experiments. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Hessian–Fisher identity of Proposition 4.2, together with the streaming SCFD sketch used to exploit it. The identity says that at an exactly solved, realizable inner optimum the Hessian of the inner reverse-KL objective equals α times the discounted trajectory Fisher information matrix, F_{θ} = E_{τ∼$p^{{π_θ}}$}[Σ_{t=1}^{∞} $γ^{{t-1}}$∇ log π_θ(a_t|s_t)∇ log π_θ(a_t|s_t)⊤]. This converts the implicit hypergradient's inverse-Hessian–vector product into a damped inverse-Fisher–vector product, which is then approximated without dense storage by representing the empirical Fisher as X⊤X, streaming the weighted score rows through SCFD, and computing the inverse-vector product from the sketch's low-rank factors (Algorithm 3).
What would settle it
In a tabular MDP with an expressive softmax policy, solve the inner problem exactly, then compare the analytic Hessian of L_inner at θ*(φ) with α times the analytic trajectory Fisher matrix; any nonzero difference would refute Proposition 4.2. On LQR, track the relative Frobenius error ∥bF_θ − $α^{{-1}}$∇²L_inner∥ / ∥$α^{{-1}}$∇²L_inner∥ while the inner PPO solve converges; if the error does not shrink toward zero, the practical approximation is unanchored.
Extended reading notes
Core claim
The paper establishes that, under exact inner optimality and the realizability condition D_KL(eπ_{θ*(φ)}∥e p_φ)=0, the Hessian of the inner loss in bilevel IRL satisfies ∇²_θ L_inner(θ*(φ), φ)=α F_{θ*(φ)}, where F is the discounted trajectory Fisher information matrix and α is the fixed temperature of the maximum-entropy trajectory model. It then uses this identity to rewrite the implicit hypergradient so that the hard step is solving (αF + λI)v = g for the outer gradient g, and supplies Monte Carlo estimators for each factor from agent and expert trajectories (Equations 6–8). To make the solve scalable, the paper represents the empirical Fisher as X⊤X with rows given by weighted policy-score vectors and streams these rows into a Spectral Compensation Frequent Directions sketch, which maintains a compact spectral approximation and returns v without ever forming the d_θ×d_θ matrix. The claim is that this yields a practical Fisher-based implicit hypergradient for ML-IRL whose storage is O(m d_θ), and the experiments on CartPole and LQR support its competitiveness against a single-loop baseline.
Load-bearing premise
The proportionality between the inner Hessian and the Fisher matrix is guaranteed only when the inner problem is solved exactly and the policy class can exactly realize the Boltzmann trajectory distribution; in the experiments the inner problem is solved approximately by PPO/SAC/REINFORCE with parametric policies, so the identity is used outside its provable regime and the resulting approximation error is neither bounded nor measured.
Editorial extensions
If this is right
- Hypergradients for ML-IRL become computable from sampled trajectories without constructing the inner Hessian or the dense Fisher matrix.
- Curvature storage drops from O(d_θ²) to O(m d_θ), so the method scales to larger policy parameterizations.
- The sketch size m and damping λ become the practical tuning knobs that trade memory, runtime, and curvature fidelity.
- On the tested environments, Fisher-based IRL is competitive with single-loop ML-IRL in policy quality and reward ranking within a matched budget, and sketching can improve wall-clock time and peak memory.
Reading between the lines
- A bound on ∥∇²L_inner − αF∥ in terms of the KL gap D_KL(eπ_{θ*(φ)}∥e p_φ) would turn the identity from a limit statement into a certified approximation, and the experiments do not currently provide that certificate.
- The same reverse-KL Hessian–Fisher structure likely appears in other bilevel RL settings such as iterative RLHF reward steering, so the sketching-based hypergradient may transfer beyond IRL.
- The nonmonotonic effect of sketch size on final loss suggests the sketch approximation error interacts with the damping and the optimization noise; a controlled study isolating these factors would clarify when larger sketches help.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies maximum-likelihood IRL as a bilevel program and derives an implicit hypergradient for the outer reward parameters. The central theoretical contribution is Proposition 4.2, which states that at an exactly solved, exactly realizable inner optimum the Hessian of the inner objective equals α times the discounted trajectory Fisher information matrix. Based on this identity, the authors propose replacing the inverse-Hessian-vector product in the implicit hypergradient with a damped inverse-Fisher-vector product, and then approximate that product with a streaming SCFD sketch that avoids forming the dense Fisher matrix. The method is evaluated on CartPole and LQR against the single-loop ML-IRL baseline, with experiments comparing explicit-Fisher and sketched-Fisher variants in terms of outer loss, policy quality, reward ranking, wall-clock time, and peak memory.
Significance. If the Hessian–Fisher identity and the sketched solver were both correct, the paper would offer a meaningful computational improvement for bilevel IRL: the hypergradient would be computable with O(m d_θ) storage instead of O(d_θ^2), and the derivation is non-circular in that the identity is proven from the reverse-KL structure rather than assumed. The appendix proofs (Propositions A.1, B.1–B.3, Corollary B.1) are internally consistent and the sign conventions chain correctly through the hypergradient. The clear statement of the realizability assumption is also a strength. However, the practical algorithms rely on approximate inner solutions and parametric policies for which the theorem's assumptions are not met, and, more seriously, the row scaling used to form the sketched Fisher matrix is algebraically inconsistent with the empirical Fisher definition. These issues are load-bearing for the paper's central claims.
major comments (3)
- [Section 5.2, Eq. (9)–(10) and Algorithm 2] The stacked-row construction is algebraically inconsistent with the empirical Fisher definition. Equation (9) defines rows x_t(τ) = sqrt(α/N_agent) γ^{t−1} ∇θ log πθ(a_t|s_t), so X^T X = (α/N_agent) Σ_{τ,t} γ^{2(t−1)} g g^T, whereas Equation (5) together with Proposition 4.2 gives α F̂_θ = (α/N_agent) Σ_{τ,t} γ^{t−1} g g^T. These coincide only when γ=1 or when every trajectory has a single step. Algorithm 2 line 10 feeds exactly these rows into the SCFD sketch, so the system actually solved is (X^T X + λI)v = g with the wrong discount weighting, not the damped trajectory-Fisher system claimed in Equation (10). Consequently, the numerical results cannot validate the proposed Fisher hypergradient unless the implementation used a different row definition, in which case the pseudocode and equations must be corrected.
- [Section 4, Proposition 4.2; Algorithms 1–2] The Hessian–Fisher identity is proven under exact inner optimality and exact realizability (D_KL = 0 at the inner optimum). In the experiments, the inner problem is solved approximately with PPO/SAC/REINFORCE and parametric policies that generically cannot realize the Boltzmann trajectory distribution; the paper provides no bound on the resulting bias in the Hessian approximation and reports no measurement of D_KL or of the error between ∇²_θ L_inner(θ*, φ) and αF_θ*. Because this identity is the entire theoretical justification for replacing the Hessian with αF in Equation (7), the gap between the theorem's assumptions and the algorithmic practice is load-bearing.
- [Section 6, Tables 2–4] Several computational-efficiency and performance entries are based on a single run, e.g., Table 3, CartPole Explicit Fisher (1464.41 s per outer iteration, Runs=1) and Table 3, LQR Fisher with Sketching m=256 (1316.88 s, Runs=1); Table 4 likewise reports single-run entries for some settings. The claimed speedups and memory gains in Table 1 are ratios of these noisy measurements. The paper should report the number of seeds for every entry and provide confidence intervals or paired-run comparisons; as presented, the empirical support for the efficiency and quality claims is underpowered.
minor comments (3)
- [Section 4, Proposition 4.2] The assumption 'the policy class is rich enough so that D_KL(eπ_{θ*(φ)} || e p_φ) = 0' is redundant given the preceding assumption that the inner problem is solved exactly; stating the assumption as 'there exists θ with zero KL divergence' would be clearer and would also make the role of realizability more transparent.
- [Section 6.1, Figure 2] The text says the sketched solver remains stable at λ=10^{-3} and that this value is fixed, but it is not stated whether λ was tuned separately for each sketch size; the conclusion that smaller sketches are more stable would be stronger if the damping was held constant across all compared configurations.
- [Section 6 and Appendix C] The experiments do not specify the hyperparameters of the inner RL solver (e.g., PPO or SAC settings, network architectures, optimization details) nor the discount factor γ used in each environment; without these details, the reported results cannot be reproduced.
Circularity Check
No significant circularity: the Hessian–Fisher identity is derived from the KL realizability assumption, the hypergradient estimators are sample-based and not fitted to test rewards, and the only self-citation is a non-load-bearing related-work pointer.
full rationale
The central identity ∇²θ Linner = αF is derived in Appendix B from the explicit assumption D_KL(ep_{θ*(φ)} || ep_φ)=0, not postulated. The proof uses the trajectory score identity and the constancy of ℓ+α at the realized optimum, so the Hessian–Fisher proportionality follows from the KL structure rather than being equivalent to its own conclusion. Equation 4 is the standard implicit-function hypergradient, and Equations 6–8 are Monte Carlo and score-function estimators whose definitions do not presuppose the outer objective. The SCFD sketch is imported from Chen et al. (2020) as an external algorithm, and the choice of damping λ and sketch size m on validation losses is ordinary hyperparameter selection, not fitting a prediction. The only self-citation (Shustova et al., including co-author Sheshukova) appears in related work on bandit sketching and is non-load-bearing; the paper's derivation does not rest on it. Separately, the reviewer's flagged algebraic mismatch in Eq. 9—rows use γ^{t-1} outside the square root, so XᵀX carries γ^{2(t-1)} weights rather than the γ^{t-1} of Eq. 5—is an internal correctness concern about the sketch construction, not a circularity: it does not make the derivation equivalent to its inputs. The paper is therefore self-contained on its stated assumptions, and the score reflects a minor self-citation and boundary case rather than reduction-by-construction.
Assumptions & free parameters
free parameters (4)
- Reward temperature α =
not specified (set by hand)
- Damping λ in Fisher system =
λ=10^{-3} for sketching experiments; λ=1 appears best for explicit in Figure 1
- Sketch size m =
m=8 (CartPole), m=32-256 (LQR), m=64 best in Figure 2
- Learning rate η_φ and inner-loop budget =
not reported
assumptions (6)
- domain assumption Exact inner optimality and realizability: D_KL(eπ_{θ*(φ)} || e p_φ)=0
- standard math Interchange of differentiation, expectation, and absolutely convergent discounted sums
- standard math Score-function identity and conditional independence: E[∇logπ(a_k|s_k)|past]=0
- domain assumption Empirical Fisher estimator approximates true discounted trajectory Fisher
- domain assumption The policy class is rich enough to represent the Boltzmann trajectory distribution
- domain assumption SCFD sketch provides a sufficiently accurate inverse-Fisher-vector product
Cite this review
Pith. "Pith review of Efficient Hypergradient Descent for Inverse Reinforcement Learning." pith.science (2026). https://pith.science/paper/VSYNMMNE
@misc{pith2026260811052,
author = {Pith},
title = {Pith review of: Efficient Hypergradient Descent for Inverse Reinforcement Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/VSYNMMNE}},
note = {Machine review of arXiv:2608.11052}
}
read the original abstract
Inverse reinforcement learning (IRL) aims to recover a reward function under which the resulting policy reproduces the behavior observed in expert demonstrations. A natural approach is to formulate IRL as a bilevel optimization problem, in which the inner level corresponds to policy optimization under the learned reward and the outer level measures the discrepancy between the induced policy and expert data. However, this formulation is computationally challenging in practice because the outer update requires a hypergradient involving an inverse-Hessian-vector product for the inner objective. We address this challenge by showing that, at the inner optimum, the Hessian of the inner objective is proportional to the Fisher information matrix of the policy, yielding a structured Fisher-based hypergradient closely related to Natural Hypergradient Descent. To address the resulting scalability bottleneck associated with large Fisher matrices, we approximate the required inverse-Fisher-vector product using a streaming spectral sketch, avoiding explicit construction of the Fisher matrix. We evaluate our approach against a first-order stochastic bilevel baseline across discrete- and continuous-control environments. The results demonstrate competitive policy performance and strong reward-ranking quality, while Fisher sketching reduces curvature-storage complexity and can improve computational efficiency relative to an explicit Fisher solver.
Figures
Reference graph
Works this paper leans on
-
[1]
Therefore, αDKL(epπθ ∥epϕ) =E τ∼epπθ " ∞X t=1 (αlogπ θ(at |s t)−r ϕ(st, at)) # +αlogZ ϕ
Moreover, logepϕ(τ) = logeρ(τ) + 1 α ∞X t=1 rϕ(st, at)−logZ ϕ. Therefore, αDKL(epπθ ∥epϕ) =E τ∼epπθ " ∞X t=1 (αlogπ θ(at |s t)−r ϕ(st, at)) # +αlogZ ϕ. 12 Published as a conference paper at ICOMP 2026 The normalization term is independent ofθ. Applying Lemma 4.1 to the expectation shows that minimizing the inner KL divergence is equivalent to minimizing E...
work page 2026
-
[2]
∞X t=1 γt−1 logπ θ(at |s t) # =−E τ∼p expert
Hence the bilevel KL problem in Equation 1 and the discounted bilevel problem in Equation 3 have the same minimizers. Proposition B.1(Discounted outer-objective gradient).The discounted outer objective satisfies ∇θLouter(θ) =−E τ∼p expert " ∞X t=1 γt−1∇θ logπ θ(at |s t) # . Proof.The expert trajectory distribution does not depend onθ. Therefore, using the...
work page 2026
-
[6]
Deyi Kong, Zaiwei Chen, Shuzhong Zhang, and Shancong Mou. Natural hypergradient de- scent: Algorithm design, convergence analysis, and parallel implementation.arXiv preprint arXiv:2602.10905,
-
[9]
W. J. A. van Heeswijk. Natural policy gradients in reinforcement learning explained.arXiv preprint arXiv:2209.01820,
-
[12]
Therefore, dθ⋆ dϕ ϕ =− " ∂g ∂θ θ⋆(ϕ),ϕ #−1 ∂g ∂ϕ θ⋆(ϕ),ϕ . Since ∂g ∂θ = ∂2Linner ∂θ 2 , ∂g ∂ϕ = ∂2Linner ∂θ∂ϕ , we get dθ⋆ dϕ ϕ =− " ∂2Linner ∂θ 2 θ⋆(ϕ),ϕ #−1 ∂2Linner ∂θ∂ϕ θ⋆(ϕ),ϕ . Substituting this expression into the derivative of the outer objective gives d eLouter dϕ ϕ =− ∂Louter ∂θ θ⋆(ϕ) " ∂2Linner ∂θ 2 θ⋆(ϕ),ϕ #−1 ∂2Linner ∂θ∂ϕ θ⋆(ϕ),ϕ . 11 Publi...
work page 2026
-
[15]
∞X t=1 gt(τ)g t(τ) ⊤ # . Finally, applying Lemma 4.1 componentwise gives Eτ∼epπθ
The casek < tfollows by transposition. Therefore, Eτ∼epπθ Sθ(τ)S θ(τ) ⊤ =E τ∼epπθ " ∞X t=1 gt(τ)g t(τ) ⊤ # . Finally, applying Lemma 4.1 componentwise gives Eτ∼epπθ " ∞X t=1 gtg⊤ t # =E τ∼p πθ " ∞X t=1 γt−1gtg⊤ t # . Combining the preceding identities and evaluating atθ=θ ⋆(ϕ)yields ∇2 θLinner(θ⋆(ϕ), ϕ) =αE τ∼p πθ " ∞X t=1 γt−1gt(τ)g t(τ) ⊤ # θ=θ⋆(ϕ) =αF ...
work page 2026
-
[2014]
doi: 10.1109/CDC.2014.7040156. Souradip Chakraborty, Amrit Bedi, Alec Koppel, Huazheng Wang, Dinesh Manocha, Mengdi Wang, and Furong Huang. Parl: A unified framework for policy alignment in reinforcement learning from human feedback. InInternational Conference on Learning Representations, volume 2024, pp. 24410–24449,
-
[2015]
Yingxiao Huo, Satya Prakash Dash, Radu Stoican, Samuel Kaski, and Mingfei Sun. Rank-1 ap- proximation of inverse fisher for natural policy gradients in deep reinforcement learning.arXiv preprint arXiv:2601.18626,
Show all 15 references
-
[2017]
Bilevel reinforcement learning via the development of hyper-gradient without lower-level convexity.arXiv preprint arXiv:2405.19697,
Yan Yang, Bin Gao, and Ya-xiang Yuan. Bilevel reinforcement learning via the development of hyper-gradient without lower-level convexity.arXiv preprint arXiv:2405.19697,
-
[2018]
Frequent directions: Simple and deterministic matrix sketching.arXiv preprint arXiv:1501.01711,
Mina Ghashami, Edo Liberty, Jeff M Phillips, and David P Woodruff. Frequent directions: Simple and deterministic matrix sketching.arXiv preprint arXiv:1501.01711,
-
[2019]
Reinforcement learning and control as probabilistic inference: Tutorial and review
Sergey Levine. Reinforcement learning and control as probabilistic inference: Tutorial and review. arXiv preprint arXiv:1805.00909,
-
[2020]
Explaining and preventing alignment collapse in iterative rlhf.arXiv preprint arXiv:2605.04266,
Etienne Gauthier, Francis Bach, and Michael I Jordan. Explaining and preventing alignment collapse in iterative rlhf.arXiv preprint arXiv:2605.04266,
-
[2022]
Then the gradient of the induced outer objective eLouter(ϕ) :=L outer(θ⋆(ϕ)) is given by ∇ϕ eLouter ϕ =− ∂2Linner ∂ϕ∂θ θ⋆(ϕ),ϕ " ∂2Linner ∂θ 2 θ⋆(ϕ),ϕ #−1 ∇θLouter|θ⋆(ϕ)
10 Published as a conference paper at ICOMP 2026 A HYPERGRADIENT DERIVATIONS Proposition A.1(Implicit hypergradient for bilevel IRL).Assume thatθ ⋆(ϕ)is a differentiable local solution of the inner problem, θ⋆(ϕ)∈arg min θ Linner(θ, ϕ), and that the Hessian ∂2Linner ∂θ 2 θ⋆(ϕ)...
2026
-
[2025]
Scalable linucb: Low-rank design matrix updates for recommenders with large action spaces.arXiv preprint arXiv:2510.19349,
Ekaterina Shustova, Marina Sheshukova, Sergey Samsonov, and Evgeny Frolov. Scalable linucb: Low-rank design matrix updates for recommenders with large action spaces.arXiv preprint arXiv:2510.19349,
-
[2026]
Approximation methods for bilevel programming.arXiv preprint arXiv:1802.02246,
Saeed Ghadimi and Mengdi Wang. Approximation methods for bilevel programming.arXiv preprint arXiv:1802.02246,
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.