{"id":"ce90d8c3-11ae-43b0-ae92-2ff1d4ca234b","arxiv_id":"2608.11052","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"For maximum-likelihood IRL, the inner-problem Hessian at a realizable optimum equals the temperature-scaled trajectory Fisher matrix, which enables a scalable sketched hypergradient method.","lead":"This paper derives a Fisher-based hypergradient for inverse reinforcement learning, replacing the expensive inner-Hessian inverse with a discounted trajectory Fisher matrix and a streaming sketch. The method performs competitively with an existing bilevel IRL baseline on CartPole and LQR while reducing memory and, at moderate sketch sizes, wall-clock time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation 9's row scaling is wrong: rows use γ^{t-1} outside the square root, so XᵀX encodes γ^{2(t-1)} weights, not the γ^{t-1} in Eq. 5; the sketched Fisher system is not the claimed Hessian–Fisher system.","rationale":"The paper's central contribution is an efficient Fisher-based hypergradient, and the streaming sketch is the scalability mechanism. The identity α \\hat{F}=XᵀX is the bridge between the Fisher matrix and the sketch. A direct expansion of Eq. 9 shows this identity is false for γ<1. This is a more decisive objection than the reader's concern about realizability, because it undermines the algorithm's correctness even when Proposition 4.2 holds. The empirical results are therefore not attributable to the described method without clarification. The reader's conditional verdict is superseded; the current manuscript should be rejected or, at minimum, the equation corrected and all experiments re-verified.","tokens_in":13655,"tokens_out":19726,"duration_ms":173477,"concrete_test":"Take γ=0.9 and a two-step trajectory with unit-norm policy-score vectors g1 and g2. Compute XᵀX using rows as defined in Eq. 9 and compare with α \\hat{F} from Eq. 5. The diagonal weights show a ratio of 0.81 between the two steps in XᵀX versus 0.9 in α \\hat{F}, a 10% relative discrepancy that grows with horizon. Re-run the same comparison on CartPole-like long horizons to quantify the bias. If the authors intended sqrt(γ^{t-1}) in Eq. 9, the fix is a one-character correction and all sketch experiments must be re-run; if not, the SCFD solver is solving the wrong linear system.","verdict_should_be":"REJECT","load_bearing_attack":"In Section 5.2, the paper claims that stacking x_t(τ) = sqrt(α/N_agent) γ^{t-1} ∇θ log πθ(a_t|s_t) as rows of X gives α \\hat{F}_θ = XᵀX, where \\hat{F}_θ is the discounted trajectory Fisher of Eq. 5. But XᵀX = (α/N_agent) Σ_{τ,t} γ^{2(t-1)} g_t g_tᵀ, whereas α \\hat{F}_θ = (α/N_agent) Σ_{τ,t} γ^{t-1} g_t g_tᵀ. These coincide only when γ=1 or every trajectory has a single step. For γ<1, later timesteps are downweighted quadratically, so the sketch is approximating a different matrix and Eq. 10 is not the damped Fisher system. This is an internal algebraic inconsistency, independent of the realizability assumption in Proposition 4.2. Since Algorithm 2's core step (SOLVE on the sketched X) relies on this identity, the described algorithm does not compute the proposed Fisher-based hypergradient. The experimental section reports this sketch as the method, so the numerical results cannot validate the theoretical claim unless the implementation used a different row definition (e.g., sqrt(γ^{t-1})).","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies maximum-likelihood IRL as a bilevel program and derives an implicit hypergradient for the outer reward parameters. The central theoretical contribution is Proposition 4.2, which states that at an exactly solved, exactly realizable inner optimum the Hessian of the inner objective equals α times the discounted trajectory Fisher information matrix. Based on this identity, the authors propose replacing the inverse-Hessian-vector product in the implicit hypergradient with a damped inverse-Fisher-vector product, and then approximate that product with a streaming SCFD sketch that avoids forming the dense Fisher matrix. The method is evaluated on CartPole and LQR against the single-loop ML-IRL baseline, with experiments comparing explicit-Fisher and sketched-Fisher variants in terms of outer loss, policy quality, reward ranking, wall-clock time, and peak memory.","tokens_in":13962,"tokens_out":8631,"duration_ms":84262,"significance":"If the Hessian–Fisher identity and the sketched solver were both correct, the paper would offer a meaningful computational improvement for bilevel IRL: the hypergradient would be computable with O(m d_θ) storage instead of O(d_θ^2), and the derivation is non-circular in that the identity is proven from the reverse-KL structure rather than assumed. The appendix proofs (Propositions A.1, B.1–B.3, Corollary B.1) are internally consistent and the sign conventions chain correctly through the hypergradient. The clear statement of the realizability assumption is also a strength. However, the practical algorithms rely on approximate inner solutions and parametric policies for which the theorem's assumptions are not met, and, more seriously, the row scaling used to form the sketched Fisher matrix is algebraically inconsistent with the empirical Fisher definition. These issues are load-bearing for the paper's central claims.","major_comments":[{"comment":"The stacked-row construction is algebraically inconsistent with the empirical Fisher definition. Equation (9) defines rows x_t(τ) = sqrt(α/N_agent) γ^{t−1} ∇θ log πθ(a_t|s_t), so X^T X = (α/N_agent) Σ_{τ,t} γ^{2(t−1)} g g^T, whereas Equation (5) together with Proposition 4.2 gives α F̂_θ = (α/N_agent) Σ_{τ,t} γ^{t−1} g g^T. These coincide only when γ=1 or when every trajectory has a single step. Algorithm 2 line 10 feeds exactly these rows into the SCFD sketch, so the system actually solved is (X^T X + λI)v = g with the wrong discount weighting, not the damped trajectory-Fisher system claimed in Equation (10). Consequently, the numerical results cannot validate the proposed Fisher hypergradient unless the implementation used a different row definition, in which case the pseudocode and equations must be corrected.","section":"Section 5.2, Eq. (9)–(10) and Algorithm 2"},{"comment":"The Hessian–Fisher identity is proven under exact inner optimality and exact realizability (D_KL = 0 at the inner optimum). In the experiments, the inner problem is solved approximately with PPO/SAC/REINFORCE and parametric policies that generically cannot realize the Boltzmann trajectory distribution; the paper provides no bound on the resulting bias in the Hessian approximation and reports no measurement of D_KL or of the error between ∇²_θ L_inner(θ*, φ) and αF_θ*. Because this identity is the entire theoretical justification for replacing the Hessian with αF in Equation (7), the gap between the theorem's assumptions and the algorithmic practice is load-bearing.","section":"Section 4, Proposition 4.2; Algorithms 1–2"},{"comment":"Several computational-efficiency and performance entries are based on a single run, e.g., Table 3, CartPole Explicit Fisher (1464.41 s per outer iteration, Runs=1) and Table 3, LQR Fisher with Sketching m=256 (1316.88 s, Runs=1); Table 4 likewise reports single-run entries for some settings. The claimed speedups and memory gains in Table 1 are ratios of these noisy measurements. The paper should report the number of seeds for every entry and provide confidence intervals or paired-run comparisons; as presented, the empirical support for the efficiency and quality claims is underpowered.","section":"Section 6, Tables 2–4"}],"minor_comments":[{"comment":"The assumption 'the policy class is rich enough so that D_KL(eπ_{θ*(φ)} || e p_φ) = 0' is redundant given the preceding assumption that the inner problem is solved exactly; stating the assumption as 'there exists θ with zero KL divergence' would be clearer and would also make the role of realizability more transparent.","section":"Section 4, Proposition 4.2"},{"comment":"The text says the sketched solver remains stable at λ=10^{-3} and that this value is fixed, but it is not stated whether λ was tuned separately for each sketch size; the conclusion that smaller sketches are more stable would be stronger if the damping was held constant across all compared configurations.","section":"Section 6.1, Figure 2"},{"comment":"The experiments do not specify the hyperparameters of the inner RL solver (e.g., PPO or SAC settings, network architectures, optimization details) nor the discount factor γ used in each environment; without these details, the reported results cannot be reproduced.","section":"Section 6 and Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The row-scaling error in Section 5.2 is likely fixable by replacing γ^{t−1} with sqrt(γ^{t−1}) in Eq. (9) and Algorithm 2, but the current pseudocode and the reported experiments are mutually inconsistent as written; if the implementation actually used the corrected square-root scaling, the text must be aligned with the implementation and the experiments re-reported. The realizability gap between Proposition 4.2 and the approximate inner solvers is a more fundamental concern that would benefit from either an error bound or an empirical measurement of the approximation error. Given that the appendix proofs are internally sound, I do not see grounds for rejection, but the manuscript needs substantial revision before the claims are credible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The theoretical centerpiece — that at a realizable inner optimum the Hessian of the reverse-KL ML-IRL objective is α times the discounted trajectory Fisher — is derived cleanly and is genuinely new in this setting. But the sketching section has an algebraic error that breaks the main algorithmic claim. Equation 9 defines rows as sqrt(α/N) γ^{t-1} g_t, so XᵀX carries weights γ^{2(t-1)}, not the γ^{t-1} that appear in the empirical Fisher of Eq. 5. Algorithm 2 therefore solves a different damped system than the one advertised. This is not a subtle conditioning issue; it is an internal inconsistency between the stated identity and the implementation.\n\nWhat the paper does well: it proves Proposition 4.2, gives a coherent derivation of the explicit-Fisher hypergradient, and is transparent about the assumptions. The appendix proofs are consistent as far as I can tell. The SCFD adaptation is a reasonable idea, though it follows existing sketching work rather than breaking new ground.\n\nSoft spots beyond the scaling error: the realizability assumption (D_KL = 0) is strong and unverified in the experiments, where PPO/SAC/REINFORCE solve the inner problem approximately with parametric policies that generally cannot realize the Boltzmann trajectory distribution. There is no bias bound or measurement. The empirical evaluation covers only CartPole and LQR, some efficiency numbers come from single runs, and no code or data are provided. These would matter less if the algorithm were correct, but combined with the scaling error they leave the practical claims unsubstantiated.\n\nWho is this paper for? Someone working on bilevel IRL or natural hypergradients might benefit from the explicit-Fisher analysis and the identity. The sketching part as written should not be used until the row scaling is fixed.\n\nRecommendation: this deserves a serious referee — the core idea is worth engaging and the error is fixable — but in its current form it should not be accepted. I would send it back for major revision.","headline":"The Hessian-Fisher identity is a real result, but the sketching algorithm has a scaling error that invalidates the reported experiments.","tokens_in":14507,"tokens_out":4471,"would_cite":false,"duration_ms":35928,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"At an exactly solved inner optimum of bilevel IRL, the inner Hessian equals α times the trajectory Fisher information matrix, turning the implicit hypergradient into a damped inverse-Fisher solve computable from sampled trajectories with…","keywords":["inverse reinforcement learning","bilevel optimization","hypergradient descent","Fisher information matrix","matrix sketching","implicit differentiation","maximum-likelihood IRL","spectral sketch"],"falsifier":"In a tabular MDP with an expressive softmax policy, solve the inner problem exactly, then compare the analytic Hessian of L_inner at θ*(φ) with α times the analytic trajectory Fisher matrix; any nonzero difference would refute Proposition 4.2. On LQR, track the relative Frobenius error ∥bF_θ − $α^{{-1}}$∇²L_inner∥ / ∥$α^{{-1}}$∇²L_inner∥ while the inner PPO solve converges; if the error does not shrink toward zero, the practical approximation is unanchored.","tokens_in":13476,"feed_emoji":"🎯","tokens_out":7267,"duration_ms":59817,"temperature":0.7,"pith_summary":"The paper targets maximum-likelihood inverse reinforcement learning, where a reward is recovered by solving a bilevel problem: an inner entropy-regularized policy optimization under the learned reward, and an outer fit to expert demonstrations. Its central claim is that at an exactly solved inner optimum the Hessian of the inner reverse-KL objective is proportional to the policy-induced trajectory Fisher information matrix, with proportionality constant α. That identity turns the expensive inverse-Hessian–vector product of the implicit hypergradient into a damped inverse-Fisher–vector product that can be solved from sampled trajectories. To avoid storing the dense Fisher matrix, the paper streams weighted policy-score vectors into a spectral sketch, reducing curvature storage from O(d_θ²) to O(m d_θ). The authors report that the resulting method matches a single-loop ML-IRL baseline on CartPole and LQR in policy quality and reward ranking while lowering memory and sometimes runtime.","feed_headline":"Fisher matrix replaces Hessian in IRL hypergradient","feed_subtitle":"A spectral sketch cuts curvature storage to O(md), so bilevel IRL becomes cheap.","key_machinery":"The load-bearing object is the Hessian–Fisher identity of Proposition 4.2, together with the streaming SCFD sketch used to exploit it. The identity says that at an exactly solved, realizable inner optimum the Hessian of the inner reverse-KL objective equals α times the discounted trajectory Fisher information matrix, F_{θ} = E_{τ∼$p^{{π_θ}}$}[Σ_{t=1}^{∞} $γ^{{t-1}}$∇ log π_θ(a_t|s_t)∇ log π_θ(a_t|s_t)⊤]. This converts the implicit hypergradient's inverse-Hessian–vector product into a damped inverse-Fisher–vector product, which is then approximated without dense storage by representing the empirical Fisher as X⊤X, streaming the weighted score rows through SCFD, and computing the inverse-vector product from the sketch's low-rank factors (Algorithm 3).","core_discovery":"The paper establishes that, under exact inner optimality and the realizability condition D_KL(eπ_{θ*(φ)}∥e p_φ)=0, the Hessian of the inner loss in bilevel IRL satisfies ∇²_θ L_inner(θ*(φ), φ)=α F_{θ*(φ)}, where F is the discounted trajectory Fisher information matrix and α is the fixed temperature of the maximum-entropy trajectory model. It then uses this identity to rewrite the implicit hypergradient so that the hard step is solving (αF + λI)v = g for the outer gradient g, and supplies Monte Carlo estimators for each factor from agent and expert trajectories (Equations 6–8). To make the solve scalable, the paper represents the empirical Fisher as X⊤X with rows given by weighted policy-score vectors and streams these rows into a Spectral Compensation Frequent Directions sketch, which maintains a compact spectral approximation and returns v without ever forming the d_θ×d_θ matrix. The claim is that this yields a practical Fisher-based implicit hypergradient for ML-IRL whose storage is O(m d_θ), and the experiments on CartPole and LQR support its competitiveness against a single-loop baseline.","pith_inferences":["A bound on ∥∇²L_inner − αF∥ in terms of the KL gap D_KL(eπ_{θ*(φ)}∥e p_φ) would turn the identity from a limit statement into a certified approximation, and the experiments do not currently provide that certificate.","The same reverse-KL Hessian–Fisher structure likely appears in other bilevel RL settings such as iterative RLHF reward steering, so the sketching-based hypergradient may transfer beyond IRL.","The nonmonotonic effect of sketch size on final loss suggests the sketch approximation error interacts with the damping and the optimization noise; a controlled study isolating these factors would clarify when larger sketches help."],"forward_implications":["Hypergradients for ML-IRL become computable from sampled trajectories without constructing the inner Hessian or the dense Fisher matrix.","Curvature storage drops from O(d_θ²) to O(m d_θ), so the method scales to larger policy parameterizations.","The sketch size m and damping λ become the practical tuning knobs that trade memory, runtime, and curvature fidelity.","On the tested environments, Fisher-based IRL is competitive with single-loop ML-IRL in policy quality and reward ranking within a matched budget, and sketching can improve wall-clock time and peak memory."],"supporting_citations":[{"why":"Defines the single-loop maximum-likelihood IRL formulation and serves as the baseline against which Fisher-based hypergradients are compared.","marker":"Zeng et al. (2022)"},{"why":"Introduces Natural Hypergradient Descent, the Fisher-surrogate idea that this paper adapts to reverse-KL inner objectives.","marker":"Kong et al. (2026)"},{"why":"Provides the maximum-entropy trajectory model whose reverse-KL inner objective underlies the Hessian–Fisher identity.","marker":"Levine (2018)"},{"why":"Documents the inverse-Hessian–vector-product bottleneck in implicit hypergradients that the proposed method avoids.","marker":"Ghadimi & Wang (2018)"},{"why":"Supplies the SCFD streaming spectral sketch used to approximate the inverse-Fisher–vector product without dense storage.","marker":"Chen et al. (2020)"}],"fun_headline_variants":["Streaming Fisher sketch speeds bilevel IRL hypergradient","Fisher-based hypergradient with spectral sketch for scalable IRL","Bilevel IRL hypergradient via streaming Fisher sketch","Spectral sketch makes Fisher-based IRL hypergradient cheap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proportionality between the inner Hessian and the Fisher matrix is guaranteed only when the inner problem is solved exactly and the policy class can exactly realize the Boltzmann trajectory distribution; in the experiments the inner problem is solved approximately by PPO/SAC/REINFORCE with parametric policies, so the identity is used outside its provable regime and the resulting approximation error is neither bounded nor measured.","fun_headline_variants_meta":{"raw":{"variants":["Streaming Fisher sketch speeds bilevel IRL hypergradient","Fisher-based hypergradient with spectral sketch for scalable IRL","Bilevel IRL hypergradient via streaming Fisher sketch","Spectral sketch makes Fisher-based IRL hypergradient cheap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000559,"raw_usage":{"total_tokens":2681,"prompt_tokens":996,"completion_tokens":1685,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":612,"completion_tokens_details":{"reasoning_tokens":1615}},"tokens_in":612,"tokens_out":1685,"duration_ms":11046,"temperature":1.0,"reasoning_tokens":1615,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:18:42.315374+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In a tabular MDP with an expressive softmax policy, solve the inner problem exactly, then compare the analytic Hessian of L_inner at θ*(φ) with α times the analytic trajectory Fisher matrix; any nonzero difference would refute Proposition 4.2. On LQR, track the relative Frobenius error ∥bF_θ − $α^{{-1}}$∇²L_inner∥ / ∥$α^{{-1}}$∇²L_inner∥ while the inner PPO solve converges; if the error does not shrink toward zero, the practical approximation is unanchored.","supporting_citations":[],"review_version":1}