Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

Predictive inverse dynamics models (PIDMs) outperform behavior cloning precisely when conditioning on a predicted future state removes more action variance than the state predictor's bias adds; the paper proves the tradeoff and measures it.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 06:52 UTC pith:3XUM2LMR

load-bearing objection The paper's variance-reduction theory collapses on its own training distribution, but the empirical phenomenon is real and worth a reviewer's time. the 4 major comments →

arxiv 2601.21718 v3 pith:3XUM2LMR submitted 2026-01-29 cs.LG cs.AI

When Does Predictive Inverse Dynamics Outperform Behavior Cloning?

Lukas Sch\"afer , Pallavi Choudhury , Abdelhak Lemkhenter , Chris Lovett , Somjit Nath , Luis Fran\c{c}a , Matheus Ribeiro Furtado de Mendon\c{c}a , Alex Lamb
show 5 more authors
Riashat Islam Siddhartha Sen John Langford Katja Hofmann Sergio Valcarcel Macua
This is my paper
classification cs.LG cs.AI MSC 68T0562F12
keywords imitation learningbehavior cloninginverse dynamics modelspredictive inverse dynamicsbias-variance tradeoffsample efficiencyfuture-state predictionoffline learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks why predictive inverse dynamics models (PIDMs) need fewer expert demonstrations than plain behavior cloning (BC) in offline imitation learning. It establishes that PIDMs introduce a bias-variance tradeoff: conditioning the action-prediction network on a predicted future state removes the part of action variance that is explained by where the agent will be, while the imperfect state predictor contributes its own bias and variance. The central identity is exact for optimal estimators: the error gap between BC and PIDM equals the expected conditional variance of the action given the future state, a nonnegative quantity. From this, the paper derives conditions on state-predictor bias under which PIDM is at least as sample-efficient as BC, and shows empirically that the predicted gap appears in 2D navigation (BC needing up to five times more demonstrations) and in a complex 3D video game (66% more samples). A reader would care because this is the first mechanism-level explanation for a set of empirical wins that were previously unexplained.

Core claim

The core discovery is a formal decomposition of why conditioning on the future helps. Let mu(s) be the optimal behavior-cloning estimator of the action and xi(s,s') the optimal inverse-dynamics estimator conditioned on a future state. The paper proves that the expected prediction error of BC minus that of PIDM equals E_s[Var_{s'|s}(E[a|s,s'])], which is always nonnegative: knowing the future state can only sharpen the action estimate. This is Theorem 1. When the future state comes from a learned predictor rather than the ground-truth distribution, the gap becomes this variance reduction plus an estimator-variance difference plus a bias penalty (Corollary 1). Using the information inequality

What carries the argument

The load-bearing object is the conditional-action variance identity Delta = E_s[Var_{s'|s}(E[a|s,s'])], obtained from the law of total variance; it quantifies how much of the action's uncertainty is resolved by knowing the future state. The argument then leans on a Fisher-information inequality (the IDM's Fisher information is at least the BC's, by Jensen's inequality after marginalizing the future state) and on a bias-variance decomposition of expected prediction error that tracks the additional bias from an approximate state predictor.

Load-bearing premise

The theory is derived for point estimators of the expected action under squared loss, with i.i.d. samples and asymptotic efficiency, and the paper explicitly treats the extension to distributional policies as a hypothesis (Section 4, Conclusion); if that transfer fails, the measured gains could stem from representation learning or architecture rather than the claimed bias-variance tradeoff.

What would settle it

Run the same 2D navigation comparison with demonstrations from a deterministic expert policy, so Delta(s) is near zero at every state; the theory predicts PIDM should not beat BC in sample efficiency, so observing a large advantage in that setting would falsify the mechanism.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If correct, PIDM is guaranteed to have no higher asymptotic prediction error than BC, with the gap equal to the future-state conditional variance of actions.
  • Sample-efficiency gains are guaranteed whenever the state predictor's bias is small relative to that variance reduction, giving a concrete design criterion for when to prefer PIDM.
  • Additional data sources that improve the state predictor or reduce IDM variance widen the gap, formally motivating action-free and non-expert data.
  • The theory identifies where gains should concentrate—states with high action variability, such as near goals—and the paper's policy visualizations confirm IDMs attend to future states there.
  • In real-world small-data settings, the predicted advantage transfers: BC needs 66% more samples in the 3D task.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the variance-reduction term depends only on the conditioning distribution, so the benefit should be robust to the choice of IDM architecture; a diffusion- or transformer-based IDM should show the same ordering if the mechanism transfers.
  • A practical diagnostic follows: estimate Delta(s) on a dataset before choosing BC versus PIDM; high Delta near task-critical states predicts PIDM gains, while a deterministic expert with near-zero Delta predicts no advantage.
  • Because the state predictor can be trained on action-free data, PIDM could be combined with self-supervised world models to push sample efficiency further; the paper's instance-based predictor is a minimal instantiation, so learned predictors with lower bias should only improve gains.
  • The Fisher-information argument suggests the advantage is not architectural but structural: any policy that marginalizes a future state inherits the lower variance, so the decomposition itself, not the encoder, drives the gain.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a theoretical explanation for the empirical success of predictive inverse dynamics models (PIDMs) over behavior cloning (BC) in offline imitation learning. It formalizes a PIDM as a state predictor followed by an inverse dynamics model (IDM) conditioned on the predicted future state, and derives: Theorem 1, an exact identity for the prediction-error gap between optimal BC and IDM estimators conditioned on the true future state, Δ = E_s[Var_{s'|s}(E[a|s,s'])] ≥ 0; Corollary 1, a bias-variance decomposition for finite-sample estimators introducing a variance-reduction term Δ and a state-predictor bias β; Theorems 2–3 and Corollary 2, sample-efficiency ratio conditions claiming PIDM is at least as sample-efficient as BC. Experiments in 2D navigation and a 3D video game report that BC requires up to 5× more demonstrations than PIDM. The paper concludes that PIDM's advantage is explained by this bias-variance tradeoff.

Significance. If the theory were correct, it would provide a principled explanation for PIDM's sample-efficiency gains and useful guidance for when future-state conditioning helps. The paper also contains an extensive empirical study: multiple 2D tasks with human and planner demonstrations, a 3D video-game task, ablations of state-predictor bias, and qualitative visualizations of which states the IDM attends to. Theorem 1 itself is a clean, exact identity for the oracle setting in which the IDM is conditioned on the true future state. However, the central claimed mechanism is not supported by the paper's own formal data model: the predicted future state is drawn independently of the action given the current state, so it carries no information about the action. This load-bearing gap invalidates the main theoretical results and, consequently, the paper's central contribution as stated.

major comments (4)
  1. [Section 4, D_{\hat p,m} definition] The data distribution used for the IDM in Corollary 1 and Theorem 2 draws \hat s_{t+k} from \hat p(·|s_t) independently of a_t given s_t. Consequently E_{D_{\hat p,m}}[a_t | s_t,\hat s_{t+k}] = E[a_t|s_t] = μ(s_t), so the population-optimal IDM is constant in \hat s and the variance-reduction term Δ from Theorem 1 is absent from this training distribution. The proof of Corollary 1 nevertheless evaluates EPE(\hat ξ_{\hat p}) under the true (s_t,s_{t+k}) distribution (Eq. 31), which is not the distribution of the deployed PIDM. This invalidates Eq. (7) and the sample-efficiency results built on it.
  2. [Section 4, bias term in Corollary 1] For a consistent IDM estimator trained on D_{\hat p,m}, the bias term b^2_ξ converges to E_{s_t,s_{t+k}}[(μ(s_t)-ξ(s_t,s_{t+k}))^2] = Δ, not to the 'bias induced by an approximate state predictor.' Substituting into Eq. (7) gives \hat Δ_{\hat p} ≈ δ asymptotically: the Δ and β terms cancel. Thus the equations do not express the claimed bias-variance tradeoff; under the formal model the net advantage from future-state conditioning is zero in the population limit.
  3. [Section 4, Lemma 2] The proof of F_ξ/F_μ ≥ 1 assumes 'the BC policy can be obtained as the marginal of the IDM policy' under p^*. This is not stated as an assumption and does not hold for the independently parameterized BC and IDM networks used in the experiments. More importantly, the relevant marginal for the PIDM is under \hat p, not p^*. Because \hat s is independent of a_t given s_t under D_{\hat p,m}, the Fisher information of the IDM likelihood need not dominate that of the BC model; simple parametric examples give ratio 1 or less. Lemma 2 therefore does not support Theorem 3 and Corollary 2.
  4. [Section 5.1 state predictors vs. theory] The instance-based state predictor (Eq. 11) and the 3D random-demonstration predictor are deterministic lookups into the training demonstrations, not draws from a fixed \hat p(·|s_t). The theory in Section 4 does not model these mechanisms. In particular, the instance-based lookup can return the true future state from the same demonstration as s_t, which is correlated with a_t; this is a qualitatively different source of information absent from D_{\hat p,m}. The empirical gains may therefore be explained by the lookup mechanism or by representation learning rather than by the claimed bias-variance tradeoff.
minor comments (6)
  1. [Theorem 3] The symbol ε is used both for the error level and for ε − E[Var(a_t|s_t)]; this is confusing and should be disambiguated.
  2. [Section 4, Eq. (8)] The statement of 'n and m denote the minimum number of samples required to achieve error level ε' is circular with the subsequent derivation; please clarify the definition of the ratio.
  3. [Lemma 2] The marginality assumption on the BC and IDM parameterizations should be stated explicitly, or the lemma restricted to the oracle setting where it holds by construction.
  4. [Table 1 and Figure 4] Efficiency ratios are point estimates; confidence intervals or significance testing would strengthen the claims of 2–5× gains.
  5. [Corollary 1, paragraph after Eq. (7)] The phrase 'assuming both estimators have similar intrinsic bias (as required for a fair comparison)' is not formalized or verified empirically.
  6. [Appendix A.3] Typo: 'We are now ready to proof Theorem 2' should read 'prove Theorem 2.'

Circularity Check

0 steps flagged

No significant circularity; derivation is self-contained, with only non-load-bearing self-citations.

full rationale

The theoretical chain does not reduce to its inputs by construction. Theorem 1 is an exact law-of-total-variance identity applied to optimal conditional estimators; Corollary 1 is a standard bias-variance decomposition; Lemma 2 is a Jensen/data-processing inequality for the score of a marginal; and Theorems 2-3 are algebraic manipulations of the textbook information inequality. No fitted constants enter the predicted error gap, and the empirical study does not fit the theory's parameters to the data; instead it measures quantities such as Delta(s) and state-prediction error and checks correlations. The paper's own self-citations to prior PIDM work (e.g., Tot et al. 2025, Schäfer et al. 2025) are descriptive background and are not load-bearing in the derivation. A separate correctness concern exists: the formal dataset D_{\hat p,m} draws \hat s_{t+k} independently of a_t given s_t, so the population optimum for an IDM conditioned on \hat s is just \mu(s_t), and the Theorem 1 variance-reduction term does not automatically transfer to that data distribution. This is a modeling/proof gap rather than a circular reduction, and the paper explicitly flags related simplifying assumptions (i.i.d. samples, point estimators, and the hypothesis that the mechanism extends to distributional policies). Therefore the paper is self-contained in the circularity sense; score 2 reflects only minor, non-load-bearing self-citation.

Axiom & Free-Parameter Ledger

2 free parameters · 6 axioms · 0 invented entities

The derivation itself uses standard statistical machinery (law of total variance, Cramer-Rao, Fisher information monotonicity) and introduces no fitted constants; the main extra assumptions are modeling choices: i.i.d./asymptotic approximation, representability of BC as a marginal of IDM, and the hypothesized transfer to log-loss policies. The empirical method introduces the design parameter k and a stochastic demonstration-index selector in 3D.

free parameters (2)
  • lookahead horizon k = k=1 for 2D; training k in {1,6,11,16,21,26} and eval k=1 for 3D
    PIDM's bias-variance tradeoff depends on k through the distribution of future states; k was chosen by the authors based on state-predictor-error analysis in Appendix D, not derived from the theory.
  • 3D state-predictor demonstration index = randomly sampled based on seed
    In the 3D 'Tour' task the state predictor returns time-shifted states from one training demonstration chosen randomly by seed; this choice affects the predicted future state and hence the bias of the PIDM, and is a free design choice of the empirical method.
axioms (6)
  • standard math Law of total variance and iterated expectations hold for the data distribution.
    Used in Theorem 1 to decompose Var(a|s) into conditional variance plus variance of conditional mean.
  • standard math Cramer-Rao information inequality (Lemma 1) for n i.i.d. samples.
    Foundational for Theorem 2's sample-efficiency ratio; relies on regularity and differentiability of the parametric family.
  • domain assumption The BC policy is the exact marginal of the IDM policy, so Fisher information is non-decreasing under conditioning (Lemma 2).
    Requires the BC policy to be representable as the marginal of the IDM policy and the model family to be closed under marginalization; only approximately true for function approximators in practice.
  • domain assumption Trajectory data can be treated as i.i.d. samples (fast-mixing chain approximation).
    Stated in Section 4; real MDP transitions are sequential and not i.i.d., and the authors call this 'a reasonable approximation for fast mixing chains.'
  • domain assumption Asymptotic efficiency and 1/sqrt(n) convergence extend to the small-data regime (Berry-Esseen argument).
    Appendix A.6 argues asymptotic variance predicts finite-time behavior; this is an extrapolation, not a theorem, and is the basis for applying Theorem 2 to the experiments.
  • ad hoc to paper Bias-variance decomposition for squared loss extends to log-loss/distributional policies.
    Section 4 and Conclusion: the theory is for point estimators with squared loss; the authors explicitly hypothesize that Bregman divergence extensions make the same tradeoff hold for diffusion/transformer policies. The experiments use neural policies trained in this regime.

pith-pipeline@v1.3.0-alltime-deepseek · 22249 in / 15667 out tokens · 154789 ms · 2026-08-03T06:52:22.044877+00:00 · methodology

0 comments
read the original abstract

Behavior cloning (BC) is a practical offline imitation learning method, but it often fails when expert demonstrations are limited. Recent works have introduced a class of architectures named predictive inverse dynamics models (PIDMs) that combine a future-state predictor with an inverse dynamics model. While PIDMs often outperform BC, the reasons behind their benefits remain unclear. In this paper, we provide a theoretical explanation: PIDMs introduce a tradeoff. Conditioning the IDM on the predicted future state can significantly reduce variance, but the prediction itself introduces additional bias and variance. We establish conditions for PIDMs to achieve higher sample efficiency and lower prediction error than BC, with the gap widening when additional data sources are available. We validate the theoretical insights empirically in 2D navigation tasks, where BC requires up to five times (three times on average) more demonstrations than PIDM to reach comparable performance. Results are also illustrated in a complex 3D environment in a modern video game with high-dimensional visual inputs and stochastic transitions, where BC requires over 66\% more samples than PIDM.

Figures

Figures reproduced from arXiv: 2601.21718 by Abdelhak Lemkhenter, Alex Lamb, Chris Lovett, John Langford, Katja Hofmann, Luis Fran\c{c}a, Lukas Sch\"afer, Matheus Ribeiro Furtado de Mendon\c{c}a, Pallavi Choudhury, Riashat Islam, Sergio Valcarcel Macua, Siddhartha Sen, Somjit Nath.

Figure 1
Figure 1. Figure 1: (a) Visualization of selected milestones from the "Tour" task in a 3D video game with stochastic transitions and real-time inference. (b) Sample efficiency curves (mean ± std) for PIDM and BC, with BC needing 66% more samples to achieve 80% success rate. Second, we provide empirical evidence that the predicted sample efficiency gains apply to more general conditions, including the small-data regime. We per… view at source ↗
Figure 2
Figure 2. Figure 2: (a) BC learns a policy conditioned on the current state. (b) IDM learns a policy conditioned on the current and future state k steps ahead. (c) Forward models predict a future state (representation) given a state and action. Note (b) IDM and (c) forward models can serve as auxiliary objectives to learn an encoder that provides effective state representations. (d) PIDM represents an alternative to BC consis… view at source ↗
Figure 3
Figure 3. Figure 3: Visualization of 2D navigation environment. (a) Tasks require the agent (blue box) to navigate to reach the goals (red boxes) in a particular order. (b) - (e) Visualizations of all four tasks and the traces of the 50 trajectories within the datasets. PIDM to be more sample efficient than BC, the bias induced by an approximate state predictor must not exceed the vari￾ance reduction achieved by conditioning … view at source ↗
Figure 4
Figure 4. Figure 4: Performance per number of training demonstrations for BC and PIDM in four tasks trained on human datasets. Lines and shading correspond to the average and standard deviation across 20 seeds. We further visualize the number of samples required by PIDM and BC to reach 90% of the highest achievable performance with vertical dotted lines. ing processed by a pre-trained vision encoder. The policy then receives … view at source ↗
Figure 7
Figure 7. Figure 7: IDM policies for future states in each cardinal and diagonal direction in Four room and Multiroom. The IDM policy only attends to future states in states with large ∆(s). efficiencies for PIDM over BC, as predicted by the analysis in Section 4. We find BC requires up to 5× more demonstra￾tions than PIDM to achieve comparable performance, and 3× on average across tasks. When training on less diverse demonst… view at source ↗
Figure 8
Figure 8. Figure 8: Impact of state predictor bias on PIDM sample efficiency. (a) Visualization of human (blue) and A* planner (orange) datasets. (b) State prediction error of the instance-based state predictor for both datasets. (c) Sample efficiency curve for PIDM trained on both datasets. (d) Correlation between state prediction error and rollout performance for PIDM on all datasets and tasks. states that are closer to the… view at source ↗
Figure 9
Figure 9. Figure 9: Visualized state-wise EPE gaps ∆(s) from Equation (13) computed for each dataset. We observe large gaps in states surrounding the goals where human actions are more diverse. B.2. IDM Policy Visualizations in all Tasks (a) Four room (b) Zigzag (c) Maze (d) Multiroom [PITH_FULL_IMAGE:figures/full_fig_p016_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Visualization of IDM policies when queried for representative states and possible future states in each cardinal and diagonal direction for all four tasks. Predicted actions spread out in states where the dataset exhibits large ∆(s). C. Experiments under Deterministic Target Policy Our main experiments in the 2D navigation environment use human demonstrations for all tasks. Human demonstrations are natura… view at source ↗
Figure 11
Figure 11. Figure 11: Traces of the 50 human (blue) and A* planner (orange) trajectories within all four 2D navigation tasks [PITH_FULL_IMAGE:figures/full_fig_p017_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Performance per number of training demonstrations for PIDM in four tasks trained on human and A* planner demonstrations. Lines and shading correspond to the average and standard deviation across 20 and 50 seeds, respectively. 1 2 5 10 20 30 40 45 Training trajectories 5 10 15 20 25 30 State Prediction Error Human Planner (a) Four room 1 2 5 10 20 30 40 45 Training trajectories Human Planner (b) Zigzag 1 2… view at source ↗
Figure 13
Figure 13. Figure 13: Error of the 2D navigation state predictor (as defined in Equation (11)) on held-out trajectories when trained on the human and A* planner demonstrations. Lines and shading correspond to the average and standard deviation across 50 seeds that determine the sampling of demonstrations used for training, respectively. C.3. Evaluation Results for 2D Navigation with Deterministic Target Policy PIDM with human … view at source ↗
Figure 14
Figure 14. Figure 14: Sample efficiency of PIDM and BC in Multiroom, trained on A* planner data. error compared to ground-truth future states in held-out trajectories for each dataset and varying number of demonstrations. We consider multiple sizes n ∈ {1, 2, 5, 10, 20, 30, 40, 45}. For each n, we randomly sample 50 different subsets of the 50 available trajectories that are used to learn the state predictor (i.e. the instance… view at source ↗
Figure 15
Figure 15. Figure 15: Variance of future states and state prediction error for varying k in the human datasets of all four 2D navigation tasks. The state predictor has been trained on 40 demonstrations and evaluated on the remaining 10 held-out trajectories. values of k can make the state prediction task more difficult, potentially leading to higher state prediction error and thus reduced performance. Motivated by these consid… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. FF-JEPA: Long-Horizon Planning in World Models with Latent Planners

    cs.AI 2026-06 unverdicted novelty 6.0

    FF-JEPA introduces a two-model hierarchical structure with an action-free latent planner to decompose long-horizon planning into short subgoals in latent world models.

  2. Latent Geometry Beyond Search: Amortizing Planning in World Models

    cs.RO 2026-05 unverdicted novelty 6.0

    In regularized latent spaces of world models, planning can be amortized into a goal-conditioned inverse dynamics model that matches CEM performance at 100-130x lower per-decision cost.

  3. Latent Geometry Beyond Search: Amortizing Planning in World Models

    cs.RO 2026-05 unverdicted novelty 6.0

    A Goal-Conditioned Inverse Dynamics Model amortizes planning in pretrained world model latents, matching or exceeding CEM in seven of eight settings at 100-130x lower per-decision cost.

Reference graph

Works this paper leans on

5 extracted references · 1 linked inside Pith · cited by 2 Pith papers

  1. [1]

    State encoder: MLP(1024, 512, 512), Policy: MLP(512, 256)

  2. [2]

    State encoder: MLP(1024, 2048, 1024, 512, 512), Policy: MLP(512, 512, 256) We also tried two learning rates per algorithm, namely linear decay 1e-3 →1e -6 and 5e-5 for PIDM, and linear decay 1e-4 →1e -6 and 1e-4 for BC, with decay for 60,000 steps. Other hyperparameters that remained constant where: training lasted 22 When does predictive inverse dynamics...

    2048

  3. [5]

    theia-base-patch16-224-cddsv

    MLP(1024, 2048, 512) From this search, we identified a single network architecture that performed best for BC and among the best for PIDM to keep for consistent comparisons thereafter. The architecture consists of network block MLP(512, 1024, 256) followed by batch normalization before MLP(256, 2) with the last 2D layer outputting the action logits. We ap...

    2048

  4. [768]

    While the embedding of the current and future frames are the input to the state encoder of the PIDM

    This embedding vector of the current frame is the input to the BC policy. While the embedding of the current and future frames are the input to the state encoder of the PIDM. F.4.2. HYPERPARAMETERSEARCH To ensure fair comparison and some degree of generalization, we conducted a comparable hyperparameter search for both BC and PIDM in a different more comp...

  5. [2023]

    ∂ ∂ξ lnπ µ(ξ)(at |s t) 2# ,(43) Fξ ≜E

    URL https://proceedings.mlr.press/ v202/mhammedi23a.html. Osa, T., Pajarinen, J., Neumann, G., Bagnell, J. A., Abbeel, P., Peters, J., et al. An algorithmic perspective on imita- tion learning.Foundations and Trends® in Robotics, 7 (1-2):1–179, 2018. 9 When does predictive inverse dynamics outperform behavior cloning? Pan, Y ., Cheng, C.-A., Saigol, K., L...

    Pith/arXiv arXiv 2018