REVIEW 4 major objections 4 minor 10 references
Confounded Causal Imitation Learning with Instrumental Variables
T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A residual test on expert trajectories can identify a valid instrumental variable and remove multi-timestep confounding from imitation learning.
desk verdict Genuinely new IV-selection result for multi-timestep confounded IL, but the simulator-based learner is biased as written and the proofs are missing. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the Auxiliary-Based testing Criterion (AB Criterion): for a candidate past state $s_k$, define the auxiliary residual $R_{s_t,a_t||s_k} := a_t - l(s_t)$ with $\mathbb{E}[R \mid s_k] = 0$ and $l(\cdot) \neq 0$, and declare $s_k$ valid exactly when $R_{s_t,a_t||s_k}$ is independent of $s_k$. This criterion converts the classical, unobservable IV validity assumptions (relevance, exclusion restriction, and exogeneity) into a single conditional-independence statement that can be checked from observed state-action trajectories. The same residual object feeds both policy-learning stages: the simulator-based stage regresses expert actions on confounder-free simulated states, and the offline stage uses the instrument $s_k$ as a discriminator input in a minimax objective.
What would settle it
Generate a synthetic C2L process with known confounder duration $\tau=3$ and non-Gaussian confounders, run the FindValidIV procedure on many trajectories, and check whether the selected candidate is always the true valid state $s_{t-3}$; if the test frequently selects a nearer state such as $s_{t-1}$ or $s_{t-2}$, the sufficiency direction of the identification theorem is falsified. A second check: in the simulator-based stage, compare the policy recovered by regressing expert actions on clean simulated states against the known true policy; any systematic residual bias would contradict the claim that this regression is unbiased.
Extended reading notes
Core claim
The paper establishes that, for its C2L model, a candidate past state $s_k$ is a valid instrumental variable relative to the current state-action pair $(s_t, a_t)$ exactly when an auxiliary residual $R_{s_t,a_t||s_k} = a_t - l(s_t)$, defined so that $\mathbb{E}[R \mid s_k] = 0$ and $l \neq 0$, is independent of $s_k$. This equivalence is proven for linear C2L models under a partial non-Gaussianity assumption (Theorem 2) and for nonlinear C2L models under a non-degenerate cross-derivative assumption (Theorem 3). The paper then uses this AB Criterion in Stage I to search past states for a valid IV via HSIC-based independence tests, and in Stage II to train the policy either with a simulator or through an offline minimax objective that uses the identified IV as a conditioning variable.
Load-bearing premise
The nonlinear identification theorem requires that the second-order partial derivative of the log joint density of the estimated residual and the candidate state be nonzero, a condition that cannot be verified from data and whose failure collapses the sufficiency direction of the result.
Editorial extensions
If this is right
- If the identification theorems hold, imitation learners no longer need to know the confounding duration $\tau$ in advance; they can search past states and select the correct instrument from data.
- The C2L framework generalizes the single-timestep Temporally Correlated Noise model, so existing methods that assume a fixed two-step confounder can be replaced by an adaptive procedure when confounders persist longer.
- The AB Criterion gives a practically computable test: any off-the-shelf independence test (the paper uses HSIC) can separate valid from invalid candidate instruments in finite samples.
- The offline variant means confounding-robust imitation learning remains possible in domains where no simulator is available, as long as a candidate history of past states can be observed.
- In experiments across LunarLander, HalfCheetah, and AntBulletEnv, the identified IV leads to lower action-prediction error and higher policy return than behavioral cloning and the TCN-based baselines.
Reading between the lines
- Beyond the paper, the same auxiliary-residual criterion could be applied to other observational causal-effect estimation tasks where an unknown number of lagged confounders render naive instruments invalid, provided the temporal structure is Markovian.
- The paper's conclusion notes sensitivity to time-varying confounding; a natural extension would be a version of the AB Criterion that re-checks instrument validity online as the confounding mechanism changes.
- Because the method relies on higher-order statistics (non-Gaussianity) for linear identification, it may struggle in near-Gaussian settings; a practical extension could combine the independence test with a Gaussianity pre-test to flag when identification is degenerate.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Confounded Causal Imitation Learning (C2L), a two-stage framework for learning expert policies from demonstrations when unmeasured confounders influence states and actions over multiple timesteps. Stage I defines an Auxiliary-Based testing Criterion (AB Criterion) and claims necessary and sufficient conditions for identifying a valid instrumental variable (IV) from observational state-action data, under linear non-Gaussian (Theorem 2) and nonlinear (Theorem 3) assumptions. Stage II provides two policy learning algorithms: a simulator-based method (Algorithm 3) that generates confounder-free synthetic states and regresses expert actions on them, and an offline minimax method (Algorithm 4) that uses the identified IV as a discriminator input. Experiments on LunarLander, HalfCheetah, and AntBulletEnv report high IV identification accuracy and improved policy performance compared with BC, DoubIL, and ResiduIL. The paper is well-motivated by the limitation of prior TCN models to single-timestep confounding, but the central theoretical results are unverifiable as submitted, and the simulator-based learner has an internal inconsistency that undermines its unbiasedness claim.
Significance. If the theoretical results and algorithms are correct, the paper would make a meaningful contribution by extending causal imitation learning to settings with persistent multi-timestep latent confounding, where the simple 'past state as IV' heuristic fails. The necessary and sufficient identification criterion based on an auxiliary residual is a novel and potentially useful idea, and the proposed two-stage framework is clearly structured. The offline algorithm (Algorithm 4) is plausibly grounded in existing minimax conditional moment estimation, and the experimental evaluation covers several environments, confounding durations, and confounder distributions. However, the paper's load-bearing theorems are not checkable because all proofs are deferred to a missing appendix, and the simulator-based learner (Algorithm 3) appears to be biased as stated, which directly contradicts the paper's claim of recovering the expert policy without confounding bias. The paper therefore requires substantial revision before its contributions can be accepted.
major comments (4)
- [Section 3] The sentence 'Please note that all proofs are provided in the Appendix' appears in the manuscript, but no appendix is included. Theorems 1-3 and Lemmas 1-2 are central to the claimed identification results, yet their correctness cannot be assessed without proofs or at least detailed proof sketches. The paper should include the full proofs in the submission, or clearly state that they are available in a supplementary document and summarize the key argument in the main text.
- [Section 4.2, Algorithm 3] The unbiasedness claim for the simulator-based policy learner is not derived and conflicts with the generative model in Eq. (1). The method minimizes E[||a_t - π(\tilde s_t)||^2] over synthetic states \tilde s_t generated from the biased policy π1. The least-squares minimizer is E[a_t | \tilde s_t], which equals E[π(s_t) | \tilde s_t] + E[h(u_t,...,u_{t-τ}) | \tilde s_t] + E[e_a | \tilde s_t]. Independence of \tilde s_t from the confounders given s_k does not imply that the second term is zero (it is generally a nonzero constant), and the first term is not equal to π(\tilde s_t) unless π is linear and E[s_t | \tilde s_t] = \tilde s_t. The statement that 'these states \tilde s_t and actions a_t are confounder-free' conflates marginal independence of \tilde s_t with the required conditional-mean equality; the observed actions still carry the confounded structure. The learned policy is therefore a biased projection, not the expert policy. The authors should either prove unbiasedness under explicit additional assumptions (e.g., linear π, zero-mean h, and E[s_t|\tilde s_t] = \tilde s_t) or remove/revise this algorithm.
- [Section 3, Assumption 2] Assumption 2 requires the second-order partial derivative of the log joint density of R_{s_t,a_t||s_k} and s_k to be nonzero. This condition is stated without justification, is not verifiable from data in the experiments, and the sufficiency direction of Theorem 3 collapses if it fails. The paper should provide a substantive explanation of why this non-degeneracy holds for the C2L model or in the benchmark environments, or offer a testable proxy condition that practitioners can check.
- [Section 5.1 and Section 3] The default experimental configuration sets the confounders to follow a Gaussian distribution, yet Theorem 2 relies on Assumption 1 (Partial Non-Gaussianity), which requires at least one confounder to be non-Gaussian or the state noise e_s of s_k to be non-Gaussian. The paper does not state whether the state noise is non-Gaussian in the default Gaussian-confounder experiments, so it is unclear whether the theory applies to the main reported setting. Please clarify the noise distributions used and, if needed, add experiments that explicitly match the conditions of Assumption 1.
minor comments (4)
- [Section 4.2, Algorithm 3] The notation 'esk+1:t ∼ Qt−1 k bT (sk, π1(sk))' is unclear and appears to be missing product or sequential composition notation; please define it precisely, e.g., as a product of transition kernels over time steps.
- [Section 3, Assumption 2] In the denominator of the partial derivative, '∂Rst,at||sk ∂sk' contains an extra subscript 'sk' on the second differential; the intended notation is likely '∂R_{s_t,a_t||s_k} ∂s_k'.
- [Section 4.2] The phrase 'these states est and actions at are confounder-free' is imprecise: the actions a_t are observed expert actions that still contain the confounding term h(...). Reword to state that the simulated states \tilde s_t are independent of the confounders given s_k, which is the actual property used.
- [Section 4.2, Eq. (5)] The minimax objective for the offline learner is justified only by citing Dikkala et al. (2020) and Swamy et al. (2022); a brief derivation or an explicit statement of how the multi-timestep C2L model fits into that framework would improve readability.
Circularity Check
No significant circularity: IV identification is derived from the structural model rather than fitted, and no load-bearing self-citation is present.
full rationale
Walking the derivation chain, the central identification results (Theorems 1-3) are not equivalent to their inputs by construction. Definition 1 defines the auxiliary residual R = a_t - l(s_t) with the constraint E[R | s_k] = 0, and the AB criterion then requires full independence of R and s_k, which is a strictly stronger condition; the theorems connect this condition to the standard IV assumptions through the structural model in Eq. (1) and the non-Gaussianity assumptions, rather than by renaming a fitted quantity. Algorithm 2 estimates the residual and applies an HSIC independence test, so no fitted constant is relabeled as a prediction. The paper also does not rely on load-bearing self-citation: its cited foundations (Swamy et al. 2022, Dikkala et al. 2020, Zhang et al. 2018) are external prior work, and the authors do not invoke their own prior theorems as a black box. Non-circular weaknesses exist and should not be mistaken for circularity: Definition 1 leaves the function l unspecified, making the estimation of R an implementation gap; Assumption 2 is an unverifiable non-degeneracy condition that is not tested in the experiments; and Section 4.2's simulator variant claims that 'these states est and actions at are confounder-free because est ⊥⊥ut...ut−τ | sk' while immediately noting that the observed actions 'maintain their confounding structure,' so the unbiasedness of Algorithm 3's objective E[||a_t - π(ẽ_s_t)||²] is underived under Eq. (1). These are correctness or derivation gaps, not reductions of the output to the input, and they do not raise the circularity score.
Assumptions & free parameters
free parameters (3)
- w (maximum candidate IV count)
- alpha (HSIC significance level)
- learning rate eta (offline algorithm)
assumptions (6)
- domain assumption Standard IV conditions A1-A3 hold for the true lag st-tau
- domain assumption Partial non-Gaussianity (Assumption 1): at least one latent confounder or the state noise is non-Gaussian
- ad hoc to paper Non-Degenerate Cross-Derivative (Assumption 2)
- domain assumption Latent confounders affect exactly tau consecutive actions and dynamics are Markovian (Eq. 1, Fig. 3)
- standard math Asymptotic infinite sample and twice-differentiable positive densities
- domain assumption The simulator transitions match the true dynamics (for Algorithm 3)
Cite this review
Pith. "Pith review of Confounded Causal Imitation Learning with Instrumental Variables." pith.science (2026). https://pith.science/paper/EHLFF72P
@misc{pith2026250717309,
author = {Pith},
title = {Pith review of: Confounded Causal Imitation Learning with Instrumental Variables},
year = {2026},
howpublished = {\url{https://pith.science/paper/EHLFF72P}},
note = {Machine review of arXiv:2507.17309}
}
read the original abstract
Imitation learning from demonstrations usually suffers from the confounding effects of unmeasured variables (i.e., unmeasured confounders) on the states and actions. If ignoring them, a biased estimation of the policy would be entailed. To break up this confounding gap, in this paper, we take the best of the strong power of instrumental variables (IV) and propose a Confounded Causal Imitation Learning (C2L) model. This model accommodates confounders that influence actions across multiple timesteps, rather than being restricted to immediate temporal dependencies. We develop a two-stage imitation learning framework for valid IV identification and policy optimization. In particular, in the first stage, we construct a testing criterion based on the defined pseudo-variable, with which we achieve identifying a valid IV for the C2L models. Such a criterion entails the sufficient and necessary identifiability conditions for IV validity. In the second stage, with the identified IV, we propose two candidate policy learning approaches: one is based on a simulator, while the other is offline. Extensive experiments verified the effectiveness of identifying the valid IV as well as learning the policy.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[13]
Gokul Swamy, Sanjiban Choudhury, Zhiwei Steven Wu, and J
URL https://arxiv.org/abs/2102.02872. Gokul Swamy, Sanjiban Choudhury, Zhiwei Steven Wu, and J. Andrew Bagnell. Of moments and matching: Trade-offs and treatments in imitation learning. CoRR, abs/2103.03236,
-
[14]
Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap
URL https://arxiv.org/abs/2103.03236. 11 Gokul Swamy, Sanjiban Choudhury, Drew Bagnell, and Steven Wu. Causal imitation learning under temporally correlated noise. In International Conference on Machine Learning , pages 20877–20890. PMLR,
-
[16]
On Covariate Shift of Latent Confounders in Imitation and Reinforcement Learning
URL https://arxiv.org/abs/2110.06539. Jur Van Den Berg, Stephen Miller, Daniel Duckworth, Humphrey Hu, Andrew Wan, Xiao-Yu Fu, Ken Goldberg, and Pieter Abbeel. Superhuman performance of surgical tasks by robots using iterative learning from human-guided demonstrations. In 2010 IEEE international conference on robotics and automation, pages 2074–2081. IEEE,
-
[1990]
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540,
-
[1999]
Proximal policy optimization algorithms
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347,
-
[2010]
URL http://arxiv.org/abs/ 1011.0686. Stefan Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences, 3(6): 233–242,
-
[2014]
URL http://arxiv.org/abs/1406.5979. Stéphane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. No-regret reductions for imitation learning and structured prediction. CoRR, abs/1011.0686,
-
[2018]
Andrew Bennett, Nathan Kallus, and Tobias Schnabel
URL http://arxiv.org/ abs/1812.03079. Andrew Bennett, Nathan Kallus, and Tobias Schnabel. Deep generalized method of moments for instrumental variable analysis. Advances in neural information processing systems, 32,
Show all 10 references
-
[2020]
Clark Glymour, Kun Zhang, and Peter Spirtes
URL https://arxiv.org/abs/2003.00806. Clark Glymour, Kun Zhang, and Peter Spirtes. Review of causal discovery methods based on graphical models. Frontiers in genetics, 10:524,
2003 arXiv
-
[2021]
Felipe Codevilla, Eder Santana, Antonio M López, and Adrien Gaidon
URL https://arxiv.org/abs/2105.10148. Felipe Codevilla, Eder Santana, Antonio M López, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous driving. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9329–9338,
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.