REVIEW 4 major objections 4 minor 15 references
Double Machine Learning for Conditional Moment Restrictions: IV Regression, Proximal Causal Learning and Beyond
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A two-stage estimator with a new Neyman-orthogonal score solves conditional moment restrictions at the minimax-optimal root-N rate.
desk verdict The root-N rate theorem is unsupported because the proposed score is identically zero at the true parameter, so J0 and E[ψψ^T] vanish and the DML conditions fail. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Neyman-orthogonal score $\psi(D; f, (s, g)) = (s(c) - g(f, c))^2$. Here $s(c)$ estimates $E[Y \mid C]$, and $g(f, c)$ estimates $E[f(X) \mid C]$, so the squared residual compares two conditional expectations rather than $Y$ and a prediction; at the truth the two coincide, making the expected score zero and its first derivative with respect to $(s, g)$ zero. Cross-fitting then trains these nuisance estimates on the complement of each fold before the score is minimised over $f_\theta$, which is the mechanism that converts first-stage estimation error into a second-order term and yields the $O(N^{-1/2})$ rate.
What would settle it
Compute $J_0 = \partial_\theta E[(s_0(C) - g_0(f_\theta, C))^2]$ at $\theta = \theta_0$ for any CMR with $s_0(C) = E[Y \mid C] = E[f_0(X) \mid C]$; the derivative is zero, so the bounded-singular-values condition of Theorem 6 fails. The same holds for the covariance $E[\psi \psi^T]$, whose entries vanish at $\theta_0$.
Extended reading notes
Core claim
The paper's central claim is that the plug-in bias of two-stage CMR estimation can be removed without influence-function calculations by replacing the standard squared loss $(Y - g(f, c))^2$ with $(s(c) - g(f, c))^2$, where $s_0(c) = E[Y \mid C]$ and $g_0(f, c) = E[f(X) \mid C]$ are nuisance functions. Since $s_0(c) = g_0(f_0, c)$ at the true parameter, the score vanishes at the truth and its Gateaux derivative with respect to the nuisance parameters is zero, which is Neyman orthogonality. Under a finite-dimensional parameterisation of $f_0$ and regularity conditions, Theorem 6 asserts that the cross-fitted estimator $\hat{\theta}$ is concentrated in an $N^{-1/2}$ neighbourhood of $\theta_0$ and satisfies $\sqrt{N}(\hat{\theta} - \theta_0) \to N(0, \sigma^2)$, matching the minimax parametric rate; the variance has the usual sandwich form. The paper also shows that the Jacobian non-singularity required by the double machine learning identification condition implies bounded ill-posedness of the underlying CMR problem.
Load-bearing premise
The rate guarantee depends on the Jacobian of the expected score and the score covariance being non-degenerate at the true parameter, but for the proposed squared score both are identically zero at the truth because $s_0(c) = g_0(f_0, c)$.
Editorial extensions
If this is right
- If the rate claim holds, DML-CMR offers a single debiasing recipe for any conditional moment restriction problem, not just IV and proximal causal learning, with neural-network nuisance estimators.
- First-stage nuisance estimators only need to achieve the $o(N^{-1/4})$ convergence rate, a condition satisfied by many standard learners, so the method inherits the debiasing promise of double machine learning for nonlinear problems.
- Because Jacobian non-singularity implies bounded ill-posedness, the paper connects the usual identifiability assumption in conditional moment problems to the double machine learning rate theorem.
- Experimental comparisons on ticket demand, MNIST-augmented, IHDP, PM-CMR, and dSprites benchmarks show lower mean squared error than existing IV and proximal causal learning methods, with especially strong gains under weak instruments.
- The computationally efficient version without cross-fitting, CE-DML-CMR, performs comparably on low-dimensional datasets, suggesting the score itself provides most of the debiasing benefit there.
Reading between the lines
- For nonparametric $f_0$, the same Neyman-orthogonal score could be analysed through orthogonal statistical learning bounds; the paper names this as future work, and a concrete outcome would be rate guarantees that depend on function-class complexity rather than parameter dimension.
- For proximal causal learning, the score's orthogonality could be ported to bridge-function estimators that currently require explicit influence-function derivations, potentially yielding doubly robust estimators for continuous treatments in a unified way.
- A direct transfer to other conditional moment problems, such as demand estimation with endogeneity or treatment choice, is plausible because the score only requires estimating $E[Y \mid C]$ and $E[f(X) \mid C]$ as nuisance functions.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes DML-CMR, a two-stage estimator for conditional moment restrictions built on the double/debiased machine learning framework. The estimator uses the score function ψ(D; f, (s, g)) = (s(c) − g(f, c))², where s(c) = E[Y|C=c] and g(f, c) = E[f(X)|C=c] are treated as nuisance parameters, and it employs K-fold cross-fitting so that nuisance estimators are trained on the complement of the fold used for the second-stage loss. Under a parametric assumption on the function of interest, the paper claims in Theorem 6 that the resulting estimator is concentrated in an N^{−1/2} neighbourhood of the true parameter and satisfies asymptotic normality, and it also claims this rate is minimax optimal. The experiments compare DML-CMR against existing IV regression and proximal causal learning methods on synthetic, semi-synthetic, and high-dimensional benchmarks.
Significance. If the central convergence claim were valid, the paper would make a useful contribution: it provides a concrete, non-adversarial DML-style algorithm for general conditional moment problems, it supports both neural and tree-based nuisance estimation, and the experimental evaluation is broad, includes external baselines, and is backed by released code. The connection drawn between DML's identification condition and bounded ill-posedness is also potentially interesting. However, the load-bearing theoretical result is not established: the proposed squared score is degenerate at the true parameter, so the Jacobian and score-covariance conditions required by the DML theorem fail. Because the claimed O(N^{−1/2}) rate is the paper's primary theoretical contribution, the soundness of the central claim is in question.
major comments (4)
- [Section 4.1, Eq. (5) and Appendix C.1, proof of Theorem 6] The score function is identically zero at the true parameter. Since s0(C) = E[Y|C] = E[f0(X)|C] = g0(f0, C), we have ψ(D; f0, (s0, g0)) = (s0(C) − g0(f0, C))² = 0 almost surely. Consequently the Jacobian J0 = ∂θ′ E[(s0(C) − g0(fθ′, C))²]|θ′=θ0 = 2E[(s0(C) − g0(fθ0, C))(−∂θ g0(fθ0, C))] = 0, and the covariance E[ψ(D; f0, (s0, g0))ψ(D; f0, (s0, g0))T] = E[ψ²] = 0. This directly contradicts Condition 4(d), which requires the singular values of J0 to be bounded away from zero, and Condition 4(e), which requires the score covariance to have all eigenvalues strictly positive. Moreover, the variance formula σ² = J0^{-1}E[ψψT](J0^{-1})T in Theorem 6 is undefined because J0 is not invertible. Thus the central N^{-1/2} convergence claim is not proved.
- [Appendix C.1, verification of Condition (e)] The trace-trick argument for positive definiteness is invalid. The proof notes that each outer product ψψT has eigenvalue ∥ψ∥² ≥ 0 and concludes that E[ψψT] is positive definite, but non-negativity of eigenvalues is not positivity. For the proposed scalar score, ψ = 0 on the support at the true parameter, so the only eigenvalue is 0 and the covariance matrix is the zero matrix. This is not a technical gap but a direct failure of Condition 4(e).
- [Section 4.3, Condition 4(d) and Section 4.4, Proposition 9] The identification condition is also internally inconsistent with the squared score. A first-order Taylor expansion gives E[ψ(D; fθ, (s0, g0))] = J0(θ − θ0) + O(∥θ − θ0∥²), and since J0 = 0, the leading term vanishes and ∥E[ψ]∥ = O(∥θ − θ0∥²). No constant c0 > 0 can therefore satisfy 2∥E[ψ]∥ ≥ c0∥θ − θ0∥ in a neighbourhood of θ0. Proposition 9, which derives bounded ill-posedness from Condition 4(d), rests on the same impossible assumption.
- [Appendix C.1, Lemma 5 and Condition 4(f)] Even setting aside the degeneracy of the score, the proof that Condition 4(f) holds is not correct as written. Lemma 5 bounds the squared norm ∥bs − s0∥₂² by c1(δN(S*)² + √(log(1/ζ)/N)), but the proof of Theorem 6 converts this into ∥bs − s0∥₂ ≤ O(δN + N^{-1/2}). Taking square roots of the lemma's bound yields O(N^{-1/4} log^{1/4}) at best when δN = o(N^{-1/4}), which is not o(N^{-1/4}) as Condition 4(f) requires. The nuisance-rate condition therefore does not follow from the stated lemma.
minor comments (4)
- [Section 4.2, Eq. (6)] Equation (6) has a typo: the second-stage loss should read bEk[(bsk(c) − bgk(fbθ, c))²], with the closing parenthesis and square placed correctly.
- [Figure 3 caption] The caption reads "Real-World Datsets"; this should be "Real-World Datasets".
- [Appendix G.2] The text refers to "Lemma 3.3" and "Theorem 3.4 and 3.5", but these numbered results do not exist in the paper; the intended references are likely Lemma 5 and Theorem 6 (or Corollary 7).
- [Algorithm 1] The notation bEck_i in line 11 of Algorithm 1 is not defined; the paper should define it as the empirical expectation over the mini-batch sampled from fold Ik.
Circularity Check
No circular derivation: DML-CMR's objective and comparisons are self-contained, and the main theoretical flaw is a degeneracy/correctness failure rather than circularity; the only self-citation concern is the reuse of the authors' own DML-IV IV-regression results, which is not load-bearing for the general CMR claim.
full rationale
Walking the paper's claimed derivation chain, the central claim is that the DML-CMR estimator converges at rate O(N^{-1/2}) by applying Theorem 3.3 of Chernozhukov et al. (2018) to the score ψ(D; f, (s,g)) = (s(c)-g(f,c))². This is not a circular reduction in the sense defined here: the estimator is a self-contained minimization objective, and the empirical comparisons are against external baselines (DeepIV, DeepGMM, KIV, DFIV, NMMR, etc.), with new PCL experiments and code. The paper does contain a serious technical degeneracy: because s0(C)=E[Y|C]=E[f0(X)|C]=g0(f0,C), the score is identically zero at the true parameter, so the Jacobian J0 and the score covariance E[ψψ^T] are zero rather than non-singular/positive-definite as required by Conditions 4(d) and 4(e). However, this is a false-premise/correctness problem, not circularity: the rate theorem does not reduce to its inputs or to a fitted parameter; it is an imported theorem whose key conditions are asserted and are in fact unsatisfiable for this score. The paper also reuses IV-regression results from the authors' prior DML-IV paper and states that DML-CMR is identical to DML-IV for IV regression, which is a minor self-citation, but it is not load-bearing for the general CMR contribution and is clearly disclosed. Under the hard rules, no specific prediction reduces by construction to its input, so the honest finding is no significant circularity; the score of 2 reflects the minor self-citation overlap rather than a circular derivation.
Assumptions & free parameters
assumptions (4)
- domain assumption Assumption 1: the true function is finite-dimensional and parameterised, f0 = fθ0 with Θ compact and θ0 in the interior.
- domain assumption Assumption 2: realisability and boundedness of g0, s0, f0 in their function classes and |Y| ≤ B.
- domain assumption Critical radius δ_N = o(N^{-1/4}) for nuisance estimators bs and bg.
- ad hoc to paper J0 has bounded singular values and E[ψψ^T] at θ0 is positive definite.
Cite this review
Pith. "Pith review of Double Machine Learning for Conditional Moment Restrictions: IV Regression, Proximal Causal Learning and Beyond." pith.science (2026). https://pith.science/paper/MG2NPIDH
@misc{pith2026250614950,
author = {Pith},
title = {Pith review of: Double Machine Learning for Conditional Moment Restrictions: IV Regression, Proximal Causal Learning and Beyond},
year = {2026},
howpublished = {\url{https://pith.science/paper/MG2NPIDH}},
note = {Machine review of arXiv:2506.14950}
}
abstract
Solving conditional moment restrictions (CMRs) is a key problem considered in statistics, causal inference, and econometrics, where the aim is to solve for a function of interest that satisfies some conditional moment equalities. Specifically, many techniques for causal inference, such as instrumental variable (IV) regression and proximal causal learning (PCL), are CMR problems. Most CMR estimators use a two-stage approach, where the first-stage estimation is directly plugged into the second stage to estimate the function of interest. However, naively plugging in the first-stage estimator can cause heavy bias in the second stage. This is particularly the case for recently proposed CMR estimators that use deep neural network (DNN) estimators for both stages, where regularisation and overfitting bias is present. We propose DML-CMR, a two-stage CMR estimator that provides an unbiased estimate with fast convergence rate guarantees. We derive a novel learning objective to reduce bias and develop the DML-CMR algorithm following the double/debiased machine learning (DML) framework. We show that our DML-CMR estimator can achieve the minimax optimal convergence rate of $O(N^{-1/2})$ under parameterisation and mild regularity conditions, where $N$ is the sample size. We apply DML-CMR to a range of problems using DNN estimators, including IV regression and proximal causal learning on real-world datasets, demonstrating state-of-the-art performance against existing CMR estimators and algorithms tailored to those problems.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[10]
K. Muandet, A. Mehrjou, S. K. Lee, and A. Raj. Dual instrumental variable regression. Advances in Neural Information Processing Systems , 2020-December, 10
work page 2020
- [11]
- [12]
-
[13]
I. Shpitser and J. Pearl. Complete identification methods for the causal hierarchy. Journal of Machine Learning Research, 9(64):1941–1979,
work page 1941
-
[15]
L. Xu, Y. Chen, S. Srinivasan, N. de Freitas, A. Doucet, and A. Gretton. Learning deep features in instrumental variable regression. ICLR 2021 - 9th International Conference on Learning Representations, 10
work page 2021
-
[2009]
ISSN 0090-5364. doi: 10.1214/08-AOS620. J. M. Blair, C. A. Edwards, and J. H. Johnson. Rational chebyshev approximations for the inverse of the error function. Mathematics of Computation , 30(136):827, 10
-
[2010]
URL http://yann. lecun.com/exdb/mnist/. L. Liao, Y. L. Chen, Z. Yang, B. Dai, Z. Wang, and M. Kolar. Provably efficient neural estimation of structural equation model: An adversarial approach. Advances in Neural Information Processing Systems, 2020-December, 7
work page 2020
-
[2011]
N. Dikkala, G. Lewis, L. Mackey, and V. Syrgkanis. Minimax estimation of conditional moment models. In Advances in Neural Information Processing Systems , volume 2020- December. Advances in Neural Information Processing Systems, 6
work page 2020
Show all 15 references
-
[2016]
Mackey, V
52 DML for Conditional Moment Restrictions L. Mackey, V. Syrgkanis, and D. Zadik. Orthogonal machine learning: Power and limitations. 35th International Conference on Machine Learning, ICML 2018 , 13:9112–9124, 11
2018
-
[2017]
Y. Luo, M. Spindler, and J. K¨ uck. High-dimensional l 2 boosting: Rate of convergence. arXiv preprint arXiv:1602.08927 ,
-
[2018]
Chernozhukov, W
V. Chernozhukov, W. K. Newey, V. Quintas-Martinez, and V. Syrgkanis. Automatic debiased machine learning via neural nets for generalized linear regression. arXiv preprint arXiv:2104.14737, 4
-
[2019]
I. Bica, J. Jordon, and M. van der Schaar. Estimating the effects of continuous-valued interventions using generative adversarial networks. Advances in Neural Information Processing Systems, 2020-December, 2
2020
-
[2020]
Loshchilov and F
I. Loshchilov and F. Hutter. Decoupled weight decay regularization. 7th International Conference on Learning Representations, ICLR 2019 , 11
2019
-
[2022]
D. J. Im, K. Cho, and N. Razavian. Causal effect variational autoencoder with uniform treatment. arXiv preprint arXiv:2111.08656 , 11
-
[2023]
URL https://mathworld.wolfram.com/ AsymptoticNotation.html. P. G. Wright. The tariff on animal and vegetable oils. https://doi.org/10.1086/254144, 38: 619–620, 10
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.