pith. sign in

arxiv: 2402.05384 · v2 · submitted 2024-02-08 · 📊 stat.ME

Efficient Nonparametric Inference for Mediation Analysis with Nonignorable Missing Confounders

Jiawei Shan , Wei Li , Chunrong Ai This is my paper

Pith reviewed 2026-05-24 03:34 UTC · model grok-4.3

classification 📊 stat.ME
keywords mediation analysisnonignorable missing datashadow variablessemiparametric efficiencysieve estimationcausal inferenceefficient influence functiondebiased machine learning
0
0 comments X

The pith

SIO estimator reaches the semiparametric efficiency bound for mediation effects despite nonignorable missing confounders

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper develops a shadow variable framework that identifies mediation effects when confounders are missing nonignorably. It introduces the Sieve-based Iterative Outward estimator and proves it attains the semiparametric efficiency bound under stated conditions while explicitly quantifying the efficiency loss caused by the missingness via the efficient influence function. This supplies both identification and asymptotically normal inference, including a debiased machine learning version, for settings where standard missing-at-random assumptions fail.

Core claim

Under a general shadow variable framework allowing shadow variables to be chosen from observed covariates or external auxiliary data, the Sieve-based Iterative Outward estimator is locally efficient, attains the semiparametric efficiency bound, and yields asymptotic normality despite the ill-posed inverse problem; the efficiency loss attributable to missingness is quantified through the efficient influence function, and a debiased machine learning procedure is provided for estimation and inference.

What carries the argument

The Sieve-based Iterative Outward (SIO) estimator, which iteratively applies sieve approximations to recover the mediation functionals from the observed data under the shadow-variable identification conditions.

If this is right

  • Mediation effects remain identifiable and estimable nonparametrically when suitable shadow variables are available.
  • Asymptotic normality holds for the SIO estimator despite the ill-posed inverse problem.
  • The efficiency loss due to nonignorable missingness is explicitly quantified by the efficient influence function.
  • A debiased machine learning procedure yields practical estimation and inference for the mediation effects.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same shadow-variable construction could be applied to other causal functionals such as direct and indirect effects in longitudinal settings.
  • Survey designers might deliberately collect low-cost auxiliary variables to serve as shadow variables and thereby reduce the sample-size penalty from missing confounders.
  • Replacing the sieve step with other flexible nonparametric estimators could extend the method to very high-dimensional covariate spaces while preserving the efficiency bound.

Load-bearing premise

Suitable shadow variables exist that satisfy the identification conditions for the mediation effects in the presence of nonignorable missing confounders.

What would settle it

A dataset or simulation in which candidate shadow variables violate the required completeness or conditional independence conditions and the resulting mediation-effect estimates fail to converge to the true value.

Figures

Figures reproduced from arXiv: 2402.05384 by Chunrong Ai, Jiawei Shan, Wei Li.

Figure 1
Figure 1. Figure 1: Causal directed acyclic graphs illustrating Examples [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The boxplots of estimates of the different methods among 500 replications under sample sizes [PITH_FULL_IMAGE:figures/full_fig_p015_2.png] view at source ↗
read the original abstract

Mediation analysis is widely used for exploring treatment mechanisms; however, it faces challenges when nonignorable missing confounders are present. Efficient inference of mediation effects and the efficiency loss due to nonignorable missingness have been rarely studied in the literature because of the difficulties arising from the ill-posed inverse problem. In this paper, we propose a general shadow variable framework for identifying mediation effects, allowing shadow variables to be selected from either observed covariates or externally collected auxiliary data. We then propose a Sieve-based Iterative Outward (SIO) approach for estimation. We establish large-sample theory, particularly asymptotic normality, for the proposed estimator despite the ill-posedness of the problem. We show that our estimator is locally efficient and attains the semiparametric efficiency bound under certain conditions. Building on the efficient influence function, we explicitly quantify the efficiency loss attributable to missingness and propose a debiased machine learning approach for estimation and inference. We examine the finite-sample performance of the proposed approach using extensive simulation studies and showcase its practical applicability through an empirical analysis of CFPS data.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 0 minor

Summary. The paper proposes a shadow variable framework to identify mediation effects under nonignorable missing confounders, develops a Sieve-based Iterative Outward (SIO) estimator, establishes large-sample theory including asymptotic normality despite the ill-posed inverse problem, shows that the estimator is locally efficient and attains the semiparametric efficiency bound under certain conditions, quantifies efficiency loss via the efficient influence function, and proposes a debiased machine learning approach, with supporting simulations and an application to CFPS data.

Significance. If the identification arguments and regularity conditions hold, the work would fill a notable gap in mediation analysis by enabling efficient nonparametric inference in the presence of a common practical complication (nonignorable missingness). The flexible shadow-variable construction (from covariates or auxiliary data) and the explicit quantification of efficiency loss are potentially useful contributions to semiparametric causal inference.

major comments (1)
  1. [Abstract] Abstract: the central efficiency claim (local efficiency and attainment of the semiparametric bound) is stated to hold only 'under certain conditions' on the shadow variables and on the sieve/iteration control of the ill-posed inverse problem; without explicit verification that the chosen shadow variables satisfy the required completeness or that the sieve dimension grows at the precise rate needed to offset ill-posedness, the efficiency result remains conditional on technical assumptions that are difficult to check in finite samples or real data.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their detailed review and constructive comments on our manuscript. We address the single major comment below.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central efficiency claim (local efficiency and attainment of the semiparametric bound) is stated to hold only 'under certain conditions' on the shadow variables and on the sieve/iteration control of the ill-posed inverse problem; without explicit verification that the chosen shadow variables satisfy the required completeness or that the sieve dimension grows at the precise rate needed to offset ill-posedness, the efficiency result remains conditional on technical assumptions that are difficult to check in finite samples or real data.

    Authors: We agree that the local efficiency and attainment of the semiparametric efficiency bound are established under explicit technical conditions, which are necessary given the ill-posed inverse problem. These conditions—including the completeness of the shadow variable (Assumption 3) and the precise growth rates for the sieve dimension and iteration number to offset ill-posedness (Conditions C1–C4 in Section 4)—are stated in the identification and asymptotic theory sections. The abstract accurately qualifies the result with 'under certain conditions' to reflect this. Completeness is an identification assumption that, like many in causal inference (e.g., positivity or exclusion restrictions), is assessed via substantive knowledge rather than direct statistical testing; the manuscript discusses selecting shadow variables from covariates or auxiliary data on this basis. The sieve tuning parameters are chosen to satisfy the rate conditions, with practical guidance via cross-validation in the implementation. Simulations verify performance when conditions hold, and the CFPS application uses contextually motivated shadow variables. We will add a brief remark in the revised discussion section on practical assessment of these conditions without altering the abstract, as the current wording is precise. revision: partial

Circularity Check

0 steps flagged

No circularity: new identification and estimation framework stands independently.

full rationale

The paper introduces a novel shadow-variable identification strategy for mediation effects under nonignorable missingness and derives the SIO estimator plus its asymptotic properties from semiparametric efficiency theory. No quoted step reduces a claimed prediction or efficiency result to a fitted parameter or self-citation by construction; the efficiency bound is attained under explicitly stated regularity conditions rather than tautologically. The derivation chain is self-contained against external benchmarks in nonparametric statistics.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 1 invented entities

Review performed from abstract only; full details on modeling assumptions, sieve basis choices, and regularity conditions unavailable.

axioms (2)
  • domain assumption Existence of shadow variables sufficient for identification of mediation effects under nonignorable missingness
    Central to the proposed identification strategy described in the abstract.
  • domain assumption Regularity conditions allowing asymptotic normality despite the ill-posed inverse problem
    Invoked to support the large-sample theory claims.
invented entities (1)
  • Shadow variable framework no independent evidence
    purpose: Identifying mediation effects when confounders are missing nonignorably
    General framework allowing shadow variables from observed covariates or external auxiliary data.

pith-pipeline@v0.9.0 · 5719 in / 1389 out tokens · 30780 ms · 2026-05-24T03:34:30.053993+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

48 extracted references · 48 canonical work pages

  1. [1]

    and Chen, X

    Ai, C. and Chen, X. (2003). Efficient estimation of models with conditional moment restrictions containing unknown functions. Econometrica , 71(6):1795--1843

    work page 2003

  2. [2]

    and Chen, X

    Ai, C. and Chen, X. (2007). Estimation of possibly misspecified semiparametric conditional moment restriction models with different conditioning variables. Journal of Econometrics , 141(1):5--43

    work page 2007

  3. [3]

    and Chen, X

    Ai, C. and Chen, X. (2012). The semiparametric efficiency bound for models of sequential moment restrictions containing unknown functions. Journal of Econometrics , 170(2):442--457

    work page 2012

  4. [4]

    Ai, C., Linton, O., and Zhang, Z. (2020). A simple and efficient estimation method for models with NonIgnorable missing data. Statistica Sinica , 30(4):1949--1970

    work page 2020

  5. [5]

    Baron, R. M. and Kenny, D. A. (1986). The moderator mediator variable distinction in social psychological research: Conceptual , strategic, and statistical considerations. Journal of Personality and Social Psychology , 51(6):1173--1182

    work page 1986

  6. [6]

    Bennett, A., Kallus, N., Mao, X., Newey, W., Syrgkanis, V., and Uehara, M. (2022). Inference on strongly identified functionals of weakly identified functions. arXiv preprint arXiv:2208.08291

    work page arXiv 2022

  7. [7]

    Blundell, R., Chen, X., and Kristensen, D. (2007). Semi-nonparametric IV estimation of shape-invariant engel curves. Econometrica , 75(6):1613--1669

    work page 2007

  8. [8]

    and Haan, P

    Breunig, C. and Haan, P. (2021). Nonparametric regression with selectively missing covariates. Journal of Econometrics , 223(1):28--52

    work page 2021

  9. [9]

    Breunig, C., Mammen, E., and Simoni, A. (2018). Nonparametric estimation in case of endogenous selection. Journal of Econometrics , 202(2):268--285

    work page 2018

  10. [10]

    Carrasco, M., Florens, J.-P., and Renault, E. (2007). Linear inverse problems in structural econometrics estimation based on spectral decomposition and regularization. In Heckman, J. J. and Leamer, E. E., editors, Handbook of Econometrics , volume 6, pages 5633--5751. Elsevier

    work page 2007

  11. [11]

    Chen, K. (2001). Parametric models for response-biased sampling. Journal of the Royal Statistical Society Series B: Statistical Methodology , 63(4):775--789

    work page 2001

  12. [12]

    Chen, Q. (2021). Robust and optimal estimation for partially linear instrumental variables models with partial identification. Journal of Econometrics , 221(2):368--380

    work page 2021

  13. [13]

    Chen, X. (2007). Large sample sieve estimation of semi-nonparametric models. In Heckman, J. J. and Leamer, E. E., editors, Handbook of Econometrics , volume 6, pages 5549--5632. Elsevier

    work page 2007

  14. [14]

    and Pouzo, D

    Chen, X. and Pouzo, D. (2012). Estimation of nonparametric conditional moment models with possibly nonsmooth generalized residuals. Econometrica , 80(1):277--321

    work page 2012

  15. [15]

    Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., Newey, W., and Robins, J. (2018). Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal , 21(1):C1--C68

    work page 2018

  16. [16]

    P., and Renault, E

    Darolles, S., Fan, Y., Florens, J. P., and Renault, E. (2011). Nonparametric instrumental regression. Econometrica , 79(5):1541--1565

    work page 2011

  17. [17]

    D'Haultfoeuille, X. (2010). A new instrumental method for dealing with endogenous selection. Journal of Econometrics , 154(1):1--15

    work page 2010

  18. [18]

    D'Haultfoeuille, X. (2011). On the completeness condition in nonparametric instrumental problems. Econometric Theory , 27(3):460--471

    work page 2011

  19. [19]

    W., and Ridder, G

    Hirano, K., Imbens, G. W., and Ridder, G. (2003). Efficient estimation of average treatment effects using the estimated propensity score. Econometrica , 71(4):1161--1189

    work page 2003

  20. [20]

    Huber, M. (2014). Identifying causal mechanisms (primarily) based on inverse probability weighting: Identifying causal mechanisms. Journal of Applied Econometrics , 29(6):920--943

    work page 2014

  21. [21]

    Imai, K., Keele, L., and Tingley, D. (2010a). A general approach to causal mediation analysis. Psychological Methods , 15(4):309--334

    work page

  22. [22]

    Imai, K., Keele, L., and Yamamoto, T. (2010b). Identification, inference and sensitivity analysis for causal mediation effects. Statistical Science , 25(1):51--71

    work page

  23. [23]

    H., Ma, Z., McHugh, M

    Kennedy, E. H., Ma, Z., McHugh, M. D., and Small, D. S. (2017). Non-parametric methods for doubly robust estimation of continuous treatment effects. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 79(4):1229--1245

    work page 2017

  24. [24]

    Kott, P. S. (2014). Calibration weighting when model and calibration variables can differ. In Mecatti, F., Conti, P. L., and Ranalli, M. G., editors, Contributions to Sampling Statistics , pages 1--18, Cham . Springer International Publishing

    work page 2014

  25. [25]

    Kress, R. (1989). Linear Integral Equations , volume 82 of Applied Mathematical Sciences . Springer New York , New York, NY

    work page 1989

  26. [26]

    Li, W., Miao, W., and Tchetgen Tchetgen, E. (2023). Non-parametric inference about mean functionals of non-ignorable non-response data without identifying the joint distribution. Journal of the Royal Statistical Society Series B: Statistical Methodology , 85(3):913--935

    work page 2023

  27. [27]

    and Zhou, X.-H

    Li, W. and Zhou, X.-H. (2017). Identifiability and estimation of causal mediation effects with missing data. Statistics in Medicine , 36(25):3948--3965

    work page 2017

  28. [28]

    Little, R. J. A. and Rubin, D. B. (2002). Statistical Analysis with Missing Data . Wiley Series in Probability and Statistics . Wiley , 1 edition

    work page 2002

  29. [29]

    Liu, Y., Yang, X., Wu, Y., Xu, Y., Zhong, Y., and Yang, S. (2023). The relationship between job satisfaction and depressive symptoms among chinese adults aged 35 60 years: The mediating role of subjective well-being and life satisfaction. International Journal of Environmental Research and Public Health , 20(3):2023

    work page 2023

  30. [30]

    T., and Geng, Z

    Miao, W., Liu, L., Tchetgen, E. T., and Geng, Z. (2015). Identification, doubly robust estimation, and semiparametric efficiency theory of nonignorable missing data with a shadow variable. arXiv preprint arXiv:1509.02556

    work page arXiv 2015

  31. [31]

    and Tchetgen Tchetgen, E

    Miao, W. and Tchetgen Tchetgen, E. (2018). Identification and inference with nonignorable missing covariate data. Statistica Sinica , 28(4):2049--2067

    work page 2018

  32. [32]

    Newey, W. K. (1994). The asymptotic variance of semiparametric estimators. Econometrica , 62(6):1349--1382

    work page 1994

  33. [33]

    Newey, W. K. (1997). Convergence rates and asymptotic normality for series estimators. Journal of Econometrics , 79(1):147--168

    work page 1997

  34. [34]

    Newey, W. K. and Powell, J. L. (2003). Instrumental variable estimation of nonparametric models. Econometrica , 71(5):1565--1578

    work page 2003

  35. [35]

    Pearl, J. (2001). Direct and indirect effects. In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence , UAI '01, pages 411--420, San Francisco, CA, USA . Morgan Kaufmann Publishers Inc

    work page 2001

  36. [36]

    Ramalho, E. A. and Smith, R. J. (2013). Discrete choice non-response. The Review of Economic Studies , 80(1):343--364

    work page 2013

  37. [37]

    Robins, J. M. and Greenland, S. (1992). Identifiability and exchangeability for direct and indirect effects. Epidemiology , 3(2):143--155

    work page 1992

  38. [38]

    Rubin, D. B. (1976). Inference and missing data. Biometrika , 63(3):581--592

    work page 1976

  39. [39]

    Santos, A. (2011). Instrumental variable methods for recovering continuous linear functionals. Journal of Econometrics , 161(2):129--146

    work page 2011

  40. [40]

    and Liu, L

    Sun, Z. and Liu, L. (2021). Semiparametric inference of causal effect with nonignorable missing confounders. Statistica Sinica , 31(4):1669--1688

    work page 2021

  41. [41]

    Tang, G., Little, R. J. A., and Raghunathan, T. E. (2003). Analysis of multivariate missing data with nonignorable nonresponse. Biometrika , 90(4):747--764

    work page 2003

  42. [42]

    and Shpitser, I

    Tchetgen Tchetgen, E. and Shpitser, I. (2012). Semiparametric theory for causal mediation analysis: Efficiency bounds, multiple robustness and sensitivity analysis. The Annals of Statistics , 40(3):1816--1845

    work page 2012

  43. [43]

    and Vanderweele, T

    Valeri, L. and Vanderweele, T. J. (2013). Mediation analysis allowing for exposure-mediator interactions and causal interpretation: Theoretical assumptions and implementation with SAS and SPSS macros. Psychological Methods , 18(2):137--150

    work page 2013

  44. [44]

    Vansteelandt, S., Bekaert, M., and Lange, T. (2012). Imputation strategies for the estimation of natural direct and indirect effects. Epidemiologic Methods , 1(1):131--158

    work page 2012

  45. [45]

    Wang, S., Shao, J., and Kim, J. K. (2014). An instrumental variable approach for identification and estimation with nonignorable nonresponse. Statistica Sinica , 24(3):1097--1116

    work page 2014

  46. [46]

    Yang, S., Wang, L., and Ding, P. (2019). Causal inference with confounders missing not at random. Biometrika , 106(4):875--888

    work page 2019

  47. [47]

    and Ma, Y

    Zhao, J. and Ma, Y. (2022). A versatile estimation procedure without estimating the nonignorable missingness mechanism. Journal of the American Statistical Association , 117(540):1916--1930

    work page 2022

  48. [48]

    Zuo, S., Ghosh, D., Ding, P., and Yang, F. (2022). Mediation analysis with the mediator and outcome missing not at random. arXiv preprint arXiv:2212.05577

    work page arXiv 2022