A Human-Augmenting Agentic Workflow for Observational Causal Inference

Adrien Alexandre; Lars Olds; Nathan Kallus; Winston Chou; Yi Zhang

REVIEW 5 major objections 5 minor 26 references

A structured human-in-the-loop workflow, not the language model itself, is what makes AI-assisted observational causal inference reliable.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 04:41 UTC pith:BREO6HM7

load-bearing objection A useful, honestly-scoped engineering paper about a human-in-the-loop LLM scaffold for causal inference; the central ablation is plausible but the evidence is under-reported in the text. the 5 major comments →

arxiv 2607.22443 v1 pith:BREO6HM7 submitted 2026-07-24 stat.CO stat.AP

A Human-Augmenting Agentic Workflow for Observational Causal Inference

Winston Chou , Adrien Alexandre , Lars Olds , Yi Zhang , Nathan Kallus This is my paper

classification stat.CO stat.AP

keywords observational causal inferenceLLM agentshuman-in-the-loopagentic workflowdouble machine learningdoubly robust estimationpropensity score diagnosticsevaluation without ground truth

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Observational causal inference (OCI) is a specialized, judgment-heavy task where AI data-analysis agents fail when given only a prompt. This paper argues that embedding a general-purpose language model in a structured workflow — templated notebooks, deterministic estimators, design diagnostics, a critic persona, and human inspection — turns it into a reliable estimator. The evidence is a capability ladder: from prompt-only to the full scaffold, the full workflow dominates all intermediate tiers on both accuracy and calibration across three OCI tasks. On a disguised version of a classic job-training study, only the fully scaffolded agent achieves reasonable accuracy. The paper concludes that the workflow, not the underlying model, is the source of reliability.

Core claim

The paper's central claim is that scaffolding is the causal ingredient for successful LLM-based observational causal inference. Concretely, on synthetic benchmark data from a 2016 causal inference competition, the critic's diagnostic grading separates 192 satisfactory estimates from 39 unsatisfactory ones, with the satisfactory group showing much lower RMSE and better-calibrated confidence intervals. In the capability-ladder ablation, the full oci-agent workflow beats every less structured tier, including letting the model write and run its own code, on both accuracy and interval score. On a disguised version of the classic job-training study — where a leakage test rules out memorization — o

What carries the argument

The actor-critic loop: the actor runs a templated notebook that applies doubly robust estimators (AIPW) or double machine learning on a partially linear model, plus design diagnostics such as covariate balance and propensity overlap; the critic then inspects the output, grades it satisfactory/unsatisfactory, and can trigger corrective actions like trimming or changing the estimand. The workflow publishes intermediate artifacts — plans, specs, executed notebooks, critic reports — so humans can audit and re-run every step. The key identity carrying the argument is the critic's diagnostic grading: the separation between satisfactory and unsatisfactory estimates is what licenses the claim that t

Load-bearing premise

The paper's central premise is that the measured superiority of the scaffolded workflow is genuinely caused by the workflow's structure, even though most supporting figures are presented qualitatively and only two numeric error values are reported.

What would settle it

Re-running the disguised job-training study with a different language model and the same five-tier capability ladder: if the prompt + code + double-machine-learning tier already achieved median absolute percentage error comparable to the full scaffold (near 52% and 19%), the claim that the full workflow is necessary would be falsified. Alternatively, a replication where the critic's satisfactory/unsatisfactory labels do not track RMSE on a fresh set of datasets would falsify the claim that the diagnostics certify reliability.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

If the scaffold is the source of reliability, then deploying LLMs for OCI without such a workflow is not advisable; current frontier models alone do not consistently produce trustworthy causal estimates.
The critic's diagnostic grading appears to act as a certificate of reliability: on the benchmark data, estimates it flags as satisfactory are markedly more accurate and better calibrated than those it flags as unsatisfactory.
On the disguised job-training study, the paper shows that the model is not reciting the memorized experimental answer; the leakage test's slope of about 0.8 indicates the scaffolded agent responds to injected shifts in the true effect.
For heterogeneous treatment effects and multiple continuous treatments, the same pattern holds: pure prompting is uncorrelated with ground truth, while the full scaffold recovers the known contrasts or coefficients.
Human oversight is not merely a safety net; in an internal case study, the critic caught poor overlap that a baseline regression missed, which changed the estimand and reduced the estimated effect by a factor of four.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The paper's evaluation design suggests a general recipe for assessing agents on tasks without ground truth: combine synthetic benchmarks that inject known effects with process audits that check whether the agent's own diagnostics predict its errors. This could transfer to other expert domains.
If the workflow, not the model, is what matters, then progress in LLM capabilities may not automatically improve OCI performance; instead, investment in better diagnostics and critic agents could have outsized returns.
A natural next test would be to run the same capability ladder on other high-stakes statistical tasks — survival analysis, missing-data imputation, or time-series policy evaluation — to see whether the scaffold's advantage is universal or specific to causal inference.
The paper's strong claim about the critic's separation of reliable from unreliable estimates could be stress-tested by adversarially injecting failures into known analyses and measuring whether the critic flags them; the paper does not report such a sensitivity analysis.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

A useful, honestly-scoped engineering paper about a human-in-the-loop LLM scaffold for causal inference; the central ablation is plausible but the evidence is under-reported in the text.

read the letter

The paper's real contribution is the scaffold pattern itself: an actor-critic loop around deterministic, vetted OCI tools (AIPW, DoubleML), with inspectable artifacts for humans. That is a sensible and timely idea, and the paper is honest that the statistics underneath are standard. The capability-ladder ablation—prompt-only up to the full scaffold—is the right way to ask whether the workflow, rather than the LLM or the estimator choice, is doing the work. The disguised-LaLonde leakage test is a thoughtful check against memorization; it is the kind of detail that makes me trust the authors are thinking about actual failure modes.

The soft spots are real but not fatal. The biggest one is evidentiary: the paper's central claims—that the scaffolded tier dominates all others on accuracy and calibration, that it recovers ground truth in nine of ten ACIC datasets—rest on figures without the underlying numbers, error bars, or per-dataset breakdowns. With n=10 datasets and n=20 bootstrap replications, "dominates all layers on both axes" is not established without some sense of spread. The single reported LaLonde median APE numbers (52% PSID, 19% CPS) are useful but do not tell me whether the next-best tier is close or far. The ACIC 2018 HTE evaluation is compromised by the missing school random effects; the authors admit this, but it means that particular benchmark mostly shows the scaffold does not collapse, not that it excels. The ATE/ATT tension in Section 5.2.2 is less serious than the stress-test note suggests: the footnote actually explains that Figure 4 deliberately evaluates ATE because ATT is largely identifiable, so the caption is inconsistent but the reasoning is clear. Minor wording fix, not a flaw.

I disagree with the harshest reading of the stress-test note. The paper does not overclaim the statistics—it says the scaffold "recovers" ground truth in nine of ten datasets, not that it does so with formal significance. The leakage test is a good-faith attempt to rule out memorization, and the open-source repository with shipped DGPs is a real asset. The methods are standard, but the paper is not pretending otherwise; the contribution is the workflow and its evaluation, and that is legitimate.

Who is this for? Applied data scientists building or evaluating agentic analysis tools, and researchers working on LLM scaffolding for tabular tasks. It is not a methodology paper. With the missing numbers and pinned repo hashes, it would be a reasonable, citable engineering contribution. I would accept it for peer review, with the clear request that the authors publish per-dataset results, error bars, and a reproducible pipeline.

Recommendation: send to review. The central claim is plausible and the open-source artifacts make it checkable. Ask for the numeric tables and the ATE/ATT caption fix, then it is a solid paper.

Referee Report

5 major / 5 minor

Summary. The paper introduces oci-agent, an open-source, human-in-the-loop agentic workflow for observational causal inference under unconfoundedness. The workflow combines templated analysis notebooks, deterministic double/debiased machine learning tools, and an actor-critic agent loop that produces inspectable intermediate artifacts for human oversight. The main empirical claim is that the full scaffolded workflow outperforms less structured LLM baselines (prompt-only, prompt+data, prompt+code, prompt+code+DML) on accuracy and calibration across three tasks: binary-treatment ATE/ATT estimation, heterogeneous treatment effects, and multiple continuous treatments via partially linear models. Evidence is drawn from ACIC 2016, the LaLonde study with a leakage test, ACIC 2018 HTE data, and custom DGPs, plus internal Netflix case studies. The paper argues that scaffolding and deterministic tools, rather than the LLM alone, are what make reliable OCI estimates possible.

Significance. If the central claim is substantiated, the paper is practically significant: it demonstrates a concrete, deployed pattern for making frontier LLMs reliable for a specialized, judgment-heavy statistical task, and it ships an open-source package, reproducible DGPs, and a leakage-testing protocol. The emphasis on inspectability and human oversight is timely. However, the evidence as presented is largely qualitative: most key figures are described verbally or show point estimates without numeric backing, error bars, or statistical comparisons. Given the small sample sizes (10 datasets in the main ablation, 20 replications in LaLonde), the strong dominance claims are not currently backed by the reported evidence. This is an evidence-weight problem rather than a fundamental flaw in the methodology, and it should be fixable with additional reporting and more careful claim-calibration.

major comments (5)

[§5.2.3, Fig. 5 (also Figs. 3, 4, 7, 9, 10, 12)] The central empirical claim—that the scaffolded workflow 'dominates all layers on both axes'—is not supported by the evidence as reported. The paper provides only plotted point estimates; no per-dataset or per-replication numeric values, standard deviations, confidence intervals, or significance tests are given. With n=10 ACIC datasets, the difference between the scaffolded arm and prompt+code+DML may easily fall within sampling noise. The paper should include full numeric tables, identify the one dataset in which the scaffold fails to recover ground truth, and provide uncertainty quantification (e.g., paired bootstrap, sign test) for all dominance claims.
[§5.2.4, Fig. 7] The LaLonde ablation is the strongest evidence for the workflow, but the manuscript reports only median APEs (52% on PSID, 19% on CPS). The distribution of APE over the 20 bootstrap replications, the injected shift schedule, and the performance of the next-best tier are omitted. As a result, the statement that 'only the fully scaffolded oci-agent achieves reasonable accuracy' cannot be evaluated—the apparent advantage over prompt+code+DML could be driven by a few replications. Please report per-replication data and a formal comparison between the scaffolded arm and the most competitive baseline.
[§6.2, Figs. 9–10] The HTE evaluations are presented only graphically. For ACIC 2018, the authors acknowledge that the synthetic school random effects could not be reproduced, yet the text claims coverage of the CATE for every level of C1 without reporting numeric coverage or estimates. For the custom within-group-contrast DGP, the per-arm RMSE and interval-score values are not tabulated. Without these numbers, the conclusion that the scaffolded agent 'recovers the ground truth' in the HTE setting is unverifiable. Provide numeric CATE estimates, coverage rates, and uncertainty measures for each C1 level and each arm.
[§5.2.4, Fig. 6] The leakage test is a thoughtful guard against memorization, but the evidence is a single OLS slope ('≈0.8') with no standard error, confidence interval, or scatterplot data. Since this is the only direct evidence that the model is not reciting the published $1,794 value, please report the full regression output (slope, intercept, SE, CI, R²) and specify the shift protocol. Also clarify whether the slope is computed across all 20 replications or within groups.
[§5.2.2 and §8] The paper claims that the critic's diagnostics 'consistently' separate reliable from unreliable estimates based on Figure 4. This is a strong operating-characteristic claim. With 231 datasets split into satisfactory (192) and unsatisfactory (39), the RMSE and coverage values for the three groups should be tabulated, and a formal test of separation (e.g., a permutation test or a comparison of confidence intervals) should be reported. Without this, the statement that the diagnostic suite 'consistently' distinguishes reliable estimates is not quantified.

minor comments (5)

[§5.2.2, footnote 4] The footnote is confusing: it says the ATT is largely identifiable while the ATE is not, and then says the evaluation focuses on the ATE. Clarify that the ATE is the deliberately harder target and that the diagnostics are expected to fail more often on it.
[§5.1] The main text says the agent trims units with propensity scores outside [0.1, 0.9], but Figure 2 evaluates stability over [0.005, 0.995]. Clarify whether the default trimming range is [0.1, 0.9] and Figure 2 is a sensitivity analysis over a wider set of thresholds.
[§7.1] Typo: 'otucome' should be 'outcome'.
[Figures 5, 7, 9, 10, 12] Several figures lack axis labels or unit definitions (e.g., Figure 5 top panel says 'ground truth' but does not specify the estimand or scale). Add captions and legends that make each panel interpretable without referring to the main text.
[§5.2.4] The note that the critic withholds certification of the PSID task in all but one replication is important; it should be in the main text, not only as an aside, since it qualifies the 'reasonable accuracy' claim for the harder LaLonde comparison.

Circularity Check

0 steps flagged

No meaningful circularity; evaluations rest on external benchmarks and a direct leakage test; minor self-citations are not load-bearing.

full rationale

The paper's central claim is that the scaffolded oci-agent workflow outperforms less structured baselines, and this is tested empirically against external ground truth: ACIC 2016 synthetic DGPs, the LaLonde experimental benchmark, and custom DGPs shipped in the public repository. The AIPW and DML estimators are standard external methods (refs 4, 7, 19). The leakage test directly addresses memorization of the LaLonde result by injecting shifts into the true effect and testing correlation with estimates; this is a genuine falsification test that does not reduce to the paper's own definitions. The critic's satisfactory/unsatisfactory grading rests on pre-specified design diagnostics (e.g., standardized mean difference, propensity overlap) from external literature (refs 2, 9), not on the outcome, so the demonstrated separation in RMSE/coverage is a validation, not a definitional artifact. Self-citations (refs 8, 17, 22) are present but only point to blog posts or prior implementation notes for standard methodology; no load-bearing argument in the evaluation chain reduces to a self-citation. The reported limitations—missing per-dataset numeric tables, no error bars or significance tests—are an evidence-reporting weakness, not circularity. Consequently, the analysis is self-contained against external benchmarks and no step fits the patterns of circular reasoning.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The paper does not introduce new statistical estimands or entities; it assembles known methods into a workflow. The main assumptions are standard causal identification and the transferability of benchmarks. The key burden is the absence of underlying numeric data and error bars for the crucial figures, not the introduction of an ad hoc entity.

free parameters (2)

Overlap trimming threshold = 0.005 and 0.995 in Figure 2
The workflow trims observations with propensity scores outside pre-specified bounds, and Figure 2 shows stability after trimming to [0.005, 0.995]. The default range [0.1, 0.9] is a choice from Crump et al., but the sensitivity range used for the headline statement is a judgment call in the paper.
Balance threshold = SMD < 0.2
The covariate balance diagnostic uses a fixed 0.2 standardized mean difference threshold from Austin and Stuart.

axioms (5)

domain assumption Unconfoundedness holds for the observed covariates.
This is the standard identification assumption for OCI, stated in Sections 1 and 5.
domain assumption The target trial emulation framework applies to the Netflix case studies.
The paper assumes that posing a target trial and measuring a retention outcome two months after an exposure window is a valid design for the internal analyses, without external validation.
domain assumption The ACIC 2016 and ACIC 2018 data generating processes are representative of real OCI tasks.
The evaluations rely on synthetic data that may not capture the full difficulty of real-world confounding and overlap.
domain assumption The LaLonde experimental estimate of $1,794 is a valid ground truth.
The disguised-LaLonde evaluation scores against the experimental estimate, implicitly assuming the experimental setting transfers to the observational comparison groups.
domain assumption The LLM's output is stochastic across runs.
The ten ACIC datasets and twenty bootstrap replications are small samples, and no seeds are provided to assess run-to-run variance.

pith-pipeline@v1.3.0-alltime-deepseek · 10215 in / 6890 out tokens · 52999 ms · 2026-08-01T04:41:26.834348+00:00 · methodology

0 comments

read the original abstract

Data analysis agents are becoming increasingly common tools for applied and scientific research. Yet, for highly specialized tasks such as Observational Causal Inference (OCI), human oversight remains necessary to ensure the validity of results. We introduce `oci-agent`, an open-source Python package that implements a human-in-the-loop agentic workflow for observational causal inference. `oci-agent` is designed to automate vital but laborious aspects of applied causal inference, such as covariate balance checking, propensity score trimming, and sensitivity analysis, so that humans can focus on more nuanced tasks, such as framing questions, scrutinizing assumptions, and evaluating diagnostics and results. We initially open-sourced `oci-agent` in June 2026 with support for doubly robust learning of the average treatment effect of a single binary treatment. Since then, we have added support for heterogeneous treatment effect estimation and for multiple continuous treatments via partially linear models. In this paper, we describe the principles behind `oci-agent` and offer internal Netflix case studies and evaluations on public data of its capabilities. Across numerous evaluations, `oci-agent` outperforms less structured baselines while remaining competitive with hand-tuned benchmarks. `oci-agent` is used extensively for causal inference at Netflix and has orchestrated more than 100 analyses per month since its release in June.

Figures

Figures reproduced from arXiv: 2607.22443 by Adrien Alexandre, Lars Olds, Nathan Kallus, Winston Chou, Yi Zhang.

**Figure 1.** Figure 1: The oci-agent workflow. AIPW score ˆ𝜙𝑖 = 𝜇ˆ1 (𝑋𝑖) −𝜇ˆ0 (𝑋𝑖) + 𝑇𝑖(𝑌𝑖 − 𝜇ˆ1 (𝑋𝑖)) 𝑒ˆ(𝑋𝑖) − (1 −𝑇𝑖) (𝑌𝑖 − 𝜇ˆ0 (𝑋𝑖)) 1 − 𝑒ˆ(𝑋𝑖) (1) and reports 𝜏ˆ = 𝑛 −1 Í 𝑖 ˆ𝜙𝑖 with uncertainty estimates. Under the standard unconfoundedness assumptions, this estimator is doubly robust for the ATE, meaning that it is consistent when either 𝜇𝑡 or 𝑒 are consistent. When the diagnostics imply insufficient overlap, the workflow t… view at source ↗

**Figure 2.** Figure 2: Estimated ATE on the overlapping population versus trimming [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Figure 3: RMSE and 95% CI coverage of oci-agent’s ATT estimates versus the 44 ACIC 2016 competitor methods. Data Competition [11]. Since then, we have also added evaluations on non-synthetic data based on the classic LaLonde study [18]. 5.2.1 Accuracy against ACIC 2016 benchmarks. Our first evaluation runs our binary treatment notebook on 231 datasets generated by randomly sampling three datasets for each of the 77 … view at source ↗

**Figure 6.** Figure 6: Leakage test on the disguised LaLonde data ( [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗

**Figure 5.** Figure 5: Capability-ladder ablation on ten ACIC 2016 datasets. Top: [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗

**Figure 8.** Figure 8: Internal HTE case study (anonymized): cumulative treatment [PITH_FULL_IMAGE:figures/full_fig_p006_8.png] view at source ↗

**Figure 7.** Figure 7: Disguised-LaLonde ablation: ATT absolute % error versus the [PITH_FULL_IMAGE:figures/full_fig_p006_7.png] view at source ↗

**Figure 9.** Figure 9: HTE on the ACIC 2018 data (best vs. worst). The near-zero, [PITH_FULL_IMAGE:figures/full_fig_p007_9.png] view at source ↗

**Figure 11.** Figure 11: Internal PLM case study (anonymized): treatment coefficients [PITH_FULL_IMAGE:figures/full_fig_p008_11.png] view at source ↗

**Figure 12.** Figure 12: contrasts pure prompting versus our scaffolding on 20 estimates of 𝜃 from 11 datasets generated from this DGP. As the top panel shows, the model requires scaffolding to produce reliable results; without scaffolding, it produces estimates that are wholly uncorrelated with ground truth. The bottom panel confirms that the scaffolded agent achieves the best accuracy and interval score across the full capabili… view at source ↗

discussion (0)

Reference graph

Works this paper leans on

26 extracted references · 3 canonical work pages · 1 internal anchor

[1]

2026.How Anthropic Enables Self-Service Data Analytics with Claude

Anthropic. 2026.How Anthropic Enables Self-Service Data Analytics with Claude. Retrieved July 22, 2026 from https://claude.com/blog/how-anthropic-enables- self-service-data-analytics-with-claude Published June 3, 2026

2026
[2]

Austin and Elizabeth A

Peter C. Austin and Elizabeth A. Stuart. 2015. Moving towards Best Practice When Using Inverse Probability of Treatment Weighting (IPTW) Using the Propensity Score to Estimate Causal Treatment Effects in Observational Studies.Statistics in Medicine34, 28 (2015), 3661–3679. doi:10.1002/sim.6607

doi:10.1002/sim.6607 2015
[3]

Kurz, and Martin Spindler

Philipp Bach, Victor Chernozhukov, Malte S. Kurz, and Martin Spindler. 2022. DoubleML: An Object-Oriented Implementation of Double Machine Learning in Python.Journal of Machine Learning Research23, 53 (2022), 1–6. https: //www.jmlr.org/papers/v23/21-0862.html

2022
[4]

Heejung Bang and James M. Robins. 2005. Doubly Robust Estimation in Missing Data and Causal Inference Models.Biometrics61, 4 (2005), 962–973. doi:10.1111/ j.1541-0420.2005.00377.x

arXiv 2005
[5]

2019.EconML: A Python Package for ML-Based Heterogeneous Treatment Effects Estimation

Keith Battocchi, Eleanor Dillon, Maggie Hei, Greg Lewis, Paul Oka, Miruna Oprescu, and Vasilis Syrgkanis. 2019.EconML: A Python Package for ML-Based Heterogeneous Treatment Effects Estimation. Retrieved July 22, 2026 from https: //github.com/py-why/EconML

2019
[6]

Carlos Carvalho, Avi Feller, Jared Murray, Spencer Woody, and David Yeager
[7]

Newey, and James M

Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Chris- tian Hansen, Whitney K. Newey, and James M. Robins. 2018. Double/Debiased Machine Learning for Treatment and Structural Parameters.The Econometrics Journal21, 1 (2018), C1–C68. doi:10.1111/ectj.12097

doi:10.1111/ectj.12097 2018
[8]

2026.A Human-Augmenting Agentic Workflow for Causal Inference

Winston Chou, Adrien Alexandre, Lars Olds, Yi Zhang, Garrett Hagemann, and Nathan Kallus. 2026.A Human-Augmenting Agentic Workflow for Causal Inference. Retrieved July 22, 2026 from https://netflixtechblog.com/a-human-augmenting- agentic-workflow-for-causal-inference-4623f0a9c5af Published June 8, 2026

2026
[9]

Crump, V

Richard K. Crump, V. Joseph Hotz, Guido W. Imbens, and Oscar A. Mitnik
[10]

Dehejia and Sadek Wahba

Rajeev H. Dehejia and Sadek Wahba. 2002. Propensity Score-Matching Methods for Nonexperimental Causal Studies.The Review of Economics and Statistics84, 1 (2002), 151–161. doi:10.1162/003465302317331982

work page doi:10.1162/003465302317331982 2002
[11]

Vincent Dorie, Jennifer Hill, Uri Shalit, Marc Scott, and Dan Cervone. 2019. Automated versus Do-It-Yourself Methods for Causal Inference: Lessons Learned from a Data Analysis Competition.Statist. Sci.34, 1 (2019), 43–68. doi:10.1214/18- STS667

doi:10.1214/18- 2019
[12]

Walzner, and Alok Gupta

Andreas Fügener, Dominik D. Walzner, and Alok Gupta. 2025. Roles of Artificial Intelligence in Collaboration with Humans: Automation, Augmentation, and the Future of Work.Management Science72, 1 (2025), 538–557. doi:10.1287/mnsc. 2024.05684 Published online October 15, 2025; assigned to the January 2026 issue

arXiv 2025
[13]

Tilmann Gneiting and Adrian E. Raftery. 2007. Strictly Proper Scoring Rules, Prediction, and Estimation.J. Amer. Statist. Assoc.102, 477 (2007), 359–378. doi:10.1198/016214506000001437

doi:10.1198/016214506000001437 2007
[14]

Hernán and James M

Miguel A. Hernán and James M. Robins. 2016. Using Big Data to Emulate a Target Trial When a Randomized Trial Is Not Available.American Journal of Epidemiology183, 8 (2016), 758–764. doi:10.1093/aje/kwv254

doi:10.1093/aje/kwv254 2016
[15]

Paul W. Holland. 1986. Statistics and Causal Inference.J. Amer. Statist. Assoc.81, 396 (1986), 945–960. doi:10.1080/01621459.1986.10478354

arXiv 1986
[16]

Guido W. Imbens. 2004. Nonparametric Estimation of Average Treatment Effects under Exogeneity: A Review.The Review of Economics and Statistics86, 1 (2004), 4–29. doi:10.1162/003465304323023651

work page doi:10.1162/003465304323023651 2004
[17]

Apoorva Lal and Winston Chou. 2026. Estimating Representative Causal Effects with Double Machine Learning. arXiv:2506.07462 [econ.EM] doi:10.48550/arXiv. 2506.07462 Version 4, revised June 19, 2026; to appear in the proceedings of ECML PKDD 2026

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2506.07462 2026
[18]

Robert J. LaLonde. 1986. Evaluating the Econometric Evaluations of Training Programs with Experimental Data.American Economic Review76, 4 (1986), 604–620. https://www.jstor.org/stable/1806062

arXiv 1986
[19]

Robins, Andrea Rotnitzky, and Lue Ping Zhao

James M. Robins, Andrea Rotnitzky, and Lue Ping Zhao. 1994. Estimation of Regression Coefficients When Some Regressors Are Not Always Observed.J. Amer. Statist. Assoc.89, 427 (1994), 846–866. doi:10.1080/01621459.1994.10476818

arXiv 1994
[20]

Rosenbaum and Donald B

Paul R. Rosenbaum and Donald B. Rubin. 1983. The Central Role of the Propensity Score in Observational Studies for Causal Effects.Biometrika70, 1 (1983), 41–55. doi:10.1093/biomet/70.1.41

doi:10.1093/biomet/70.1.41 1983
[21]

Donald B. Rubin. 1974. Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies.Journal of Educational Psychology66, 5 (1974), 688–701. doi:10.1037/h0037350

doi:10.1037/h0037350 1974
[22]

2025.Heterogeneous Treatment Effects at Netflix

Scott Seyfarth, Adrien Alexandre, Carlos Velasco Rivera, Colin Gray, Tomoya Sasaki, Shusei Eshima, Sambhav Jain, Simon Ejdemyr, Mihir Tendulkar, Matthew Wardrop, and Winston Chou. 2025.Heterogeneous Treatment Effects at Netflix. Retrieved July 22, 2026 from https://netflixtechblog.medium.com/heterogeneous- treatment-effects-at-netflix-da5c3dd58833 Publish...

2025
[23]

Robert L. Winkler. 1972. A Decision-Theoretic Approach to Interval Estimation.J. Amer. Statist. Assoc.67, 337 (1972), 187–191. doi:10.1080/01621459.1972.10481224

arXiv 1972
[24]

2026.Inside OpenAI’s In-House Data Agent

Bonnie Xu, Aravind Suresh, and Emma Tang. 2026.Inside OpenAI’s In-House Data Agent. Retrieved July 22, 2026 from https://openai.com/index/inside-our- in-house-data-agent/ Published January 29, 2026

2026
[2009]

doi:10.1093/biomet/asn055

Dealing with Limited Overlap in Estimation of Average Treatment Ef- fects.Biometrika96, 1 (2009), 187–199. doi:10.1093/biomet/asn055

doi:10.1093/biomet/asn055 2009
[2019]

doi:10.1353/obs

Assessing Treatment Effect Variation in Observational Studies: Results from a Data Challenge.Observational Studies5, 2 (2019), 21–35. doi:10.1353/obs. 2019.0000

arXiv 2019

[1] [1]

2026.How Anthropic Enables Self-Service Data Analytics with Claude

Anthropic. 2026.How Anthropic Enables Self-Service Data Analytics with Claude. Retrieved July 22, 2026 from https://claude.com/blog/how-anthropic-enables- self-service-data-analytics-with-claude Published June 3, 2026

2026

[2] [2]

Austin and Elizabeth A

Peter C. Austin and Elizabeth A. Stuart. 2015. Moving towards Best Practice When Using Inverse Probability of Treatment Weighting (IPTW) Using the Propensity Score to Estimate Causal Treatment Effects in Observational Studies.Statistics in Medicine34, 28 (2015), 3661–3679. doi:10.1002/sim.6607

doi:10.1002/sim.6607 2015

[3] [3]

Kurz, and Martin Spindler

Philipp Bach, Victor Chernozhukov, Malte S. Kurz, and Martin Spindler. 2022. DoubleML: An Object-Oriented Implementation of Double Machine Learning in Python.Journal of Machine Learning Research23, 53 (2022), 1–6. https: //www.jmlr.org/papers/v23/21-0862.html

2022

[4] [4]

Heejung Bang and James M. Robins. 2005. Doubly Robust Estimation in Missing Data and Causal Inference Models.Biometrics61, 4 (2005), 962–973. doi:10.1111/ j.1541-0420.2005.00377.x

arXiv 2005

[5] [5]

2019.EconML: A Python Package for ML-Based Heterogeneous Treatment Effects Estimation

Keith Battocchi, Eleanor Dillon, Maggie Hei, Greg Lewis, Paul Oka, Miruna Oprescu, and Vasilis Syrgkanis. 2019.EconML: A Python Package for ML-Based Heterogeneous Treatment Effects Estimation. Retrieved July 22, 2026 from https: //github.com/py-why/EconML

2019

[6] [6]

Carlos Carvalho, Avi Feller, Jared Murray, Spencer Woody, and David Yeager

[7] [7]

Newey, and James M

Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Chris- tian Hansen, Whitney K. Newey, and James M. Robins. 2018. Double/Debiased Machine Learning for Treatment and Structural Parameters.The Econometrics Journal21, 1 (2018), C1–C68. doi:10.1111/ectj.12097

doi:10.1111/ectj.12097 2018

[8] [8]

2026.A Human-Augmenting Agentic Workflow for Causal Inference

Winston Chou, Adrien Alexandre, Lars Olds, Yi Zhang, Garrett Hagemann, and Nathan Kallus. 2026.A Human-Augmenting Agentic Workflow for Causal Inference. Retrieved July 22, 2026 from https://netflixtechblog.com/a-human-augmenting- agentic-workflow-for-causal-inference-4623f0a9c5af Published June 8, 2026

2026

[9] [9]

Crump, V

Richard K. Crump, V. Joseph Hotz, Guido W. Imbens, and Oscar A. Mitnik

[10] [10]

Dehejia and Sadek Wahba

Rajeev H. Dehejia and Sadek Wahba. 2002. Propensity Score-Matching Methods for Nonexperimental Causal Studies.The Review of Economics and Statistics84, 1 (2002), 151–161. doi:10.1162/003465302317331982

work page doi:10.1162/003465302317331982 2002

[11] [11]

Vincent Dorie, Jennifer Hill, Uri Shalit, Marc Scott, and Dan Cervone. 2019. Automated versus Do-It-Yourself Methods for Causal Inference: Lessons Learned from a Data Analysis Competition.Statist. Sci.34, 1 (2019), 43–68. doi:10.1214/18- STS667

doi:10.1214/18- 2019

[12] [12]

Walzner, and Alok Gupta

Andreas Fügener, Dominik D. Walzner, and Alok Gupta. 2025. Roles of Artificial Intelligence in Collaboration with Humans: Automation, Augmentation, and the Future of Work.Management Science72, 1 (2025), 538–557. doi:10.1287/mnsc. 2024.05684 Published online October 15, 2025; assigned to the January 2026 issue

arXiv 2025

[13] [13]

Tilmann Gneiting and Adrian E. Raftery. 2007. Strictly Proper Scoring Rules, Prediction, and Estimation.J. Amer. Statist. Assoc.102, 477 (2007), 359–378. doi:10.1198/016214506000001437

doi:10.1198/016214506000001437 2007

[14] [14]

Hernán and James M

Miguel A. Hernán and James M. Robins. 2016. Using Big Data to Emulate a Target Trial When a Randomized Trial Is Not Available.American Journal of Epidemiology183, 8 (2016), 758–764. doi:10.1093/aje/kwv254

doi:10.1093/aje/kwv254 2016

[15] [15]

Paul W. Holland. 1986. Statistics and Causal Inference.J. Amer. Statist. Assoc.81, 396 (1986), 945–960. doi:10.1080/01621459.1986.10478354

arXiv 1986

[16] [16]

Guido W. Imbens. 2004. Nonparametric Estimation of Average Treatment Effects under Exogeneity: A Review.The Review of Economics and Statistics86, 1 (2004), 4–29. doi:10.1162/003465304323023651

work page doi:10.1162/003465304323023651 2004

[17] [17]

Apoorva Lal and Winston Chou. 2026. Estimating Representative Causal Effects with Double Machine Learning. arXiv:2506.07462 [econ.EM] doi:10.48550/arXiv. 2506.07462 Version 4, revised June 19, 2026; to appear in the proceedings of ECML PKDD 2026

work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2506.07462 2026

[18] [18]

Robert J. LaLonde. 1986. Evaluating the Econometric Evaluations of Training Programs with Experimental Data.American Economic Review76, 4 (1986), 604–620. https://www.jstor.org/stable/1806062

arXiv 1986

[19] [19]

Robins, Andrea Rotnitzky, and Lue Ping Zhao

James M. Robins, Andrea Rotnitzky, and Lue Ping Zhao. 1994. Estimation of Regression Coefficients When Some Regressors Are Not Always Observed.J. Amer. Statist. Assoc.89, 427 (1994), 846–866. doi:10.1080/01621459.1994.10476818

arXiv 1994

[20] [20]

Rosenbaum and Donald B

Paul R. Rosenbaum and Donald B. Rubin. 1983. The Central Role of the Propensity Score in Observational Studies for Causal Effects.Biometrika70, 1 (1983), 41–55. doi:10.1093/biomet/70.1.41

doi:10.1093/biomet/70.1.41 1983

[21] [21]

Donald B. Rubin. 1974. Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies.Journal of Educational Psychology66, 5 (1974), 688–701. doi:10.1037/h0037350

doi:10.1037/h0037350 1974

[22] [22]

2025.Heterogeneous Treatment Effects at Netflix

Scott Seyfarth, Adrien Alexandre, Carlos Velasco Rivera, Colin Gray, Tomoya Sasaki, Shusei Eshima, Sambhav Jain, Simon Ejdemyr, Mihir Tendulkar, Matthew Wardrop, and Winston Chou. 2025.Heterogeneous Treatment Effects at Netflix. Retrieved July 22, 2026 from https://netflixtechblog.medium.com/heterogeneous- treatment-effects-at-netflix-da5c3dd58833 Publish...

2025

[23] [23]

Robert L. Winkler. 1972. A Decision-Theoretic Approach to Interval Estimation.J. Amer. Statist. Assoc.67, 337 (1972), 187–191. doi:10.1080/01621459.1972.10481224

arXiv 1972

[24] [24]

2026.Inside OpenAI’s In-House Data Agent

Bonnie Xu, Aravind Suresh, and Emma Tang. 2026.Inside OpenAI’s In-House Data Agent. Retrieved July 22, 2026 from https://openai.com/index/inside-our- in-house-data-agent/ Published January 29, 2026

2026

[25] [2009]

doi:10.1093/biomet/asn055

Dealing with Limited Overlap in Estimation of Average Treatment Ef- fects.Biometrika96, 1 (2009), 187–199. doi:10.1093/biomet/asn055

doi:10.1093/biomet/asn055 2009

[26] [2019]

doi:10.1353/obs

Assessing Treatment Effect Variation in Observational Studies: Results from a Data Challenge.Observational Studies5, 2 (2019), 21–35. doi:10.1353/obs. 2019.0000

arXiv 2019