Data Fusion for Partial Identification of Causal Effects

Alexander Volfovsky; Cynthia Rudin; Harsh Parikh; Quinn Lanners

REVIEW 2 major objections 4 minor 1 cited by

Data Fusion for Partial Identification of Causal Effects

T0 review · 2 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proves that causal effects remain bounded even when both unmeasured confounding and study-exchangeability assumptions fail, and supplies estimators that are root-n normal.

desk verdict A useful and novel sensitivity framework for simultaneous NUC and exchangeability violations, but the main theorem is missing a dominance condition and needs a fix before the bounds are valid. read the letter →

arxiv 2505.24296 v1 pith:OZJCIKS6 submitted 2025-05-30 stat.ME cs.LGecon.EM

Quinn Lanners , Cynthia Rudin , Alexander Volfovsky , Harsh Parikh This is my paper

classification stat.MEcs.LGecon.EM

keywords datafusionpartialidentificationsensitivityanalysiscausalinferencedoublemachinelearningunobservedconfoundingstudyexchangeabilitybreakdownfrontier

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Data fusion usually needs one of two assumptions to hold: the observational data are unconfounded, or the experimental and observational study populations are exchangeable. This paper tackles the common case where both fail. It introduces two interpretable sensitivity parameters, ρ for unobserved confounding in the observational arm and γ for study-exchangeability violations, and proves that, for any compatible pair (ρ, γ), the true average treatment effect lies between bounds built from observed data. It then supplies bias-corrected, root-n-normal estimators for those bounds using double machine learning, and demonstrates them on Project STAR, where the positive effect of small classes survives substantial simultaneous violations of both assumptions. If correct, the approach turns the question 'is the effect positive?' into a map of how much assumption violation would be needed to overturn the answer.

What carries the argument

Two estimable bounding functions carry the argument. $v(x,t,\gamma)=(1+\gamma)\mu(x,1,t)$ uses the randomized arm's observed outcome mean and γ to bound the observational arm's potential outcome; $w(x,t,\rho)=e_t(x,0)\mu(x,0,t)+(1-e_t(x,0))(1+\rho)\mu(x,0,t)$ uses the observational arm's propensity score, outcome mean, and ρ. Lemma 1 takes the maximum of the two lower bounds and the minimum of the two upper bounds, producing sharp conditional bounds whenever the two intervals overlap, which is exactly the compatibility condition. To make these nondifferentiable max/min bounds estimable, the paper replaces them with a Boltzmann operator $\lambda_1 v+\lambda_2 w$ with softmax weights parameterized by α; this yields smooth bounds whose efficient influence function can be derived, and cross-fitting with estimated study-selection, propensity, and outcome-regression nuisance functions gives doubly robust, $\sqrt{n}$-normal estimators.

What would settle it

Simulate a known data-generating process with a binary covariate subgroup small enough to be invisible in the average compatibility test but with the pointwise condition violated for that subgroup; compute the averaged bounds and check whether they contain the subgroup's true conditional ATE. If they exclude it, the claim that Theorem 1's conditions deliver coverage for every supported x fails.

Watch

Extended reading notes

Core claim

The central claim is Theorem 1: under treatment positivity, study positivity, and internal validity of the experiment, for any sensitivity parameters ρ ≥ 0 and γ ≥ 0 satisfying the two compatibility inequalities pointwise, the average treatment effect satisfies $E[l(X,1,\rho,\gamma)-u(X,0,\rho,\gamma)] \le \tau \le E[u(X,1,\rho,\gamma)-l(X,0,\rho,\gamma)]$, where $l$ and $u$ are the lower and upper bounds constructed in Lemma 1. Lemma 1 bounds the unobservable conditional potential outcome $E[Y(t)|X=x,S=0]$ by intersecting an RCT-derived interval $[(1-\gamma)\mu(x,1,t),(1+\gamma)\mu(x,1,t)]$ with an observational-data interval $[w(x,t,-\rho),w(x,t,\rho)]$. Theorem 2 states that, for fixed smoothing parameter α, the cross-fitted bias-corrected estimators of the smoothed versions of these bounds are $\sqrt{n}$-consistent and asymptotically normal, with variance given by the efficient influence function. The paper also shows that the smoothed bounds converge to the sharp max/min bounds as α → ∞.

Load-bearing premise

The load-bearing premise is that the compatibility inequalities $v(x,t,-\gamma) \le w(x,t,\rho)$ and $w(x,t,-\rho) \le v(x,t,\gamma)$ hold for every covariate value x and treatment arm t under the true data-generating process; the paper tests them only on average over X and with a resampling test that ignores nuisance-parameter uncertainty.

Editorial extensions

If this is right

For any (ρ, γ) pair that passes the compatibility check, a researcher can report a confidence interval for the ATE and read off whether the sign is conclusive, tentative, or inconclusive.
The breakdown frontier divides the (ρ, γ) plane into compatible and incompatible regions, so the same plot warns when the assumed violations are too small to explain observed differences between the experimental and observational studies.
Because nuisance functions can be estimated by flexible machine learning at $n^{-1/4}$ rates, the bounds remain asymptotically normal without parametric outcome models.
Applied to Project STAR, the analysis says that study-exchangeability violations under about 5% do not overturn the positive class-size effect even with substantial unobserved confounding, and that older kindergarten entrants show a more robust effect.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper, the pointwise compatibility condition is stronger than the average test that the paper implements; violations concentrated in a subgroup could make the reported ATE bounds fail to cover that subgroup's true conditional effect even when the average test passes.
Beyond the paper, a full bootstrap incompatibility test that re-estimates nuisance functions in every resample would likely classify fewer (ρ, γ) pairs as incompatible, so the published incompatible region is probably wider than a fully uncertainty-aware procedure would produce.
Beyond the paper, the frontier boundary can be read as a quantitative 'least required violation' index—a number policy analysts could compare across studies—though the paper does not develop that metric.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The headline is that this paper fills a real gap — providing sensitivity bounds when both unobserved confounding and study exchangeability fail — but its main theorem is not true as stated. That's fixable, but it's load-bearing.

The novelty is genuine. Yu et al. (2024) assumes study exchangeability; Parikh et al. (2023) requires you to know which assumption fails. This paper gives two interpretable parameters and derives bounds using a mix of experimental and observational data. The double machine learning estimator based on the efficient influence function is a serious piece of work, and the breakdown frontier visualization is a practical tool. The Project STAR analysis is relevant and mostly well-executed.

The problem is in Lemma 1 and Theorem 1. They claim coverage of the true ATE whenever the compatibility inequalities hold. The compatibility conditions only ensure the two intervals derived from rho and gamma intersect. They don't ensure the true conditional mean is inside that intersection. To get coverage you also need rho and gamma to be at least the true sup deviations, as defined in Section 4. The proof itself uses those sup definitions, so the lemma is missing an explicit dominance assumption. The stress-test counterexample is correct: with mu=10, e_t=0.5, and true counterfactual 0.1, the true rho is 0.99; choosing rho=0.1 and gamma=0.6 passes the compatibility inequalities, but the lower bound exceeds the true outcome. As stated, the theorem is false.

This is substantial but repairable. The framework works if the bounds are restated as valid for user-chosen (rho, gamma) that dominate the true pointwise violations. That's not cosmetic; it changes how practitioners should interpret the bounds.

Other issues are minor by comparison. The claim that shifting outcomes is without loss of generality is wrong, since rho and gamma are relative to the outcome scale. The incompatibility test is an average test that ignores nuisance uncertainty; the authors admit it is conservative. The simulations would benefit from baseline comparisons. None of these undermine the core idea.

Who should read this? Anyone working on sensitivity analysis for data fusion, and editors deciding whether to invest referee time. It deserves a serious referee — the gap is real and much of the machinery is sound — but the authors need to fix Theorem 1 before the bounds are used. I would not cite it in its current form.

Referee Report

2 major / 4 minor

Summary. The paper proposes a partial identification framework for the average and conditional average treatment effects when both no-unmeasured-confounding in the observational study and study exchangeability between the RCT and observational data fail. It introduces two interpretable sensitivity parameters, rho and gamma, derives bounds on potential outcome means conditional on covariates, smooths the max/min operators with a Boltzmann approximation, constructs efficient influence-function-based doubly robust estimators, and operationalizes the results as breakdown-frontier plots. The framework is applied to synthetic data and to Project STAR, where the authors report robustness of the positive class-size effect to simultaneous violations of both assumptions.

Significance. If the central identification results were correct, the paper would fill a real gap in the data-fusion sensitivity-analysis literature by allowing both study-exchangeability and no-unmeasured-confounding to fail simultaneously. The interpretability of rho and gamma as relative deviation bounds is attractive, and the breakdown-frontier visualization is useful and well-motivated. The paper is also unusually complete in its supplementary material: the EIF derivation in Appendix C is detailed, the algorithms are specified in full, and the empirical section is reproducible in principle. These strengths make the manuscript worth serious attention. However, the central theorems on which all downstream claims rest are, as stated, not correct, and the practical diagnostic for incompatibility is not aligned with the pointwise conditions that the theorems require.

major comments (2)

[Section 4, Lemma 1 and Theorem 1 (and Lemma 2)] The correction is local in the sense that the proof already contains the necessary bounding argument, but the theorem statements and the interpretation of the breakdown-frontier regions need to be changed.
[Appendix D.2 and Remark 1] This is a load-bearing issue for the empirical contribution, since the breakdown frontier plots in Section 6 are built on this test.

minor comments (4)

[Appendix B.1] There is a typo in the proof: 'We then define w(x, t, rho) = (1 + rho) mu(x, 0, t)(1 - et(x, 0))' should use e_{1-t}(x,0) rather than (1 - et(x,0)) for clarity, although they are equal for binary treatment.
[Appendix B.1] The phrase 'By the internal calidity of the RCT treatment randomization' contains a typo: 'calidity' should be 'validity'.
[Section 5.2 and Equation (2)] The displayed EIF in the main text and the algorithmic version in Equation (2) are both very complex; it would help the reader if the authors explicitly stated that the two forms are algebraically identical, and if they provided a short intuitive explanation of each term in the main text rather than only in the appendix.
[Section 4, definitions of rho and gamma] The definitions of rho and gamma as suprema over x and t sit uneasily with their subsequent treatment as user-chosen sensitivity parameters. If they are user-chosen, the theorems need the bounding assumption; if they are the true suprema, then the grid of values in the breakdown frontier is not a set of sensitivity parameters. The authors should clarify this distinction explicitly in Section 4.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: sensitivity parameters are user-specified, the bounds follow from their defining inequalities, and the only self-citation is non-load-bearing.

full rationale

The paper's central derivation is not circular. The sensitivity parameters rho and gamma are introduced in Section 4 as quantities defined through the very counterfactual deviations they are meant to bound; however, they are not fitted to the data and then relabeled as predictions. For a fixed user-selected (rho, gamma), the functions v, w, l, and u are explicit closed-form functions of observable conditional means and propensities, and Lemma 1's interval is obtained by rearranging the defining inequalities gamma >= |1 - E[Y(t)|S=0]/E[Y(t)|S=1]| and rho >= |1 - E[Y(t)|S=0,T=1-t]/E[Y(t)|S=0,T=t]|. This is a conditional sensitivity bound, not an empirical prediction. The doubly robust estimator in Section 5 is a standard one-step EIF construction whose asymptotic normality is imported from Kennedy (2024) and Chernozhukov et al. (2018), both external references. The only self-citation with overlapping authors is Parikh et al. (2023), used to motivate the incompatible region at (0,0); it is not load-bearing for the main bounds. The incompatibility test in Appendix D.2 is a diagnostic that reuses the same nuisance estimates, but the paper explicitly acknowledges that it ignores nuisance uncertainty and may over-label pairs as incompatible; that is a stated limitation, not a fitted input passed off as an independent result. One should note as a correctness issue (not circularity) that Theorem 1's compatibility conditions alone do not guarantee coverage for arbitrary user-chosen (rho, gamma) unless those parameters also dominate the true pointwise deviations; this is a gap in the theorem statement, not a circular derivation.

Assumptions & free parameters 3 free parameters · 6 assumptions · 0 invented entities

The framework rests on standard causal assumptions (A1-A3), positivity of outcomes, and user-specified sensitivity parameters. No new entities, forces, or conserved quantities are introduced. The main free choices are the sensitivity parameters and the smoothing parameter, which are the intended inputs of a sensitivity analysis.

free parameters (3)

rho (unmeasured confounding sensitivity parameter) = Not fitted; user-specified, explored over [0, 0.2] in experiments
Quantifies the relative deviation between counterfactual outcome means in the observational arm. The central bounds are indexed by rho, so the conclusions are conditional on the chosen value.
gamma (study exchangeability sensitivity parameter) = Not fitted; user-specified, explored over [0, 0.2] in experiments
Quantifies the relative deviation between potential outcome means across study arms. The central bounds are indexed by gamma.
alpha (Boltzmann smoothing parameter) = 10
Smooths the max/min operators to enable EIF derivation. Larger alpha approximates the sharp bounds better but increases estimator instability. The asymptotic theory is for fixed alpha.

assumptions (6)

domain assumption SUTVA and consistency
Stated in Section 2; needed to link observed outcomes to potential outcomes.
domain assumption A1: Treatment positivity
Stated in Section 2; ensures overlap in treatment probabilities within each study arm.
domain assumption A2: Study positivity
Stated in Section 2; ensures overlap between experimental and observational covariate distributions.
domain assumption A3: Internal validity of the RCT
Stated in Section 2; used to replace E[Y(t)|X,S=1] with the observed μ(x,1,t) in Equation (1).
domain assumption Outcome space is bounded and positive
Stated in Section 2; positivity is used in the proofs of Lemma 1 to multiply and divide by outcome means without sign changes. The paper's claim that a shift makes this without loss of generality is incorrect for the relative sensitivity parameters.
standard math Nuisance estimators converge at n^{-1/4} rates
Invoked in the proof of Theorem 2 (Appendix B.2) to control the second-order remainder term, following Chernozhukov et al. (2018) and Kennedy (2024).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Data Fusion for Partial Identification of Causal Effects." pith.science (2026). https://pith.science/paper/OZJCIKS6

@misc{pith2026250524296,
  author       = {Pith},
  title        = {Pith review of: Data Fusion for Partial Identification of Causal Effects},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OZJCIKS6}},
  note         = {Machine review of arXiv:2505.24296}
}

read the original abstract

Data fusion techniques integrate information from heterogeneous data sources to improve learning, generalization, and decision making across data sciences. In causal inference, these methods leverage rich observational data to improve causal effect estimation, while maintaining the trustworthiness of randomized controlled trials. Existing approaches often relax the strong no unobserved confounding assumption by instead assuming exchangeability of counterfactual outcomes across data sources. However, when both assumptions simultaneously fail - a common scenario in practice - current methods cannot identify or estimate causal effects. We address this limitation by proposing a novel partial identification framework that enables researchers to answer key questions such as: Is the causal effect positive or negative? and How severe must assumption violations be to overturn this conclusion? Our approach introduces interpretable sensitivity parameters that quantify assumption violations and derives corresponding causal effect bounds. We develop doubly robust estimators for these bounds and operationalize breakdown frontier analysis to understand how causal conclusions change as assumption violations increase. We apply our framework to the Project STAR study, which investigates the effect of classroom size on students' third-grade standardized test performance. Our analysis reveals that the Project STAR results are robust to simultaneous violations of key assumptions, both on average and across various subgroups of interest. This strengthens confidence in the study's conclusions despite potential unmeasured biases in the data.

Figures

Figures reproduced from arXiv: 2505.24296 by the authors.

**Figure 2.** Breakdown frontier plots for various synthetic datasets. Figure titles indicate the relation [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** Breakdown frontier plots for Project STAR (a) population ATE and (b) subgroup-specific [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Bounds on EP [Y (t) | X = x, S = 0] from [v(x, t, −0), v(x, t, 0)] and [w(x, t, −0), w(x, t, 0)] when µ(x, 1, t) ̸= µ(x, 0, t). Towards building up an understanding through a series of visuals, we depict this scenario in a toy example using [PITH_FULL_IMAGE:figures/fu…

**Figure 5.** Figure 5: Bounds on EP [Y (t) | X = x, S = 0] from [v(x, t, −γ), v(x, t, γ)] and [w(x, t, −0), w(x, t, 0)] when µ(x, 1, t) ̸= µ(x, 0, t) and γ is large enough that v(x, t, −γ) < w(x, t, 0). When γ and ρ are both greater than zero, they can still be incompatible if either v(x, t,…

**Figure 6.** Figure 6: Bounds on EP [Y (t) | X = x, S = 0] from [v(x, t, −γ), v(x, t, γ)] and [w(x, t, −ρ), w(x, t, ρ)] when µ(x, 1, t) ̸= µ(x, 0, t). In (a), ρ and γ are not large enough for the bounds to intersect. Whereas, in (b), ρ and γ are made large enough for the bounds to intersect.…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Partial identification via conditional linear programs: estimation and policy learning
stat.ME 2025-06 conditional novelty 7.0 of 10

Two debiased estimators, one based on linear programming solutions and one on entropic smoothing, provide asymptotic confidence intervals for covariate-dependent partial identification bounds and support policy learning.

Reference graph

Works this paper leans on

18 extracted references · 17 canonical work pages · cited by 1 Pith paper

[1]

We introduce interpretable sensitivity parameters to enable partial identification of treatment effects in Section 4

Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: Our contributions are clearly outlined at the end of our Introduction. We introduce interpretable sensitivity parameters to enable partial identification of treatment effects in Section 4. We procee...

work page
[2]

Limitations

Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We include a limitations and future work section in the Conclusion (Section 7). We reference the potential issue with small sample sizes and instability at the end of Section 5. We reference limitations with the test we use to de...

work page
[3]

The derivation of the efficient influence function is in Appendix C Guidelines: • The answer NA means that the paper does not include theoretical results

Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: Each lemma and theorem includes the necessary assumptions and the full proofs are all included in Appendix B. The derivation of the efficient influence function is in Appen...

work page
[4]

In particular, Appendix E has algorithms for our estimators and constructing breakdown frontier plots values

Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: All algorithms and experim...

work page
[5]

Guidelines: • The answer NA means that paper does not include experiments requiring code

Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: Anonymized code and data included in submission and will be included as GitHub link in camera-ready version...

work page
[6]

Guidelines: • The answer NA means that the paper does not include experiments

Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: Included in Appendix G. Guidelines: • The answer NA means that the paper does not include experiments. • The ...

work page
[7]

Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] 17 Justification: While we do not use traditional error bars, we quantify uncertainty through confidence regions in the breakdown frontier plots....

work page
[8]

Guidelines: • The answer NA means that the paper does not include experiments

Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: Information on compute resources used is included in Appendix G. Guidelines: • The answer NA means that...

work page

Show all 18 references

[9]

As a sensitivity analysis framework, our method poses minimal risk of societal harm

Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: This research does not involve human subjects or confidential data. As a sensit...
[10]

However, we do not explicitly discuss potential negative societal impacts

Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [No] Justification: We discuss the potential positive societal impacts of encouraging greater exploration of the robustness of...
[11]

Guidelines: • The answer NA means that the paper poses no such risks

Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: As a methodolog...
[12]

Guidelines: • The answer NA means that the paper does not use existing assets

Licenses for existing assets 19 Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: The Project S...

1995
[13]

We include experimentation details in Appendix G and provide documentation of the code repository

New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We release code implementing our estimation procedure and experiments. We include experimentation details in Appendi...
[14]

Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...
[15]

Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...
[16]

Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the ...

2024
[18]

takes a parameter and returns its efficient influence function

and Chernozhukov et al. (2018) (Theorem 3.1). C EIF Derivation C.1 Setup Recap We start by reiterating our setup and relevant notation. Recall that we have the following variables, • X: A vector of pretreatment covariates. 28 • S ∈ {0, 1}: A binary variable indicating study as...

2018
[2018]

bias correction

established consistency guarantees for parametric estimators and (Colnet et al., 2024b) did the same for non-parametric estimators. Finally, under parametric assumptions, (Buchanan et al., 2018) derived the asymptotic variance for the tasks of generalization and (Zivich et al....

2015

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

We introduce interpretable sensitivity parameters to enable partial identification of treatment effects in Section 4

Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: Our contributions are clearly outlined at the end of our Introduction. We introduce interpretable sensitivity parameters to enable partial identification of treatment effects in Section 4. We procee...

work page

[2] [2]

Limitations

Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We include a limitations and future work section in the Conclusion (Section 7). We reference the potential issue with small sample sizes and instability at the end of Section 5. We reference limitations with the test we use to de...

work page

[3] [3]

The derivation of the efficient influence function is in Appendix C Guidelines: • The answer NA means that the paper does not include theoretical results

Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] Justification: Each lemma and theorem includes the necessary assumptions and the full proofs are all included in Appendix B. The derivation of the efficient influence function is in Appen...

work page

[4] [4]

In particular, Appendix E has algorithms for our estimators and constructing breakdown frontier plots values

Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: All algorithms and experim...

work page

[5] [5]

Guidelines: • The answer NA means that paper does not include experiments requiring code

Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: Anonymized code and data included in submission and will be included as GitHub link in camera-ready version...

work page

[6] [6]

Guidelines: • The answer NA means that the paper does not include experiments

Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: Included in Appendix G. Guidelines: • The answer NA means that the paper does not include experiments. • The ...

work page

[7] [7]

Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [Yes] 17 Justification: While we do not use traditional error bars, we quantify uncertainty through confidence regions in the breakdown frontier plots....

work page

[8] [8]

Guidelines: • The answer NA means that the paper does not include experiments

Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: Information on compute resources used is included in Appendix G. Guidelines: • The answer NA means that...

work page