On the Impossibility of Specification Testing of Interference Models Based on Exposure Mappings
Pith reviewed 2026-05-12 02:56 UTC · model grok-4.3
The pith
Any specification test for an exposure mapping model has worst-case Type I and Type II errors summing to one when it must have power against larger models
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
For any specification test of a given exposure mapping model that is required to have power against a strictly larger exposure mapping model, the supremum Type I error plus the supremum Type II error equals one. The result applies to all finite sample sizes, to outcomes taking values in [0,1], and to alternatives that differ from the null in the most extreme way permitted by the exposure mapping framework.
What carries the argument
An exposure mapping, which assigns each unit one of several discrete exposure levels determined by the full vector of treatment assignments across all units, together with the partial order on exposure mappings in which one mapping is larger than another when it induces a finer partition of units into exposure groups.
If this is right
- Specification tests that do not restrict the alternative class cannot reliably detect misspecified interference models.
- Useful tests require the analyst to commit in advance to a narrow pair of models rather than testing against all larger exposure mappings.
- A uniformly consistent test exists for the specific case of distinguishing no interference from a network linear-in-means model.
Where Pith is reading between the lines
- The impossibility suggests that data-driven selection among interference models will generally require domain knowledge to limit the candidate models under consideration.
- Similar limits may apply when testing other structured causal models that are defined by partitions or groupings induced by the treatment assignment.
- Researchers could explore the weakest additional assumptions on the alternative class that restore the possibility of informative tests with controlled errors.
Load-bearing premise
The test must be required to have power against every possible larger exposure mapping model, with no further restrictions placed on how those alternatives differ from the null model.
What would settle it
A concrete test statistic and rejection threshold for which there exists at least one exposure mapping null model and one strictly larger alternative model such that the worst-case Type I error plus the worst-case Type II error is strictly less than one.
read the original abstract
Researchers use interference models based on exposure mappings to facilitate estimation of causal effects in randomized experiments with interference. To test the veracity of such models, researchers can use specification tests that aim to detect departures from the stipulated model. However, existing tests suffer from poor power and are often unable to detect important model violations. The main result in this paper is to show that the specification testing problem for exposure mapping models is inherently difficult, and the poor power of existing tests is inescapable. In particular, the worst-case Type I and Type II error rates must sum to one for any specification test of such models, ruling out the existence of a uniformly consistent test. This is the worst-case overall error rate achieved by a naive test that discards all data and arbitrarily rejects the null at random; the testing problem is in this sense impossible. This negative result holds true for all exposure mappings, all sample sizes, for uniformly bounded outcomes, and for alternatives that are maximally separated from the null. While some tests can detect some type of departures from the null model, there will always be relevant departures from the null that are undetectable. Informative specification tests must therefore restrict the alternative model against which they seek to attain power for, beyond the restrictions imposed by the exposure mappings alone. We illustrate this by providing a uniformly consistent test for differentiating no-interference from a network-linear-in-means model.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper establishes a finite-sample impossibility result for specification tests of exposure mapping models in randomized experiments with interference. Any test that controls Type I error under a given exposure mapping null while attaining power against the class of all strictly larger exposure mappings has worst-case Type I error (supremum over null distributions) plus worst-case Type II error (supremum over the larger class) equal to exactly 1. This holds for every finite sample size n, uniformly bounded outcomes, and randomized designs, and is tight because the random-guess test achieves equality. The authors note that informative tests therefore require further restrictions on the alternative and supply one such example: a uniformly consistent test for no-interference versus a network linear-in-means model.
Significance. If the central impossibility result holds, it has clear significance for causal inference under interference: it shows that unrestricted power against larger exposure mappings cannot improve upon random guessing in the worst case, forcing researchers to impose additional structure on alternatives (as the authors do in their positive result). The finite-sample character, the minimal assumptions (bounded outcomes and randomized designs), and the explicit tightness example are strengths. The work directly informs the design of specification tests for network experiments that use exposure mappings.
minor comments (2)
- Abstract: the phrase 'alternatives which are maximally separated from the null' is used without an immediate cross-reference to the precise definition employed in the main impossibility theorem; adding a parenthetical pointer would improve readability.
- Introduction: the formal definition of an exposure mapping and the partial order used to define 'strictly larger' models appear only after several paragraphs of motivation; moving a concise statement of these objects earlier would help readers follow the subsequent impossibility claim.
Simulated Author's Rebuttal
We thank the referee for their positive assessment of the manuscript and recommendation for minor revision. The referee's summary accurately captures both the impossibility result and the constructive example we provide.
Circularity Check
No significant circularity; impossibility result derived directly from error-rate definitions
full rationale
The central impossibility result follows from the definitions of Type I error (supremum rejection probability under the null exposure mapping) and Type II error (supremum acceptance probability under any strictly larger exposure mapping), together with the requirement that the test must control the former while attaining power against the unrestricted larger class. For any test, adversarial distributions can be constructed (under bounded outcomes and randomized designs) such that the sum of these worst-case errors equals 1; the random-guess test achieves equality, showing the bound is tight. This argument uses only the model definitions and finite-sample error-rate suprema; it does not rely on fitted parameters, self-citations, or any ansatz imported from prior work. The subsequent positive result (uniformly consistent test for no-interference versus network linear-in-means) is constructed explicitly under an additional restriction on the alternative and likewise contains no self-referential steps.
Axiom & Free-Parameter Ledger
axioms (3)
- domain assumption Outcomes are uniformly bounded
- domain assumption The experiment is a randomized experiment
- domain assumption Interference is captured by exposure mappings
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.