Sensitivity Analysis for the Test-Negative Design

Jingshu Wang; Peng Ding; Soumyabrata Kundu; Xinran Li

arxiv: 2406.06980 · v2 · pith:QWT7L6X4new · submitted 2024-06-11 · 📊 stat.ME

Sensitivity Analysis for the Test-Negative Design

Soumyabrata Kundu , Peng Ding , Jingshu Wang , Xinran Li This is my paper

Pith reviewed 2026-05-24 00:16 UTC · model grok-4.3

classification 📊 stat.ME

keywords test-negative designsensitivity analysiscausal odds ratiovaccine effectivenessunmeasured confoundingpartial identificationCOVID-19 vaccinesobservational data

0 comments

The pith

The test-negative design identifies the causal odds ratio for vaccine effects under assumptions about health-care-seeking behavior, with sensitivity methods to bound it when those assumptions fail.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows how a perfect test-negative design permits point identification of the causal odds ratio measuring vaccination's effect on infection, but only for people who seek health care readily. It develops two distinct sensitivity approaches: one that allows the design to control only partially for the unmeasured confounder, and another that lets the confounder affect both vaccination uptake and infection risk directly. These are combined to produce narrower bounds on the true causal odds ratio, with further tightening when treatment effect heterogeneity is restricted. The methods are then used on real COVID-19 test-negative datasets to assess vaccine effectiveness while accounting for possible residual confounding. Readers care because test-negative studies are the main way post-licensure vaccine performance is tracked, so knowing how much the results can shift under plausible deviations matters for policy.

Core claim

Under a test-negative design the causal odds ratio on the subpopulation with good health-care-seeking behavior is point-identified once identification assumptions are stated; two sensitivity strategies address departures from perfect control, one by modeling partial control and the other by allowing confounding to act on both vaccination and infection; their combination supplies narrower bounds that can be sharpened by restricting treatment-effect heterogeneity.

What carries the argument

Two complementary sensitivity analysis approaches for unmeasured confounding that are combined to narrow bounds on the causal odds ratio.

If this is right

Even with only partial control for health-care-seeking behavior the causal odds ratio remains partially identified.
Allowing confounding to influence both vaccination and infection produces valid but wider bounds.
Combining the two sensitivity routes yields strictly narrower bounds than either route alone.
Imposing a bound on treatment effect heterogeneity further reduces the width of the identified set.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same bounding strategy could be used to re-analyze existing test-negative studies of seasonal influenza vaccines.
If the final bounds exclude an odds ratio of one, the data still support a protective effect even after allowing for the modeled confounding.
The subpopulation restriction to good health-care seekers means the bounds speak only to that group, not to the entire population.

Load-bearing premise

The test-negative design controls for health-care-seeking behavior as the main unmeasured confounder between vaccination and infection.

What would settle it

Apply the combined sensitivity bounds to a test-negative dataset for a vaccine whose true causal odds ratio is already known from a large randomized trial; if the true value lies outside the reported bounds, the sensitivity model is misspecified.

Figures

Figures reproduced from arXiv: 2406.06980 by Jingshu Wang, Peng Ding, Soumyabrata Kundu, Xinran Li.

**Figure 2.** Figure 2: (a) shows the vaccine efficacy estimates derived from 1 minus the observed odds [PITH_FULL_IMAGE:figures/full_fig_p029_2.png] view at source ↗

read the original abstract

The test-negative design has become popular for evaluating the effectiveness of post-licensure vaccines using observational data. In addition to its logistical convenience on data collection, the design is also believed to control for the differential health-care-seeking behavior between vaccinated and unvaccinated individuals, an important while often unmeasured confounder between the vaccination and infection. Hence, the design has been employed routinely to monitor seasonal flu vaccines and more recently to measure the COVID-19 vaccine effectiveness. Despite its popularity, the design has been questioned, in particular about its ability to fully control for the unmeasured confounding. In this paper, we explore deviations from a perfect test-negative design, and propose various sensitivity analysis methods for estimating the effect of vaccination measured by the causal odds ratio on the subpopulation of individuals with good health-care-seeking behavior. We start with point identification of the causal odds ratio under a test-negative design, comparing different forms of identification assumptions and their corresponding estimands. We then propose two approaches for conducting sensitivity analysis, addressing the influence of the unmeasured confounding in two different ways. Specifically, one approach investigates partial control for unmeasured confounding in the test-negative design, while the other examines the impact of unmeasured confounding on both vaccination and infection. Furthermore, we combine these approaches to provide narrower bounds on the true causal odds ratio, and further sharpen the bounds by restricting the treatment effect heterogeneity. Finally, we apply the proposed methods to evaluate the effectiveness of COVID-19 vaccines using observational data from test-negative designs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper gives concrete sensitivity bounds for causal odds ratios in the test-negative design by combining partial-control and unmeasured-confounding adjustments, then applies them to COVID vaccine data.

read the letter

The core contribution is a pair of sensitivity approaches for the test-negative design that start from point identification of the causal odds ratio on the good-health-care-seeking subpopulation and then relax the perfect-control assumption. One route models partial control for the unmeasured confounder; the other lets the confounder affect both vaccination and infection. Combining the two produces narrower bounds, and adding a restriction on treatment-effect heterogeneity sharpens them further. The COVID application shows the methods can be run on existing TND data sets. That structure is useful because it moves directly from standard identification results to usable bounds without requiring new data collection. The derivations appear to rest on standard causal-inference sensitivity techniques rather than circular definitions, which keeps the grounding external. The main limitation is that the resulting bounds will often remain wide unless the analyst is willing to put strong limits on the sensitivity parameters or on heterogeneity; the paper does not demonstrate that the intervals are narrow enough in typical vaccine-monitoring settings to alter policy conclusions. The application section would benefit from more detail on how the chosen parameter ranges were justified and whether the bounds change the sign or magnitude of the estimated vaccine effect. This work is aimed at methodologists and applied epidemiologists who already use or review TND studies and want practical tools for robustness checks. It is solid enough on its own terms to merit referee time rather than a desk reject, even if revisions will be needed on the width and interpretability of the bounds.

Referee Report

0 major / 2 minor

Summary. The manuscript develops sensitivity analysis methods for the test-negative design (TND) to estimate the causal odds ratio measuring vaccine effectiveness on the subpopulation with good health-care-seeking behavior. It begins with point identification of the causal odds ratio under TND assumptions, compares different forms of identification assumptions and their estimands, proposes two sensitivity approaches (one for partial control of unmeasured confounding and one for the impact of unmeasured confounding on both vaccination and infection), combines the approaches to obtain narrower bounds on the causal odds ratio, further sharpens the bounds by restricting treatment effect heterogeneity, and applies the methods to observational COVID-19 vaccine data collected via TND.

Significance. If the derivations hold, the work provides a practical framework for assessing robustness of TND-based vaccine effectiveness estimates to deviations from ideal assumptions about unmeasured confounding. Given the routine use of TND for seasonal flu and COVID-19 monitoring, tools that allow partial control, bounding, and bound sharpening via heterogeneity restrictions could meaningfully improve the interpretability of observational results. The combination of sensitivity approaches and the empirical application are strengths.

minor comments (2)

[Abstract and identification section] The abstract states that different forms of identification assumptions and their estimands are compared, but a table or explicit list contrasting the assumptions, estimands, and required conditions would improve clarity for readers.
Notation for the sensitivity parameters (e.g., those governing confounding strength and partial control) should be introduced with clear definitions and ranges early in the methods to aid reproducibility.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for their positive summary of the manuscript, recognition of its potential utility for TND-based vaccine effectiveness monitoring, and recommendation of minor revision. No specific major comments were listed in the report.

Circularity Check

0 steps flagged

No significant circularity identified

full rationale

The paper's derivation begins from standard point identification of the causal odds ratio under test-negative design assumptions on the subpopulation with good health-care-seeking behavior, then extends via two sensitivity approaches for partial control and unmeasured confounding, their combination for bounds, and sharpening via heterogeneity restrictions. These steps are constructed from first-principles causal inference without any quoted reduction of a prediction to a fitted input by construction, self-definitional equivalence, or load-bearing self-citation chain. The provided abstract and description contain no equations or citations that equate outputs to inputs internally, and the methods are presented as extensions with independent grounding rather than renaming or smuggling prior results.

Axiom & Free-Parameter Ledger

1 free parameters · 2 axioms · 0 invented entities

The paper rests on domain assumptions about the test-negative design's control for health-care-seeking confounding and introduces sensitivity parameters to quantify deviations from ideal conditions.

free parameters (1)

sensitivity parameters for confounding strength and partial control
Introduced to explore and bound the influence of unmeasured confounding in the two proposed approaches.

axioms (2)

domain assumption Test-negative design controls for differential health-care-seeking behavior between vaccinated and unvaccinated
Stated as the belief behind the design's popularity and the starting point for sensitivity analysis.
domain assumption Point identification of causal odds ratio is possible under the test-negative design with certain assumptions
Explicitly used as the initial step before sensitivity analysis.

pith-pipeline@v0.9.0 · 5803 in / 1277 out tokens · 27420 ms · 2026-05-24T00:16:45.727967+00:00 · methodology

discussion (0)

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

IndisputableMonolith/Cost/FunctionalEquation.lean washburn_uniqueness_aczel unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

We start with point identification of the causal odds ratio under a test-negative design... propose two approaches for conducting sensitivity analysis... combine these approaches to provide narrower bounds... sharpen the bounds by restricting the treatment effect heterogeneity.
IndisputableMonolith/Foundation/AbsoluteFloorClosure.lean absolute_floor_iff_bare_distinguishability unclear

?

unclear
Relation between the paper passage and the cited Recognition theorem.

Assumption 4. For some 0 ≤ δ ≤ 1, we have P(U ≠ 0 | T = 1) ≤ δ. ... Assumption 6. For some Γ ≥ 1, we have 1/Γ ≤ pzy|1/pzy|0 ≤ Γ...

What do these tags mean?

matches: The paper's claim is directly supported by a theorem in the formal canon.
supports: The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends: The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses: The paper appears to rely on the theorem as machinery.
contradicts: The paper's claim conflicts with a theorem or certificate in the canon.
unclear: Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

28 extracted references · 28 canonical work pages

[1]

M. H. Alharbi and C. M. Kribs. How influenza vaccination and virus interference may impact combined influenza-coronavirus disease burden. Journal of Mathematical Biology, 85: 0 10, 2022

work page 2022
[2]

L. R. Baden, H. M. El Sahly, B. Essink, K. Kotloff, S. Frey, R. Novak, D. Diemert, S. A. Spector, N. Rouphael, C. B. Creech, J. McGettigan, S. Khetan, N. Segall, J. Solis, A. Brosz, C. Fierro, H. Schwartz, K. Neuzil, L. Corey, P. Gilbert, H. Janes, D. Follmann, M. Marovich, J. Mascola, L. Polakowski, J. Ledgerwood, B. S. Graham, H. Bennett, R. Pajon, C. K...

work page 2021
[3]

Bareinboim and J

E. Bareinboim and J. Pearl. Controlling selection bias in causal inference. In N. D. Lawrence and M. Girolami, editors, Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, volume 22 of Proceedings of Machine Learning Research, pages 100--108, La Palma, Canary Islands, 2012

work page 2012
[4]

Cioc a nea-Teodorescu, M

I. Cioc a nea-Teodorescu, M. Nason, A. Sj \"o lander, and E. E. Gabriel. Adjustment for disease severity in the test-negative study design. American Journal of Epidemiology, 190: 0 1882--1889, 2021

work page 2021
[5]

Coronavirus disease 2019 (covid-19) treatment guidelines, 2023

COVID-19 Treatment Guidelines Panel . Coronavirus disease 2019 (covid-19) treatment guidelines, 2023. URL https://www.covid19treatmentguidelines.nih.gov/

work page 2019
[6]

N. E. Dean, J. W. Hogan, and M. E. Schnitzer. Covid-19 vaccine effectiveness and the test-negative design. New England Journal of Medicine, 385: 0 1431--1433, 2021

work page 2021
[7]

Didelez, S

V. Didelez, S. Kreiner, and N. Keiding. Graphical Models for Inference Under Outcome-Dependent Sampling . Statistical Science, 25: 0 368 -- 387, 2010

work page 2010
[8]

Ding and T

P. Ding and T. J. VanderWeele. Sensitivity analysis without assumptions. Epidemiology, 27: 0 368--377, 2016

work page 2016
[9]

E. E. Gabriel, M. C. Sachs, and A. Sj \"o lander. Causal bounds for outcome-dependent sampling in observational studies. Journal of the American Statistical Association, 117 0 (538): 0 939--950, 2022

work page 2022
[10]

Greenland, J

S. Greenland, J. Pearl, and J. M. Robins. Confounding and Collapsibility in Causal Inference . Statistical Science, 14: 0 29 -- 46, 1999

work page 1999
[11]

Guo and Z

J. Guo and Z. Geng. Collapsibility of logistic regression coefficients. Journal of the Royal Statistical Society. Series B (Methodological), 57: 0 263--267, 1995

work page 1995
[12]

Gurobi Optimizer Reference Manual , 2022

Gurobi Optimization, LLC . Gurobi Optimizer Reference Manual , 2022. URL https://www.gurobi.com

work page 2022
[13]

Haber, Q

M. Haber, Q. An, I. M. Foppa, D. K. Shay, J. M. Ferdinands, and W. A. Orenstein. A probability model for evaluating the bias and precision of influenza vaccine effectiveness estimates from case-control studies. Epidemiology and Infection, 143: 0 1417--1426, 2015

work page 2015
[14]

Huang and W.-C

T.-H. Huang and W.-C. Lee. Bounding Formulas for Selection Bias . American Journal of Epidemiology, 182: 0 868--872, 2015

work page 2015
[15]

M. L. Jackson and J. C. Nelson. The test-negative design for estimating influenza vaccine effectiveness. Vaccine, 31: 0 2165--2168, 2013

work page 2013
[16]

K. Q. Li, X. Shi, W. Miao, and E. T. Tchetgen. Double negative control inference in test-negative design studies of vaccine effectiveness. Journal of the American Statistical Association, in press, 2023

work page 2023
[17]

F. P. Polack, S. J. Thomas, N. Kitchin, J. Absalon, A. Gurtman, S. Lockhart, J. L. Perez, G. P \'e rez Marc, E. D. Moreira, C. Zerbini, et al. Safety and efficacy of the bnt162b2 mrna covid-19 vaccine. New England journal of medicine, 383: 0 2603--2615, 2020

work page 2020
[18]

P. R. Rosenbaum. Observational Studies. Springer, New York, 2 edition, 2002

work page 2002
[19]

P. R. Rosenbaum. Sensitivity analysis in observational studies. Encyclopedia of statistics in behavioral science, 2005

work page 2005
[20]

P. R. Rosenbaum and D. B. Rubin. The central role of the propensity score in observational studies for causal effects . Biometrika, 70: 0 41--55, 1983

work page 1983
[21]

X. Shi, K. Q. Li, and B. Mukherjee. Current challenges with the use of test-negative designs for modeling covid-19 vaccination and outcomes. American Journal of Epidemiology, 192: 0 328--333, 2023

work page 2023
[22]

L. H. Smith and T. J. VanderWeele. Bounding bias due to selection. Epidemiology, 30 0 (4): 0 509, 2019

work page 2019
[23]

S. G. Sullivan, S. Feng, and B. J. Cowling. Potential of the test-negative design for measuring influenza vaccine effectiveness: a systematic review. Expert review of vaccines, 13 0 (12): 0 1571--1591, 2014

work page 2014
[24]

S. G. Sullivan, E. J. Tchetgen Tchetgen, and B. J. Cowling. Theoretical basis of the test-negative study design for assessment of influenza vaccine effectiveness. American journal of epidemiology, 184 0 (5): 0 345--353, 2016

work page 2016
[25]

Effectiveness of covid-19 vaccines in ambulatory and inpatient care settings

Thompson et al. Effectiveness of covid-19 vaccines in ambulatory and inpatient care settings. New England Journal of Medicine, 385: 0 1355--1371, 2021

work page 2021
[26]

B. Wang, S. M. Dufault, D. S. Small, and N. P. Jewell. Randomization inference for cluster-randomized test-negative designs with application to dengue studies: Unbiased estimation, partial compliance, and stepped-wedge design. The Annals of Applied Statistics, 17: 0 1592--1614, 2023

work page 2023
[27]

Design of vaccine efficacy trials to be used during public health emergencies---points of considerations and key principles, 2020

World Health Organization . Design of vaccine efficacy trials to be used during public health emergencies---points of considerations and key principles, 2020

work page 2020
[28]

M. Yu, K. Q. Li, N. Jewell, E. T. Tchetgen, D. Small, X. Shi, and B. Wang. Test-negative designs with various reasons for testing: statistical bias and solution. arXiv preprint arXiv:2312.03967, 2023

work page arXiv 2023

[1] [1]

M. H. Alharbi and C. M. Kribs. How influenza vaccination and virus interference may impact combined influenza-coronavirus disease burden. Journal of Mathematical Biology, 85: 0 10, 2022

work page 2022

[2] [2]

L. R. Baden, H. M. El Sahly, B. Essink, K. Kotloff, S. Frey, R. Novak, D. Diemert, S. A. Spector, N. Rouphael, C. B. Creech, J. McGettigan, S. Khetan, N. Segall, J. Solis, A. Brosz, C. Fierro, H. Schwartz, K. Neuzil, L. Corey, P. Gilbert, H. Janes, D. Follmann, M. Marovich, J. Mascola, L. Polakowski, J. Ledgerwood, B. S. Graham, H. Bennett, R. Pajon, C. K...

work page 2021

[3] [3]

Bareinboim and J

E. Bareinboim and J. Pearl. Controlling selection bias in causal inference. In N. D. Lawrence and M. Girolami, editors, Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, volume 22 of Proceedings of Machine Learning Research, pages 100--108, La Palma, Canary Islands, 2012

work page 2012

[4] [4]

Cioc a nea-Teodorescu, M

I. Cioc a nea-Teodorescu, M. Nason, A. Sj \"o lander, and E. E. Gabriel. Adjustment for disease severity in the test-negative study design. American Journal of Epidemiology, 190: 0 1882--1889, 2021

work page 2021

[5] [5]

Coronavirus disease 2019 (covid-19) treatment guidelines, 2023

COVID-19 Treatment Guidelines Panel . Coronavirus disease 2019 (covid-19) treatment guidelines, 2023. URL https://www.covid19treatmentguidelines.nih.gov/

work page 2019

[6] [6]

N. E. Dean, J. W. Hogan, and M. E. Schnitzer. Covid-19 vaccine effectiveness and the test-negative design. New England Journal of Medicine, 385: 0 1431--1433, 2021

work page 2021

[7] [7]

Didelez, S

V. Didelez, S. Kreiner, and N. Keiding. Graphical Models for Inference Under Outcome-Dependent Sampling . Statistical Science, 25: 0 368 -- 387, 2010

work page 2010

[8] [8]

Ding and T

P. Ding and T. J. VanderWeele. Sensitivity analysis without assumptions. Epidemiology, 27: 0 368--377, 2016

work page 2016

[9] [9]

E. E. Gabriel, M. C. Sachs, and A. Sj \"o lander. Causal bounds for outcome-dependent sampling in observational studies. Journal of the American Statistical Association, 117 0 (538): 0 939--950, 2022

work page 2022

[10] [10]

Greenland, J

S. Greenland, J. Pearl, and J. M. Robins. Confounding and Collapsibility in Causal Inference . Statistical Science, 14: 0 29 -- 46, 1999

work page 1999

[11] [11]

Guo and Z

J. Guo and Z. Geng. Collapsibility of logistic regression coefficients. Journal of the Royal Statistical Society. Series B (Methodological), 57: 0 263--267, 1995

work page 1995

[12] [12]

Gurobi Optimizer Reference Manual , 2022

Gurobi Optimization, LLC . Gurobi Optimizer Reference Manual , 2022. URL https://www.gurobi.com

work page 2022

[13] [13]

Haber, Q

M. Haber, Q. An, I. M. Foppa, D. K. Shay, J. M. Ferdinands, and W. A. Orenstein. A probability model for evaluating the bias and precision of influenza vaccine effectiveness estimates from case-control studies. Epidemiology and Infection, 143: 0 1417--1426, 2015

work page 2015

[14] [14]

Huang and W.-C

T.-H. Huang and W.-C. Lee. Bounding Formulas for Selection Bias . American Journal of Epidemiology, 182: 0 868--872, 2015

work page 2015

[15] [15]

M. L. Jackson and J. C. Nelson. The test-negative design for estimating influenza vaccine effectiveness. Vaccine, 31: 0 2165--2168, 2013

work page 2013

[16] [16]

K. Q. Li, X. Shi, W. Miao, and E. T. Tchetgen. Double negative control inference in test-negative design studies of vaccine effectiveness. Journal of the American Statistical Association, in press, 2023

work page 2023

[17] [17]

F. P. Polack, S. J. Thomas, N. Kitchin, J. Absalon, A. Gurtman, S. Lockhart, J. L. Perez, G. P \'e rez Marc, E. D. Moreira, C. Zerbini, et al. Safety and efficacy of the bnt162b2 mrna covid-19 vaccine. New England journal of medicine, 383: 0 2603--2615, 2020

work page 2020

[18] [18]

P. R. Rosenbaum. Observational Studies. Springer, New York, 2 edition, 2002

work page 2002

[19] [19]

P. R. Rosenbaum. Sensitivity analysis in observational studies. Encyclopedia of statistics in behavioral science, 2005

work page 2005

[20] [20]

P. R. Rosenbaum and D. B. Rubin. The central role of the propensity score in observational studies for causal effects . Biometrika, 70: 0 41--55, 1983

work page 1983

[21] [21]

X. Shi, K. Q. Li, and B. Mukherjee. Current challenges with the use of test-negative designs for modeling covid-19 vaccination and outcomes. American Journal of Epidemiology, 192: 0 328--333, 2023

work page 2023

[22] [22]

L. H. Smith and T. J. VanderWeele. Bounding bias due to selection. Epidemiology, 30 0 (4): 0 509, 2019

work page 2019

[23] [23]

S. G. Sullivan, S. Feng, and B. J. Cowling. Potential of the test-negative design for measuring influenza vaccine effectiveness: a systematic review. Expert review of vaccines, 13 0 (12): 0 1571--1591, 2014

work page 2014

[24] [24]

S. G. Sullivan, E. J. Tchetgen Tchetgen, and B. J. Cowling. Theoretical basis of the test-negative study design for assessment of influenza vaccine effectiveness. American journal of epidemiology, 184 0 (5): 0 345--353, 2016

work page 2016

[25] [25]

Effectiveness of covid-19 vaccines in ambulatory and inpatient care settings

Thompson et al. Effectiveness of covid-19 vaccines in ambulatory and inpatient care settings. New England Journal of Medicine, 385: 0 1355--1371, 2021

work page 2021

[26] [26]

B. Wang, S. M. Dufault, D. S. Small, and N. P. Jewell. Randomization inference for cluster-randomized test-negative designs with application to dengue studies: Unbiased estimation, partial compliance, and stepped-wedge design. The Annals of Applied Statistics, 17: 0 1592--1614, 2023

work page 2023

[27] [27]

Design of vaccine efficacy trials to be used during public health emergencies---points of considerations and key principles, 2020

World Health Organization . Design of vaccine efficacy trials to be used during public health emergencies---points of considerations and key principles, 2020

work page 2020

[28] [28]

M. Yu, K. Q. Li, N. Jewell, E. T. Tchetgen, D. Small, X. Shi, and B. Wang. Test-negative designs with various reasons for testing: statistical bias and solution. arXiv preprint arXiv:2312.03967, 2023

work page arXiv 2023