From Individual Experience to Collective Evidence: A Reporting-Based Framework for Identifying Systemic Harms

Benjamin Recht; Inioluwa Deborah Raji; Jessica Dai; Paula Gradu

REVIEW 3 major objections 6 minor 19 references

From Individual Experience to Collective Evidence: A Reporting-Based Framework for Identifying Systemic Harms

T0 review · 3 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that a group overrepresented in incident reports, under bounded reporting inequality, provably suffers a higher true rate of harm.

desk verdict A genuinely new framework for turning incident reports into anytime-valid evidence of disproportionate harm, but the harm interpretation rests on unverifiable reporting bounds and one power proof has a fixable gap. read the letter →

arxiv 2502.08166 v2 pith:4LFOCCPW submitted 2025-02-12 cs.CY cs.LG

Jessica Dai , Paula Gradu , Inioluwa Deborah Raji , Benjamin Recht This is my paper · ORCID

classification cs.CYcs.LG MSC 62L1062F03

keywords reportingdatabasessequentialhypothesistestinganytime-validinferencerelativeriskincidenceratefairnessauditingalgorithmicaccountabilitypreponderance

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a database of individual incident reports—noisy, incomplete, and biased by who happens to complain—can still serve as collective evidence about which groups are truly harmed by a system. Its central mathematical claim is that the ratio of a group's share of reports to its share of the reference population, $\mu_G/\mu^0_G$, is a valid signal for true relative risk whenever reporting behavior is not too unequal across groups. Under that condition, a sequential test that flags a group for being $\beta$ times overrepresented in reports implies the group's true relative risk is at least $\beta/b$, where $b$ bounds how much any group's report-to-incidence ratio can exceed the population average. The authors instantiate the test with two anytime-valid algorithms, prove their guarantees, and show on real vaccine and mortgage report data that known harmed subgroups are flagged quickly. If the argument holds, incident-reporting systems become a formal route from individual complaints to real-time, auditable findings of systemic harm.

What carries the argument

The load-bearing object is the preponderance ratio $\mu_G/\mu^0_G$, tested sequentially as $H^G_0: \mu_G \le \beta\mu^0_G$ against $H^G_1: \mu_G > \beta\mu^0_G$ for every group. Two mechanisms convert the test's output into statements about harm: the Bayes identity $\mu_G/\mu^0_G = \Pr[R=1 \mid G]/\Pr[R=1]$, and the report-to-incidence ratio $\rho_G$ that controls how far reporting behavior can distort the comparison. Two anytime-valid algorithms implement the test in parallel with a Bonferroni correction: a sequential Z-test whose threshold is built from a finite-time law of the iterated logarithm, and a betting-style test that accumulates $\omega^G_t = \omega^G_{t-1} + \ln(1 + \lambda^G_t (\mathbf{1}[X_t \in G] - \beta\mu^0_G))$ as an e-value, with $\lambda^G_t$ chosen by Online Newton Step.

What would settle it

Match a reporting database to independently measured outcomes—for example, VAERS myocarditis reports against insurance-claims-confirmed myocarditis among vaccinated young men—and compute each group's true relative risk together with $\rho_G/\rho$. Finding any group with high true relative risk whose report-to-incidence ratio exceeds the $b$ used to set $\beta$, and whose reports therefore fail to trigger the bound $RR_G \ge \beta/b$, would falsify the sufficiency of the stated conditions.

Watch

Extended reading notes

Core claim

The paper's central discovery is a formal bridge between the visible preponderance of reports and the unobserved distribution of true harm. Writing $\mu_G$ for the share of reports from group $G$ and $\mu^0_G$ for $G$'s share of the reference population, Bayes' rule gives $\mu_G/\mu^0_G = \Pr[R=1 \mid G]/\Pr[R=1]$. If the group's report-to-incidence ratio $\rho_G = \Pr[R=1 \mid G]/\Pr[Y=1 \mid G]$ is at most $b$ times the population ratio $\rho$, then establishing $\mu_G \ge \beta \mu^0_G$ implies the true relative risk satisfies $RR_G \ge \beta/b$. A parallel proposition converts the same overrepresentation flag into a lower bound on absolute incidence, $IR_G \ge (\beta - \gamma^{FR}_G)/(\gamma^{TR}_G - \gamma^{FR}_G)$, given assumptions on true and false reporting rates. These identities turn the act of testing whether a subgroup is overrepresented among reports into a quantitative statement about harm, and the sequential algorithms ensure the statement can be made in real time with controlled error.

Load-bearing premise

The load-bearing premise is that no subgroup's report-to-incidence ratio exceeds the population ratio by more than the assumed factor $b$; the paper itself notes that the reporting-rate parameters needed for the incidence interpretation are not directly estimable from report data, so if genuinely harmed groups report sparsely, or groups with little true harm over-report, observed preponderance will not track true harm.

Editorial extensions

If this is right

A regulator can monitor a report stream continuously and, at the moment a subgroup's preponderance crosses the threshold, assert a lower bound on that subgroup's true relative risk without waiting for a full audit.
Because both algorithms are anytime-valid, the same accumulating reports can be reused to escalate from a low $\beta$ to higher values, turning an initial alarm into a severity ranking of the flagged harm.
The Bonferroni correction over all predefined subgroups adds only a logarithmic factor to stopping time, so the method remains tractable even with hundreds of combinatorially defined groups.
On real report streams, the method identifies young men for vaccine-associated myocarditis and healthy-debt-to-income applicants denied mortgages, using fewer reports than the original post-hoc analyses.
If no group is truly overrepresented, both algorithms have a false-alarm probability bounded by $\alpha$ over indefinite monitoring.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension the paper leaves implicit is to apply the same preponderance test to other existing report streams, such as consumer-complaint databases, platform moderation appeals, or police misconduct records, and compare the flagged subgroups against disparities established by independent investigations.
Because the incidence-rate interpretation depends on $\gamma^{TR}_G$ and $\gamma^{FR}_G$, which the paper says are not directly estimable from report data, a natural complement is a calibration study that measures true and false reporting rates through follow-up surveys or incentivized reporting; those estimates would turn the incidence bound from qualitative into quantitative.
The i.i.d. reporting assumption and the fixed bound $b$ both become fragile under coordinated campaigns or publicity shocks; one extension would be to replace the fixed bound with a time-varying or robustified bound and stress-test the preponderance signal under adversarial report surges.
In the mortgage case, the authors simulate reporting models rather than observe real reports; the variation in relative risk of the first-flagged group across models suggests that a practitioner should estimate reporting behavior before interpreting a flagged group's severity as ground truth.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper you asked about is worth a serious look. It formalizes a question nobody else has formalized: can a stream of individual incident reports be used to flag subgroups that are disproportionately harmed, without knowing true harm rates? The answer it gives is 'yes, under explicit assumptions on reporting rates,' and it makes those assumptions unusually visible. Propositions 3.2 and 3.4 are clean Bayes-rule derivations, and the anytime-valid tests in Section 4 are standard but competently assembled. The real-data demonstrations on VAERS myocarditis and HMDA mortgage denials are honest about their limitations and do show early detection relative to the original published analyses. That is a real contribution.

The soft spots are proportionate. First, the power proof of Theorem 4.2 has a genuine gap: Hoeffding is applied pointwise in t, and the proof then asserts a simultaneous bound 'with probability at least 1−δ for all t' without a union bound. The log-log factor in the threshold suggests this is fixable, but as written it is not a proof. Second, the bridge from report preponderance to true harm rests on the bound b in Proposition 3.2 (or the reporting rates γTR, γFR in Prop. 3.4), and Section 3.2 concedes these are not directly estimable from report data. The paper's own Anti-Correlated mortgage row at β=1.6 shows what can happen: max ρG/ρ=2.7, so the formal guarantee is RRG ≥ 0.59, while the first flagged group has true RR 1.05. That does not contradict any theorem, but it does blunt the 'collective evidence' phrasing. The authors are transparent about this—they explicitly say the quantities are not estimable—but the usefulness claim is conditional on a practitioner having a credible external bound. Third, the vaccine experiment uses a pre-filtered myocarditis report set; the paper acknowledges the filter was motivated by prior knowledge, so it is a validation of the method conditioned on knowing what to look for, not an end-to-end discovery.

Minor: Table 2 reports averages without error bars.

Who should read this: people working on post-market surveillance, AI fairness auditing, and sequential hypothesis testing. It deserves a serious referee. With the power gap fixed and the language around harm interpretation sharpened, I'd be comfortable seeing it accepted.

Recommendation: send to peer review; expect a revision.

Referee Report

3 major / 6 minor

Summary. The paper studies the 'reporting database problem': individual reports of adverse events arrive sequentially and the goal is to identify subgroups that are disproportionately likely to experience harm. The authors formalize this as a sequential hypothesis test of whether a group's share of reports, mu_G, exceeds its base share of the reference population, mu^0_G, by a factor beta. They give two interpretation results: Proposition 3.2 converts an overrepresentation finding into a lower bound on relative risk under a bound on report-to-incidence ratios, and Proposition 3.4 converts it into a lower bound on absolute incidence under assumptions on true and false reporting rates. They propose two algorithms, a sequential Z-test and a betting-style e-value test, and prove anytime-validity and power guarantees. Experiments on VAERS vaccine reports and simulated reporting over HMDA mortgage data show that the tests can flag known high-risk groups early in some settings. Overall, the paper is a clear and mostly rigorous conditional framework: under explicit, acknowledged assumptions, report preponderance can be used for valid sequential testing of overrepresentation, with the harm interpretation following only when the reporting assumptions are credible.

Significance. If the framework is accepted, the paper makes a useful contribution to post-deployment auditing: it gives a principled way to turn individual incident reports into collective, anytime-valid statistical evidence, with honest treatment of multiple testing and explicit conditions for harm interpretation. The Bayes-rule derivations in Propositions 3.2 and 3.4 are clean, and the validity theorems (4.1 and 4.3) are standard supermartingale/Ville arguments that appear sound. The authors are transparent that the reporting-behavior parameters (e.g., rho_G/rho, gamma^TR_G, gamma^FR_G) are not directly estimable from report data, and they explicitly state the limits of their claims. The VAERS case study is a compelling demonstration of early identification of young men as a high-risk group. However, the paper's central 'usefulness' claim for detecting true harm is materially weakened by the underdetermination of the report-to-incidence bound b, and one of the paper's own mortgage experiments flags a group with near-baseline true risk at a setting where the formal guarantee is vacuous. This is a fixable but important qualification.

major comments (3)

[Section 3, Prop. 3.2; Section 5.2, Table 2] The harm interpretation is underdetermined by an unidentifiable bound b. Proposition 3.2 gives RR_G >= beta/b, but b = max_G rho_G/rho is defined in terms of Pr[R|G] and Pr[Y|G], neither of which is observable from the reporting database, and Section 3.2 concedes that the analogous reporting rates are 'not directly estimable from reporting data'. The paper's own Anti-Correlated mortgage experiment illustrates the fragility: with max_G rho_G/rho = 2.7 and beta = 1.6, the formal guarantee is only RR_G >= 0.59, and the first-flagged group has true relative risk 1.05 (Table 2). This directly undercuts the central claim that overrepresentation is 'a useful signal for Pr[Y|G]' in a wide class of applications, unless the user has external justification that b < beta. The authors should either reframe the central claim to be about testing for overrepresentation (with harm inference as a conditional interpretation), provide a sensitivity analysis or calibration procedure for b, or identify conditions under which such failures cannot occur.
[Appendix A.1, proof of Theorem 4.2] The high-probability stopping-time proof is invalid as written. The proof applies Hoeffding's inequality pointwise to obtain Pr[omega^G*_t < E[omega^G*_t] - C1 sqrt(t ln(...))] <= delta, and then asserts: 'Thus, with probability at least 1 - delta, for all t simultaneously, omega^G*_t > ...'. A pointwise tail bound does not imply a simultaneous bound over unbounded t without a union bound (which with fixed delta would diverge) or a time-uniform concentration argument. Consequently, the claim 'with probability at least 1 - delta that T <= e^O(ln(1/delta)/Delta^2_max)' is not proven. This is likely repairable (e.g., by applying the pointwise bound at the target t*, or using a union bound over dyadic blocks and absorbing log factors into the e^O), but the proof must be corrected.
[Section 5.2, Table 2] The mortgage experiments do not report the identity of the first-identified group, only its average true relative risk. The abstract claims that the method 're-identifies subgroups known to experience disproportionate harm' on both datasets, but in the mortgage setting the paper never verifies that the flagged subgroups are the known race-based groups from Martinez and Kirchner (2021). A flagged group with relative risk ~1.5-1.8 could be a large group with high denial rates for reasons unrelated to the documented racial disparities. The authors should report which subgroups are flagged under each reporting model and beta, and compare them to the known harmed groups, to make the empirical claim substantive.

minor comments (6)

[Appendix A.1, proof of Theorem 4.2] The line 'lim_{t -> infinity} Pr[T > t] = lim_{t -> infinity} exp(-t) = 1' should read '= 0', since exp(-t) decays to zero. Also, the exponent in the preceding Hoeffding bound appears to be dimensionally inconsistent: it omits the C1^2 ln(...) term that arises when squaring the deviation. The displayed inequality should be re-derived carefully.
[Section 4.1, Eq. (3) and Appendix A.1] The finite-sample validity proof in Appendix A.1 yields a threshold with leading constant sqrt(2)/2 ~ 0.707, whereas Eq. (3) with C = 1/2 and the constant 2.07 gives constant ~0.719. The paper says only that 'the statement follows from handling the numerical impact of rounding'; a short explanation of how 2.07 is obtained from the proof would improve reproducibility.
[Section 5.1] Base preponderances for intersection groups are imputed as products of marginal rates, implicitly assuming independence of age and sex in vaccination coverage. This is mentioned in the text, but its potential effect on the reported stopping dates is not discussed; a brief caveat in the results interpretation would be appropriate.
[Section 5.1, footnote 7] The claim that re-running the test with increasing beta is 'statistically valid due to the equivalence between one-sided hypothesis testing and confidence sequences' is important but terse. A reference or a one-sentence explanation of the duality would help readers appreciate why the adaptive beta procedure does not break error control.
[Section 2, Notation] The notation uses X_i for individuals and X_t for the reporting individual at time t, which is reasonable, but the transition between the two in Section 2 ('Every individual Xi belongs to at least one group') is slightly abrupt; a remark that the i.i.d. assumption applies to the reporting stream X_t would improve clarity.
[Table 2] The footnote markers in the table (e.g., '—1 —' and '297513') are visually confusing as rendered; consider using superscript letters or a separate column to indicate trials that did not stop within 40,000 steps.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the report-preponderance bounds are conditional implications of explicit reporting assumptions, not fitted to outcome labels.

full rationale

The paper's central derivations are conditional statements, not disguised inputs. Proposition 3.2 converts the assumed bound ρG ≤ b·ρ and the tested overrepresentation µG ≥ βµ0_G into the algebraic lower bound RRG ≥ β/b; it never uses the target group's true harm rate as an input. Proposition 3.4 similarly derives IRG ≥ (β−γFR_G)/(γTR_G−γFR_G) from Bayes' rule and explicitly states that γTR_G and γFR_G 'are not directly estimable from reporting data,' which is an identifiability limitation rather than circularity. The parameter β is set a priori from stated reporting assumptions, not fitted to the outcome data used for evaluation. In the experiments, the ground-truth harmed groups come from external analyses (Oster et al. for VAERS; Martinez and Kirchner for HMDA), and the HMDA ground-truth relative risks are computed from the full dataset only to evaluate the algorithms, not to choose β. The mortgage 'Anti-Correlated' row illustrates that the formal guarantee can be vacuous when the assumed bound b exceeds β, but a vacuous bound is not an equivalence between inputs and outputs; the paper is transparent about the sufficiency of these assumptions. Self-citations (e.g., Dai et al. 2024, Raji et al. 2022) appear only in related-work or policy context and are not load-bearing. The sequential-test validity proofs are adapted from external results (Ville's inequality, Koolen, Balsubramani, Waudby-Smith and Ramdas, Vovk and Wang, Chugg et al.) and are reproduced in the appendix rather than imported as black boxes. No step in the derivation chain reduces by construction to its own inputs.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The paper introduces a framework and definitions (report-to-incidence ratio, preponderance test) but no new physical or causal entities; the key unobservables (rho, gammaTR, gammaFR) are already present in the reporting model rather than newly postulated. The central assumptions are modeled as user-specified bounds, not as entities with independent evidence.

free parameters (4)

beta (overrepresentation threshold) = 2.0, 2.5, 3.0 (vaccine); 1.2, 1.4, 1.6, 1.8 (mortgage)
Central test threshold; user-set based on assumed reporting behavior, not fitted to outcome. Different beta change both stopping time and the implied severity threshold.
alpha (test level) = 0.1 in all experiments
Standard significance level; user-set.
minimum stopping time = 25 (vaccine), 50 (mortgage)
Heuristic minimum applied to the asymptotically-valid Z-test to reduce early spurious rejections; not derived from the theory.
b (report-to-incidence ratio bound) = assumed 1.25 in the Section 3.1 example; implied 1, 7/6, 4/3, 3/2 in mortgage experiments
Used in Proposition 3.2 to convert a preponderance flag into a lower bound on relative risk; not estimated from data.

assumptions (6)

standard math Ville's inequality and sub-Gaussian/Hoeffding concentration bounds
Used to prove Theorem 4.1 and Theorem 4.2 in Appendix A.
standard math Online Newton Step regret bound (Cutkosky and Orabona 2018)
Used to set lambda_t in the betting-style test (Equation 5) and to bound its expected stopping time.
domain assumption Assumption 2.1: base preponderances mu0_G are known for every group G
The hypothesis test compares report preponderance to this quantity; in practice it is estimated or imputed, which introduces error not modeled in the main theory.
domain assumption Reports are i.i.d. across time from an underlying reporting distribution
Section 2, footnote 1; the authors say in Appendix B it can be relaxed, but the main analysis assumes it.
domain assumption Reporting-behavior bounds: rho_G <= b*rho in Prop 3.2; gammaTR_G > gammaFR_G and bounded gamma values in Prop 3.4
These are unverifiable from reporting data alone and are what allow the conversion from report preponderance to true risk; if violated, the harm interpretation of a flag can fail.
domain assumption In the vaccine case study, intersection vaccination rates are imputed by multiplying marginal age and sex rates
Section 5.1: 'We thus impute the vaccination rates for intersections of subgroups by multiplying the known marginal rates.' This assumes independence of age and sex in vaccination coverage.

how reviews work

0 comments

Cite this review

Pith. "Pith review of From Individual Experience to Collective Evidence: A Reporting-Based Framework for Identifying Systemic Harms." pith.science (2026). https://pith.science/paper/4LFOCCPW

@misc{pith2026250208166,
  author       = {Pith},
  title        = {Pith review of: From Individual Experience to Collective Evidence: A Reporting-Based Framework for Identifying Systemic Harms},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4LFOCCPW}},
  note         = {Machine review of arXiv:2502.08166}
}

read the original abstract

When an individual reports a negative interaction with some system, how can their personal experience be contextualized within broader patterns of system behavior? We study the reporting database problem, where individual reports of adverse events arrive sequentially, and are aggregated over time. In this work, our goal is to identify whether there are subgroups--defined by any combination of relevant features--that are disproportionately likely to experience harmful interactions with the system. We formalize this problem as a sequential hypothesis test, and identify conditions on reporting behavior that are sufficient for making inferences about disparities in true rates of harm across subgroups. We show that algorithms for sequential hypothesis tests can be applied to this problem with a standard multiple testing correction. We then demonstrate our method on real-world datasets, including mortgage decisions and vaccine side effects; on each, our method (re-)identifies subgroups known to experience disproportionate harm using only a fraction of the data that was initially used to discover them.

Figures

Figures reproduced from arXiv: 2502.08166 by the authors.

**Figure 2.** Number of reports (t) it takes for each algorithm to reject the null hypothesis for any group (i.e. first identification of harm), over 100 random permutations of COVID-19 vaccine report database. Tests are run with β = 2. Each point on the plot reflects the number of trials (out of 100) in which a rejection has occured by time t. To explore the robustness of these results, we also run synthetic experiments, perm… view at source ↗

**Figure 3.** Impact of multiple hypothesis correction on stopping time across algorithms. As in Figure [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 15 canonical work pages

[1]

ωG⋆ t ≤ E[ωG⋆ t ] − C1 s t ln (2 + log2(t))2 δ # ≤ Pr

For the second claim, we apply Hoeffding’s inequality again to see that for all t, Pr " ωG⋆ t ≤ E[ωG⋆ t ] − C1 s t ln (2 + log2(t))2 δ # ≤ Pr " ωG⋆ t ≤ E[ωG⋆ t ] − s t 2 ln 1 δ # ≤ δ. Thus, with probability at least 1 − δ, for all t simultaneously, ωG⋆ t > tµG⋆ − C1 r t ln (2+log2(t))2 δ . The algorithm stops at time t if and only if tµG⋆ − C1 s t ln (2 +...

work page 2008
[5]

Ira Globus-Harris, Michael Kearns, and Aaron Roth

URL https://news.bloomberglaw.com/daily-labor-report/ workday-ai-biased-against-black-disabled-applicants-suit-says . Ira Globus-Harris, Michael Kearns, and Aaron Roth. An algorithmic framework for bias bounties. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency , pages 1106– 1124,

work page 2022
[9]

Towards ai accountability infrastructure: Gaps and opportunities in ai audit tooling

Victor Ojewale, Ryan Steed, Briana Vecchione, Abeba Birhane, and Inioluwa Deborah Raji. Towards ai accountability infrastructure: Gaps and opportunities in ai audit tooling. arXiv preprint arXiv:2402.17861,

arXiv
[10]

Myocarditis cases reported after mrna-based covid-19 vaccination in the us from december 2020 to august 2021.Jama, 327(4):331–340,

Matthew E Oster, David K Shay, John R Su, Julianne Gee, C Buddy Creech, Karen R Broder, Kathryn Edwards, Jonathan H Soslow, Jeffrey M Dendy, Elizabeth Schlaudecker, et al. Myocarditis cases reported after mrna-based covid-19 vaccination in the us from december 2020 to august 2021.Jama, 327(4):331–340,

work page 2020
[14]

Fairness-aware model-agnostic positive and unlabeled learning

Ziwei Wu and Jingrui He. Fairness-aware model-agnostic positive and unlabeled learning. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency , pages 1698–1708,

work page 2022
[18]

HG 0 holds] ≤ X G s.t

Finally, by union bound we get the desired guarantee: Pr[∃t : ∃G ∈ GFlag s.t. HG 0 holds] ≤ X G s.t. HG 0 holds Pr[∃t : ωG t > log (|G|/α)] ≤ |G s.t. HG 0 holds| ·α/|G| ≤ α. Before proving Theorem 4.4, we first state and prove some helper results. 10In particular, it is imposed that λG t be ’predictable’ which precisely implies that it is fixed given Ft. ...

work page 2018
[19]

Since T is a positive integer, we can express the expected stopping time as E[T ] = ∞X t=1 Pr[T > t] = ∞X t=1 Pr[¬E1 ∧

24 Expected Stopping Time. Since T is a positive integer, we can express the expected stopping time as E[T ] = ∞X t=1 Pr[T > t] = ∞X t=1 Pr[¬E1 ∧ . . .∧ ¬Et] ≤ ∞X t=1 Pr[¬Et] = ∞X t=1 Pr[∀G, ωG t < ln(|G|/α)] ≤ ∞X t=1 Pr h ωG⋆ t < ln(|G|/α) i ≤ max 24 ω2⋆ , 22 · ln(|G|/α) ω⋆ + ∞X t=1 exp − (1 − βµ0 G⋆ )2 2 · ω2 ⋆ · t (9) = max 24 ω2⋆ , 22 · ln(|G|/α) ω⋆ +...

work page 2024
[1994]

Statistical inference for fairness auditing

John J Cherian and Emmanuel J Cand` es. Statistical inference for fairness auditing. arXiv preprint arXiv:2305.03712,

arXiv

Show all 19 references

[2003]

Theorem 4.1 (Validity)

19 A Omitted Proofs A.1 Omitted proofs for Sequential Z-test We prove Theorem 4.1, restated below. Theorem 4.1 (Validity). Running Algorithm 1 with θt(α) as in Equation (3), setting C = 1/2, and ωG t updated as in Equation (2), guarantees that the probability that GFlag will e...

1939
[2011]

Challenging the Machine: Contestability in Government AI Systems

Susan Landau, James X Dempsey, Ece Kamar, Steven M Bellovin, and Robert Pool. Challenging the Machine: Contestability in Government AI Systems. arXiv preprint arXiv:2406.10430 ,

arXiv
[2015]

Why we need to know more: Exploring the state of ai incident documen- tation practices

18 Violet Turri and Rachel Dzombak. Why we need to know more: Exploring the state of ai incident documen- tation practices. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society , pages 576–583,

2023
[2016]

Multicalibration: Calibration for the (computationally-identifiable) masses

Ursula H´ ebert-Johnson, Michael Kim, Omer Reingold, and Guy Rothblum. Multicalibration: Calibration for the (computationally-identifiable) masses. In International Conference on Machine Learning , pages 1939–1948. PMLR,

1939
[2017]

[2014]); we reproduce the argument in the context of our work below, though we emphasize that we do not claim the proof technique as ours

(which itself draws from Balsub- ramani [2014], and is a refinement of Jamieson et al. [2014]); we reproduce the argument in the context of our work below, though we emphasize that we do not claim the proof technique as ours. Proof of Theorem 4.1. It is sufficient to show that...

2014
[2018]

Learning with multi-group guarantees for clusterable subpop- ulations

Jessica Dai, Nika Haghtalab, and Eric Zhao. Learning with multi-group guarantees for clusterable subpop- ulations. arXiv preprint arXiv:2410.14588 ,

arXiv
[2019]

Contestability in al- gorithmic systems

Kristen Vaccaro, Karrie Karahalios, Deirdre K Mulligan, Daniel Kluttz, and Tad Hirsch. Contestability in al- gorithmic systems. In Companion Publication of the 2019 Conference on Computer Supported Cooperative Work and Social Computing , pages 523–527,

2019
[2021]

URLhttps: //www.cnbc.com/2021/05/10/pfizer-covid-vaccine-fda-clears-use-in-kids-ages-12-to-15. html. Mayme Marshall, Ian D Ferguson, Paul Lewis, Preeti Jaggi, Christina Gagliardo, James Stewart Collins, Robin Shaughnessy, Rachel Caron, Cristina Fuss, Kathleen Jo E Corbin, et a...

2021
[2022]

Outsider oversight: Designing a third party audit ecosystem for ai governance

Inioluwa Deborah Raji, Peggy Xu, Colleen Honigsberg, and Daniel Ho. Outsider oversight: Designing a third party audit ecosystem for ai governance. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society , pages 557–571,

2022
[2023]

Multiple testing under negative dependence

Ziyu Chi, Aaditya Ramdas, and Ruodu Wang. Multiple testing under negative dependence. arXiv preprint arXiv:2212.09706,

arXiv
[2024]

Akshay Balsubramani

URL https://documents.un.org/doc/undoc/ltd/n24/065/92/pdf/ n2406592.pdf. Akshay Balsubramani. Sharp finite-time iterated-logarithm martingale concentration. arXiv preprint arXiv:1405.2639,

arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

ωG⋆ t ≤ E[ωG⋆ t ] − C1 s t ln (2 + log2(t))2 δ # ≤ Pr

For the second claim, we apply Hoeffding’s inequality again to see that for all t, Pr " ωG⋆ t ≤ E[ωG⋆ t ] − C1 s t ln (2 + log2(t))2 δ # ≤ Pr " ωG⋆ t ≤ E[ωG⋆ t ] − s t 2 ln 1 δ # ≤ δ. Thus, with probability at least 1 − δ, for all t simultaneously, ωG⋆ t > tµG⋆ − C1 r t ln (2+log2(t))2 δ . The algorithm stops at time t if and only if tµG⋆ − C1 s t ln (2 +...

work page 2008

[2] [5]

Ira Globus-Harris, Michael Kearns, and Aaron Roth

URL https://news.bloomberglaw.com/daily-labor-report/ workday-ai-biased-against-black-disabled-applicants-suit-says . Ira Globus-Harris, Michael Kearns, and Aaron Roth. An algorithmic framework for bias bounties. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency , pages 1106– 1124,

work page 2022

[3] [9]

Towards ai accountability infrastructure: Gaps and opportunities in ai audit tooling

Victor Ojewale, Ryan Steed, Briana Vecchione, Abeba Birhane, and Inioluwa Deborah Raji. Towards ai accountability infrastructure: Gaps and opportunities in ai audit tooling. arXiv preprint arXiv:2402.17861,

arXiv

[4] [10]

Myocarditis cases reported after mrna-based covid-19 vaccination in the us from december 2020 to august 2021.Jama, 327(4):331–340,

Matthew E Oster, David K Shay, John R Su, Julianne Gee, C Buddy Creech, Karen R Broder, Kathryn Edwards, Jonathan H Soslow, Jeffrey M Dendy, Elizabeth Schlaudecker, et al. Myocarditis cases reported after mrna-based covid-19 vaccination in the us from december 2020 to august 2021.Jama, 327(4):331–340,

work page 2020

[5] [14]

Fairness-aware model-agnostic positive and unlabeled learning

Ziwei Wu and Jingrui He. Fairness-aware model-agnostic positive and unlabeled learning. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency , pages 1698–1708,

work page 2022

[6] [18]

HG 0 holds] ≤ X G s.t

Finally, by union bound we get the desired guarantee: Pr[∃t : ∃G ∈ GFlag s.t. HG 0 holds] ≤ X G s.t. HG 0 holds Pr[∃t : ωG t > log (|G|/α)] ≤ |G s.t. HG 0 holds| ·α/|G| ≤ α. Before proving Theorem 4.4, we first state and prove some helper results. 10In particular, it is imposed that λG t be ’predictable’ which precisely implies that it is fixed given Ft. ...

work page 2018

[7] [19]

Since T is a positive integer, we can express the expected stopping time as E[T ] = ∞X t=1 Pr[T > t] = ∞X t=1 Pr[¬E1 ∧

24 Expected Stopping Time. Since T is a positive integer, we can express the expected stopping time as E[T ] = ∞X t=1 Pr[T > t] = ∞X t=1 Pr[¬E1 ∧ . . .∧ ¬Et] ≤ ∞X t=1 Pr[¬Et] = ∞X t=1 Pr[∀G, ωG t < ln(|G|/α)] ≤ ∞X t=1 Pr h ωG⋆ t < ln(|G|/α) i ≤ max 24 ω2⋆ , 22 · ln(|G|/α) ω⋆ + ∞X t=1 exp − (1 − βµ0 G⋆ )2 2 · ω2 ⋆ · t (9) = max 24 ω2⋆ , 22 · ln(|G|/α) ω⋆ +...

work page 2024

[8] [1994]

Statistical inference for fairness auditing

John J Cherian and Emmanuel J Cand` es. Statistical inference for fairness auditing. arXiv preprint arXiv:2305.03712,

arXiv