Aggregated Individual Reporting for Post-Deployment Evaluation

Benjamin Recht; Inioluwa Deborah Raji; Irene Y. Chen; Jessica Dai

REVIEW 3 major objections 5 minor 1 cited by

Aggregated Individual Reporting for Post-Deployment Evaluation

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that post-deployment AI evaluation should run on aggregated reports from affected people, and that such a mechanism, called AIR, is a practical path to surfacing harms that static benchmarks miss.

desk verdict A clear, honest position paper that formalizes aggregated individual reporting as a framework for post-deployment evaluation; the practical-pathway claim rests on unproven reporting participation, but the authors openly flag it. read the letter →

arxiv 2506.18133 v2 pith:TUX4M5AX submitted 2025-06-22 cs.CY

Jessica Dai , Inioluwa Deborah Raji , Benjamin Recht , Irene Y. Chen This is my paper · ORCID

classification cs.CY

keywords aggregatedindividualreportingpost-deploymentevaluationAIaccountabilityalgorithmicharmdiscoverypublicmechanismsdemocraticcrowdsourcedfeedbacksequentialaggregation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that post-deployment evaluation of AI systems should treat individual experiences of harm as core evidence, and that aggregating those experiences over time into a shared framework is a practical path to doing so. It proposes aggregated individual reporting (AIR): a mechanism in which anyone affected by a deployed system can submit a report about a problematic interaction, the reports are aggregated over time to produce a fine-grained evaluation, and the evaluation triggers predetermined action when patterns emerge. The authors claim this approach surfaces 'unknown unknowns' that centralized benchmarks and incident databases miss, and that aggregation is necessary (though not necessarily sufficient) for moving from anecdotes to high-level action. They position AIR as the missing post-deployment piece of 'democratic AI' debates, which have focused on upfront design rather than ongoing accountability. The paper does not report an experiment; it makes a conceptual and methodological case for building such systems.

What carries the argument

The central object is the AIR mechanism itself: a three-component structure—individual reporting, aggregation for evaluation, and evaluation-conditional action—plus the actors (evaluated system, affected population, mechanism administrator) and a design space of organizational, reporting, and aggregation choices. The argument's engine is the contrast between per-report resolution, which dominates current systems like complaint databases and bug bounties, and aggregate pattern detection, which the paper says is required for harms that only appear at group level, such as disproportionate impact on a subgroup. The paper also introduces an evaluation condition: a pre-specified aggregate outcome that, if reached, triggers downstream action; in the running methodological example this is a sequential hypothesis test that rejects the null of no overrepresented harm.

What would settle it

Deploy a pilot AIR for a real system with a known but subtle failure mode that affects a small subgroup, then measure whether the aggregate evaluation reaches its pre-specified trigger within a defined time window; if reports never accumulate enough to trigger action because participation is near zero or too biased, the claim that AIR is a practical pathway is empirically refuted.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is a definitional claim: post-deployment evaluation must include the perspectives of those affected by a deployed system, and a mechanism that collects individual reports, aggregates them over time, and makes downstream action conditional on the aggregate evaluation is a concrete way to meet that obligation. The paper names this aggregated individual reporting (AIR) and specifies three load-bearing components: individual reporting by the affected population; aggregation for evaluation, so that individual experiences become collective evidence; and evaluation-conditional action, so that the mechanism can actually change system behavior or pressure its operator. It argues that existing approaches—incident databases, usage clustering, chat rankings, bug bounties, red-teaming—each cover only part of this loop, and it defends the claim with examples from vaccine safety, aviation, consumer finance, and social-media-driven rollbacks. The paper also contends that AIR closes a gap in 'democratic AI' scholarship by giving the public an ongoing, post-deployment channel to revoke consent, not just a pre-deployment seat at the design table.

Load-bearing premise

The framework's usefulness depends on enough members of the affected population actually submitting informative, honest reports; the paper itself concedes that people may not report if the process is burdensome or unknown, and if participation is too low, aggregation cannot reveal patterns or trigger action.

Editorial extensions

If this is right

If AIR is implemented, failures that are too niche to go viral—like the mental-health spiral case the paper contrasts with a widely reported rollback—could still surface and trigger action.
Independent third-party aggregation produces statistical evidence that can empower affected groups and pressure operators, as the paper argues happened with crowdsourced rideshare wage data.
AIR fills the post-deployment accountability gap in participatory and pluralistic AI proposals, which currently concentrate on model development and training objectives.
AIR is complementary to, not a replacement for, incident databases, usage analytics, and flaw disclosure; each covers a different part of the evaluation ecosystem.
The framework gives high-level policy mandates, such as those in recent AI regulations, a concrete structure for collecting and acting on public feedback.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same mechanism could be piloted in low-stakes settings first—a recommendation system or a customer-service chatbot—to empirically measure reporting rates and bias before applying it to consequential domains.
Aggregation methods from sequential statistics could be coupled with natural-language processing of free-text reports, a combination the paper gestures at but leaves as an open research question.
If participation is the load-bearing premise, AIR design should treat recruitment and retention as first-class problems, borrowing from study-recruitment literature rather than assuming reports arrive automatically and representatively.
The framework may generalize beyond AI systems to any deployed algorithmic or organizational process where affected people can describe harm; the paper's own examples already span vaccines and consumer finance.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read this one if you work on AI evaluation or governance. The paper's contribution is definitional: it names and combines three components—individual reporting, aggregation for evaluation, and evaluation-conditional action—into a single 'AIR' framework, and it gives a useful taxonomy of administrators (first/second/third party). That is genuinely new as a package, even though each ingredient exists in VAERS, ASRS, CFPB, incident databases, and user-driven auditing work. The authors are clear about what they are not claiming: they do not originate reporting, and they carefully distinguish AIR from flaw disclosure and from incident databases.

The paper is strongest where you'd expect a position paper to be strong: it argues coherently that individual reports surface unknown unknowns, that aggregation is necessary for actionability (with the ChatGPT sycophancy episode as a vivid case), and that post-deployment input is a missing piece in the 'democratic AI' conversation. The writing is clear and the limitations section is unusually candid—they list noise/gaming, non-participation, first-party incentive misalignment, and third-party sustainability as real failure modes.

The soft spot is the one the stress-test note identifies, but I don't think the paper needs to be knocked for it. The central practical claim—that AIR is 'a practical pathway'—requires enough informative reports at sufficient volume, and the paper concedes in §6.1(1b) that people may not submit even if the mechanism exists. The supporting cases don't close that gap: VAERS has structured clinical reporting behind it, CFPB offers per-complaint remedy, and the ChatGPT case ran on Twitter virality rather than a purpose-built AIR. So the empirical premise is open. But the paper's goal is to define the framework and lay out design decisions and open questions, not to pilot it. Framed that way, the limitation is proportionate. There's also a broken citation placeholder ('??') in §5.2, and a few typos ('fo', 'as as').

Who is this for? Researchers and policymakers thinking about post-deployment monitoring and public-input mechanisms. It's a serious, well-situated piece of conceptual work, and it deserves a real referee rather than a desk reject. I'd suggest the editor send it to review with the expectation that the authors trim the 'practical pathway' claim to match the evidence, or at least sharpen what would count as validation. I'd cite it if I were writing about post-deployment evaluation.

Referee Report

3 major / 5 minor

Summary. This position paper proposes a framework called Aggregated Individual Reporting (AIR) for post-deployment evaluation of AI systems. An AIR mechanism is defined by three components: individual reporting by members of an affected population, aggregation of reports over time for fine-grained evaluation, and evaluation-conditional action by a mechanism administrator. The paper argues that individual reports can surface unknown unknowns, that aggregation is a prerequisite for high-level action, that such mechanisms complete a missing piece in discussions of 'democratic' AI, and that AIR is a practical pathway to post-deployment evaluation. It supports these claims with examples (the GPT-4o sycophancy rollback, VAERS, the CFPB complaint database, RegretsReporter, Fairfare), contrasts AIR with existing incident databases and flaw disclosure mechanisms, and outlines a detailed set of design decisions and open research questions in Section 5.

Significance. The paper's main contribution is conceptual rather than empirical: it offers a clear, coherent definition of a distinct mechanism, locates it precisely with respect to existing work on incident databases, user-driven auditing, and policy mandates, and connects it to normative arguments about accountability and democratic legitimacy. The design-decision tables in Figures 3 and 4 and the candid treatment of limitations in Section 6 are valuable starting points for a research agenda. The significance would be substantial if the practical-pathway claim were established, because AIR could provide shared vocabulary and structure for public-input-driven post-deployment evaluation. The paper is transparent that the key feasibility questions are open; this honesty is a strength, but it also means the central practical claim is currently asserted rather than demonstrated. I found no circularity in the framework's definition or internal inconsistency in its main argument.

major comments (3)

[Abstract and §6.1(1b)] The central claim that AIR is 'a practical pathway' to post-deployment evaluation rests on the load-bearing premise that affected people will submit enough informative reports, but that premise is explicitly left open. Section 6.1(1b) states that 'People may not submit reports even if the mechanism technically exists—e.g., if reporting is too burdensome, or if affected populations are unaware of the option to report,' and Section 6.2 concludes that 'the success of AIRs is a fundamentally empirical question.' As written, the abstract and Section 1 assert a stronger practical-pathway claim than the evidence supports. I recommend either (a) providing at least one end-to-end case in which a formal AIR-style mechanism for an AI system surfaced non-viral harms and led to action, or (b) reframing the contribution as a conceptual framework and research agenda, with feasibility explicitly labeled as an open empirical question.
[§3.2] The claim that 'aggregation is a necessary (though potentially not sufficient) condition for taking high-level action from reports' is stronger than the presented evidence. The case studies of CFPB, VAERS, RegretsReporter, Fairfare, and the Twitter quasi-aggregation show that aggregation can accompany or enable action in some settings, but they do not establish necessity; one can imagine high-level action taken in response to a single severe or uniquely diagnostic report. Since this necessity claim anchors the third component of the definition (evaluation-conditional action) and the distinction from incident databases, I suggest either replacing 'necessary' with 'an important enabler' or providing a counterfactual-based argument for why no high-level action can occur without aggregation.
[§3.1 and Figure 2] The motivating GPT-4o case is presented as evidence that individual reporting can surface harms, but the actual mechanism was informal Twitter virality combined with a quasi-aggregation by the timeline algorithm, not a purpose-built AIR; the paper acknowledges this in Section 3.1 but continues to use the example as a success story. Similarly, VAERS operates with regulatory backing and structured clinical reporting, and CFPB offers per-complaint remedy, neither of which is guaranteed in a first-party AI reporting context. The gap between these precedents and the proposed AI setting is precisely the empirical risk identified in Section 6.1(1b), and it should be addressed directly in the argument for practical viability rather than only in the limitations section.

minor comments (5)

[§5.2] The sentence on motivations and incentives contains an incomplete citation: 'theoretical models (e.g., ??).' This should be replaced with the intended reference or removed.
[Throughout] There are several typos and spacing errors: Section 4 has 'mechanisms fo' for 'mechanisms for' and 'explicitliy' for 'explicitly'; Section 5.1 has 'reponse' and 'ued'; footnote 3 has 'are are only briefly mentioned'; and the text contains stray spaces before commas (e.g., 'Intuitively ,').
[Figure 2] In the VAERS row, the entry under 'Individual report information' reads 'specific adverse event (e.g., ) and demographic information'; the parenthetical is empty and should be completed or removed.
[§6.1(1)] The sentence 'By design, feedback collected by AIRs is “one-sided,’ in the sense that, and moreover, relies on usage or adoption at scale' is ungrammatical and should be rewritten to state what 'one-sided' means.
[Abstract and §1] The phrase 'the scope of our proposed aggregated individual reporting mechanism is a practical path' is awkward; 'scope' seems to be the wrong word here, and 'practical path' is asserted before the evidence is discussed. Consider writing 'the proposed mechanism is a candidate pathway' to match the paper's own epistemic stance.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the paper is a position argument, not a derivation, and its only author self-citation is illustrative rather than load-bearing.

full rationale

This is a position paper with no mathematical derivation, fitted parameters, or quantitative predictions, so the fitting-to-input and prediction-renaming forms of circularity do not apply. The AIR definition names three components (individual reporting, aggregation for evaluation, evaluation-conditional action), and the paper explicitly applies this definition to existing systems such as VAERS, ASRS, CFPB, RegretsReporter, and Fairfare, while also distinguishing AIR from incident databases and flaw disclosure mechanisms. The claim that aggregation enables high-level action is supported by external case comparisons (e.g., VAERS aggregate monitoring versus CFPB per-complaint resolution), not by the definition alone; the paper also states that aggregation is 'necessary (though potentially not sufficient)' and concedes real-world feasibility is open. The central 'practical pathway' claim is explicitly hedged: Section 6.1(1b) concedes that 'People may not submit reports even if the mechanism technically exists,' and the conclusion states 'the success of AIRs is a fundamentally empirical question.' The only same-author self-citation is Dai et al. [2025], used as a running example of a compatible methodological proposal and as a hypothetical loan-allocation illustration; removing it would not change the argument's structure, so it is not load-bearing. One manuscript defect is a dangling '??' reference in Section 5.2 ('via theoretical models (e.g., ??)'), but that is a completeness issue, not circularity. No equations, no imported uniqueness theorems, and no ansatz smuggled in via citation were found. The score reflects one minor, non-load-bearing self-citation rather than any substantive circular step.

Assumptions & free parameters 0 free parameters · 5 assumptions · 1 invented entities

All components of the central proposal rest on assumptions about reporting behavior and the relationship between reports and system harm. These are not proven or empirically tested in the paper; the authors identify several as open questions in Section 6.1. The free-parameter list is empty because no quantitative model or fit is presented.

assumptions (5)

domain assumption Individual reports from the affected population contain valid, useful information about real harms caused by the evaluated system.
The paper's framework depends on the premise that reports reflect genuine problems rather than mere noise, and that reporters have unique knowledge of failures (Sections 3.1 and 6.1).
domain assumption Members of the affected population will submit reports in sufficient quantity and quality for aggregation to identify patterns.
The entire mechanism requires reporting participation; the paper itself flags this as an open concern in Section 6.1 (1b) but does not resolve it.
domain assumption Aggregation of reports is necessary (though possibly not sufficient) for downstream action and accountability.
Section 3.2 argues this from case studies; it is an empirical premise, not a proven theorem.
domain assumption The analogy between democratic governance and AI system governance is valid enough that post-deployment consent and accountability are meaningful requirements.
The normative argument in Section 4 relies on treating users like citizens and AI operators like governments; this is a contested analogy.
domain assumption Existing post-deployment approaches (incident databases, Clio, bug bounties) are insufficient for fine-grained, actionable evaluation, so a new mechanism adds value.
Section 2.2 claims AIRs enable distinct evaluations; this is a comparison claim based on current systems, not independence.

invented entities (1)

Aggregated individual reporting (AIR) mechanism
purpose: Structured channel for members of the public to report problematic experiences with a deployed AI system, aggregate them over time, and trigger evaluation-conditional action.
Introduced as an idealized framework (Section 2.1); no deployed implementation exists, so there is no falsifiable handle outside the paper.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Aggregated Individual Reporting for Post-Deployment Evaluation." pith.science (2026). https://pith.science/paper/TUX4M5AX

@misc{pith2026250618133,
  author       = {Pith},
  title        = {Pith review of: Aggregated Individual Reporting for Post-Deployment Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TUX4M5AX}},
  note         = {Machine review of arXiv:2506.18133}
}

read the original abstract

The need for developing model evaluations beyond static benchmarking, especially in the post-deployment phase, is now well-understood. At the same time, concerns about the concentration of power in deployed AI systems have sparked a keen interest in 'democratic' or 'public' AI. In this work, we bring these two ideas together by proposing mechanisms for aggregated individual reporting (AIR), a framework for post-deployment evaluation that relies on individual reports from the public. An AIR mechanism allows those who interact with a specific, deployed (AI) system to report when they feel that they may have experienced something problematic; these reports are then aggregated over time, with the goal of evaluating the relevant system in a fine-grained manner. This position paper argues that individual experiences should be understood as an integral part of post-deployment evaluation, and that the scope of our proposed aggregated individual reporting mechanism is a practical path to that end. On the one hand, individual reporting can identify substantively novel insights about safety and performance; on the other, aggregation can be uniquely useful for informing action. From a normative perspective, the post-deployment phase completes a missing piece in the conversation about 'democratic' AI. As a pathway to implementation, we provide a workflow of concrete design decisions and pointers to areas requiring further research and methodological development.

Figures

Figures reproduced from arXiv: 2506.18133 by the authors.

**Figure 1.** The components of our framework and how they interact. 2 Defining aggregated individual reporting We begin by describing our idealized vision of a mechanism for aggregated individual reporting, illustrated in [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

**Figure 2.** Examples of how AIRs could be set up for a variety of applications. Here, we elide the corresponding aggregation methods in order to focus on the application itself; note, however, that the “evaluation conditions” are aggregate system failures rather than per-report problems. 4 [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Organizational and interaction-focused questions. include more information depending on the application, such as medical history (for a vaccine or pharmaceutical system) or financial background information (for a loan allocation system). Reporting behavior. Due to the nature of reporting data it is, intrinsically, essential to understand reporting behavior: what factors affect the decision to submit a report, and, c… view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Methodology-focused questions. at different rates? Do different types of issues lead to different reporting behaviors? In Dai et al. [2025], the choice is to commit to a set of (quantitative) assumptions about the extent to which reporting rates can vary, and incorpora…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Position: Use Sparse Autoencoders to Discover Unknowns
cs.LG 2025-06 conditional novelty 6.0 of 10

Sparse autoencoders are best used to discover unknown concepts, not to act on known concepts.

Reference graph

Works this paper leans on

28 extracted references · 20 canonical work pages · cited by 1 Pith paper

[6]

W. H. Deng, B. Guo, A. Devrio, H. Shen, M. Eslami, and K. Holstein. Understanding practices, challenges, and opportunities for user-engaged algorithm auditing in industry practice. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems , pages 1–18,

work page 2023
[7]

DeVos, A

A. DeVos, A. Dhabalia, H. Shen, K. Holstein, and M. Eslami. Toward user-driven algorithm auditing: Investigating users’ strategies for uncovering harmful algorithmic behavior. In Proceedings of the 2022 CHI conference on human factors in computing systems, pages 1–19,

work page 2022
[11]

Globus-Harris, M

18 I. Globus-Harris, M. Kearns, and A. Roth. An algorithmic framework for bias bounties. InProceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency , pages 1106–1124,

work page 2022
[12]

Globus-Harris, D

I. Globus-Harris, D. Harrison, M. Kearns, P. Perona, and A. Roth. Diversified ensembling: An experiment in crowdsourced machine learning. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 529–545,

work page 2024
[14]

Huang, D

S. Huang, D. Siddarth, L. Lovitt, T. I. Liao, E. Durmus, A. Tamkin, and D. Ganguli. Collective Constitutional AI: Aligning a Language Model with Public Input. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 1395–1417,

work page 2024
[16]

URL https://www.theverge.com/2021/7/7/22567640/ youtube-algorithm-suggestions-radicalization-mozilla . A. Littwin. Why process complaints: Then and now consumer. Temple Law Review, 87:895,

work page 2021
[18]

Movva, K

R. Movva, K. Peng, N. Garg, J. Kleinberg, and E. Pierson. Sparse autoencoders for hypothesis generation. arXiv preprint arXiv:2502.04382,

arXiv
[19]

URL https://oecd.ai/en/incidents. V. Ojewale, R. Steed, B. Vecchione, A. Birhane, and I. D. Raji. Towards ai accountability infrastruc- ture: Gaps and opportunities in ai audit tooling. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pages 1–29,

work page 2025

Show all 28 references

[20]

Expanding on what we missed with sycophancy

OpenAI. Expanding on what we missed with sycophancy. OpenAI Blog, May 2025a. URL https: //openai.com/index/expanding-on-sycophancy/. 20 OpenAI. Sycophancy in gpt-4o: what happened and what we’re doing about it, April 2025b. URL https://openai.com/index/sycophancy-in-gpt-4o/ . ...

2020
[21]

Ovadya, L

A. Ovadya, L. Thorburn, K. Redman, F. Devine, S. Milli, M. Revel, A. Konya, and A. Kasirzadeh. Toward democracy levels for ai. arXiv preprint arXiv:2411.09222,

arXiv
[22]

I. D. Raji, P. Xu, C. Honigsberg, and D. Ho. Outsider oversight: Designing a third party audit ecosystem for ai governance. In Proceedings of the 2022 AAAI/ACM Conference on AI, Ethics, and Society, pages 557–571,

2022
[23]

V. N. Rao, E. Agarwal, S. Dalal, D. Calacci, and A. Monroy-Hernández. Quallm: An llm-based framework to extract quantitative insights from online forums. arXiv preprint arXiv:2405.05345,

arXiv
[24]

B. Recht. A bureaucratic theory of statistics. arXiv preprint arXiv:2501.03457,

arXiv
[25]

P. Reviews. Post on x. https://x.com/ReviewsPossum/status/1917292836082397355, April
[27]

Sorensen, L

T. Sorensen, L. Jiang, J. D. Hwang, S. Levine, V. Pyatkin, P. West, N. Dziri, X. Lu, K. Rao, C. Bhaga- vatula, et al. Value kaleidoscope: Engaging AI with pluralistic human values, rights, and duties. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38,...

arXiv
[87]

Feffer, M

M. Feffer, M. Skirpan, Z. Lipton, and H. Heidari. From preference elicitation to participatory ml: A critical survey & guidelines for future research. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pages 38–48,

2023
[1689]

Longpre, K

S. Longpre, K. Klyman, R. E. Appel, S. Kapoor, R. Bommasani, M. Sahar, S. McGregor, A. Ghosh, B. Blili-Hamelin, N. Butters, et al. In-house evaluation is not enough: Towards robust third-party flaw disclosure for general-purpose ai. arXiv preprint arXiv:2503.16861,

arXiv
[1948]

Post on x

Williawa. Post on x. https://x.com/williawa/status/1916935712550551743, April
[1990]

Post on x

Frye. Post on x. https://x.com/___frye/status/1916346474893656572, April
[1997]

Regulation (eu) 2024/1689 of the european parliament and of the council of 13 june 2024 laying down harmonised rules on artificial intelligence (artificial intelligence act),

European Parliament. Regulation (eu) 2024/1689 of the european parliament and of the council of 13 june 2024 laying down harmonised rules on artificial intelligence (artificial intelligence act),

2024
[1999]

Slattery , A

P. Slattery , A. K. Saeri, E. A. Grundy , J. Graham, M. Noetel, R. Uuk, J. Dao, S. Pour, S. Casper, and N. Thompson. The ai risk repository: A comprehensive meta-review, database, and taxonomy of risks from artificial intelligence. arXiv preprint arXiv:2408.12622,

arXiv
[2002]

C. Bertram. Jean Jacques Rousseau. In E. N. Zalta and U. Nodelman, editors, The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University , Summer 2023 edition,

2023
[2012]

URL https: //www.consumerfinance.gov/data-research/consumer-complaints/ . J. Dai and E. Fleisig. Mapping social choice theory to RLHF. arXiv preprint arXiv:2404.13038,

arXiv
[2021]

Post on x

19 Laura. Post on x. https://x.com/IndieLauraSDG/status/1913102627837206587, April
[2022]

Calacci, V

D. Calacci, V. N. Rao, S. Dalal, C. Di, K.-W. Pua, A. Schwartz, D. Spitzberg, and A. Monroy-Hernández. Fairfare: A tool for crowdsourcing rideshare data to empower labor organizers. arXiv preprint arXiv:2502.11273,

arXiv
[2023]

S. Bharath. Post on x. https://x.com/Siddharth87/status/1916999455146185022, April
[2024]

Ahmad, S

L. Ahmad, S. Agarwal, M. Lampe, and P. Mishkin. Openai’s approach to external red teaming for ai models and systems. arXiv preprint arXiv:2503.16431,

arXiv
[2025]

URL https://www.nytimes.com/2025/06/13/technology/ chatgpt-ai-chatbots-conspiracies.html . E. Ho. Post on x. https://x.com/Eddie_Ho2025/status/1913596114156028013, April

2025

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [6]

W. H. Deng, B. Guo, A. Devrio, H. Shen, M. Eslami, and K. Holstein. Understanding practices, challenges, and opportunities for user-engaged algorithm auditing in industry practice. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems , pages 1–18,

work page 2023

[2] [7]

DeVos, A

A. DeVos, A. Dhabalia, H. Shen, K. Holstein, and M. Eslami. Toward user-driven algorithm auditing: Investigating users’ strategies for uncovering harmful algorithmic behavior. In Proceedings of the 2022 CHI conference on human factors in computing systems, pages 1–19,

work page 2022

[3] [11]

Globus-Harris, M

18 I. Globus-Harris, M. Kearns, and A. Roth. An algorithmic framework for bias bounties. InProceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency , pages 1106–1124,

work page 2022

[4] [12]

Globus-Harris, D

I. Globus-Harris, D. Harrison, M. Kearns, P. Perona, and A. Roth. Diversified ensembling: An experiment in crowdsourced machine learning. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 529–545,

work page 2024

[5] [14]

Huang, D

S. Huang, D. Siddarth, L. Lovitt, T. I. Liao, E. Durmus, A. Tamkin, and D. Ganguli. Collective Constitutional AI: Aligning a Language Model with Public Input. In The 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 1395–1417,

work page 2024

[6] [16]

URL https://www.theverge.com/2021/7/7/22567640/ youtube-algorithm-suggestions-radicalization-mozilla . A. Littwin. Why process complaints: Then and now consumer. Temple Law Review, 87:895,

work page 2021

[7] [18]

Movva, K

R. Movva, K. Peng, N. Garg, J. Kleinberg, and E. Pierson. Sparse autoencoders for hypothesis generation. arXiv preprint arXiv:2502.04382,

arXiv

[8] [19]

URL https://oecd.ai/en/incidents. V. Ojewale, R. Steed, B. Vecchione, A. Birhane, and I. D. Raji. Towards ai accountability infrastruc- ture: Gaps and opportunities in ai audit tooling. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pages 1–29,

work page 2025