Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling

Abeba Birhane; Briana Vecchione; Inioluwa Deborah Raji; Ryan Steed; Victor Ojewale

Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.17861 v3 pith:ROILCGCX submitted 2024-02-27 cs.CY

Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling

Victor Ojewale , Ryan Steed , Briana Vecchione , Abeba Birhane , Inioluwa Deborah Raji This is my paper

classification cs.CY

keywords audittoolspractitionersaccountabilityauditsbeyondeffortsevaluation

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

0 comments

read the original abstract

Audits are critical mechanisms for identifying the risks and limitations of deployed artificial intelligence (AI) systems. However, the effective execution of AI audits remains incredibly difficult, and practitioners often need to make use of various tools to support their efforts. Drawing on interviews with 35 AI audit practitioners and a landscape analysis of 435 tools, we compare the current ecosystem of AI audit tooling to practitioner needs. While many tools are designed to help set standards and evaluate AI systems, they often fall short in supporting accountability. We outline challenges practitioners faced in their efforts to use AI audit tools and highlight areas for future tool development beyond evaluation -- from harms discovery to advocacy. We conclude that the available resources do not currently support the full scope of AI audit practitioners' needs and recommend that the field move beyond tools for just evaluation and towards more comprehensive infrastructure for AI accountability.

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

White Box Evidence Packages for Policy Audit Reports
cs.CY 2026-07 conditional novelty 6.0

In a 60-case controlled audit study, adding white-box model evidence to an LLM auditor increased citation volume but weakened passage grounding and raised evidence misuse, while a shuffled control showed reports can s...