REVIEW 3 major objections 5 minor 58 references
No single U.S. disclosure regime makes federal AI fully visible; each exposes a different slice, and no persistent identifiers link records across filings or over time.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 04:59 UTC pith:OOZ3MOXT
load-bearing objection The qualitative finding—three federal transparency regimes are fragmented and the AI Use Case Inventory is the best reform lever—is credible and useful, but the headline "80–97% unlinked" numbers rest on a linking pipeline with unmeasured recall and should be treated as upper bounds, not measurements. the 3 major comments →
Triangulating Across U.S. Federal AI Transparency Regimes
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central discovery is that the three U.S. federal AI transparency regimes are structurally non-interoperable: they use different vocabularies, operate on different cycles, and share no common identifiers. The paper finds that between 80% and 97% of records in each corpus have no confirmed cross-regime link, and even a permissive entity-resolution pipeline confirms few same-system matches. Case studies show that cross-regime linking can reconstruct data flows and reveal mission creep, but that even linked records understate the systems' scope and risk. The paper argues these gaps are structural rather than accidental, tracing each regime's disclosure weaknesses to its administrative purpos
What carries the argument
The key mechanism is a cross-document entity resolution pipeline that links records across the three regimes via candidate retrieval (TF-IDF, agency name, and named-entity overlap), probabilistic scoring, LLM reranking of pairs as 'same system,' 'related,' or 'no link,' and agglomerative clustering. Central to the argument is the distinction between same-system links and data-dependency edges, which lets the authors map data supply chains rather than isolated models. Also central is the AI Use Case Inventory itself—an annually published, statutorily required list of federal AI systems—which the paper identifies as the best existing mechanism for public transparency and the target of its poli
Load-bearing premise
The central quantitative claim—that 80–97% of records have no confirmed cross-regime link—depends on the linking pipeline's thresholds and LLM reranking being reliable, but the thresholds were calibrated on only 8 known explicit cross-references, related-class precision was 51% in a small single-annotator sample, and no recall is reported.
What would settle it
Take a random sample of AI systems identified in the 2025 AI Use Case Inventory and manually search the SORN and ICR corpora for any records referring to the same system, using full-text search independent of the linking pipeline; if a large fraction (e.g., more than half) of the systems have hidden cross-regime references, the claim of structural non-interoperability would be overestimated. Alternatively, run a sensitivity analysis with more permissive linking thresholds and see whether the overlap rates change materially.
If this is right
- The public currently cannot track a specific government AI system across filings or over time, undermining accountability for high-impact systems.
- The gaps are structural, not accidental, so piecemeal fixes to individual regimes will not solve the problem.
- Reforming the AI Use Case Inventory—broad definition, persistent identifiers with cross-references, and risk-management transparency—would substantially improve public oversight.
- Even with perfect linking, current disclosures are insufficient; the case studies show systems operating with outdated privacy notices and missing risk-assessment information.
- The linking methodology provides a reusable way to triangulate across records that were not designed to interoperate.
Where Pith is reading between the lines
- Persistent identifiers for AI use cases could become a backbone for linking other government data sources beyond these three regimes, potentially enabling automated oversight.
- The data-supply-chain view suggests that oversight should focus on flows of personal data into AI systems, not just on individual models; this could inspire similar analyses of procurement or grant records.
- The 80–97% unlinked figures might understate fragmentation if the linking pipeline misses true links, or overstate it if false positives inflate confirmed-link counts; a recall study would tighten the estimate.
- The paper's recommendations for the AI Use Case Inventory could be tested before 2027, when the statutory requirement may lapse; an archival public interface would preserve the data for future accountability.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper examines three U.S. federal AI transparency regimes — System of Records Notices (SORNs), Information Collection Requests (ICRs), and the AI Use Case Inventory — and asks how well they individually and jointly disclose government AI use. The authors collect and normalize these corpora, classify AI relevance via zero-shot GPT-5.4-mini prompts, and link records across regimes using a retrieval/scoring/LLM-reranking/clustering pipeline. They present two case studies (Automated Targeting System, Mobile Fortify), report that 80–97% of records have no confirmed cross-regime link, diagnose structural weaknesses (vocabulary divergence, mission creep, documentation lag), and make policy recommendations centered on the AI Use Case Inventory. They also release a public web tool (govAIfiles.org) and code/data.
Significance. If the qualitative findings hold, the paper makes a substantial contribution to the AI governance literature: it is among the first to systematically triangulate three federal disclosure regimes, and its two case studies show concretely how linking records can surface information that no single regime provides. The public search tool and the released pipeline are reproducible assets that will benefit researchers and journalists. The policy recommendations are grounded and actionable. However, the quantitative claim of very low cross-regime overlap — a load-bearing part of the paper's central thesis — is not yet established at the same level of rigor as the case-study analysis.
major comments (3)
- [§4, §7] The headline '80–97% of records have no confirmed cross-regime link' is an upper bound on confirmed links, not a measurement of fragmentation, because the linker's recall is never reported. Thresholds in §4 were calibrated on only 8 known SORN–ICR cross-references; the same-system precision is 85% on 100 pairs and related-class precision is 51%, also on a single annotator's 100 pairs. Without recall, 'no confirmed link' conflates true non-links with pipeline misses. The text in §7 and the Abstract should be reworded (e.g., 'no confirmed link') and recall should be estimated on a labeled set, or the quantitative support for the 'limited overlap' conclusion should be downgraded.
- [§4, Table 1, §7] There are internal count inconsistencies that undermine the reported percentages. §4 gives 591 AI-related SORNs in the cross-product and 4,558 AI Use Case Inventory entries; Table 1 reports 554 AI-related SORNs and total inventory records of 710+2,071+3,542=6,323; §7 uses 554 SORNs and 5,268 inventory entries. These mismatches are not cosmetic: they change the denominators for the overlap statistics. The paper should reconcile these counts or explain which corpus version each number refers to, and the percentages in §7 and Figure 8 should be recomputed consistently.
- [§6, §7] The ATS case study relies heavily on the 'related' category (166 ICRs naming ATS as downstream processor, and the 15,000 related pairs in §7). Since the related class has only 51% precision in the evaluation, roughly half of these labels may be false positives. The case study's data-dependency picture is therefore plausible but not yet quantitatively reliable. The authors should either report a higher-precision human validation for the ATS subgraph or explicitly present the 166/15,000 figures as pipeline suggestions requiring further verification.
minor comments (5)
- [Abstract] Typo: 'Our case studies finds' should be 'Our case studies find.'
- [§4] Typo: 'probablistic scoring' should be 'probabilistic scoring.'
- [§7] 'The ICR corpus tells a similar same story' is awkward; suggest 'The ICR corpus tells a similar story.'
- [Figure 8 caption] The caption notes that only records in the reranker candidate pool are included; this is important because the no-link percentages in the text appear to apply to all records. The text should clarify whether the 80–97% figures are computed over the full AI-related corpora or only over candidate-pool records.
- [§8] The citation '40 U.S.C. 2022' is formatted unlike other legal references; consider using the standard public-law citation used elsewhere in the reference list.
Circularity Check
No significant circularity: the fragmentation findings and policy recommendations are empirical and interpretive outputs, not restatements of the paper's inputs.
full rationale
The paper's central claim is that the three U.S. federal disclosure regimes are structurally non-interoperable and that no single regime fully reveals government AI use. None of the load-bearing quantities is defined in terms of that conclusion. The 80–97% no-link figures are outputs of a separately specified retrieval/scoring/reranking pipeline (Eqs. 1–2, Section 4) rather than inputs; the linking thresholds were calibrated on 8 known explicit SORN–ICR cross-references, but this is a small labeled set for threshold selection and does not make the low overlap rate true by construction. The AI-relevance labels come from hand-validated zero-shot classifiers with reported precision (69.7% SORN, 75% ICR) and prompt definitions stated independently of the conclusions. The ATS and Mobile Fortify case studies are built on direct quotation of SORNs, ICRs, PIAs, and inventory records, and the policy recommendations are grounded in administrative history and observable inventory fields rather than in the linking outputs. The appendix itself flags that its keyword vocabulary is 'heuristic rather than exhaustive' and that document-type patterns 'are not guaranteed to be exhaustive,' which are self-acknowledged measurement limits rather than circular steps. The only self-citation (Friedler & Selbst 2025) supports a peripheral characterization of OMB memoranda and is not load-bearing. Concerns about unmeasured recall and 51% related-class precision are correctness/validity risks, not circularity: an inaccurate measurement is not a definitional reduction, and the paper does not derive its conclusion from the fitted thresholds by construction.
Axiom & Free-Parameter Ledger
free parameters (4)
- Retrieval score weights (0.5 TF-IDF, 0.2 agency, 0.3 named entity) =
0.5 / 0.2 / 0.3; top-20 candidates
- Composite score weights and thresholds (0.25/0.25/0.30/0.15/0.05; high≥0.62, low<0.30) =
0.25/0.25/0.30/0.15/0.05; 0.62; 0.30
- Clustering linkage threshold (group-average, 0.7) =
0.7
- Longitudinal fuzzy-match thresholds (composite 65, name-only 70) =
65 / 70
axioms (4)
- domain assumption The three regimes—SORNs, ICRs, and the AI Use Case Inventory—are the appropriate and sufficient transparency infrastructure for studying federal AI disclosure.
- domain assumption Corpus completeness: the Federal Register API returned all SORNs 2010–2025, the reginfo.gov scrape captured all ICRs Jan 2018–Mar 2026, and agency inventory files are complete and truthful.
- domain assumption Zero-shot LLM classifications with the supplied prompts approximate true AI relevance.
- domain assumption Cross-document entity resolution based on textual similarity, fuzzy names, and LLM reranking identifies the same underlying AI system.
Cite this review
Pith. "Pith review of Triangulating Across U.S. Federal AI Transparency Regimes." pith.science (2026). https://pith.science/paper/OOZ3MOXT
@misc{pith2026260729540,
author = {Pith},
title = {Pith review of: Triangulating Across U.S. Federal AI Transparency Regimes},
year = {2026},
howpublished = {\url{https://pith.science/paper/OOZ3MOXT}},
note = {Machine review of arXiv:2607.29540}
}
read the original abstract
Federal AI systems can deny benefits or flag individuals for deportation, but the public disclosures meant to make those systems visible are fragmented and unevenly detailed. This paper examines three existing U.S. federal transparency regimes---System of Records Notices (SORNs), Information Collection Requests (ICRs), and the AI Use Case Inventory---and asks how well they, individually and together, describe government AI use. We find that no single regime fully reveals how the government constructs or deploys AI: each discloses different aspects of a system, and the current disclosure infrastructure makes it very challenging for the public to track specific AI systems across regulatory regimes and over time. Persistent identifiers are absent, granularity varies widely, and the annual AI Use Case Inventory cycle means federal agencies can deploy systems months before appearing in any official record. Using hand-validated zero-shot classification and cross-document entity resolution, we contribute a triangulation method that links disclosures across all three regimes and present two case studies. Our case studies finds that linking records provides greater insight into government AI use, but even linked records would constitute insufficient oversight compared to what public reporting has revealed about the same systems. We trace each regime's disclosure weaknesses to its original administrative purpose, showing these gaps are structural, and offer recommendations focused on the AI Use Case Inventory as the mechanism best suited for public-facing transparency: (1) a broad and consistently applied AI system definition, (2) persistent system identifiers with cross-references to related disclosures, and (3) restored public visibility into risk management processes.
Figures
Reference graph
Works this paper leans on
-
[1]
2025 , month =
Alder, Madison and Heilweil, Rebecca , title =. 2025 , month =
2025
-
[2]
Berkeley Technology Law Journal , year=
The OMB Artificial Intelligence Memoranda , author=. Berkeley Technology Law Journal , year=
-
[3]
404 Media , year =
Cox, Joseph and Cleek, Ashley , title =. 404 Media , year =
-
[4]
1973 , number =
Records, Computers, and the Rights of Citizens: Report of the Secretary's Advisory Committee on Automated Personal Data Systems , institution =. 1973 , number =
1973
-
[5]
Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency , pages=
One Bad NOFO? AI Governance in Federal Grantmaking , author=. Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency , pages=
2025
-
[6]
2007 , howpublished =
Stanley, Jay , title =. 2007 , howpublished =
2007
-
[7]
Wang, Nina and McDonald, Allison and Bateyko, Daniel and Tucker, Emily , title =
-
[8]
Proceedings Of The 2022 ACM Conference On Fairness, Accountability, And Transparency , pages=
Regulating facial processing technologies: Tensions between legal and technical considerations in the application of illinois bipa , author=. Proceedings Of The 2022 ACM Conference On Fairness, Accountability, And Transparency , pages=
2022
-
[9]
Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency , pages=
A sociotechnical audit: Assessing police use of facial recognition , author=. Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency , pages=
2023
-
[10]
2025 , month =
Laperruque, Jake , title =. 2025 , month =
2025
-
[11]
2025 , month =
Joffe-Block, Jude , title =. 2025 , month =
2025
-
[12]
2025 , month =
Stewart, Josie and Du, Michelle and Turner Lee, Nicol , title =. 2025 , month =
2025
-
[13]
investigating the public demand for bans, audits, and public database registrations , author=
Regulating ai-based remote biometric identification. investigating the public demand for bans, audits, and public database registrations , author=. Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency , pages=
2024
-
[14]
Conference on fairness, accountability and transparency , pages=
Gender shades: Intersectional accuracy disparities in commercial gender classification , author=. Conference on fairness, accountability and transparency , pages=. 2018 , organization=
2018
-
[15]
University of Pittsburgh Law Review , author=
The Privacy Act of 1974: The American Bill of Rights on Data and Its Unfinished Business , volume=. University of Pittsburgh Law Review , author=. 2024 , pages=
1974
-
[16]
1995 , note =
Paperwork Reduction Act of 1995 , author =. 1995 , note =
1995
-
[17]
1974 , note =
Privacy Act of 1974 , author =. 1974 , note =
1974
-
[18]
2020 , month =
Executive Order 13960:. 2020 , month =
2020
-
[19]
An institutional view of algorithmic impact assessments , author=. Harv. JL & Tech. , volume=. 2021 , publisher=
2021
-
[20]
It's None of Their Damn Business
" It's None of Their Damn Business": Privacy and Disclosure Control in the US Census, 1790-2020 , author=. Population and development review , volume=
2020
-
[21]
2026 , month =
Disclosed government AI use increased by 70 author =. 2026 , month =
2026
-
[22]
AI Now , volume=
Algorithmic impact assessments: a practical Framework for Public Agency , author=. AI Now , volume=
-
[23]
Proceedings of the 2021 ACM conference on fairness, accountability, and transparency , pages=
Algorithmic impact assessments and accountability: The co-construction of impacts , author=. Proceedings of the 2021 ACM conference on fairness, accountability, and transparency , pages=
2021
-
[24]
Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , pages=
Governing algorithmic systems with impact assessments: Six observations , author=. Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society , pages=
2021
-
[25]
Disparate impact in big data policing , author=. Ga. L. Rev. , volume=. 2017 , publisher=
2017
-
[26]
Proceedings of the conference on fairness, accountability, and transparency , pages=
Actionable recourse in linear classification , author=. Proceedings of the conference on fairness, accountability, and transparency , pages=
-
[27]
Fordham L
The intuitive appeal of explainable machines , author=. Fordham L. Rev. , volume=. 2018 , publisher=
2018
-
[28]
Nature machine intelligence , volume=
The global landscape of AI ethics guidelines , author=. Nature machine intelligence , volume=. 2019 , publisher=
2019
-
[29]
2024 , month =
Memorandum. 2024 , month =
2024
-
[30]
2025 , month = feb, type =
Memorandum. 2025 , month = feb, type =
2025
-
[31]
, title =
Sunstein, Cass R. , title =. Harvard Law Review , year =
-
[32]
computing , volume=
A systematic review and comparative analysis of cross-document coreference resolution methods and tools , author=. computing , volume=. 2017 , publisher=
2017
-
[33]
Findings of the Association for Computational Linguistics: ACL 2025 , pages=
Legalcore: A dataset for event coreference resolution in legal documents , author=. Findings of the Association for Computational Linguistics: ACL 2025 , pages=
2025
-
[34]
Proceedings of the 27th conference on computational natural language learning (CoNLL) , pages=
Cross-document event coreference resolution: Instruct humans or instruct gpt? , author=. Proceedings of the 27th conference on computational natural language learning (CoNLL) , pages=
-
[35]
Berkeley Tech
Trademarks as surveillance transparency , author=. Berkeley Tech. LJ , volume=. 2021 , publisher=
2021
-
[36]
Discover Artificial Intelligence , volume=
Public sector AI transparency standard: UK Government seeks to lead by example , author=. Discover Artificial Intelligence , volume=. 2022 , publisher=
2022
-
[37]
2019 , number =
Recommendation of the Council on Artificial Intelligence , institution =. 2019 , number =
2019
-
[38]
2022 , number =
Blueprint for an. 2022 , number =
2022
-
[39]
Proceedings of the ACM on Human-Computer Interaction , volume=
Privacy legislation as business risks: How GDPR and CCPA are represented in technology companies' investment risk disclosures , author=. Proceedings of the ACM on Human-Computer Interaction , volume=. 2023 , publisher=
2023
-
[40]
Regulation , volume =
Shapiro, Stuart , title =. Regulation , volume =
-
[41]
2012 , type =
Shapiro, Stuart , title =. 2012 , type =
2012
-
[42]
, title =
Samaha, Adam M. , title =. Duke Law Journal , volume =
-
[43]
, title =
Levy, Andrew L. , title =. Journal of Law and Commerce , volume =
-
[44]
1995 , note =
Paperwork Reduction Act , howpublished =. 1995 , note =
1995
-
[45]
2023 , publisher=
Recoding America: why government is failing in the digital age and how we can do better , author=. 2023 , publisher=
2023
-
[46]
Findings of the Association for Computational Linguistics: ACL 2024 , pages=
Knowledge-driven cross-document relation extraction , author=. Findings of the Association for Computational Linguistics: ACL 2024 , pages=
2024
-
[47]
John Marshall Law Review , year =
Ehlke, Richard , title =. John Marshall Law Review , year =
-
[48]
University of Pittsburgh Law Review , year =
Zang, Dongsheng , title =. University of Pittsburgh Law Review , year =
-
[49]
Maryland Law Review , year =
Hartzog, Woodrow , title =. Maryland Law Review , year =
-
[50]
2025 , howpublished =
Gellman, Robert , title =. 2025 , howpublished =
2025
-
[51]
AI & Society , year =
Hickok, Merve , title =. AI & Society , year =
-
[52]
New Media & Society , year =
Ananny, Mike and Crawford, Kate , title =. New Media & Society , year =
-
[53]
2020 , howpublished =
2020
-
[54]
2023 , howpublished =
2023
-
[55]
Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) , pages=
Enhancing cross-document event coreference resolution by discourse structure and semantic information , author=. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) , pages=
2024
-
[56]
2007 , month = aug, type =
2007
-
[57]
2012 , month = may, type =
2012
-
[58]
404 Media , year =
Cox, Joseph , title =. 404 Media , year =
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.