REVIEW 4 major objections 4 minor 1 cited by
AI risk reports overlook fraud and manipulation
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
An analysis of 460,000 Hugging Face model cards reveals that developer-reported risks focus on technical issues and largely miss fraud and manipulation harms that show up in real incidents.
T0 review reviewed 2026-08-05 challenge →
load-bearing objection Big new dataset, but the headline gap about fraud/manipulation can't be judged from the abstract alone; the extraction and mapping methods decide it. the 4 major comments →
The AI Model Risk Catalog: What Developers and Researchers Miss About Real-World AI Harms
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
By extracting risk mentions from nearly 460,000 model cards and clustering them into a catalog of about 3,000 unique risks, the paper shows that developer-reported risks concentrate on bias, safety, and other technical failures, while researcher-identified risks emphasize societal concerns. When both sets are compared with a database of real-world AI incidents, fraud and manipulation appear frequently in the incident record yet are nearly absent from both developer and researcher risk lists. The central discovery is that the two communities have complementary blind spots that together leave a common class of harms unaddressed in risk documentation.
What carries the argument
The AI Model Risk Catalog, constructed from deduplicated risk mentions across hundreds of thousands of model cards, serves as the instrument for comparing developer-reported, researcher-identified, and real-world incident risks. The comparison itself—mapping mentions and categories across the three sources—is the mechanism that exposes the gap.
Load-bearing premise
The reported gap depends on the extracted risk mentions being a faithful representation of what developers wrote across all model cards, and on the two reference collections being complete and unbiased; if either fails, the gap could shrink or disappear.
What would settle it
Re-analyzing a random sample of model cards with a different extraction and categorization method, and finding that fraud and manipulation are mentioned at rates comparable to the incident database, would undercut the central claim.
If this is right
- Model documentation standards could include structured fields for human-interaction risks like fraud and manipulation, forcing developers to address them explicitly.
- Academic risk taxonomies could be extended by mining real-world incident databases to ensure neglected harm categories are represented.
- The catalog could serve as a practical checklist for developers writing model cards, reducing the likelihood of overlooking common harms.
- Risk reporting guidelines could be revised to require consideration of systemic and interaction-driven harms at the design stage rather than after deployment.
- If widely adopted, the catalog would make cross-platform risk comparisons easier and more consistent over time.
Where Pith is reading between the lines
- The gap may be even larger than reported because model cards are self-reported and developers may omit risks they do not recognize, suggesting the 3,000 mentions are an upper bound on awareness, not a complete picture.
- The neglect of fraud and manipulation could reflect incentive structures: developers focus on measurable technical metrics, while researchers prioritize topics with established academic vocabularies, leaving interaction-driven harms in a blind zone.
- A testable extension would apply the same catalog construction to newer model cards or other AI platforms to see whether the blind spot persists or closes as awareness grows.
- The comparison method could be generalized to audit other risk domains, such as autonomous systems or recommendation engines, where developer self-reports may similarly diverge from real-world incident patterns.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper analyzes nearly 460,000 Hugging Face model cards, extracts approximately 3,000 unique risk mentions into an "AI Model Risk Catalog," and compares these mentions with two external resources: the MIT Risk Repository (researcher-identified risks) and the AI Incident Database (real-world incidents). The authors report that developers concentrate on technical issues such as bias and safety, researchers emphasize broader social impacts, and both groups pay little attention to fraud and manipulation, which they argue are common real-world harms. The paper proposes clearer structured risk reporting to address this gap. A link to the catalog and appendix is provided.
Significance. If the findings hold, the paper would provide a large-scale, comparative evidence base on how AI risks are documented across developers, researchers, and real-world incidents. The proposed catalog could be a useful community resource for improving model-card risk reporting. The scale (460,000 cards) and the tripartite comparison are notable strengths. However, the abstract alone does not establish the validity of the extraction pipeline, the selection biases of the reference repositories, or the consistency of the category mapping. No inter-annotator agreement, error bounds, or sensitivity analyses are reported. Therefore, the significance is currently conditional on the availability and quality of a fully described methodology, which is not visible in the abstract.
major comments (4)
- [Abstract] The abstract states that ~3,000 unique risk mentions were extracted from ~460,000 model cards, but it does not describe the extraction method, deduplication criterion, or category taxonomy. Without inter-annotator agreement, a manual validation sample, or any error estimate, the count is not interpretable. This is load-bearing because the subsequent comparison assumes that the 3,000 mentions faithfully represent what developers wrote across all cards. If the extraction or deduplication is biased, the reported gaps could be artifacts. Please provide details of the pipeline and a validation study.
- [Abstract] The comparison with the MIT Risk Repository and the AI Incident Database assumes that these are unbiased reference sets for researcher-identified and real-world harms, respectively. The abstract gives no justification for this assumption. In particular, the AI Incident Database may itself underreport fraud and manipulation incidents due to reporting biases or definitions; if so, the claim that these harms are 'common' is not established. The paper should address selection bias in the reference sets or triangulate with additional incident collections and research taxonomies.
- [Abstract] The central claim that 'both groups paid little attention to fraud and manipulation' depends entirely on how risk mentions are mapped to categories. The abstract does not define the boundaries of 'fraud/manipulation' or state how terms such as 'scam,' 'deceptive content,' 'social engineering,' or 'prompt injection for fraud' were categorized. If these were mapped to other buckets or collapsed in deduplication, the gap could be an artifact. Please provide the full category definitions, the mapping rules, and a sensitivity analysis that varies the mapping.
- [Abstract] The conclusion that 'researchers emphasized broader social impacts' is drawn from the MIT Risk Repository, which is an academic construct. The abstract does not establish that this repository is representative of researcher attention, nor does it compare like with like: model-card texts are unstructured developer self-reports, whereas the MIT repository is a curated, likely normalized taxonomy. This asymmetry in data generation may drive the reported differences. The paper should discuss or adjust for this asymmetry.
minor comments (4)
- [Abstract] Please report exact counts with confidence intervals or rounding rules instead of 'around 3,000' and 'nearly 460,000'.
- [Abstract] Clarify what fields of Hugging Face model cards were analyzed (e.g., model card text, limitations sections, or metadata) and whether the analysis was restricted to English-language cards.
- [Abstract] The term 'real-world harms' from the AI Incident Database conflates reported incidents with actual harm rates; the AIID has known documentation and selection biases. Please state this as a limitation.
- [Abstract] The URL for the catalog should be checked for stability and ideally archived (e.g., via Zenodo or OSF) to support reproducibility.
Circularity Check
No circularity detected in the abstract-only evidence; the catalog is built from independent model-card text and compared against external repositories.
full rationale
The paper's central claim is an empirical comparison: it extracts ~3,000 unique risk mentions from ~460,000 Hugging Face model cards, builds a catalog, and compares it with the independent MIT Risk Repository and AI Incident Database. Each input is distinct: model-card text, researcher-identified risks, and real-world incident records. The conclusion that developers and researchers underreport fraud/manipulation is a comparative empirical finding, not a derivation that reduces to its own inputs. There are no equations, no fitted parameters renamed as predictions, and no self-citation chain visible in the abstract. The concerns raised by the skeptic—unvalidated extraction methodology, category mapping, and completeness of external repositories—are validity and replication concerns, not circularity. Per the hard rules, missing methodology does not constitute circularity unless a specific reduction can be exhibited, and none can be from the provided text. Therefore the appropriate score is 0.
Axiom & Free-Parameter Ledger
axioms (3)
- domain assumption The 3,000 unique risk mentions extracted from 460,000 model cards faithfully represent what developers report.
- domain assumption The MIT Risk Repository and the AI Incident Database are valid, comprehensive reference sets.
- domain assumption Model card text on Hugging Face is representative of AI developer risk reporting broadly.
Cite this review
Pith. "Pith review of The AI Model Risk Catalog: What Developers and Researchers Miss About Real-World AI Harms." pith.science (2026). https://pith.science/paper/DH4FIC2B
@misc{pith2026250816672,
author = {Pith},
title = {Pith review of: The AI Model Risk Catalog: What Developers and Researchers Miss About Real-World AI Harms},
year = {2026},
howpublished = {\url{https://pith.science/paper/DH4FIC2B}},
note = {Machine review of arXiv:2508.16672}
}
read the original abstract
We analyzed nearly 460,000 AI model cards from Hugging Face to examine how developers report risks. From these, we extracted around 3,000 unique risk mentions and built the \emph{AI Model Risk Catalog}. We compared these with risks identified by researchers in the MIT Risk Repository and with real-world incidents from the AI Incident Database. Developers focused on technical issues like bias and safety, while researchers emphasized broader social impacts. Both groups paid little attention to fraud and manipulation, which are common harms arising from how people interact with AI. Our findings show the need for clearer, structured risk reporting that helps developers think about human-interaction and systemic risks early in the design process. The catalog and paper appendix are available at: https://social-dynamics.net/ai-risks/catalog.
Forward citations
Cited by 1 Pith paper
-
Evaluation Cards: An Interpretive Layer for AI Evaluation Reporting
EvalCards is a composable reporting schema and monitoring tool for AI evaluations, derived from 52 papers and 10 interviews, and applied to 5,816 models and 101,843 results to surface reporting gaps.
This paper was first reviewed by deepseek-v4-flash on August 5, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.