pith. sign in

arxiv: 2605.16566 · v1 · pith:5APGFOLQnew · submitted 2026-05-15 · 💻 cs.CY · cs.HC

Characterizing AI Fact-Checkers and Their Contributions on Community Notes

Yilin Gong , Siqi Wu This is my paper

Pith reviewed 2026-05-19 21:00 UTC · model grok-4.3

classification 💻 cs.CY cs.HC
keywords AI fact-checkingCommunity NotesX platformcrowdsourced fact-checkinghelpfulness ratingshuman-AI collaborationveracity
0
0 comments X

The pith

AI-generated notes on Community Notes are less likely to be rated as helpful than those from human experts but more helpful than laypeople's notes.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper examines the role of AI in X's Community Notes by analyzing 20 identified AI writers from September 2025 to May 2026. It shows these AI systems account for a growing share of notes, respond quickly, and cover posts that humans miss. However, their notes receive fewer helpful classifications than expert human contributions while surpassing those from average users. This matters for understanding how AI can augment but not fully replace human judgment in verifying online information.

Core claim

The study characterizes AI fact-checkers on Community Notes using volume, velocity, variety, and veracity. AI writers make up 14.2% of submitted notes overall, increasing to 44.8% recently, and submit notes within minutes of availability. They contribute to 16.8% of fact-checked posts, mostly new ones without human input. AI notes have a higher share of helpful ratings relative to submissions but are less likely to be helpful than expert human notes and more likely than laypeople notes. Both AI and humans show first-mover advantages in attracting ratings.

What carries the argument

The distinction between AI writers, human experts, and laypeople based on note helpfulness ratings and submission patterns on Community Notes.

Load-bearing premise

The identification of exactly 20 AI writers is accurate and the helpfulness ratings measure true quality differences rather than rater biases or platform effects.

What would settle it

Repeating the analysis after the platform changes its AI API or identification methods to check if the relative helpfulness of AI notes changes.

Figures

Figures reproduced from arXiv: 2605.16566 by Siqi Wu, Yilin Gong.

Figure 1
Figure 1. Figure 1: Daily volume of notes by writer category. Plotted lines are seven-day rolling averages. After mid-March 2026, [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Distributions of response times by writer category. Solid lines represent the time from post creation to [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Temporal trends of median inter-arrival times by writer category. Lines show seven-day rolling averages on a [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Venn diagram for X posts checked by AI writers, human experts, and laypeople. 2025-09 2025-11 2026-01 2026-03 2026-05 Date 0 20 40 60 80 100 Percentage (%) coverage rate discovery rate duplicate rate [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Temporal trends of coverage, discovery, and duplication rates, computed using only the [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Temporal trends in the AI share of submitted notes, ratings received within 48 hours, and CRH notes. Lines [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Distributions of writer CRH rates for AI writers, human experts, and laypeople with at least five submitted [PITH_FULL_IMAGE:figures/full_fig_p009_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Ratios of ratings received by AI notes relative to matched human notes, controlling for the fact-checked post. [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Win rates in head-to-head comparisons between AI and human notes, stratified by writer category and [PITH_FULL_IMAGE:figures/full_fig_p010_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Summary of the 20 AI writers. Each circle represents one writer. The x-axis shows median response time to [PITH_FULL_IMAGE:figures/full_fig_p011_10.png] view at source ↗
read the original abstract

Recent advances in artificial intelligence (AI) have made timely, scalable, and effective fact-checking increasingly feasible. One such deployment is X's Community Notes, which provides the AI Note Writer API to enable end-to-end automated generation of contextual information. We present the first empirical analysis of AI fact-checkers and their contributions on Community Notes, examining four key dimensions: volume, velocity, variety, and veracity. We find that, between September 2, 2025 and May 9, 2026, 20 AI writers account for 14.2% of all submitted notes, with their daily share rising rapidly to 44.8% lately. AI writers are highly responsive, typically submitting notes within minutes of posts becoming available via the API. They also expand coverage, contributing notes to 16.8% of fact-checked posts, of which 74.4% are not checked by humans. Over time, AI writers become more prolific and responsive, with increasing coverage and discovery rates. Despite these advantages, their veracity remains mixed. Collectively, AI writers contribute a higher share of helpful notes while receiving a smaller share of human ratings, relative to their share of submitted notes. Controlling for the fact-checked post and note submission order, both AI and human writers exhibit a first-mover advantage, with earlier notes attracting more ratings. More importantly, AI-generated notes are less likely to be classified as helpful than those written by human experts, though they outperform those written by laypeople. Our findings provide new insights into the practical capabilities and limitations of AI-driven fact-checking, with implications for the design and governance of human--AI collaborative crowdsourced context systems.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper presents the first empirical analysis of AI fact-checkers on X's Community Notes platform via the AI Note Writer API. Analyzing data from September 2, 2025 to May 9, 2026, it claims that 20 AI writers account for 14.2% of submitted notes (rising to 44.8% daily share), exhibit high responsiveness (submitting within minutes), expand coverage to 16.8% of fact-checked posts (74.4% unique to AI), show increasing productivity over time, and contribute a higher share of helpful notes relative to their submission volume but receive fewer ratings. After controlling for post identity and submission order, AI notes are less likely to be rated helpful than those by human experts but outperform laypeople, with both AI and human writers showing first-mover advantages.

Significance. If the AI-writer identification and helpfulness controls prove robust, the study supplies timely, concrete data on volume, velocity, variety, and veracity of AI contributions to a real-world crowdsourced fact-checking system. The reported trends in coverage expansion, responsiveness gains, and the comparative helpfulness ordering (AI vs. experts vs. laypeople) after first-mover controls would inform platform governance and human-AI collaboration design. The observational scale over an eight-month window is a strength.

major comments (3)
  1. Data collection and identification of AI writers: The abstract and methods summary report 20 specific AI writers but supply no explicit rule set, behavioral thresholds, API logs, or validation procedure for tagging them. This partitioning is load-bearing for every volume, velocity, and veracity claim; without it, contamination or selection effects cannot be quantified.
  2. Helpfulness analysis and controls (results section): The central ordering—that AI notes are less likely to be classified helpful than human-expert notes yet outperform laypeople, after controlling for post identity and submission order—assumes the binary helpful label reflects content merit rather than rater or platform biases against machine-generated style or source cues. No test or discussion of residual style/source effects is provided, rendering the comparison uninterpretable if the assumption fails.
  3. Veracity and rating-share claims: The statement that AI writers 'contribute a higher share of helpful notes while receiving a smaller share of human ratings, relative to their share of submitted notes' requires the same robust partitioning and bias controls as the expert/layperson comparison; the current description leaves both unverified.
minor comments (2)
  1. Clarify how 'human experts' and 'laypeople' are operationalized in the dataset (e.g., via user metadata, note history, or rating patterns) to allow replication.
  2. The time window (September 2025–May 2026) and 'lately' phrasing for the 44.8% share should be tied to a specific figure or table for precision.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their constructive feedback on our manuscript analyzing AI fact-checkers on Community Notes. We address each of the major comments below, indicating revisions where appropriate to strengthen the paper.

read point-by-point responses

  1. Referee: Data collection and identification of AI writers: The abstract and methods summary report 20 specific AI writers but supply no explicit rule set, behavioral thresholds, API logs, or validation procedure for tagging them. This partitioning is load-bearing for every volume, velocity, and veracity claim; without it, contamination or selection effects cannot be quantified.

    Authors: We agree that explicit details on the identification of AI writers are crucial for the validity of our claims. In the revised manuscript, we will include a dedicated subsection in the Methods describing the rule set and behavioral thresholds used to identify the 20 AI writers. This will encompass their interaction with the AI Note Writer API, patterns in submission timing and volume, and any cross-validation procedures employed. We will also quantify potential selection effects and discuss limitations to allow readers to assess contamination risks. revision: yes

  2. Referee: Helpfulness analysis and controls (results section): The central ordering—that AI notes are less likely to be classified helpful than human-expert notes yet outperform laypeople, after controlling for post identity and submission order—assumes the binary helpful label reflects content merit rather than rater or platform biases against machine-generated style or source cues. No test or discussion of residual style/source effects is provided, rendering the comparison uninterpretable if the assumption fails.

    Authors: The referee raises an important point about potential biases in helpfulness ratings. Our current analysis controls for post identity and submission order to isolate the effect of writer type. However, we recognize that style or source cues could influence ratings. In the revision, we will add a discussion of this limitation and propose future work to test for such effects, perhaps through controlled experiments or additional covariates if data permits. We maintain that the observed differences reflect real-world platform dynamics, but we will make the assumptions more explicit. revision: partial

  3. Referee: Veracity and rating-share claims: The statement that AI writers 'contribute a higher share of helpful notes while receiving a smaller share of human ratings, relative to their share of submitted notes' requires the same robust partitioning and bias controls as the expert/layperson comparison; the current description leaves both unverified.

    Authors: We will enhance the description of the veracity analysis in the revised manuscript by providing more details on how the shares of helpful notes and ratings are computed, ensuring consistency with the partitioning used in the helpfulness comparisons. We will also incorporate additional controls and present the results with greater transparency regarding potential biases, thereby verifying the claims more robustly. revision: yes

Circularity Check

0 steps flagged

No circularity: purely observational empirical analysis with no derivations or fitted predictions

full rationale

The paper reports direct counts, shares, and statistical comparisons (e.g., helpfulness rates after controlling for post identity and submission order) drawn from observed Community Notes data. No equations, models, or first-principles derivations are present; the central claims about AI vs. human note performance are empirical observations, not quantities that reduce to inputs by construction. No self-citations or uniqueness theorems are invoked as load-bearing steps. This is a standard non-circular empirical characterization study.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The analysis rests on platform-provided data and the assumption that AI writers can be reliably identified and that helpfulness ratings serve as a valid quality proxy.

axioms (2)
  • domain assumption AI writers can be accurately distinguished from human writers using available metadata or behavior patterns.
    Central to separating the 20 AI writers and attributing 14.2% of notes.
  • domain assumption Helpfulness ratings reflect note quality independent of writer type or submission timing biases.
    Used to compare veracity across AI, expert human, and layperson notes.

pith-pipeline@v0.9.0 · 5833 in / 1208 out tokens · 23000 ms · 2026-05-19T21:00:05.996153+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

32 extracted references · 32 canonical work pages · 3 internal anchors

  1. [1]

    CHI , year=

    Birds of a feather don’t fact-check each other: Partisanship and the evaluation of news in Twitter’s Birdwatch crowdsourced fact-checking program , author=. CHI , year=

    work page

  2. [2]

    CHI , year=

    Fueling volunteer growth: The case of Wikipedia administrators , author=. CHI , year=

    work page

  3. [3]

    Nature Machine Intelligence , year=

    Factuality challenges in the era of large language models and opportunities for fact-checking , author=. Nature Machine Intelligence , year=

    work page

  4. [4]

    Nature Communications , year=

    Community-based fact-checking reduces the spread of misleading posts on X (formerly Twitter) , author=. Nature Communications , year=

    work page

  5. [5]

    TheWebConf , year=

    Supernotes: Driving consensus in crowd-sourced fact-checking , author=. TheWebConf , year=

    work page

  6. [6]

    PNAS , year=

    Fact-checking information from large language models can decrease headline discernment , author=. PNAS , year=

    work page

  7. [7]

    Journal of Experimental Psychology: General , year=

    Algorithm aversion: People erroneously avoid algorithms after seeing them err , author=. Journal of Experimental Psychology: General , year=

    work page

  8. [8]

    arXiv preprint arXiv:2504.09865 , year=

    Labeling messages as AI-generated does not reduce their persuasive effects , author=. arXiv preprint arXiv:2504.09865 , year=

    work page arXiv

  9. [9]

    The Effects of Request Alerts on the Diversity and Visibility of Community Notes

    The effects of request alerts on the diversity and visibility of community notes , author=. arXiv preprint arXiv:2604.17042 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    Journal of Online Trust and Safety , year=

    Scaling human judgment in Community Notes with LLMs , author=. Journal of Online Trust and Safety , year=

    work page

  11. [11]

    AI Fact-Checking in the Wild: A Field Evaluation of LLM-Written Community Notes on X

    AI fact-checking in the wild: A field evaluation of LLM-written community notes on X , author=. arXiv preprint arXiv:2604.02592 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    CSCW , year=

    Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AI , author=. CSCW , year=

    work page

  13. [13]

    CHI , year=

    Beyond community notes: A framework for understanding and building crowdsourced context systems for social media , author=. CHI , year=

    work page

  14. [14]

    AI Feedback Enhances Community-Based Content Moderation through Engagement with Counterarguments

    AI feedback enhances community-based content moderation through engagement with counterarguments , author=. arXiv preprint arXiv:2507.08110 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [15]

    Technology in Society , year=

    Fact-checking in the age of AI: Reducing biases with non-human information sources , author=. Technology in Society , year=

    work page

  16. [16]

    Psychological Bulletin , year=

    AI aversion or appreciation? A capability--personalization framework and a meta-analytic review , author=. Psychological Bulletin , year=

    work page

  17. [17]

    SIGIR , year=

    Efficiency and effectiveness of LLM-based summarization of evidence in crowdsourced fact-checking , author=. SIGIR , year=

    work page

  18. [18]

    TACL , year=

    Can LLMs automate fact-checking article writing? , author=. TACL , year=

    work page

  19. [19]

    arXiv preprint arXiv:2602.08945 , year=

    GitSearch: Enhancing community notes generation with gap-informed targeted search , author=. arXiv preprint arXiv:2602.08945 , year=

    work page arXiv

  20. [20]

    arXiv:2502.08841 , year=

    Delayed takedown of illegal content on social media makes moderation ineffective , author=. arXiv preprint arXiv:2502.08841 , year=

    work page arXiv

  21. [21]

    arXiv preprint arXiv:2511.02615 , year=

    Community Notes are vulnerable to rater bias and manipulation , author=. arXiv preprint arXiv:2511.02615 , year=

    work page arXiv

  22. [22]

    Nature Human Behaviour , year=

    When combinations of humans and AI are useful: A systematic review and meta-analysis , author=. Nature Human Behaviour , year=

    work page

  23. [23]

    arXiv preprint arXiv:2210.15723 , year=

    Birdwatch: Crowd wisdom and bridging algorithms can inform understanding and reduce the spread of misinformation , author=. arXiv preprint arXiv:2210.15723 , year=

    work page arXiv

  24. [24]

    FAccT , year=

    The impact and opportunities of generative AI in fact-checking , author=. FAccT , year=

    work page

  25. [25]

    ICWSM , year=

    Variation across scales: Measurement fidelity under twitter data sampling , author=. ICWSM , year=

    work page

  26. [26]

    ACL , year=

    Beyond the crowd: LLM-augmented community notes for governing health misinformation , author=. ACL , year=

    work page

  27. [27]

    NAACL , year=

    Evaluating evidence attribution in generated fact checking explanations , author=. NAACL , year=

    work page

  28. [28]

    2025 , howpublished =

    World's first AI Community Note , author =. 2025 , howpublished =

    work page 2025

  29. [29]

    TheWebConf , year=

    A fact-checking framework with denoising evidence retrieval and LLM-based debate verification , author=. TheWebConf , year=

    work page

  30. [30]

    arXiv preprint arXiv:2509.11052 , year=

    Commenotes: Synthesizing organic comments to support community-based fact-checking , author=. arXiv preprint arXiv:2509.11052 , year=

    work page arXiv

  31. [31]

    Journal of Online Trust and Safety , year=

    Insights from a comparative study on the variety, velocity, veracity, and viability of crowdsourced and professional fact-checking services , author=. Journal of Online Trust and Safety , year=

    work page

  32. [32]

    arXiv preprint arXiv:2403.11169 , year=

    Correcting misinformation on social media with a large language model , author=. arXiv preprint arXiv:2403.11169 , year=

    work page arXiv