The Effects of Request Alerts on the Diversity and Visibility of Community Notes

Siqi Wu; Yilin Gong

arxiv: 2604.17042 · v1 · submitted 2026-04-18 · 💻 cs.CY · cs.HC

The Effects of Request Alerts on the Diversity and Visibility of Community Notes

Yilin Gong , Siqi Wu This is my paper

Pith reviewed 2026-05-10 06:06 UTC · model grok-4.3

classification 💻 cs.CY cs.HC

keywords community notesrequest alertscrowdsourced fact-checkingtopic diversitynote visibilitymixed-effects modelpivot penaltycontent inequality

0 comments

The pith

Request alerts on Community Notes raise the odds a note is rated helpful by 8.4 to 20.2 percentage points.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests whether request alerts on the X platform change how contributors write Community Notes. It compares 54,874 notes from 318 frequent writers, some written when alerts were active and some when they were not. Writers respond to alerts by covering more varied and more political posts than they normally do. At the platform level the same shift funnels more notes into the already largest category, widening differences in topic coverage. Notes written under alerts are also more likely to pass the helpfulness threshold that makes them visible to readers, though the gain shrinks when the topic lies far from a writer's usual subjects.

Core claim

Request alerts act as an interface cue that prompts writers to fact-check more diverse and political content than they otherwise would; the same cue raises the probability that a note receives a helpful rating by 8.4 to 20.2 percentage points after writer- and topic-level random effects are controlled; the visibility advantage is smaller for topics distant from a writer's prior activity; and the individual-level diversification nevertheless increases aggregate concentration in the Politics and Conflict category.

What carries the argument

Within-writer comparison of notes written under inferred request alerts versus notes written without alerts, estimated with a mixed-effects model that includes random intercepts for each writer and each topic.

If this is right

Writers expand the range of posts they choose to fact-check when alerts appear.
The same notes are more likely to become visible to the public.
Collective coverage becomes more concentrated in the Politics and Conflict category despite the individual-level broadening.
Visibility gains are reduced when writers pivot to topics outside their established interests.
Alerts can be used to steer crowdsourced fact-checking toward posts that users have explicitly requested.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Similar alert mechanisms could be tested on other crowdsourced moderation platforms to see whether the visibility and concentration patterns repeat.
Designers might combine alerts with topic-specific prompts to offset the observed increase in content inequality.
The pivot-penalty finding suggests that repeated exposure to alerts on new topics could gradually reduce the visibility gap over time.

Load-bearing premise

The presence or absence of a request alert at the exact moment each note was written can be inferred accurately enough that misclassified notes do not distort the measured differences.

What would settle it

A direct log of alert display status for each note at the time it was authored, or a field experiment that randomly shows or withholds alerts and then measures subsequent helpfulness ratings.

read the original abstract

Several major social media platforms have shifted toward crowdsourced fact-checking systems like Community Notes to combat misinformation at scale. However, these systems face criticism regarding which content is scrutinized and how visible that scrutiny is. To address these concerns, X allows users to request community notes for specific posts. When sufficient requests accumulate, X displays an alert, formalizing an interface cue intended to guide contributor behavior. In this study, we examine the effectiveness of request alerts. We infer the presence of request alerts at the time each note was written and identify 318 top writers who were repeatedly exposed to these alerts. Through analyzing their contributed 54,874 English notes written with and without request alerts, we find that at the individual level, writers fact-check more diverse and more political content under alerts. Nonetheless, at the collective level, these shifts direct contributions toward the already dominant Politics and Conflict category, thereby increasing content inequality within the Community Notes ecosystem. Finally, using a mixed-effects model that controls for both writer- and topic-level random effects, we estimate that notes written under alerts are between 8.4 and 20.2 percentage points more likely to be classified as helpful and thus visible to the public, compared to non-alerted notes. This visibility gain diminishes as topics diverge further from writers' prior interests, demonstrating a pivot penalty effect. Taken together, our findings show that request alerts function as an effective interface cue that increases both topical diversity and note visibility in Community Notes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

Request alerts nudge writers toward more visible notes but also concentrate output in dominant topics, with the main uncertainty around how accurately the alerts were detected from the data.

read the letter

The paper's core finding is that notes written when request alerts were active are 8 to 20 percentage points more likely to be rated helpful, while individual writers cover more diverse and political ground but the overall distribution of notes tilts further toward Politics and Conflict. They reach this with a mixed-effects regression on 54,874 notes from 318 frequent contributors, including writer and topic random effects plus an interaction for how far a note sits from a writer's past focus. That setup lets them separate personal habits from the alert effect and document the pivot penalty. The dataset size and the use of real platform behavior are the strongest parts; most prior work on Community Notes has been smaller or more descriptive. The alert inference step is the clearest weakness. The abstract states they infer presence for each note but gives no validation against ground-truth timestamps, no sensitivity tests, and no description of how request accumulation was reconstructed. If that binary indicator is noisy in ways that track topic popularity or writer activity, the coefficient on alerts could be biased even with the random effects. The full paper may fill this in, but the abstract leaves it open. Readers working on platform moderation, interface design, or observational studies of crowdsourced systems would find the numbers and the inequality result useful. It is worth sending to referees because the question is practical and the data are large, provided the methods section gets a close look on the treatment construction and robustness checks.

Referee Report

2 major / 2 minor

Summary. The paper claims that request alerts in Community Notes increase individual-level topical diversity and political content among top writers, but lead to greater collective concentration in the Politics category. Using a mixed-effects model with writer and topic random effects on 54,874 notes, it estimates an 8.4 to 20.2 percentage point increase in the likelihood of notes being rated helpful and visible under alerts, with a diminishing effect for topics distant from writers' prior interests.

Significance. If the results hold, they demonstrate the power of interface cues to guide crowdsourced moderation, boosting note visibility while potentially exacerbating content imbalances. The analysis benefits from controlling for writer- and topic-specific heterogeneity via random effects and leverages a substantial dataset of repeated exposures. This contributes to understanding platform interventions in fact-checking ecosystems.

major comments (2)

[Abstract] The central visibility estimate of 8.4–20.2 percentage points is obtained from a mixed-effects model using a binary alert-presence indicator. The paper indicates that alert presence is inferred for each note but provides no validation details, sensitivity analyses, or reconstruction method for request accumulation. This inference is load-bearing for the claim, as misclassification correlated with topic or writer factors could bias results despite the random effects controls.
[Results section] The finding that alerts increase content inequality by shifting toward the dominant Politics and Conflict category should specify the exact measures of diversity and inequality employed, and demonstrate robustness to alternative topic classifications or time trends.

minor comments (2)

Clarify in the abstract or methods whether the 8.4–20.2 range represents confidence interval bounds, results from multiple models, or another derivation.
[Abstract] The total number of notes on the platform during the study period or the selection criteria for the 318 top writers could be briefly noted for better context on generalizability.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive and detailed comments. We address each major comment below and describe the revisions we will make to strengthen the manuscript.

read point-by-point responses

Referee: [Abstract] The central visibility estimate of 8.4–20.2 percentage points is obtained from a mixed-effects model using a binary alert-presence indicator. The paper indicates that alert presence is inferred for each note but provides no validation details, sensitivity analyses, or reconstruction method for request accumulation. This inference is load-bearing for the claim, as misclassification correlated with topic or writer factors could bias results despite the random effects controls.

Authors: We agree that greater transparency on the alert inference procedure is warranted. The binary indicator is constructed by reconstructing request accumulation at the moment each note was written, using the platform's documented threshold for displaying an alert. In the revised manuscript we will add a dedicated Methods subsection that (i) details the exact reconstruction algorithm, (ii) reports a validation exercise on a manually inspected subsample where alert status can be directly verified, and (iii) presents sensitivity analyses that vary the accumulation threshold and test for differential misclassification by topic or writer experience. These checks will be shown to leave the 8.4–20.2 pp visibility estimate materially unchanged, consistent with the protection already afforded by the writer- and topic-level random effects. revision: yes
Referee: [Results section] The finding that alerts increase content inequality by shifting toward the dominant Politics and Conflict category should specify the exact measures of diversity and inequality employed, and demonstrate robustness to alternative topic classifications or time trends.

Authors: We accept that the diversity and inequality metrics must be stated explicitly and subjected to additional robustness tests. In the revision we will define individual topical diversity as the Shannon entropy of each writer's per-note topic distribution and the political shift as the change in the share of notes assigned to the Politics and Conflict category. Collective inequality will be quantified by the Herfindahl-Hirschman Index of topic shares together with the Gini coefficient across all categories. We will add (a) results under alternative topic models (different numbers of topics and a supervised classifier trained on a hand-labeled subset) and (b) specifications that include time-period fixed effects or linear time trends to address possible secular changes in content production. These analyses will appear in the main Results section and an expanded appendix. revision: yes

Circularity Check

0 steps flagged

No circularity: regression estimates are model outputs on observed data

full rationale

The paper's central claim is produced by fitting a mixed-effects model to 54,874 observed notes, using an inferred binary alert indicator as a predictor and helpful classification as the outcome. The alert indicator is constructed from request-accumulation data independently of the helpfulness label; the coefficient (8.4–20.2 pp) is therefore an estimated parameter, not a quantity that equals any input by definition. No equations, self-citations, or prior-work ansatzes are invoked to force the result. The derivation chain is a standard statistical estimation step whose output is not equivalent to its inputs.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The analysis rests on the assumption that request-alert presence can be reliably inferred from platform data at note-writing time and that the mixed-effects model adequately controls for unobserved writer and topic heterogeneity.

free parameters (1)

random effects variances in mixed model
The model includes writer-level and topic-level random intercepts whose variances are estimated from the data.

axioms (1)

domain assumption Request alert status can be accurately inferred from available platform signals at the moment of note creation.
The paper states it infers alert presence but the abstract gives no validation procedure.

pith-pipeline@v0.9.0 · 5561 in / 1365 out tokens · 25573 ms · 2026-05-10T06:06:22.255008+00:00 · methodology

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Characterizing AI Fact-Checkers and Their Contributions on Community Notes
cs.CY 2026-05 unverdicted novelty 7.0

AI writers account for 14.2% of Community Notes submissions with high responsiveness and coverage but lower helpfulness classification rates than human experts.