REVIEW 4 major objections 6 minor 33 references
Reddit Rules and Rulers: Quantifying the Link Between Rules and Perceptions of Governance across Thousands of Communities
T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read This paper claims that the rules a Reddit community publishes are measurably connected to how members talk about their moderators, with participation, formatting, and commercialization rules linked to the most positive sentiment.
desk verdict A substantial descriptive study of Reddit rules whose headline longitudinal claim ('rule additions improve governance perceptions') is likely a compositional artifact and needs a rewrite. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the labeled rule period: a block of time in which a community's published sidebar rule set is constant, reconstructed from archive snapshots with an average start-and-end uncertainty of about seventeen days. Each rule period is labeled by a 17-attribute taxonomy of tone (prescriptive versus restrictive), target (post content, post format, user-related), and topic (such as spam, commercialization, tagging and flairing, and brigading), and is paired with the community's governance-perception score, the fraction of governance-discussing posts and comments in that period with positive, neutral, or negative sentiment. Cross-sectional comparisons use inverse probability of treatment weighting, a procedure that reweights communities by their propensity to have the rule so that treated and untreated groups resemble the overall population, while the longitudinal analysis compares the twelve months before a rule addition with successive two-month windows after it.
What would settle it
A direct comparison would settle it: survey a random sample of members across these same communities with a pre-registered governance-satisfaction instrument, and re-run the rule-type and rule-addition analyses on the survey responses; if the +0.61 percentage-point improvement after rule additions and the associations with participation and formatting rules disappear, the public-discussion proxy is what produced the paper's pattern.
Extended reading notes
Core claim
The paper claims to be the first large-scale empirical link between specific rule types and community members' expressed perceptions of governance on Reddit. Its central finding is that rule choice is not arbitrary: even after matching communities on topic and size, communities that publish rules about who is allowed to participate, about post formatting and tags, and about commercial activity show more positive and less negative governance sentiment than comparable communities without those rules, while rules phrased restrictively and rules mentioning bans or brigading are associated with more negative sentiment. The companion longitudinal result is that adding a new rule is followed by a small immediate improvement in governance sentiment, with positive sentiment up 0.61 percentage points and negative sentiment down 0.77 points, but this effect is no longer distinguishable from baseline after about six months. The paper interprets the fade as evidence that part of a rule change's benefit comes from the signal that moderators are responding to their community, not only from the rule's enforcement.
Load-bearing premise
Every result rests on treating the sentiment of public posts and comments about governance as an accurate measure of how community members really feel about their moderators, and the paper itself notes that public discussion may not match privately held attitudes.
Editorial extensions
If this is right
- Moderators who want better governance sentiment have a concrete shortlist: adopt rules about who may participate, about post formatting and tagging, and about commercial activity, and phrase rules prescriptively where possible.
- A single rule addition is not a durable governance fix; the paper's estimate is an immediate improvement of about 0.61 percentage points in positive sentiment that statistically disappears after roughly six months.
- Because user-related rules are rare, appearing in about 21% of communities, yet are strongly associated with positive sentiment, there is room for many communities to adopt this rule type and potentially shift their governance perception.
- Platforms could build rule 'starter packs' and topic- and size-based rule recommendations from the released timeline data, since the paper shows that rule sets are patterned by community topic and size.
Reading between the lines
- If the six-month decay is a signaling effect rather than an enforcement effect, which the paper suggests but does not test, then visible moderator engagement such as announcements or rule refreshers might reproduce the benefit without accumulating permanent rules.
- The observational design cannot distinguish whether participation rules improve sentiment by pre-filtering norm-compatible users or by changing existing members' behavior; a randomized field experiment varying rule wording across similar communities would separate the two.
- The released timeline data would also support studies of rule removal and of consistency between posted rules and enforcement, both of which this paper leaves out.
- A concrete testable extension is that communities adding a rule right after a visible incident should show a larger immediate sentiment shift than communities adding a rule proactively, because the responsiveness signal is stronger.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper studies Reddit community rules at scale: it reconstructs rule timelines from Wayback Machine snapshots for 5,225 communities over 2018–2023, classifies 67,545 unique rules into a 17-attribute taxonomy via a GPT-4o retrieval-augmented classifier, describes rule prevalence by community type, tests associations between rule presence and a sentiment-based measure of governance perceptions using IPTW, and reports pre-post changes in these perceptions around rule additions. The central empirical claims are that rules about participation, post formatting, and commercialization are associated with more positive governance perceptions, and that rule additions are followed by a temporary improvement that fades after about six months.
Significance. If the associations and temporal claims survive closer inspection, this would be the largest-scale evidence linking specific rule types to governance sentiment, and the longitudinal finding (if not a measurement artifact) would be novel and actionable. The paper's strengths include a large public dataset of rule timelines, a careful taxonomy development with human IRR and model evaluation, bootstrap uncertainty estimates, and a mostly well-executed IPTW adjustment with a published balance table. The descriptive results in §4 are solid and reproducible. The main risk is the validity of the outcome measure for the causal claims, because the measure is a fraction of governance-discussing posts and comments.
major comments (4)
- [§6 and Figure 6] The pre-post analysis is vulnerable to a compositional artifact. The outcome is the fraction of posts/comments classified as discussing governance that are positive/neutral/negative (§3.3). A rule addition is itself a governance event and is likely to generate posts/comments about the new rule (e.g., "what does rule 7 mean?"). If those posts are disproportionately neutral or positive, the reported +0.61 pp positive and -0.77 pp negative changes can occur without any change in sentiment toward ongoing governance. The paper's own §7.1 states that "we also measure discussions about governance and moderators generally, not rules specifically." The concurrent +0.71 pp neutral increase at 2–4 months is consistent with this mechanism. The authors should re-estimate the analysis excluding posts/comments that mention the added rule (or otherwise rule-related discussion), or show that the result is robust in a sample of governance discussion that is unrelated to the specific rule change.
- [Appendix C, specifically C.13 (Divisive Content)] The IPTW balance check reports SMD = 0.96 for community size in the control group, far above the 0.25 threshold the paper uses. The text in §3.4 states "no SMD exceeds 1.00" as a reassurance, but an SMD of 0.96 means the weighted control group is essentially unadjusted on community size, so the Divisive Content comparison in Figure 5m should not be interpreted as confounder-adjusted. This does not affect the central rule types, but the summary statement that balance is achieved in 235 of 238 cases is misleading without flagging the magnitude of this failure.
- [§5 and §3.3] The cross-sectional associations share the same denominator issue. Communities with User-Related or Post Format rules may discuss those rules more often, so the fraction of governance-discussing posts that is positive/neutral may differ for reasons unrelated to overall governance satisfaction. Because the abstract singles out "rules addressing who participates" as a key finding, and the automated F1 for User-Related rules is only 0.61 (Table 1), the authors should provide a robustness check that controls for the volume of governance discussion or restricts the outcome to governance discussion that does not reference the rule category itself.
- [Abstract and §6] The language of "impact" and "immediately improve" overstates what a pre-post observational design can establish. The paper's own §7.1 notes that difference-in-difference designs would be needed to handle unobserved confounding. I recommend rewording the abstract and §6 to describe associations and temporal sequences rather than causal effects, unless the authors add a control group (e.g., matched communities without rule changes) or another design that supports causal interpretation.
minor comments (6)
- [Abstract and Appendix C.18] The abstract reports 67,545 unique rules across 5,225 communities, while the dataset datasheet (Appendix C.18) reports 73,087 unique rules across 6,120 communities; these numbers should be reconciled.
- [Table 1 and Table 3] Given that User-Related and Peer Engagement are central to the findings, it would be helpful to report per-class precision and recall, not only F1, for these categories.
- [§3.2] The sentence "Our taxonomy simplifies the Fiesler et al. (2018) taxonomy, with with a slightly smaller set" contains a duplicated word.
- [§3.3] The phrase "For each community's rules periods" should be clarified to indicate whether the unit of analysis is the community-period, since communities can appear in multiple rules periods.
- [§5] The text says "after adjusting for confounding factors" but does not specify whether the unit of analysis is a community or a community-period; this matters because the same community can contribute multiple rules periods to the IPTW analysis.
- [§3.3 and §4.3] The outcome measure and community-topic labels both come from Weld et al. (2024), which is self-cited from the same group. This is not circular because the rule taxonomy and associations are new, but the paper would be strengthened by acknowledging this shared-source dependency and, ideally, by an independent replication of the sentiment pipeline.
Circularity Check
No circularity found: the rule taxonomy, IPTW associations, and longitudinal before-after analyses are derived from new empirical pipelines rather than from the outcome measure or from fitted inputs.
full rationale
The derivation chain is self-contained with respect to circularity. The rule taxonomy is developed from grounded coding of a development sample and evaluated against human labels with reported kappa/F1, not derived from the governance-sentiment outcome. The GPT-4o classifier is fit to human rule labels, not to perceptions of governance, so the rule classifications used in Sections 4 and 5 are not fitted inputs to the outcome. The governance-perception measure is taken from Weld et al. (2024), a same-group prior method, but it is used as an external measurement instrument: the paper's claims are conditional on that instrument and do not redefine the outcome in terms of the rules being tested. IPTW weights in Section 5 depend only on community topic and size covariates, not on the outcome or on rule-type parameters, so the reported associations are not forced by construction. The longitudinal analysis in Section 6 computes simple before-after differences of the same outcome across rule-change events and does not solve for coefficients that would mechanically produce the reported positive/negative changes. The skeptic's compositional-denominator concern is a measurement-validity threat rather than an equation-level equivalence: it would require evidence that rule additions alter the mix of governance discussion in a way that the sentiment classifier maps to the reported signs, which the paper does not assume. The self-citations for the outcome measure and topic labels create a dependency, but they are not load-bearing circular reductions: no central claim is established by citing the prior paper in place of an argument, and the rule taxonomy, the associations, and the temporal findings are new empirical results built on top of those measurements.
Assumptions & free parameters
free parameters (3)
- Edit distance threshold for rule deduplication =
2 characters
- Post-change window width for longitudinal analysis =
2 months
- IPTW propensity score logistic regression coefficients =
not reported in paper
assumptions (4)
- domain assumption Public posts and comments discussing governance, classified by the Weld et al. (2024) pipeline, reflect community members' perceptions of governance.
- domain assumption The GPT-4o retrieval-augmented classifier labels rules with sufficient accuracy that misclassification does not materially bias the reported associations.
- domain assumption IPTW adjusting for community size and topic adequately controls confounding in the §5 comparisons.
- domain assumption Wayback Machine snapshots reconstruct community rules periods with negligible error.
Cite this review
Pith. "Pith review of Reddit Rules and Rulers: Quantifying the Link Between Rules and Perceptions of Governance across Thousands of Communities." pith.science (2026). https://pith.science/paper/T6UWOYTX
@misc{pith2026250114163,
author = {Pith},
title = {Pith review of: Reddit Rules and Rulers: Quantifying the Link Between Rules and Perceptions of Governance across Thousands of Communities},
year = {2026},
howpublished = {\url{https://pith.science/paper/T6UWOYTX}},
note = {Machine review of arXiv:2501.14163}
}
read the original abstract
Rules are a critical component of the functioning of nearly every online community, yet it is challenging for community moderators to make data-driven decisions about what rules to set for their communities. The connection between a community's rules and how its membership feels about its governance is not well understood. In this work, we conduct the largest-to-date analysis of rules on Reddit, collecting a set of 67,545 unique rules across 5,225 communities which collectively account for more than 67% of all content on Reddit. More than just a point-in-time study, our work measures how communities change their rules over a 5+ year period. We develop a method to classify these rules using a taxonomy of 17 key attributes extended from previous work. We assess what types of rules are most prevalent, how rules are phrased, and how they vary across communities of different types. Using a dataset of communities' discussions about their governance, we are the first to identify the rules most strongly associated with positive community perceptions of governance: rules addressing who participates, how content is formatted and tagged, and rules about commercial activities. We conduct a longitudinal study to quantify the impact of adding new rules to communities, finding that after a rule is added, community perceptions of governance immediately improve, yet this effect diminishes after six months. Our results have important implications for platforms, moderators, and researchers. We make our classification model and rules datasets public to support future research on this topic.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Would answering this research question advance science without violating social contracts, such as violating pri- vacy norms, perpetuating unfair profiling, exacerbating the socio-economic divide, or implying disrespect to so- cieties or cultures? Yes, we believe that our work makes important contributions to the field of computational so- cial science wi...
-
[2]
Do your main claims in the abstract and introduction ac- curately reflect the paper’s contributions and scope? Yes, all claims made are concretely supported
-
[3]
Do you clarify how the proposed methodological ap- proach is appropriate for the claims made? Yes, our methodology is clear and rational given claims made
-
[4]
Do you clarify what are possible artifacts in the data used, given population-specific distributions? Yes, All artifacts are stated and addressed
-
[5]
Did you describe the limitations of your work? Yes, Lim- itations are discussed in 7.1
-
[6]
Did you discuss any potential negative societal impacts of your work? Yes, negative impacts are discussed in §3.5
-
[7]
Did you discuss any potential misuse of your work? Yes, misuse is discussed in §3.5
-
[8]
This is described in detail in Ap- pendix C.18
Did you describe steps taken to prevent or mitigate po- tential negative outcomes of the research, such as data and model documentation, data anonymization, respon- sible release, access control, and the reproducibility of findings? Yes, data is aggregated and anonymized to re- move sources of harm. This is described in detail in Ap- pendix C.18
Show all 33 references
-
[9]
Have you read the ethics review guidelines and ensured that your paper conforms to them? Yes, this paper con- forms to ethics review guidelines
-
[10]
Did you clearly state the assumptions underlying all the- oretical results? Not applicable
-
[11]
Have you provided justifications for all theoretical re- sults? Not applicable
-
[12]
Did you discuss competing hypotheses or theories that might challenge or complement your theoretical results? Not applicable
-
[13]
Have you considered alternative mechanisms or explana- tions that might account for the same outcomes observed in your study? Yes, alternative mechanisms are discussed in §7.1
-
[14]
Did you address potential biases or limitations in your theoretical framework? Yes, bias and limitations are dis- cussed in §7.1
-
[15]
Have you related your theoretical results to the existing literature in social science? Not applicable
-
[16]
Did you discuss the implications of your theoretical re- sults for policy, practice, or further research in the social science domain? Yes, we discuss implications and future directions in §7
-
[17]
Did you state the full set of assumptions of all theoretical results? Not applicable
-
[18]
Did you include complete proofs of all theoretical re- sults? Not applicable
-
[19]
Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? Code and data are available online at bdata.cs.washington.edu/mod-perceptions
-
[20]
Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? Yes, all train- ing and prompting details are specified in Appendix B
-
[21]
Did you report error bars (e.g., with respect to the ran- dom seed after running experiments multiple times)? Yes, bootstrapped 95% confidence intervals are reported for all results
-
[22]
Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? Yes, compute details are given in §3
-
[23]
Do you justify how the proposed evaluation is sufficient and appropriate to the claims made? Yes, discussed in §3
-
[24]
Do you discuss what is “the cost“ of misclassification and fault (in)tolerance? Yes, the cost of misclassification is addressed in §7.1
-
[25]
All are cited
If your work uses existing assets, did you cite the cre- ators? Yes, our work makes extensive use of existing models, among other assets. All are cited
-
[26]
Did you mention the license of the assets? Assets were shared without formal licensing beyond a request to cite relevant papers, which we comply with
-
[27]
Did you include any new assets in the supplemental ma- terial or as a URL? Code and data are available online at bdata.cs.washington.edu/mod-perceptions
-
[28]
Did you discuss whether and how consent was ob- tained from people whose data you’re using/curating? Yes, source data licenses are discussed
-
[29]
Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? Yes, the risks of deanonymization and improper conduct are discussed
-
[30]
If you are curating or releasing new datasets, did you dis- cuss how you intend to make your datasets FAIR? Yes, FAIR guidelines are present in Appendix C.18
-
[31]
Restrictive
If you are curating or releasing new datasets, did you create a Datasheet for the Dataset? Yes, Datasheet is in- cluded in Appendix C.18 Appendix A Codebook We provide a copy of our codebook, with further details and examples, as shared among authors for study reproduction. Th...
-
[2018]
Reddit Rules! Characterizing an Ecosystem of Governance. ICWSM. Frey, S.; and Sumner, R. W. 2018. Emergence of integrated insti- tutions in a large population of self-governing communities. PLoS ONE, 14. Frey, S.; Zhong, Q.; Bulat, B.; Weisman, W. D.; Liu, C.; Fuji- moto, S.; ...
2018 arXiv
-
[2021]
TheWebConf
Conversations Gone Alright: Quantifying and Predicting Prosocial Outcomes in Online Conversations. TheWebConf. Baumgartner, J.; Zannettou, S.; Keegan, B.; Squire, M.; and Black- burn, J. 2020. The Pushshift Reddit Dataset. arXiv:2001.08435. Chandrasekharan, E.; Samory, M.; Jha...
2020 arXiv
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.