{"id":"06744b18-44be-4e5e-ba55-a9626623c1bd","arxiv_id":"2501.14163","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Using 67,545 rules from 5,225 Reddit communities, the paper identifies rule types associated with more positive governance perceptions and finds the positive effect of adding rules wears off after about six months.","lead":"This study analyzed 67,545 rules from 5,225 Reddit communities over five years, linking rule types and phrasing to how members publicly discuss their moderators. Certain rules, such as those about who may participate and how posts are formatted, accompany more positive governance perceptions, and adding rules gives a temporary boost that fades after about six months.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's 'after a rule is added, governance perceptions immediately improve' claim may be a compositional artifact: the outcome is the fraction of governance-discussing posts, and rule additions themselves generate governance discussion, shifting that fraction without any real sentiment change.","rationale":"I read the paper as a careful, well-executed observational study; the descriptive findings (§4) are robust and the rule taxonomy and classification pipeline are a useful contribution. The reader's conditional verdict is appropriate. My stress-test focuses on a specific, internal mechanism within the outcome measure that the reader flagged broadly: because the outcome in §5 and §6 is a fraction of governance-discussing posts, the treatment (a rule addition) can directly affect the denominator by generating new governance discussion about the rule itself. This is not just a general 'public vs. private' validity concern; it is a compositional feedback loop that can manufacture the exact pattern reported in Figure 6—positive rising, negative falling, neutral rising for a few months. The paper's own limitation statement in §7.1 strengthens this reading. A count-based or per-capita outcome would settle the point. I agree with the reader that the IPTW balance failures and classifier F1 for User-Related rules temper confidence, but those are secondary; the denominator-feedback issue is the single most load-bearing gap because it threatens the headline longitudinal claim while also having implications for the cross-sectional 'who participates' finding. The conditional verdict stands, with the added requirement that the authors test for this compositional artifact.","tokens_in":25186,"tokens_out":13188,"duration_ms":120048,"concrete_test":"Using the released dataset, compute for each of the 6,645 rule-addition events the per-community count of Weld et al.-classified governance posts/comments (the denominator) in the 12-month pre-baseline and each 2-month post window. First test whether the denominator spikes immediately after rule additions; if it does, re-estimate §6 using sentiment counts normalized by total community posts/comments (or per capita) rather than by governance-discussion count. If the 'improvement' shrinks below significance or reverses under the count-based outcome, the fraction-based finding is a compositional artifact and the abstract's 'impact' claim must be reframed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is not just that public discussion reflects private attitudes (§7.1), but that the dependent variable used in §5 and §6—the fraction of posts/comments discussing governance that are positive or negative—is not itself changed by the treatment. In §3.3 and §6, the outcome is defined as a fraction whose denominator is the set of posts/comments the Weld et al. pipeline classifies as 'discussing governance.' Rule additions are governance events: they generate posts and comments about the new rule ('what does rule 7 mean?', 'why did mods add this?'), which are exactly what the pipeline selects. The paper's own §7.1 acknowledges it 'measure[s] discussions about governance and moderators generally, not rules specifically.' If a rule addition increases the volume of governance discussion and shifts its composition toward neutral or positive meta-commentary, the reported +0.61 pp positive and –0.77 pp negative changes in §6 can occur even if the underlying sentiment of regular community activity is unchanged. The concurrent +0.71 pp rise in neutral sentiment at 2–4 months (§6) is consistent with this mechanism. The same denominator problem threatens the cross-sectional results: communities with User-Related rules (classification F1 = 0.61) or Post Format rules may discuss those rules more, inflating positive or neutral fractions for reasons unrelated to governance quality. This is testable and, if confirmed, would invalidate the abstract's causal 'impact' wording.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies Reddit community rules at scale: it reconstructs rule timelines from Wayback Machine snapshots for 5,225 communities over 2018–2023, classifies 67,545 unique rules into a 17-attribute taxonomy via a GPT-4o retrieval-augmented classifier, describes rule prevalence by community type, tests associations between rule presence and a sentiment-based measure of governance perceptions using IPTW, and reports pre-post changes in these perceptions around rule additions. The central empirical claims are that rules about participation, post formatting, and commercialization are associated with more positive governance perceptions, and that rule additions are followed by a temporary improvement that fades after about six months.","tokens_in":25458,"tokens_out":6886,"duration_ms":62583,"significance":"If the associations and temporal claims survive closer inspection, this would be the largest-scale evidence linking specific rule types to governance sentiment, and the longitudinal finding (if not a measurement artifact) would be novel and actionable. The paper's strengths include a large public dataset of rule timelines, a careful taxonomy development with human IRR and model evaluation, bootstrap uncertainty estimates, and a mostly well-executed IPTW adjustment with a published balance table. The descriptive results in §4 are solid and reproducible. The main risk is the validity of the outcome measure for the causal claims, because the measure is a fraction of governance-discussing posts and comments.","major_comments":[{"comment":"The pre-post analysis is vulnerable to a compositional artifact. The outcome is the fraction of posts/comments classified as discussing governance that are positive/neutral/negative (§3.3). A rule addition is itself a governance event and is likely to generate posts/comments about the new rule (e.g., \"what does rule 7 mean?\"). If those posts are disproportionately neutral or positive, the reported +0.61 pp positive and -0.77 pp negative changes can occur without any change in sentiment toward ongoing governance. The paper's own §7.1 states that \"we also measure discussions about governance and moderators generally, not rules specifically.\" The concurrent +0.71 pp neutral increase at 2–4 months is consistent with this mechanism. The authors should re-estimate the analysis excluding posts/comments that mention the added rule (or otherwise rule-related discussion), or show that the result is robust in a sample of governance discussion that is unrelated to the specific rule change.","section":"§6 and Figure 6"},{"comment":"The IPTW balance check reports SMD = 0.96 for community size in the control group, far above the 0.25 threshold the paper uses. The text in §3.4 states \"no SMD exceeds 1.00\" as a reassurance, but an SMD of 0.96 means the weighted control group is essentially unadjusted on community size, so the Divisive Content comparison in Figure 5m should not be interpreted as confounder-adjusted. This does not affect the central rule types, but the summary statement that balance is achieved in 235 of 238 cases is misleading without flagging the magnitude of this failure.","section":"Appendix C, specifically C.13 (Divisive Content)"},{"comment":"The cross-sectional associations share the same denominator issue. Communities with User-Related or Post Format rules may discuss those rules more often, so the fraction of governance-discussing posts that is positive/neutral may differ for reasons unrelated to overall governance satisfaction. Because the abstract singles out \"rules addressing who participates\" as a key finding, and the automated F1 for User-Related rules is only 0.61 (Table 1), the authors should provide a robustness check that controls for the volume of governance discussion or restricts the outcome to governance discussion that does not reference the rule category itself.","section":"§5 and §3.3"},{"comment":"The language of \"impact\" and \"immediately improve\" overstates what a pre-post observational design can establish. The paper's own §7.1 notes that difference-in-difference designs would be needed to handle unobserved confounding. I recommend rewording the abstract and §6 to describe associations and temporal sequences rather than causal effects, unless the authors add a control group (e.g., matched communities without rule changes) or another design that supports causal interpretation.","section":"Abstract and §6"}],"minor_comments":[{"comment":"The abstract reports 67,545 unique rules across 5,225 communities, while the dataset datasheet (Appendix C.18) reports 73,087 unique rules across 6,120 communities; these numbers should be reconciled.","section":"Abstract and Appendix C.18"},{"comment":"Given that User-Related and Peer Engagement are central to the findings, it would be helpful to report per-class precision and recall, not only F1, for these categories.","section":"Table 1 and Table 3"},{"comment":"The sentence \"Our taxonomy simplifies the Fiesler et al. (2018) taxonomy, with with a slightly smaller set\" contains a duplicated word.","section":"§3.2"},{"comment":"The phrase \"For each community's rules periods\" should be clarified to indicate whether the unit of analysis is the community-period, since communities can appear in multiple rules periods.","section":"§3.3"},{"comment":"The text says \"after adjusting for confounding factors\" but does not specify whether the unit of analysis is a community or a community-period; this matters because the same community can contribute multiple rules periods to the IPTW analysis.","section":"§5"},{"comment":"The outcome measure and community-topic labels both come from Weld et al. (2024), which is self-cited from the same group. This is not circular because the rule taxonomy and associations are new, but the paper would be strengthened by acknowledging this shared-source dependency and, ideally, by an independent replication of the sentiment pipeline.","section":"§3.3 and §4.3"}],"recommendation":"major_revision","confidential_remarks":"The compositional-artifact concern in §6 is the main threat to the headline finding. The authors should be asked to address it directly with a rule-mention exclusion or an equivalent robustness check. The paper otherwise appears methodical and within the scope of the venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know before you read this one. First, it is the largest descriptive map of Reddit rules to date, with data and code released, and it makes a plausible first cut at linking rule attributes to publicly expressed governance sentiment. That part is worth your time. Second, the abstract's claim that adding a rule makes governance perceptions improve, with effects wearing off after six months, is not backed by the analysis as it stands. The outcome variable is the fraction of governance-discussing posts that are positive or negative; adding a rule is itself a governance event that generates discussion about the rule. That new discussion lands in the denominator and can shift the fraction without any change in how people feel about the community. The paper acknowledges it measures discussion about governance 'generally, not rules specifically,' but it does not confront the fact that the treatment changes the composition of the outcome's denominator.\n\nWhere the paper is solid: the descriptive prevalence results, the tone/target/topic taxonomy, and the clear documentation of methods. The IPTW cross-sectional analysis is mostly well executed, with balance achieved in 235 of 238 condition-covariate pairs. The honest limitations section is a plus.\n\nSoft spots: three unbalanced covariate pairs, including a control-group SMD of 0.96 for community size when comparing Divisive Content rules, should be addressed. User-Related rule classification F1 of 0.61 makes the strongest positive association claim rest on noisy labels. The longitudinal section has no control group, and the 'impact' language in the abstract overstates what a before-after comparison can support. The compositional issue I mentioned is the main reason to be skeptical of the §6 numbers, and it is testable: restrict the outcome to governance posts that are not about the new rule, or use matched communities with no rule change as controls.\n\nWho benefits: people studying online moderation, governance, and community management. The dataset is a useful public asset. The paper deserves a serious referee, but I would send it back conditional on a rewrite of §6 that removes causal wording, checks the denominator artifact, and reports sensitivity analyses. As it stands, the descriptive and cross-sectional findings are publishable; the temporal wear-off claim is not yet.","headline":"A substantial descriptive study of Reddit rules whose headline longitudinal claim ('rule additions improve governance perceptions') is likely a compositional artifact and needs a rewrite.","tokens_in":26039,"tokens_out":2395,"would_cite":true,"duration_ms":22178,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that the rules a Reddit community publishes are measurably connected to how members talk about their moderators, with participation, formatting, and commercialization rules linked to the most positive sentiment.","keywords":["online community governance","Reddit rules","content moderation","rule taxonomy","governance perceptions","longitudinal analysis","inverse probability of treatment weighting","computational social science"],"falsifier":"A direct comparison would settle it: survey a random sample of members across these same communities with a pre-registered governance-satisfaction instrument, and re-run the rule-type and rule-addition analyses on the survey responses; if the +0.61 percentage-point improvement after rule additions and the associations with participation and formatting rules disappear, the public-discussion proxy is what produced the paper's pattern.","tokens_in":24950,"feed_emoji":"📜","tokens_out":8495,"duration_ms":66366,"temperature":0.7,"pith_summary":"This paper tries to establish, at a scale not attempted before, that the rules a Reddit community publishes are systematically connected to how its members publicly discuss governance. It reconstructs five-plus-year rule timelines for 5,225 communities covering more than two-thirds of Reddit activity, classifies the 67,545 unique rules it finds into a 17-attribute taxonomy, and compares communities with and without each rule type after adjusting for community topic and size. The core claims are that rules about who may participate, about post formatting and tagging, and about commercial activity are associated with more positive perceptions of governance, and that adding a rule is followed by a small immediate improvement in sentiment that fades after roughly six months. If these associations are real, moderators and platforms would have an empirical basis for rule choices where currently they mostly have intuition.","feed_headline":"Reddit rule additions lift goodwill for six months","feed_subtitle":"Across 5,225 communities, participation and formatting rules track the most positive sentiment about moderators.","key_machinery":"The central object is the labeled rule period: a block of time in which a community's published sidebar rule set is constant, reconstructed from archive snapshots with an average start-and-end uncertainty of about seventeen days. Each rule period is labeled by a 17-attribute taxonomy of tone (prescriptive versus restrictive), target (post content, post format, user-related), and topic (such as spam, commercialization, tagging and flairing, and brigading), and is paired with the community's governance-perception score, the fraction of governance-discussing posts and comments in that period with positive, neutral, or negative sentiment. Cross-sectional comparisons use inverse probability of treatment weighting, a procedure that reweights communities by their propensity to have the rule so that treated and untreated groups resemble the overall population, while the longitudinal analysis compares the twelve months before a rule addition with successive two-month windows after it.","core_discovery":"The paper claims to be the first large-scale empirical link between specific rule types and community members' expressed perceptions of governance on Reddit. Its central finding is that rule choice is not arbitrary: even after matching communities on topic and size, communities that publish rules about who is allowed to participate, about post formatting and tags, and about commercial activity show more positive and less negative governance sentiment than comparable communities without those rules, while rules phrased restrictively and rules mentioning bans or brigading are associated with more negative sentiment. The companion longitudinal result is that adding a new rule is followed by a small immediate improvement in governance sentiment, with positive sentiment up 0.61 percentage points and negative sentiment down 0.77 points, but this effect is no longer distinguishable from baseline after about six months. The paper interprets the fade as evidence that part of a rule change's benefit comes from the signal that moderators are responding to their community, not only from the rule's enforcement.","pith_inferences":["If the six-month decay is a signaling effect rather than an enforcement effect, which the paper suggests but does not test, then visible moderator engagement such as announcements or rule refreshers might reproduce the benefit without accumulating permanent rules.","The observational design cannot distinguish whether participation rules improve sentiment by pre-filtering norm-compatible users or by changing existing members' behavior; a randomized field experiment varying rule wording across similar communities would separate the two.","The released timeline data would also support studies of rule removal and of consistency between posted rules and enforcement, both of which this paper leaves out.","A concrete testable extension is that communities adding a rule right after a visible incident should show a larger immediate sentiment shift than communities adding a rule proactively, because the responsiveness signal is stronger."],"forward_implications":["Moderators who want better governance sentiment have a concrete shortlist: adopt rules about who may participate, about post formatting and tagging, and about commercial activity, and phrase rules prescriptively where possible.","A single rule addition is not a durable governance fix; the paper's estimate is an immediate improvement of about 0.61 percentage points in positive sentiment that statistically disappears after roughly six months.","Because user-related rules are rare, appearing in about 21% of communities, yet are strongly associated with positive sentiment, there is room for many communities to adopt this rule type and potentially shift their governance perception.","Platforms could build rule 'starter packs' and topic- and size-based rule recommendations from the released timeline data, since the paper shows that rule sets are patterned by community topic and size."],"supporting_citations":[{"why":"Supplies the governance-perception sentiment classification pipeline used as the outcome measure and the community-topic classifications used as covariates.","marker":"Weld et al. 2024"},{"why":"Provides the base rule taxonomy that the paper extends into its 17-attribute codebook.","marker":"Fiesler et al. 2018"},{"why":"Provides a second rule taxonomy and a prior Wayback-Machine-based reconstruction of rule timelines that this study expands.","marker":"Fang, Yang, and Zhu 2023"},{"why":"Supplies the inverse probability of treatment weighting method used to adjust the cross-sectional comparisons for community topic and size.","marker":"Austin and Stuart 2015"},{"why":"Establishes the prior longitudinal analysis of how Reddit communities' rules evolve, which this paper extends to a longer period and larger community set.","marker":"Reddy and Chandrasekharan 2023"},{"why":"Provides the Pushshift-derived community metadata used for community size and activity measures.","marker":"Baumgartner et al. 2020"}],"fun_headline_variants":["Reddit rule boost fades in six months, study finds","Participation rules linked to happier Reddit communities","Adding Reddit rules helps for a while, then effect dims","First large-scale Reddit rule study reveals governance links","Rule clarity on Reddit tied to better governance perceptions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every result rests on treating the sentiment of public posts and comments about governance as an accurate measure of how community members really feel about their moderators, and the paper itself notes that public discussion may not match privately held attitudes.","fun_headline_variants_meta":{"raw":{"variants":["Reddit rule boost fades in six months, study finds","Participation rules linked to happier Reddit communities","Adding Reddit rules helps for a while, then effect dims","First large-scale Reddit rule study reveals governance links","Rule clarity on Reddit tied to better governance perceptions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1447,"prompt_tokens":1004,"completion_tokens":443,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":363}},"tokens_in":620,"tokens_out":443,"duration_ms":4637,"temperature":1.0,"reasoning_tokens":363,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:19:21.793518+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct comparison would settle it: survey a random sample of members across these same communities with a pre-registered governance-satisfaction instrument, and re-run the rule-type and rule-addition analyses on the survey responses; if the +0.61 percentage-point improvement after rule additions and the associations with participation and formatting rules disappear, the public-discussion proxy is what produced the paper's pattern.","supporting_citations":[],"review_version":1}