REVIEW 4 major objections 5 minor 25 references
Decoding the Rule Book: Extracting Hidden Moderation Criteria from Reddit Communities
T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read Moderation criteria in Reddit communities can be extracted as explicit per-phrase removal scores using an interpretable transformer, and these scores match neural classifiers while revealing how similar norms are enforced differently.
desk verdict Useful application of PAT to Reddit moderation, but the CriteriaMatrix term scores are not validated and the central claim overstates what they establish. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Partial Attention Transformer (PAT): an interpretable text classifier that, for each input, partitions the comment into two spans, encodes each with BERT, sums the span scores, and trains on the full-comment removal label. During inference, a phrase alone is fed in and its softmax output becomes the term's moderation score. This lets a single model that never sees phrase-level labels produce calibrated phrase-level probabilities, which are assembled into the CriteriaMatrix—the shared vocabulary scored for each subreddit that carries the whole comparison.
What would settle it
Audit the criteria matrix directly: ask moderators of the studied subreddits to judge whether the top-scoring phrases are genuine violations, and test the PAT scores on later, out-of-time removal data against a context-aware full-text model. If moderators mostly reject the phrases, or if phrase scores lose most of their predictive power while the full-text model does not, the matrix is a description of the classifier rather than of community criteria.
Extended reading notes
Core claim
The central claim is that moderation criteria are "score tables of lexical expressions associated with content removal" and these tables can be produced automatically from historical deletion data. Using Partial Attention Transformer (PAT), a BERT-based model trained to predict removal from a comment while forcing its prediction to decompose into scores for text spans, the authors assign each candidate phrase in a shared vocabulary a 0-1 probability that a comment containing it gets moderated in a given subreddit. The resulting CriteriaMatrix reproduces neural model performance (PAT Bipartite F1 0.80 vs BERT FT 0.81) and yields interpretable, verifiable terms. The paper argues the matrix exp
Load-bearing premise
The load-bearing premise is that moderator decisions leave a reliable surface-lexical fingerprint: a phrase's removal score, learned from a balanced 50/50 sample, faithfully represents the community's real enforcement criteria rather than the model's own biases.
Editorial extensions
If this is right
- Moderators and platform analysts can audit enforcement by inspecting the top-scoring phrases rather than reading policy text.
- Enforcement differences among communities sharing the same stated rule can be quantified, enabling moderation calibration to local norms.
- The same PAT pipeline can be applied to any classification task with subdomains that may apply different criteria, not just Reddit.
- Supervised moderation models can be screened for unintended learned biases, such as flagging neutral "mod" mentions, before deployment.
- Because PAT Window gets near-BERT performance while using only short spans, term-level criteria are usable as lightweight screening rules in high-volume settings.
Reading between the lines
- Beyond the paper's claims, the criteria matrix may be as much a map of the classifier's decision boundary as of moderator intent; the authors' own spurious-correlation caveat suggests validating term scores against moderator-labeled removal reasons or prospective moderation decisions.
- The 50/50 balanced training sample likely inflates the apparent importance of rare phrases and could alter cross-community rankings; an inference is that sampling at the natural ~5% removal rate would yield a different, perhaps more policy-faithful, matrix.
- The clustering of personal attacks suggests a testable extension: if communities tolerate different attack subtypes, toxicity classifiers tuned per subtype rather than one global threshold should improve both precision and user satisfaction.
- Since the data come from 2016–2017, patterns like high Trump scores or location terms in r/Games may be time-bound; applying the method to current data would reveal drift in enforcement criteria.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes to extract explicit moderation criteria from historical Reddit moderation decisions by training a Partial Attention Transformer (PAT) per subreddit and using the model to assign 0-1 probabilities to individual vocabulary terms. These scores form a 'CriteriaMatrix' that is then used to compare community-specific enforcement patterns, to study moderator-related terms, and to cluster personal-attack language into subcategories. The authors report that PAT achieves classification performance 'comparable' to ChatGPT and 'effectively replicate[s] the performance of neural moderation models,' and they present qualitative analyses of r/fantasyfootball, r/Games, the term 'mod,' and nine personal-attack clusters as evidence of hidden, community-specific moderation criteria.
Significance. If the central claim were fully supported, the paper would make a useful contribution: a globally interpretable, lexical representation of moderation behavior that can be compared across communities, with potential practical value for moderators and for auditing automated moderation systems. The paper also has genuine strengths: it uses a widely studied moderation dataset, evaluates PAT in held-out classification tasks, reports concrete F1/AUC numbers, and explicitly acknowledges limitations such as spurious correlations and missing conversational context. However, the empirical support for the central claim is incomplete. The evaluation that is most relevant to the CriteriaMatrix—PAT's term-level inference mode—is not directly validated, and the headline performance claim is overstated relative to the paper's own Table 3. The qualitative findings are interesting but currently rest on post-hoc selection and manual inspection rather than on a validated measurement instrument.
major comments (4)
- [§3.5, Eq. (3), §5.1-5.3] The standalone term scores used to build CriteriaMatrix are computed by applying Eq. (3) to an isolated term, but PAT is trained only on the bipartite aggregation in Eq. (2). The model never sees single sequences during training, so these term-level inputs are out-of-distribution. The paper itself acknowledges in §3.5 that ρ1 may be inaccurate for very short sequences, yet PAT (Window)—the inference mode closest to term-level scoring—is evaluated by averaging over three-token windows in full comments, not by evaluating the actual standalone scores used in CriteriaMatrix. Consequently, the core interpretable output is not validated as a faithful proxy for moderators' criteria. Every downstream analysis in Sections 5.1-5.3 inherits this gap. The authors should provide direct evidence that term-level PAT scores track held-out removal rates (e.g., compare scores with empirical removal propor
- [Abstract, §4, Table 3] The claim that extracted lexical patterns 'effectively replicate the performance of neural moderation models' is not supported by the reported numbers. In Table 3, subreddit-aware BERT (FT) achieves F1 0.81 / AUC 0.90, PAT (Bipartite) achieves 0.80 / 0.89, and PAT (Window) achieves 0.69 / 0.83. Since CriteriaMatrix is built from PAT's short-span scoring mode, PAT (Window) is the relevant comparison, not PAT (Bipartite). A 12-point F1 gap and a 7-point AUC gap relative to BERT (FT) is a meaningful degradation, not 'effectively' replication. The authors should either recalibrate the wording or, better, add an evaluation that directly measures the predictive accuracy of the extracted score tables (e.g., ranking comments by their maximum term score and measuring AUC on held-out moderation outcomes).
- [§5.1, Figure 3] The community-specific norm analysis is based on an explicitly post-hoc selection of two subreddits with the largest BERT-vs-ChatGPT F1 gaps, followed by manual inspection of terms. The r/Games interpretation—that game titles/company names are removed for 'not providing additional values' and that location names reflect strict off-topic enforcement—is illustrated with a small number of examples and no systematic verification. The authors themselves note that for r/fantasyfootball they cannot directly verify whether flagged phrases occurred in rule-violating threads. These findings are plausible hypotheses, but they do not establish that the CriteriaMatrix 'reveals' enforcement criteria. To support the central claim, the authors could formulate a small number of pre-registered or at least clearly separated hypotheses and test them on held-out data, or provide structured annotation of a ra
- [§5.3, Table 5, Figure 4] The claim that personal-attack clusters reveal 'underlying subcategories of the toxic speech classification' depends on a k-means clustering with k=100 chosen without stated justification, a self-reported Silhouette score of 0.15, and manual categorization into named clusters. A Silhouette score of 0.15 indicates weak clustering structure, and the cluster names in Table 8 are acknowledged to be illustrative rather than precise. Moreover, Figure 4 plots per-subreddit score differences for six unnamed subreddits without confidence intervals or statistical tests for the claimed community-specific tolerances. These results are suggestive but should be presented as exploratory, or the authors should supply quantitative support (e.g., evaluation of cluster stability across seeds, permutation tests for subreddit-by-cluster interactions).
minor comments (5)
- [§1, Introduction] There are several broken sentences and typos, e.g., 'PAT is further used to build analysis using PAT provides various insights' and 'We hypothesize that they can be meaningfully represented.' The paper would benefit from a careful copyedit.
- [§3.6] The vocabulary construction is underspecified: 'For each n values, we selected the top 10,000 most frequent n-gram terms, based on probability scoring from an off-the-shelf large language model, Llama-3' is unclear. What 'probability scoring' is used, and why is it needed if the n-grams are selected by frequency? Also, the 'high-score span extraction threshold' is not defined, making the pipeline difficult to reproduce.
- [Table 3] The header 'PAT (Bipratite)' contains a typo; should be 'PAT (Bipartite)'.
- [Table 8] Several sample terms appear to be subword tokens (e.g., 'youll', 'you don', 'you a', 'you are'), which weakens the readability of the cluster examples. Please clarify whether these are raw tokens or detokenized phrases.
- [§5.2] The synthetic-comment experiment uses 50 ChatGPT/Claude-generated comments and finds that 16 of 60 classifiers flag all 50 as needing moderation. The paper correctly lists possible explanations, but the base rate of 'mod' removal in real data is also high (over 80% in 34% of subreddits). It would strengthen the analysis to report the empirical removal rate for 'mod' comments conditional on the specific synthetic comment templates, or to compare with a set of control terms with similarly frequent mentions of other community roles.
Circularity Check
CriteriaMatrix term scores are the fitted outputs of a classifier trained on the same removal labels they are claimed to reveal; partial circularity, mitigated by held-out classification and one external validation.
-
fitted input called prediction
[Section 3.1, Section 3.6, Section 5.1]
"We conceptualize the challenge of understanding community-specific moderation as a vocabulary scoring problem... our goal is to build a vocabulary V ... and community-specific score matrices S_i ... where each element s_j^i indicates the contribution of term v_j to moderation decisions in community i. ... Finally, we apply each PAT_i to score all terms in vocabulary, to get a score matrix M, where M_i,j indicates the score that PAT for subreddit i has assigned to a term j."
The 'contribution of a term to moderation decisions' is operationally defined as the score assigned by PAT, a model trained on the very same moderation decisions (removal labels). The CriteriaMatrix is therefore, by construction, the fitted decision surface of the classifier, not an independently measured moderator criterion. When the paper interprets high-scoring terms as evidence of community norms (e.g., r/Games locations, r/fantasyfootball phrases), it is re-describing the model's learned associations. The paper does validate PAT on held-out labels and checks real 'mod' removal rates, but those do not establish that most term scores correspond to moderators' true intent. The paper itself concedes in Section 5.4 that 'supervised models may learn spurious correlations rather than moderat
full rationale
The paper's central representation, the CriteriaMatrix, is built by applying a trained PAT model to a vocabulary and recording its softmax scores. Since PAT is trained to predict whether a comment was removed, the scores are, by definition, the model's fitted function of the training labels. Presenting these scores as 'moderation criteria' is therefore a form of fitted-input-called-prediction: the extracted criteria are the classifier's own outputs, not an external ground truth about moderator intent. This is the main circular step. However, the circularity is partial rather than total. The paper evaluates PAT's classification performance on held-out data (Table 3), which shows the learned patterns are predictive of removal outcomes in a generalizable way. The 'blaming moderators' analysis (Section 5.2) also checks actual moderation rates of comments containing 'mod', providing an external anchor for one specific finding. Additionally, the paper's Limitations section acknowledges missing context, balanced sampling, and the risk of spurious correlations, which tempers the strength of the claims. The self-citation to prior PAT work (Kim et al., 2023) is not load-bearing in a circular way; the architecture is described and evaluated rather than assumed correct via citation. Overall, the central 'criteria discovery' claim reduces in large part to the fitted model's outputs, but independent predictive checks and one external validation keep it from being fully definitional. Score 5 reflects this partial circularity.
Assumptions & free parameters
free parameters (4)
- k-means cluster count k =
100
- n-gram vocabulary threshold =
top 10,000 per n
- PAT (Window) window size =
3 tokens
- High-score span extraction threshold =
unspecified
assumptions (4)
- domain assumption Moderation decisions can be meaningfully represented by predictive lexical patterns.
- domain assumption Labels y indicate genuine moderator violations.
- domain assumption PAT term-level output rho is a well-calibrated probability of moderation.
- domain assumption Llama-3 probability scoring selects natural n-grams.
Cite this review
Pith. "Pith review of Decoding the Rule Book: Extracting Hidden Moderation Criteria from Reddit Communities." pith.science (2026). https://pith.science/paper/M3FKYAYE
@misc{pith2026250902926,
author = {Pith},
title = {Pith review of: Decoding the Rule Book: Extracting Hidden Moderation Criteria from Reddit Communities},
year = {2026},
howpublished = {\url{https://pith.science/paper/M3FKYAYE}},
note = {Machine review of arXiv:2509.02926}
}
read the original abstract
Effective content moderation systems require explicit classification criteria, yet online communities like subreddits often operate with diverse, implicit standards. This work introduces a novel approach to identify and extract these implicit criteria from historical moderation data using an interpretable architecture. We represent moderation criteria as score tables of lexical expressions associated with content removal, enabling systematic comparison across different communities. Our experiments demonstrate that these extracted lexical patterns effectively replicate the performance of neural moderation models while providing transparent insights into decision-making processes. The resulting criteria matrix reveals significant variations in how seemingly shared norms are actually enforced, uncovering previously undocumented moderation patterns including community-specific tolerances for language, features for topical restrictions, and underlying subcategories of the toxic speech classification.
Figures
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Shamim Ahmed, M Shamim Kaiser, Mohammad Shahadat Hossain, and Karl Andersson. 2024. A comparative analysis of lime and shap interpreters with explainable ml-based diabetes predictions. IEEE Access
work page 2024
-
[4]
Nadia Burkart and Marco F Huber. 2021. A survey on the explainability of supervised machine learning. Journal of Artificial Intelligence Research, 70:245--317
work page 2021
-
[5]
Eshwar Chandrasekharan, Chaitrali Gandhi, Matthew Wortley Mustelier, and Eric Gilbert. 2019. Crossmod: A cross-community learning-based system to assist reddit moderators. Proceedings of the ACM on human-computer interaction, 3(CSCW):1--30
work page 2019
-
[6]
Eshwar Chandrasekharan, Mattia Samory, Shagun Jhaver, Hunter Charvat, Amy Bruckman, Cliff Lampe, Jacob Eisenstein, and Eric Gilbert. 2018. The internet's hidden rules: An empirical study of reddit norm violations at micro, meso, and macro scales. Proceedings of the ACM on Human-Computer Interaction, 2(CSCW):1--25
work page 2018
-
[7]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171--4186
2019
-
[8]
Casey Fiesler, Jialun Jiang, Joshua McCann, Kyle Frye, and Jed Brubaker. 2018. Reddit rules! characterizing an ecosystem of governance. In Proceedings of the International AAAI Conference on Web and Social Media, volume 12
work page 2018
Show all 25 references
-
[9]
Agam Goyal, Charlotte Lambert, Yoshee Jain, and Eshwar Chandrasekharan. 2024. Uncovering the internet's hidden values: An empirical study of desirable behavior using highly-upvoted content on reddit. arXiv preprint arXiv:2410.13036
2024 arXiv
-
[10]
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783
2024 arXiv
-
[11]
Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. arXiv preprint arXiv:2203.09509
2022 arXiv
-
[12]
Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. 2023. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674
2023 arXiv
-
[13]
Youngwoo Kim, Razieh Rahimi, and James Allan. 2023. https://aclanthology.org/2023.findings-emnlp.456 Conditional natural language inference . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6833--6851, Singapore. Association for Computational Li...
2023
-
[14]
Youngwoo Kim, Razieh Rahimi, and James Allan. 2024. Discovering biases in information retrieval models using relevance thesaurus as global explanation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 19530--19547
2024
-
[15]
Mahi Kolla, Siddharth Salunkhe, Eshwar Chandrasekharan, and Koustuv Saha. 2024. Llm-mod: Can large language models assist content moderation? In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, pages 1--8
2024
-
[16]
positive reinforcement helps breed positive behavior
Charlotte Lambert, Frederick Choi, and Eshwar Chandrasekharan. 2024. " positive reinforcement helps breed positive behavior": Moderator perspectives on encouraging desirable behavior. Proceedings of the ACM on Human-Computer Interaction, 8(CSCW2):1--33
2024
-
[17]
Scott M Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30
2017
-
[18]
Yair Neuman and Yochai Cohen. 2023. Ai for identifying social norm violation. Scientific Reports, 13(1):8103
2023
-
[19]
Chan Young Park, Shuyue Stella Li, Hayoung Jung, Svitlana Volkova, Tanu Mitra, David Jurgens, and Yulia Tsvetkov. 2024. Valuescope: Unveiling implicit norms and values via return potential model of social interactions. In Findings of the Association for Computational Linguisti...
2024
-
[20]
Chan Young Park, Julia Mendelsohn, Karthik Radhakrishnan, Kinjal Jain, Tushar Kanakagiri, David Jurgens, and Yulia Tsvetkov. 2021. Detecting community sensitive norm violations in online conversations. In Findings of the Association for Computational Linguistics: EMNLP 2021, p...
2021
-
[21]
Richard Phillips, Kyu Hyun Chang, and Sorelle A Friedler. 2018. Interpretable active learning. In Conference on fairness, accountability and transparency, pages 49--61. PMLR
2018
-
[22]
Kavel Rao, Liwei Jiang, Valentina Pyatkin, Yuling Gu, Niket Tandon, Nouha Dziri, Faeze Brahman, and Yejin Choi. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.812 What makes it ok to set a fire? iterative self-distillation of contexts and rationales for disambiguating d...
2023 doi
-
[23]
why should i trust you?
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016. https://doi.org/10.1145/2939672.2939778 "why should i trust you?": Explaining the predictions of any classifier . In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining...
2016
-
[24]
Paul R \"o ttger, Bertie Vidgen, Dong Nguyen, Zeerak Waseem, Helen Margetts, and Janet Pierrehumbert. 2021. Hatecheck: Functional tests for hate speech detection models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Int...
2021
-
[25]
Galen Weld, Amy X Zhang, and Tim Althoff. 2024. Making online communities ‘better’: a taxonomy of community values on reddit. In Proceedings of the International AAAI Conference on Web and Social Media, volume 18, pages 1611--1633
2024
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.