{"id":"050542cf-f66b-43ca-941d-b3caf6d089eb","arxiv_id":"2506.07667","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Twitch's AutoMod flags only about 22% of hateful comments, misses most implicit hate, and blocks a large share of non-hateful uses of sensitive words.","lead":"This paper tested Twitch's automated moderation system by sending over 100,000 hateful and benign chat messages through private test streams. It found that the system catches only about one in five hateful messages, while blocking many harmless uses of sensitive words.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported recall numbers assume dataset offensiveness/toxicity labels are ground truth for Twitch's hate-speech policy; if they are not, the headline percentages measure a different target.","rationale":"The reader's weakest assumption is the same one I would flag: the audit measures recall against offensiveness/toxicity labels, not against a validated operationalization of Twitch's hate-speech policy. This is load-bearing because the paper's headline results are recall numbers; if the denominator is wrong, every percentage inherits the error. The direction is not obvious: more restrictive policy-valid labeling would raise recall on the remaining positives, while a broader interpretation of 'hateful conduct' could lower it. The audit's independent strengths—real siloed stream tests, repeated runs, released code and data, and consistency with Hartmann et al.—support the qualitative conclusions (weak context awareness, slur sensitivity, overblocking of sensitive language) but they do not validate the exact 22%/6%/89.5% point estimates against Twitch's policy. A small annotator validation study would settle this at low cost. Because the concern is real but addressable and the reader already marked the paper CONDITIONAL, I do not move the verdict.","tokens_in":25754,"tokens_out":8754,"duration_ms":109483,"concrete_test":"Take a stratified random sample of 500 positives and 500 negatives from each dataset (SBIC at threshold 1.0, DynaHate, ToxiGen, IHC) and have three annotators independently label each example as violating or not violating Twitch's hate-speech policy as quoted in §1, blind to dataset labels and observed moderation outcomes. Recompute overall and per-dataset recall using only examples with majority 'violates policy'. If policy-valid recall remains near 22% (and near 6-7% on ToxiGen/IHC), the central claim survives; if it rises materially, the headline overstates AutoMod's failure because the original ground truth included non-hateful offensiveness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (22% recall overall; 6-7% on ToxiGen/IHC; up to 94% bypassing moderation) is computed against dataset labels that are not validated as operationalizations of Twitch's hate-speech policy. SBIC is an offensiveness corpus; the overall evaluation uses a hand-set threshold of offensiveness score 1.0 (§3.2), and the filter subsets use 0.5 (§B.2). ToxiGen positives are selected by RoBERTa toxicity scores, not by Twitch's 'Discrimination & Slurs' policy quoted in §1. An 'unflagged' message is only a true miss if it would violate that policy; mere offensiveness, profanity, or toxicity is not enough. The paper provides no validation that dataset positives align with that policy. If a substantial share of positives are offensive but not hateful toward a protected group, then the reported recall is agreement with offensiveness labels, not hate-speech recall, and can differ materially from the policy-relevant number. This affects not only the headline 22% but also the per-filter and target-group analyses and the policy-adherence case study in §5. A secondary version of the same measurement issue is the inferred pre-filter category (§4.1, §C.4): messages absent from both the receiver bot and Pubsub are classified as service-level moderation, but transient message loss would mislabel misses as catches, changing the numerator and denominator of recall.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a large-scale algorithmic audit of Twitch's AutoMod hate-speech moderation tool. The authors created siloed test streams and, using three bots per pipeline, sent over 100,000 (they later state around 300,000) messages drawn from four hate-speech datasets (SBIC, DynaHate, ToxiGen, IHC) to measure AutoMod's moderation decisions. They report that even at the maximum filtering level AutoMod flags only 22% of hateful content overall, with particularly low recall on implicit-hate datasets (6-7% on ToxiGen and IHC). Case studies show that replacing demographic terms with explicit slurs leads to 100% removal, that semantic-preserving perturbations sharply reduce moderation rates (from 100% to 4% for some obfuscations), and that 89.5%-98.5% of benign pedagogical or empowering uses of sensitive terms are blocked at alpha=2 and alpha=4, respectively. The paper also analyzes filter-wise recall, filter precision, target-group differences, and the contribution of an inferred 'pre-filtering' stage.","tokens_in":25953,"tokens_out":5327,"duration_ms":65646,"significance":"If the findings hold, this is a valuable third-party audit of a widely deployed commercial moderation system, one that is rarely audited in the algorithmic-accountability literature. The paper's strengths include a controlled, siloed experimental design that limits harm from posting hateful content, a large message corpus, a reproducible pipeline with code and data released, and careful reporting of case studies that connect the quantitative results to plausible underlying mechanisms. The observation that AutoMod relies on explicit slurs rather than context aligns with concurrent work on commercial moderation APIs (Hartmann et al., 2025), lending external consistency. The main limitation is that the headline recall figures treat dataset offensiveness/toxicity labels as ground truth for Twitch's actual hate-speech policy, which is a real but fixable gap in the central measurement.","major_comments":[{"comment":"The headline recall numbers (22% overall; 19% on SBIC; 6-7% on ToxiGen/IHC) are computed using dataset labels as ground truth for Twitch's 'Discrimination & Slurs' policy. SBIC is an offensiveness corpus, and the chosen threshold of 1.0 (and 0.5 for filter subsets) selects examples that all annotators found offensive, not necessarily examples that violate the policy quoted in §1. ToxiGen positives are selected using RoBERTa toxicity scores, not Twitch's policy. No validation is provided that these labels align with the policy-relevant concept of hateful conduct toward a protected group. If a substantial fraction of dataset positives are offensive but not policy-relevant hate, the reported recall measures agreement with offensiveness labels rather than hate-speech recall. This is load-bearing because the abstract and introduction state the conclusion in policy terms. The authors should either (i) manually annotate a random sample of positives against Twitch's policy to demonstrate high agreement, or (ii) consistently re-frame all recall figures as 'recall with respect to dataset offensiveness/toxicity labels' and temper the policy-level conclusions in the title, abstract, and introduction.","section":"§3.2, §B.2, Table 1"},{"comment":"The 'pre-filtered' category is inferred from messages that appear in neither the receiver bot's IRC stream nor the Pubsub moderation queue. The authors note that they 'suspect' and 'speculate' this is service-level moderation, and they manually verified only 'some' such messages. Messages lost to transient connection failures or rate-limit drops would be misclassified as pre-filtered, inflating recall (because these messages are counted as moderated) and distorting the filter-wise analysis in Table 2 and Figure 3, where pre-filtering accounts for 54.8% of SSG-related moderation and large shares of other categories. The paper needs a direct control: e.g., send a set of known-benign messages through the same pipeline and measure the disappearance rate, or re-send unobserved messages to see whether they appear on retry. Without such a check, the pre-filtering mechanism itself is confounded with message loss.","section":"§4.1, §C.4, Table 2"},{"comment":"The three case studies rely on small, manually selected samples: 110 counterfactual false negatives, 20 sensitive fragments for the policy-adherence study, and 50 fragments for perturbation robustness. The paper reports point estimates (100% counterfactual recall, 89.5%/98.5% blocking of pedagogical/empowering examples, and drops to 4% for partial obfuscation) without confidence intervals, selection protocols, or the total number of generated examples actually tested. A handful of selection or labeling choices could materially change these percentages. The authors should report full sample sizes after generation, the criteria for manual selection, and bootstrap confidence intervals for the moderation rates. Without these, the case-study conclusions are indicative but not quantitatively robust; as written they are used as direct evidence for the paper's central claim that AutoMod lacks contextual understanding.","section":"§5, Tables 6-7"}],"minor_comments":[{"comment":"The abstract states 'over 107,000 comments' while §4.1 says 'around 300,000 messages' for the experiments; the total number of unique messages sent versus the number of moderation decisions logged should be reconciled, and the discrepancy explained.","section":"Abstract vs. §4.1"},{"comment":"The figure labels a send rate of 0.43 msg/s, but §C.1 describes sending 5 messages with a 4-second wait between each message plus a 3.5-second pause, which corresponds to roughly 0.26 msg/s; the figure and text should be made consistent.","section":"Figure 1"},{"comment":"The table uses two F1 definitions ('P,R F1' and 'TPR,TNR F1'); the caption should define both and explain why both are reported.","section":"Table 1"},{"comment":"The text refers to 'tmi.js servers'; this appears to be a typo, as tmi.js is a JavaScript library. The intended meaning should be clarified.","section":"§C.2"},{"comment":"The word 'compherensibility' should be 'comprehensibility'.","section":"§E.3"},{"comment":"The citation 'Gilens, Naomi and Williams, Jamie' is not a standard bibliographic entry; the court decision or the news article should be cited properly so that the legal claim can be verified.","section":"References"},{"comment":"The 'quality control' analysis uses AutoMod's own decisions to claim that the filter-specific subsets are correctly categorized; this is a useful filter-specificity sanity check, but it does not validate that the subsets align with Twitch's policy. The text should not present this as evidence for the ground-truth operationalization used in the main recall calculation.","section":"§D.8"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First: this is the first audit of Twitch AutoMod, and it's worth reading. The authors built a controlled siloed testbed, pushed ~100k+ messages through live chat, and show convincingly that AutoMod flags only a fraction of hateful content (22% overall, 6-7% on implicit-hate benchmarks), while over-blocking benign pedagogical language. The case studies—counterfactual slur replacement, perturbation robustness, policy adherence—are well designed and support the conclusion that AutoMod leans heavily on explicit slurs/profanity rather than context.\n\nThe strengths are real. The scale is substantial, the findings are consistent across four datasets, and they release code and data. The discovery of a service-level pre-filtering layer with its own blocklist biases is a nice empirical contribution. The comparison with SoTA LLMs (prompted with Twitch guidelines) puts AutoMod's performance in context.\n\nNow the soft spots. The main one is the ground-truth alignment. The recall and precision numbers treat dataset labels as though they operationalize Twitch's Discrimination & Slurs policy. SBIC is an offensiveness corpus with a hand-set threshold; ToxiGen positives are chosen by RoBERTa toxicity scores. If many positives are offensive or toxic but not actually hateful under Twitch's policy, the headline numbers measure something adjacent to hate-speech recall. The authors don't validate this alignment. I don't think it overturns the central claim—the directional finding is robust across datasets and echoed by the concurrent Hartmann et al. study—but the exact percentages (22%, 94%) should be treated as upper bounds on AutoMod's failure rate rather than precise measurements.\n\nSecondary issues: the pre-filtered category is inferred from messages missing from both receiver and Pubsub; some manual verification is done, but transient message loss would misclassify misses as catches. No confidence intervals are reported, which is a bit sloppy for point estimates on large samples. The abstract says 107k messages while the intro says 300k; that needs reconciling. The policy-adherence case study uses only 20 generated examples, so the 89.5%/98.5% figures are fragile, though the direction is clear.\n\nOverall, this is a serious, competent audit with a credible central finding. The label-alignment caveat is important but addressable. I'd send it to peer review; the authors should be asked to validate or temper the ground-truth assumptions, add uncertainty estimates, and clean up the internal inconsistencies. It deserves a place in the literature on platform accountability.","headline":"First audit of Twitch AutoMod: credible finding of low hate-speech recall and slur dependence, but the headline percentages rest on unvalidated label alignment with Twitch's policy.","tokens_in":26559,"tokens_out":3083,"would_cite":true,"duration_ms":35312,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Twitch's automated moderator AutoMod catches only 22 percent of hateful comments overall and 6 to 7 percent of implicit hate, while blocking most benign uses of sensitive terms.","keywords":["content moderation audit","hate speech detection","Twitch AutoMod","implicit hate speech","algorithm auditing","counterfactual analysis","moderation bias","live chat moderation"],"falsifier":"Re-run the audit on fresh accounts while confirming every sent message appears in at least one independent IRC log; if many 'pre-filtered' messages turn out to be undelivered, or if moderation rates change materially, the reported recall figures are not stable measurements of AutoMod.","tokens_in":25497,"feed_emoji":"🛡️","tokens_out":6344,"duration_ms":72979,"temperature":0.7,"pith_summary":"The paper reports a large-scale black-box audit of Twitch's automated moderation tool, AutoMod, conducted by sending more than 107,000 comments from four hate-speech datasets into siloed live chats. The central finding is that AutoMod, even at its strictest setting, flags only 22% of hateful comments overall and only 6 to 7% on datasets built to capture implicit hate, so most hateful content bypasses moderation. The audit also finds that swapping a demographic term for a slur triggers 100% removal, while benign pedagogical or empowering sentences containing sensitive words are blocked up to 98.5% of the time. The paper argues these patterns show AutoMod depends on slurs and profanity rather than context, and that this gap matters because Twitch's own guidelines promise context-sensitive moderation.","feed_headline":"Up to 94% of hateful comments slip past Twitch's AutoMod","feed_subtitle":"An audit of 107,000 chat messages finds implicit hate caught only 6-7% of the time while empowering posts get blocked.","key_machinery":"The argument is carried by a three-bot audit pipeline running inside siloed Twitch streams: a messenger bot sends dataset comments within chat rate limits, a receiver bot logs every non-moderated message, and a Pubsub bot records AutoMod's moderation queue events together with Twitch's internal category labels such as Ableism, Misogyny, Racism, and Homophobia. A third outcome, labeled 'pre-filtered,' covers messages visible to neither bot and is attributed to a service-level blocklist that runs before AutoMod. The pipeline is used to measure filter-wise recall and precision for four AutoMod filters by toggling them individually, and to run controlled counterfactual, policy-adherence, and perturbation case studies.","core_discovery":"The central claim is that AutoMod's moderation behavior is both under- and over-inclusive in a way that tracks surface offensive words, not hateful intent. On the four datasets, recall at maximum filtering is 22% overall, with race, ethnicity, and religion hate caught only 12.3% of the time and hate targeting mentally disabled people escaping up to 98% of the time on some datasets; on ToxiGen and the Implicit Hate Corpus, implicit-hate recall is 6 to 7%. In counterfactual testing, 110 previously unflagged hateful SBIC messages become 100% moderated once demographic terms are replaced with slurs, and in robustness testing, simple spelling or spacing changes drop moderation of sensitive fragments from 100% to as low as 4%. At the same time, 89.5% at alpha=2 and 98.5% at alpha=4 of benign model-generated sentences that use sensitive words in pedagogical or empowering contexts are blocked, contrary to Twitch's community guidelines.","pith_inferences":["A natural extension is to run the same counterfactual and perturbation battery on other platforms' moderation systems; the paper's results predict a similar slur-dependence, consistent with the concurrent findings it cites for commercial moderation APIs.","The blocklist-based 'pre-filtering' pattern implies a disparate-impact risk: groups whose slurs sit on the blocklist receive disproportionate protection, while groups whose hate is expressed implicitly are left exposed; balanced cross-community datasets could test this directly.","A practical, testable consequence is that attackers can systematically collect AutoMod's false negatives and apply small semantic-preserving edits to keep hate below the moderation threshold, since the robustness case study already demonstrates the mechanism at small scale.","If Twitch moved toward a context-aware classifier, one would expect recall on ToxiGen and the Implicit Hate Corpus to rise toward the 70 to 85% levels that modern language models reach on the same inputs; this is a direct, falsifiable prediction from the paper's comparison."],"forward_implications":["If these measurements hold, Twitch's most stringent AutoMod setting is not a safety guarantee: most explicitly hateful comments and nearly all implicit hate pass through into chat.","The counterfactual result implies AutoMod's moderation decisions are largely driven by the presence of slurs, so hate that avoids slurs is unlikely to be caught.","Benign speech about marginalized groups, including education and empowerment, faces a high risk of being held or blocked, imposing costs on the very communities the moderation policy is meant to protect.","Because switching filter levels from alpha=2 to alpha=4 changes recall by only about 1% on the Implicit Hate Corpus, turning the strictness dials up does little to close the gap on implicit hate.","The reported pre-filtering rates, up to 54.8% for SSG-related examples, mean a large share of catches occur before AutoMod, so the channel-level tool's own performance is even weaker than the headline recall suggests."],"supporting_citations":[{"why":"Supplies the SBIC real-world dataset with offensiveness scores and the 20k sample used for overall evaluation, counterfactual analysis, and a baseline classifier comparison.","marker":"[Sap et al., 2020]"},{"why":"Supplies DynaHate, the synthetic adversarial hate dataset used to measure filter recall and to analyze false positives.","marker":"[Vidgen et al., 2021b]"},{"why":"Supplies ToxiGen, the synthetic implicit-hate dataset that yields the 6% recall figure.","marker":"[Hartvigsen et al., 2022]"},{"why":"Supplies the Implicit Hate Corpus, the real-world implicit-hate dataset that yields the 7% recall figure.","marker":"[ElSherief et al., 2021]"},{"why":"Defines AutoMod's configurable filter levels and content categories that the audit configures and tests.","marker":"[Twitch, 2024a]"},{"why":"Defines chat rate limits and the moderation pyramid used to distinguish pre-filtering from AutoMod-level moderation.","marker":"[Twitch, 2025]"},{"why":"Provides the platform survey and policy framing that motivate the choice of hate speech and the audit target.","marker":"[Schaffner et al., 2024]"}],"fun_headline_variants":["Twitch AutoMod misses 94% of hate, blocks 89% of benign","Up to 94% hate slips, 89% benign blocked: Twitch audit","AutoMod double fail: 94% hate missed, 89% benign blocked","Twitch AutoMod: 94% hate passes, 89% empowerment blocked","Hate 94% unflagged, benign 89% blocked: Twitch audit"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The recall and precision measurements assume the four datasets' human labels and the chosen offensiveness thresholds are the right definition of hateful content for Twitch's policy, and that messages missing from both observation bots were actually pre-filtered by Twitch rather than lost to a connection glitch.","fun_headline_variants_meta":{"raw":{"variants":["Twitch AutoMod misses 94% of hate, blocks 89% of benign","Up to 94% hate slips, 89% benign blocked: Twitch audit","AutoMod double fail: 94% hate missed, 89% benign blocked","Twitch AutoMod: 94% hate passes, 89% empowerment blocked","Hate 94% unflagged, benign 89% blocked: Twitch audit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001281,"raw_usage":{"total_tokens":5287,"prompt_tokens":1051,"completion_tokens":4236,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":667,"completion_tokens_details":{"reasoning_tokens":4128}},"tokens_in":667,"tokens_out":4236,"duration_ms":32079,"temperature":1.0,"reasoning_tokens":4128,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:28:59.611677+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the audit on fresh accounts while confirming every sent message appears in at least one independent IRC log; if many 'pre-filtered' messages turn out to be undelivered, or if moderation rates change materially, the reported recall figures are not stable measurements of AutoMod.","supporting_citations":[],"review_version":1}