{"id":"65a04da7-d20a-41a7-a949-6a410eba678c","arxiv_id":"2605.17187","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"PluRule is a new multimodal multilingual benchmark showing that state-of-the-art vision-language models perform only marginally better than a trivial baseline at detecting specific rule violations in pluralistic online communities.","lead":"This paper introduces PluRule, a large benchmark of rule violations from thousands of Reddit communities to test whether AI can identify which specific community rule a comment breaks. A smart generalist might read it to understand the limits of current AI in handling diverse, self-governed online spaces instead of uniform global rules.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Multiple-choice formulation with explicit rule options may not faithfully proxy real moderator decision processes","rationale":"The reader's weakest assumption directly identifies the same proxy-validity issue. With full text now available, the concern remains load-bearing because the abstract's claim of 'fundamental challenge' is only as strong as the benchmark's ecological validity; no other internal inconsistency (e.g., dataset scale or model scaling trends) appears more central once the task formulation is scrutinized.","tokens_in":1702,"tokens_out":384,"duration_ms":45584,"concrete_test":"Re-run the GPT-5.2 evaluation on a 200-example subset under two conditions: (1) original multiple-choice with rules listed in the prompt, (2) open-ended generation where the model must name the violated rule (or 'none') without any rule list provided. Measure accuracy and F1; if the open-ended condition drops more than 15 points relative to multiple-choice while human moderator agreement remains stable, the original results overstate model capability under realistic conditions.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim—that pluralistic moderation is a fundamental challenge for current VLMs—depends on PluRule being a valid proxy for deployed moderation. The paper formalizes the task as multiple-choice identification of a violated rule given comment + context + (implicitly) the community's rule list. Real moderators, however, typically apply rules from memory, community guidelines, or ad-hoc lookup rather than selecting from an enumerated option set for every comment. This difference risks models succeeding or failing via surface cues (e.g., lexical overlap with listed rules or elimination) that would not exist in open-ended application. If the performance gap to the trivial baseline is driven by this artifact rather than genuine rule-understanding difficulty, the headline conclusion that the task is fundamentally hard for LMs does not follow.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces PluRule, a multimodal and multilingual benchmark with 13,371 rule violations across 1,989 Reddit communities and 2,885 rules in 9 languages. It formalizes pluralistic moderation as a multiple-choice task—given a comment, context, and rule list, identify the violated rule (or none)—and reports that state-of-the-art vision-language models, including GPT-5.2, achieve only marginal gains over a trivial baseline. The authors conclude that moderating pluralistic communities constitutes a fundamental challenge for current language models, with additional findings on model scale, context length, and rule type difficulty. Code and benchmark are released publicly.","tokens_in":1843,"tokens_out":613,"duration_ms":32547,"significance":"If the benchmark and evaluation setup are shown to be a faithful proxy, the work provides a timely, large-scale resource for studying AI moderation in community-governed platforms where norms vary. The public release of data and code is a clear strength that enables follow-up research. The reported performance gaps versus baselines highlight practical difficulties with context-dependent rules, though the strength of this implication rests on the validity of the multiple-choice framing.","major_comments":[{"comment":"Abstract and task-formalization section: the claim that the multiple-choice formulation 'mirrors how human moderators operate in the real world' is load-bearing for the central conclusion that poor model performance demonstrates a 'fundamental challenge.' Real moderators typically retrieve and apply rules from memory or guidelines rather than selecting from an explicit enumerated option set for each comment; this difference introduces the possibility that observed gaps reflect surface-level lexical matching or elimination strategies rather than genuine rule-understanding deficits. A concrete test (e.g., comparison to open-ended generation or human moderator simulation without options) is needed to establish that the performance gap is not an artifact of the evaluation design.","section":"Abstract / Task Formalization"},{"comment":"Evaluation and data-construction sections: the abstract reports clear performance gaps but provides no details on the data-labeling process, inter-annotator agreement, potential selection biases in community or rule sampling, or statistical significance tests for the reported margins over the trivial baseline. These omissions make it difficult to assess whether the headline result (GPT-5.2 only slightly above baseline) is robust or sensitive to annotation artifacts.","section":"Evaluation / Data Construction"}],"minor_comments":[{"comment":"Clarify the exact construction of the 'trivial baseline' (e.g., random selection among rules or majority-class) and report per-rule and per-language breakdowns to support the claim that universal rules are easier to detect.","section":"Results"},{"comment":"Figure and table captions should explicitly state the number of communities, rules, and languages represented in each split to improve reproducibility.","section":"Figures / Tables"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their constructive and detailed feedback. The comments highlight important aspects of task design and evaluation transparency that we have addressed through targeted revisions and clarifications.","responses":[{"response":"We agree the multiple-choice framing is a controlled proxy rather than an exact replica of moderator cognition. Moderators often consult explicit rule lists or guidelines when reviewing content, and our setup tests the core ability to match a comment to a specific rule from the community's set. We have revised the abstract and task formalization section to describe the formulation as 'a controlled multiple-choice evaluation of rule violation detection given explicit rule lists' instead of claiming it directly mirrors real-world operation. We added a limitations paragraph noting the distinction and the possibility of surface-level strategies. A full open-ended generation comparison was outside the scope of this benchmark-focused paper due to evaluation challenges with free-form outputs, but we view the current results as still informative since models show only marginal gains even with options provided.","revision_made":"partial","referee_comment":"The claim that the multiple-choice formulation 'mirrors how human moderators operate in the real world' is load-bearing. Real moderators typically retrieve and apply rules from memory or guidelines rather than selecting from an explicit enumerated option set; gaps may reflect lexical matching rather than genuine understanding. A concrete test (e.g., open-ended generation) is needed."},{"response":"We appreciate this observation and agree these details are essential for evaluating result reliability. In the revised manuscript we have substantially expanded the data construction and evaluation sections to include: a step-by-step description of the labeling process and annotator guidelines, inter-annotator agreement statistics, discussion of our stratified sampling approach for communities and rules to promote diversity, and statistical significance tests (including p-values) confirming the reported margins over the baseline. These additions directly address concerns about potential artifacts.","revision_made":"yes","referee_comment":"The abstract reports performance gaps but provides no details on the data-labeling process, inter-annotator agreement, potential selection biases in community or rule sampling, or statistical significance tests for margins over the trivial baseline. This makes it difficult to assess robustness or sensitivity to annotation artifacts."}],"tokens_in":1471,"tokens_out":472,"duration_ms":59182,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main point is that this paper releases a sizable new dataset of rule violations drawn from almost 2000 Reddit communities and shows that even strong vision-language models only edge out a trivial baseline when asked to pick which rule was broken. That scale and the multilingual coverage are the real additions here. Prior moderation work usually sticks to one platform's general rules or single-community cases, so having 2885 distinct rules across nine languages gives a clearer picture of how context-specific enforcement looks in practice. The public release of the benchmark and code is also straightforwardly helpful for anyone who wants to test new approaches on pluralistic settings. They correctly note that universal rules like civility come through more easily than niche ones, and that simply scaling models or adding context yields only small lifts. Those patterns line up with what the numbers appear to show. The softer part is the task design itself. The paper treats the multiple-choice format as a close stand-in for how moderators actually work, but real decisions often involve recalling or searching rules without an explicit option list in front of you. That difference could let models lean on surface matching or elimination rather than deeper norm understanding, which would make the performance gap look larger than the underlying difficulty. The abstract does not spell out labeling details or bias checks, so it is hard to judge how cleanly the violations were identified. Overall this is aimed at people building or evaluating moderation systems that need to respect group differences rather than apply one-size-fits-all filters. Anyone working on context-aware LLM evaluation would find the dataset worth trying. It is solid enough on the data side to go to peer review, though the authors should expect questions on whether the proxy task supports the stronger claims about fundamental limits.","headline":"PluRule builds a useful large-scale benchmark for community rule detection but the multiple-choice framing leaves open whether models are truly struggling with norms or just the test format.","tokens_in":2354,"tokens_out":418,"would_cite":true,"duration_ms":27400,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"State-of-the-art vision-language models barely outperform a trivial baseline when identifying rule violations in pluralistic communities.","keywords":["pluralistic communities","social media moderation","rule violation detection","AI benchmarks","vision-language models","Reddit","multilingual","multimodal"],"falsifier":"A direct comparison measuring how often the benchmark's multiple-choice answers match the actual violation decisions made by human moderators reviewing the same comments in their communities.","tokens_in":2605,"feed_emoji":"🤖","tokens_out":610,"duration_ms":41824,"temperature":0.7,"pith_summary":"The paper introduces PluRule, a large benchmark of rule violations drawn from thousands of Reddit communities across nine languages. It frames moderation as a multiple-choice task where a model must pick which community rule a comment breaks, if any, using the surrounding context and the community's full rule list. Tests show that even advanced models like GPT-5.2 with high reasoning score only slightly above a baseline that always answers no violation. This result matters because many platforms are shifting to community-defined norms, so automated moderation would need to respect those varying standards rather than apply fixed global rules. The work therefore positions pluralistic moderation as a core unsolved problem for current AI systems.","feed_headline":"AI models barely beat baseline on pluralistic community moderation","feed_subtitle":"New benchmark with 13k violations shows even GPT-5.2 only slightly outperforms random guessing across diverse rules and languages.","key_machinery":"The PluRule benchmark, which formalizes moderation as a multiple-choice problem of selecting the specific violated rule given a comment and its context.","core_discovery":"We introduce PluRule, a multimodal, multilingual benchmark for detecting 13,371 rule violations across 1,989 Reddit communities spanning 2,885 rules in 9 languages. Using this benchmark, we show that state-of-the-art vision-language models struggle significantly: even GPT-5.2 with high reasoning performs only slightly better than a trivial baseline.","pith_inferences":["The benchmark could be used to train models that learn community norms from examples instead of relying on explicit rule lists.","Success on this task might enable moderation tools better suited to decentralized or community-governed platforms.","The performance gap suggests exploring hybrid systems where AI assists rather than replaces human moderators in diverse groups."],"forward_implications":["Bigger models and increased context provide only marginal gains in detection accuracy.","Universal rules such as civility and self-promotion are detected more reliably than community-specific rules.","Moderation of pluralistic communities remains a fundamental challenge for current language models."],"fun_headline_variants":["PluRule exposes AI limits in pluralistic community moderation","Even GPT-5.2 only slightly beats baseline on diverse rules","Bigger models and context offer marginal moderation gains","AI finds universal rules easier to moderate than specific ones"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The multiple-choice formulation with provided context and rules accurately mirrors how human moderators operate when deciding violations in real pluralistic communities.","fun_headline_variants_meta":{"raw":{"variants":["PluRule exposes AI limits in pluralistic community moderation","Even GPT-5.2 only slightly beats baseline on diverse rules","Bigger models and context offer marginal moderation gains","AI finds universal rules easier to moderate than specific ones"]},"model":"grok-4.3","cost_usd":0.016695,"raw_usage":{"total_tokens":7022,"prompt_tokens":634,"num_sources_used":0,"completion_tokens":57,"cost_in_usd_ticks":166953000,"prompt_tokens_details":{"text_tokens":634,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":6331,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":634,"tokens_out":57,"duration_ms":102148,"temperature":1.0,"reasoning_tokens":6331,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-20T13:59:47.861322+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A direct comparison measuring how often the benchmark's multiple-choice answers match the actual violation decisions made by human moderators reviewing the same comments in their communities.","supporting_citations":[],"review_version":1}