{"id":"c4536f7f-1ea9-4d1d-a873-cbfe51a9e613","arxiv_id":"2605.27313","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"Demographic information aids perspective-aware hate speech detection in regimes of low training disagreement and high test disagreement, with a gated residual model proving effective on high-disagreement examples across MHS and POPQUORN datasets.","lead":"The paper finds that demographic information improves hate speech detection models mainly in data regimes with low training annotator disagreement but high test disagreement, plus sufficient data and demographic overlap; a gated residual model selectively adds demographics and performs better on ambiguous cases. Smart generalists might read it to see how to decide when extra annotator details are worth using in subjective AI tasks like content moderation instead of adding the","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Regime definitions may proxy for unmeasured confounders like label distribution rather than isolating disagreement/overlap effects","rationale":"The load-bearing concern is identical to the reader's weakest_assumption. Because the reader's assessment was abstract-only and the full text is now referenced, the same confounding risk remains the clearest internal threat to the regime-based claim; no stronger objection (e.g., circularity in the gated model or metric definition) is identifiable from the supplied material.","tokens_in":1684,"tokens_out":297,"duration_ms":16877,"concrete_test":"Recompute the demographic-gain tables after stratifying splits to match label distributions (e.g., via propensity weighting or exact matching on label entropy) across the low/high-disagreement bins; if the concentration of gains in the stated regimes disappears or reverses, the primary-driver assumption fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that annotator disagreement (train low, test high) and demographic overlap are the primary causal drivers of when demographics help, rather than correlated factors such as label entropy, class imbalance, or example difficulty. The abstract and described analysis identify regimes post-hoc from the same data without reporting controls or orthogonal ablations that hold other variables fixed while varying only disagreement/overlap. If disagreement simply tracks harder-to-classify examples with different label marginals, the observed concentration of gains would not generalize to the claimed data/modeling regimes.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper examines when demographic information aids perspective-aware hate speech detection. It analyzes demographic performance gains as a function of data-split properties (annotator disagreement in train vs. test, training size, demographic overlap) and modeling choices. Gains are reported to concentrate in regimes with low training disagreement, high test disagreement, fine-grained ambiguity, sufficient data, and greater overlap. Motivated by these observations, the authors introduce a gated demographic residual model that applies demographics selectively to text-only predictions. Experiments on the MHS and POPQUORN datasets indicate the gated model is particularly effective on high-disagreement or low-confidence examples. The central conclusion is that demographics should not be assumed beneficial by default but depend on joint data regime and modeling framework.","tokens_in":1813,"tokens_out":575,"duration_ms":28855,"significance":"If the empirical regime findings and gated-model gains hold after appropriate controls, the work supplies actionable guidance for when annotator demographics are worth incorporating in subjective NLP tasks rather than treated as default or noise. The gated residual design is a concrete, motivated modeling contribution that could be adopted more broadly. The paper also supplies a useful empirical decomposition of performance by disagreement and overlap, which is a strength for reproducibility and follow-up work.","major_comments":[{"comment":"§4 (Regime Analysis): The claim that demographic gains concentrate specifically in the low-train/high-test disagreement regime requires evidence that disagreement and overlap are the primary drivers rather than proxies for unmeasured factors such as label entropy or class imbalance. No ablation or matched comparison is described that holds label distribution fixed while varying only disagreement/overlap; without such controls the reported concentration may not isolate the intended causal factors.","section":"§4"},{"comment":"§5.2 (Gated Model Experiments): The effectiveness of the gated demographic residual model on high-disagreement examples is presented as supporting the regime analysis, yet the model itself is motivated post-hoc from the same data splits. It is unclear whether the gating mechanism's gains survive when the underlying regime definitions are replaced by orthogonal difficulty metrics (e.g., model confidence alone or lexical features).","section":"§5.2"}],"minor_comments":[{"comment":"Table 2 and Figure 3: Axis labels and legend entries for disagreement thresholds should be stated explicitly in the caption rather than only in the main text to improve readability.","section":"Table 2"},{"comment":"The abstract states results on MHS and POPQUORN but does not indicate whether the same train/test splits and annotation protocols are used across both; a brief clarification in §3 would help.","section":"§3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive feedback on our analysis of demographic information in perspective-aware hate speech detection. The comments highlight important opportunities to strengthen causal claims in the regime analysis and to further validate the gated model's robustness. We address each point below and will incorporate revisions to improve the manuscript.","responses":[{"response":"We agree that the current regime analysis would be strengthened by explicit controls that hold label distribution fixed. While our splits already stratify by disagreement levels and we report results across multiple datasets with varying class balances, we did not perform matched ablations isolating disagreement from entropy or imbalance. In revision we will add such matched comparisons on both MHS and POPQUORN, selecting subsets with equivalent label entropy and class distribution while varying train/test disagreement. This will help confirm whether disagreement captures perspective-related variance beyond these factors. We view this as a valuable addition rather than a fundamental flaw in the reported trends.","revision_made":"yes","referee_comment":"[§4] §4 (Regime Analysis): The claim that demographic gains concentrate specifically in the low-train/high-test disagreement regime requires evidence that disagreement and overlap are the primary drivers rather than proxies for unmeasured factors such as label entropy or class imbalance. No ablation or matched comparison is described that holds label distribution fixed while varying only disagreement/overlap; without such controls the reported concentration may not isolate the intended causal factors."},{"response":"The gated residual model was motivated by the observed regimes but is evaluated on both disagreement-based and model-confidence-based partitions, as already shown in §5.2 and the abstract. To address the concern about post-hoc motivation and orthogonal metrics, we will add results using lexical difficulty proxies (e.g., sentence length, lexical ambiguity scores) and confirm that gating still yields gains on high-difficulty subsets defined independently of the original disagreement splits. This will demonstrate that the selective demographic adjustment is not tied exclusively to the regime definitions used for motivation.","revision_made":"partial","referee_comment":"[§5.2] §5.2 (Gated Model Experiments): The effectiveness of the gated demographic residual model on high-disagreement examples is presented as supporting the regime analysis, yet the model itself is motivated post-hoc from the same data splits. It is unclear whether the gating mechanism's gains survive when the underlying regime definitions are replaced by orthogonal difficulty metrics (e.g., model confidence alone or lexical features)."}],"tokens_in":1427,"tokens_out":517,"duration_ms":19595,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The core finding is that demographic features improve hate speech classifiers mainly in data regimes with low annotator disagreement in training, high disagreement in testing, enough examples, and good demographic overlap. The gated residual model, which adds demographics selectively to a text baseline, delivers gains especially on high-disagreement or low-confidence cases. This is a practical observation for anyone building perspective-aware systems.\n\nThe joint framing of data-split properties with modeling choices is the clearest new piece. Prior work has used demographics or noted inconsistency, but tying performance gains to measurable disagreement and coverage, then motivating a selective architecture from those patterns, is a step beyond the usual \"add demographics and see\" approach. Experiments on MHS and POPQUORN back the pattern.\n\nThe main soft spot is that the regimes are defined post-hoc on the same data without reported controls that hold label entropy, class balance, or example difficulty fixed while varying only disagreement and overlap. If disagreement simply tracks harder examples with different marginals, the concentration of gains may not travel to new data the way the paper claims. The abstract does not show orthogonal ablations, so the causal story stays provisional.\n\nThis is useful for NLP groups working on subjective tasks and for moderation teams deciding whether to collect demographics. It is narrow but grounded enough to merit referee time; the question is real and the model is simple to implement. I would send it for review with a request for controls on the regime definitions.","headline":"The paper maps when demographics help in hate speech detection via disagreement-based regimes and shows a gated residual model works in those spots, but the regimes risk capturing label distribution or difficulty instead of the stated factors.","tokens_in":2262,"tokens_out":376,"would_cite":false,"duration_ms":23803,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Demographic information aids hate speech detection primarily in data regimes featuring low training disagreement and high test disagreement.","keywords":["hate speech detection","demographic information","annotator disagreement","perspective-aware modeling","gated residual model","data regimes","subjective tasks"],"falsifier":"A dataset split exhibiting low training disagreement, high test disagreement, and sufficient size where adding demographic features produces no performance gain would challenge the identified concentration of gains.","tokens_in":2589,"feed_emoji":"","tokens_out":598,"duration_ms":38710,"temperature":0.7,"pith_summary":"This paper investigates when demographic features improve performance in hate speech detection, a subjective task where annotator perspectives matter. It finds that such gains occur specifically when training data shows low annotator disagreement, test data shows high disagreement, there is sufficient training data, fine-grained ambiguity measures, and greater demographic overlap between train and test. The authors introduce a gated demographic residual model that selectively adjusts text-only predictions using demographics. This approach proves particularly effective on examples with high disagreement or low model confidence. The results indicate that demographic information should not be used by default as it can act as noise outside these conditions.","feed_headline":"Demographics aid hate speech detection under low training disagreement","feed_subtitle":"Gains concentrate with high test disagreement and gated models; demographics add noise outside these regimes.","key_machinery":"The gated demographic residual model, which selectively adjusts text-only predictions using demographic information.","core_discovery":"The paper claims that demographic gains concentrate in regimes with low training disagreement, high test disagreement, fine-grained ambiguity measurement, sufficient training data, and greater demographic overlap. A gated demographic residual model that treats demographics as a selective adjustment to text-only predictions is effective, especially on high disagreement or low confidence examples. Demographics should not be assumed useful by default; their value depends jointly on the data regime and the modeling framework.","pith_inferences":["The gated residual approach may generalize to other subjective tasks such as sentiment analysis if similar disagreement patterns hold.","Future modeling work should routinely measure and condition on annotator disagreement levels.","Selective demographic adjustment could reduce noise in low-confidence predictions across related detection tasks."],"forward_implications":["Demographic performance gains concentrate under low training disagreement and high test disagreement.","The gated model works best on high disagreement or low confidence examples.","Greater demographic overlap between train and test sets increases gains.","Sufficient training data is needed for demographics to help.","Fine-grained ambiguity measurement reveals the regimes where demographics are useful."],"fun_headline_variants":["Demographics help hate speech detection only with low training disagreement","Demographic gains concentrate in high test disagreement settings","Gated demographic model effective for ambiguous hate speech examples","Demographics not default useful for perspective aware hate speech detection"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Annotator disagreement measured by label differences and demographic overlap are the primary drivers of when demographics help, rather than other unmeasured factors such as label distribution.","fun_headline_variants_meta":{"raw":{"variants":["Demographics help hate speech detection only with low training disagreement","Demographic gains concentrate in high test disagreement settings","Gated demographic model effective for ambiguous hate speech examples","Demographics not default useful for perspective aware hate speech detection"]},"model":"grok-4.3","cost_usd":0.008359,"raw_usage":{"total_tokens":3768,"prompt_tokens":634,"num_sources_used":0,"completion_tokens":62,"cost_in_usd_ticks":83587000,"prompt_tokens_details":{"text_tokens":634,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3072,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":634,"tokens_out":62,"duration_ms":28429,"temperature":1.0,"reasoning_tokens":3072,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-29T18:54:23.774876+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A dataset split exhibiting low training disagreement, high test disagreement, and sufficient size where adding demographic features produces no performance gain would challenge the identified concentration of gains.","supporting_citations":[],"review_version":1}