{"id":"1f00526d-a882-4362-b341-2cfabd4fa500","arxiv_id":"2506.19603","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A user-level hate-monger detection model that aggregates post-level hate probabilities with ego-network features outperforms text-only and graph-only baselines across three social platforms.","lead":"This paper proposes a multimodal method that detects hate-promoting users by combining the hate scores of their posts with their social network context, and tests it on Twitter, Gab, and Parler. The approach often beats text-only and graph-only baselines, and the authors release a new user-level hate dataset from Parler.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No user-disjoint split is described between the utterance-level DistilBERT and user-level CV; if posts from test users appear in DistilBERT's training set, the reported gains in Table 5 could be leakage artifacts.","rationale":"The reader's conditional verdict is appropriate, but I think the most load-bearing issue is not the Parler annotation quality; it is the missing guarantee of user-disjoint training for the utterance-level classifier. The Parler label concern (Kappa 0.36, biased post sampling) affects one platform and is acknowledged in Appendix F; the leakage concern, if real, affects all three datasets and directly contaminates every proposed aggregation because they are all defined as functions of θ(t) (Eqs. 3-6). The reported margins on Echo and Parler are tiny (0.001 and 0.031 F1 over GAT) and within one standard deviation, so even a modest leakage effect could create them. On Gab the largest gain comes from the distributional aggregator, which uses no social context, so the abstract's claim that 'social context significantly improves' is not uniformly supported by the table. The proposed concrete test (retraining DistilBERT with user-disjoint splits and re-running Table 5) would settle whether the cross-platform claim is real. I keep the verdict unchanged (conditional) because the result may survive the test, but the condition must now include this leakage check.","tokens_in":15257,"tokens_out":7156,"duration_ms":77952,"concrete_test":"Recompute Table 5 with a strict user-level split of the utterance-level model: train DistilBERT only on posts whose authors are in the user-level training folds (with validation drawn from those same users), then compute θ(t) for held-out users and re-run all user-level methods. Compare F1 for Multimodal Relational vs GAT on Echo/Parler and Distributional vs Combined on Gab. If the proposed methods' leads shrink or reverse, leakage explains the reported gains; if the leads persist with non-overlapping bootstrap confidence intervals, the concern is resolved. Inspecting the provided code to check whether the public scripts split posts independently of users is a faster preliminary check.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that aggregating post-level predictions with social context beats text-only and graph baselines. The load-bearing condition is that user-level evaluation is free of leakage from the utterance-level classifier. Section 5.1 reports only 'Utterance-level: We fine-tuned DistilBERT per dataset (80/20 train/val split, batch 32, early stopping)' and 'User-level: 5-fold CV on the largest connected component (LCC) of each dataset'. No statement says the 80/20 post split is user-disjoint. If it is a random post split, then for a user u in a held-out user fold, many of u's posts are likely in DistilBERT's training set; θ(t) for those posts is produced by a model that has seen that exact text. Every proposed aggregator (ΘF in Eq. 3, ΘR in Eq. 4, ΘD in Eq. 5, and Θ in Eq. 6) is a function of θ(t), so distributional and relational features directly encode test-user content. GNN node features are described as 'mean pooling of DistilBERT embeddings', so if the same fine-tuned model is used the graph baselines inherit the leak, but DeGroot and Node2Vec do not. This could inflate all proposed methods on all three datasets and may explain why the Echo and Parler leads over GAT are only 0.001 and 0.031 F1, within one standard deviation, while Gab's larger gap is driven by the distributional (text-only) aggregator. The limitations section (i) acknowledges dependence on θ(t) but never verifies that θ(t) was trained without seeing test users. Without this verification, the cross-platform claim is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes multimodal, aggregative approaches for detecting hate-mongers at the user level. The authors combine utterance-level predictions (from a fine-tuned DistilBERT) into user-level scores using three aggregation schemes: fixed-threshold counting, relational aggregation over a user's ego network, and distributional aggregation over confidence bins/quantiles. These are evaluated with 5-fold cross-validation on three datasets—Echo (Twitter), Gab, and Parler—against text-only and graph-based baselines. The paper also introduces a new user-level annotation set for Parler. The central claim is that processing a user's texts in her social context significantly improves hate-monger detection relative to prior methods.","tokens_in":15574,"tokens_out":3689,"duration_ms":39007,"significance":"If the reported results are valid, the paper makes a useful practical contribution: it shows that simple, interpretable aggregations of post-level scores with network context can often outperform more complex graph neural networks for user-level hate detection. The release of code and data (Appendix G) and the novel Parler user annotations are concrete assets for the community. The comparison across three very different platforms is valuable for understanding the boundary conditions of aggregation methods. However, the evaluation has a potentially load-bearing data-leakage issue, and the headline claim of significant improvement is not backed by significance testing, so the current evidence is weaker than the paper's conclusions suggest.","major_comments":[{"comment":"The utterance-level DistilBERT is fine-tuned on an 80/20 train/val split, but the manuscript does not state whether this split is user-disjoint. Since the user-level evaluation uses 5-fold CV on the LCC and every proposed aggregator (Eqs. 3–6) is a function of the post-level scores θ(t), any posts from users in the held-out folds that appear in DistilBERT's training set would leak content directly into the user-level features. This would inflate the proposed methods and the GNN baselines that use DistilBERT embeddings, while leaving DeGroot and Node2Vec unaffected. The paper must state the split type; if it is a random post split, the experiments must be re-run with a user-disjoint utterance split to support the central comparison.","section":"Section 5.1, Appendix C"},{"comment":"The abstract and Section 5 claim that the proposed methods 'significantly improve' detection, but no significance tests are reported. The margins are often very small: on Echo, Multimodal Relational achieves F1 0.826 vs GAT 0.825, a difference of 0.001, and on Parler the lead over GAT is 0.031; given the reported standard deviations, these differences may be within noise. The authors should provide paired significance tests or confidence intervals over the five folds for the principal comparisons.","section":"Section 5.2, Table 5"},{"comment":"The Parler user-level ground truth is based on annotations with an average Cohen's Kappa of only 0.36, and the annotators were shown the 15 posts with the highest hate-prediction scores plus 5 random posts. The low inter-annotator agreement and the selection of posts by a model that the paper's own pipeline uses undermine the reliability of the Parler labels. This weakens the cross-platform generalization claim and should be addressed with a label-noise robustness analysis or an explicit discussion of how annotation uncertainty might affect the reported rankings.","section":"Appendix F"}],"minor_comments":[{"comment":"The abstract contains a typo, 'hatefulutterances', and refers to the Twitter dataset as 'X (Twitter)' while the rest of the paper uses 'Echo (Twitter)'; please make the naming consistent.","section":"Abstract"},{"comment":"The sentence 'We therefore describe the classification of of a single utterance' contains a duplicated 'of'.","section":"Section 3.1"},{"comment":"There is a typo in 'nearly matching the best performerming model'; it should read 'best-performing'.","section":"Section 5.2"},{"comment":"The Echo row for DeGroot's Diffusion reads '0.319±0.0Node2Vec', which appears to be a formatting error that merged two table cells.","section":"Table 5"},{"comment":"The sets of followers and followees, denoted ←−u and −→u, are not defined in the main text; please define them explicitly when the equation is introduced.","section":"Equation (4)"},{"comment":"The sentence 'Initial beliefs were set based on a small seed set (e.g., 5' is truncated and should be completed.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The central idea is sound and the paper is potentially valuable, but the missing user-disjoint split detail is a serious evaluation risk that must be clarified or fixed before publication. The authors should also be asked to provide significance tests and to confront the low Parler annotation reliability directly. If the leakage issue turns out to be real, the headline results may collapse, so this is a load-bearing revision rather than a cosmetic one."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper has a real contribution: a well-structured combination of relational and distributional aggregation over post-level hate probabilities, plus a new user-level Parler annotation set. The method itself is simple and efficient—logistic regression over a small feature set that includes post counts, neighbor hate proportions, and binned confidence scores. That is actually a strength. The baselines are comprehensive, the three-platform evaluation is genuinely informative, and the limitations section is honest rather than performative. The effort to annotate Parler users, even with its flaws, is a real resource for the field.\n\nBut I do not think the headline claim holds as reported. The load-bearing issue is the split between the utterance-level classifier and the user-level evaluation. Section 5.1 describes an 80/20 train/val split for DistilBERT and 5-fold CV for user-level, with no statement that the post split is user-disjoint. If posts from users in the held-out user folds appear in DistilBERT's training set, then the aggregation features—which are all functions of theta(t)—directly encode test-user content. That would inflate every proposed method, including the relational and distributional variants, and it would explain why the Echo and Parler leads over GAT are only 0.001 and 0.031 F1, well within one standard deviation. The Gab gap is driven by the distributional, text-only aggregator, which is exactly what you would expect if the leak is real. The authors need to either confirm that the splits are user-disjoint or rerun the whole evaluation with user-disjoint splits.\n\nThe second soft spot is the Parler ground truth. The average Cohen's Kappa of 0.36 is low, and the annotation protocol showed annotators the 15 posts with the highest hate-prediction scores plus only 5 random posts. That can bias the labels toward the model's own predictions. The cross-platform generalizability claim should be tempered until this is addressed.\n\nWho is this paper for? People working on user-level hate detection and anyone who wants a labeled Parler user set. If the leakage is resolved, the method deserves a close look. As is, the paper should be sent to serious referees, but they will need to demand a user-disjoint split, a significance test for the claimed improvements, and a closer look at the Parler label reliability. I would bring it to a reading group as a case study in evaluation leakage, but I would not cite it until the split is fixed.","headline":"Useful method and a new Parler resource, but the central evaluation has a leakage risk the authors have to rule out.","tokens_in":16129,"tokens_out":2038,"would_cite":false,"duration_ms":21988,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Processing a user's texts together with her social context—specifically the hatefulness of her followers and followees and the distribution of her posts' hate scores—significantly improves detection of hate-mongers on Echo, Gab, and Parler.","keywords":["hate speech detection","user-level classification","multimodal aggregation","social network analysis","relational aggregation","distributional aggregation","Parler dataset","Gab dataset"],"falsifier":"Re-annotate Parler user labels with a high-agreement protocol (e.g., expert adjudication) and rerun the 5-fold cross-validation; if Multimodal Relational Aggregation no longer exceeds the best graph baseline on Echo and Parler, the claim that social context improves user-level hate detection would be refuted for these platforms.","tokens_in":15005,"feed_emoji":"🕸️","tokens_out":4383,"duration_ms":39004,"temperature":0.7,"pith_summary":"This paper argues that detecting hate-mongers—users who spread hate—should consider both what they post and who they are connected to. The authors propose multimodal aggregation: combining a per-post hate classifier with the share of hateful friends in a user's ego network and the distribution of hate scores across the user's posts. Evaluated on three platforms (Twitter/Echo, Gab, Parler), the relational aggregation achieves the best F1 on Echo (0.826) and Parler (0.519), while distributional aggregation leads on Gab (0.540). The point is that weak signals from coded or implicit posts can be reinforced by social context, improving user-level detection over text-only and graph-only baselines.","feed_headline":"Friends' posts boost hate-monger detection on three platforms","feed_subtitle":"New aggregation of text and network edges beats text-only and graph-only models on Echo, Gab, and Parler.","key_machinery":"The central machinery is a set of aggregation functions over a per-utterance hate classifier: fixed-threshold counting (ΘF), relational aggregation (ΘR) that adds the proportion of hateful neighbors in the user's ego network, distributional aggregation (ΘD) that bins or quantizes the hate-probability distribution of the user's posts, and a combined multimodal model that learns weights over all components. These functions turn noisy per-post signals into a user-level hate score, with weights learned by logistic regression.","core_discovery":"The central claim is that a user's text history and her social neighborhood jointly determine whether she is a hate-monger, and that a linear combination of three aggregation signals—her own hateful-post count, the average hateful-post count of her followers and followees, and the distribution of her posts' hate probabilities—outperforms both text-only aggregation and network-only embeddings. The authors report that Multimodal Relational Aggregation achieves the top F1 score on Echo (0.826) and Parler (0.519), and the combined Distributional (bins+quantiles) aggregation is best on Gab (0.540), with all proposed aggregative methods competitive with or better than GNN and diffusion baselines.","pith_inferences":["If the relational advantage holds, platform moderation could prioritize accounts by a lightweight neighbor-hate score before running full graph models, reducing computational cost.","The distributional approach suggests that the shape of a user's hate-score distribution, not just its mean or count, carries signal; this could generalize to other content-moderation tasks such as detecting coordinated inauthentic behavior.","The reported Parler label noise (Cohen's Kappa 0.36) implies the absolute F1 values may be optimistic; a re-annotation study with expert adjudication would clarify the true performance ceiling.","A testable extension is applying the same aggregation functions to other user-classification problems where per-item signals are weak, such as detecting bots or coordinated harassment accounts."],"forward_implications":["User-level hate detection can be improved without expensive graph training by combining text predictions with simple neighbor statistics.","The approach transfers across platforms with very different network structures, from the small scale-free Echo network to the large, random-like Gab network.","Aggregation of weak per-post signals enables detection of coded messages, dog-whistling, and gas-lighting that single-utterance classifiers miss.","The method is scalable to large networks because aggregation features are cheap to compute and do not require message passing.","The newly released Parler user-level dataset provides a resource for further user-level hate research."],"supporting_citations":[{"why":"Provides the DeGroot diffusion baseline that our relational and distributional aggregations are compared against on hateful-user detection.","marker":"(Ribeiro et al., 2018a)"},{"why":"Supplies the GNN baselines (AGNN, GCN) and experimental settings for user-level hateful user detection.","marker":"(Das et al., 2021)"},{"why":"DistilBERT, the per-utterance hate classifier whose probabilities feed all aggregation functions.","marker":"(Sanh et al., 2019b)"},{"why":"Curates the Echo (Twitter) dataset with tweet- and user-level annotations used in evaluation.","marker":"(Arviv et al., 2021)"},{"why":"Collects the Gab dataset of 22M posts and network data used in our experiments.","marker":"(Zannettou et al., 2018)"},{"why":"Provides the raw Parler dataset from which the annotated user sample is drawn.","marker":"(Aliapoulios et al., 2021)"},{"why":"Contributes the Parler post-level annotations and the DeGroot experimental settings for the Parler baseline.","marker":"(Israeli and Tsur, 2022)"}],"fun_headline_variants":["Friends' hate posts boost hatemonger detection across platforms","Text plus social network finds hatemongers on three platforms","Multimodal aggregation beats text-only and graph-only hate detection","Combining posts and network ties identifies hatemongers better","How your friends' posts reveal if you're a hatemonger"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The Parler user-level ground-truth labels, produced by student annotators with an average Cohen's Kappa of 0.36 and sampled using the model's own hate scores, are reliable enough to support the cross-platform claim.","fun_headline_variants_meta":{"raw":{"variants":["Friends' hate posts boost hatemonger detection across platforms","Text plus social network finds hatemongers on three platforms","Multimodal aggregation beats text-only and graph-only hate detection","Combining posts and network ties identifies hatemongers better","How your friends' posts reveal if you're a hatemonger"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001336,"raw_usage":{"total_tokens":5405,"prompt_tokens":888,"completion_tokens":4517,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":4429}},"tokens_in":504,"tokens_out":4517,"duration_ms":29841,"temperature":1.0,"reasoning_tokens":4429,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:06:11.614555+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate Parler user labels with a high-agreement protocol (e.g., expert adjudication) and rerun the 5-fold cross-validation; if Multimodal Relational Aggregation no longer exceeds the best graph baseline on Echo and Parler, the claim that social context improves user-level hate detection would be refuted for these platforms.","supporting_citations":[],"review_version":1}