Pith. sign in

REVIEW 5 major objections 6 minor 3 references

Multilingual Hate Speech Detection in Social Media Using Translation-Based Approaches with Large Language Models

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that translating tweets before classification lets GPT-3.5 Turbo and Qwen 2.5 72B detect hate speech in English, Spanish, and Urdu at macro-F1 scores of 0.87, 0.85, and 0.81, and 0.88 on the pooled set, beating SVM…

desk verdict A genuinely useful trilingual dataset undermined by an internal consistency error that invalidates the headline F1 scores as reported. read the letter →

arxiv 2506.08147 v1 pith:5VTHSAMA submitted 2025-06-09 cs.CL cs.AIcs.LG

Muhammad Usman , Muhammad Ahmad , M. Shahiki Tash , Irina Gelbukh , Rolando Quintero Tellez , Grigori Sidorov This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords multilingualhatespeechdetectiontranslation-basedapproachlow-resourceUrdulargelanguagemodelsfew-shotlearningtrilingualdatasetmacroF1attentionlayer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's thesis is that translation can level the playing field for hate speech detection: translate high-resource and low-resource tweets into a shared form, then let large language models do the classifying. To test this, the authors built a trilingual dataset of 10,193 keyword-filtered tweets in English, Urdu, and Spanish, labeled by native speakers with substantial agreement (Fleiss' κ = 0.821). They report that GPT-3.5 Turbo reaches a macro F1 of 0.87 on English and 0.85 on Spanish, while Qwen 2.5 72B reaches 0.81 on Urdu and 0.88 on the pooled trilingual set, beating TF-IDF/SVM baselines (0.77–0.82) by 5 to 9 percent. If the results hold, the payoff is that Urdu, a language with almost no annotated hate-speech resources, can be brought within reach of English-level detection quality using off-the-shelf translation and prompting.

What carries the argument

Two objects carry the argument. The first is the new trilingual corpus: 10,193 tweets (3,834 English, 3,197 Urdu, 3,162 Spanish), balanced between hateful and not-hateful labels, each tweet annotated by three native speakers per language with majority voting and a global Fleiss' κ of 0.821. The second is the translation-plus-attention pipeline, which standardizes all tweets through the Google Translate API into unified per-language and joint corpora and then applies what the paper calls attention-augmented layers: multi-head self-attention with 12 heads plus a Linformer-style sparse low-rank projection, described as a contextual filter prepended before transformer or LLM encoders. Classical models receive TF-IDF features, deep models receive FastText or GloVe embeddings, and the LLMs are deployed in a few-shot setting that yields the headline numbers.

What would settle it

Re-run the evaluation with a strict protocol: an 80/20 split applied separately within each language, few-shot examples drawn only from the training split, and the test set withheld from any fine-tuning or prompt development; then check the arithmetic — an 80/20 split of 3,834 English tweets yields about 767 test tweets, whereas the paper's English confusion matrix sums to 2,039, so per-language test partitions need to be reconstructed and verified. If per-language F1 falls back to the SVM baseline range under a clean split, the claimed gains do not survive.

Watch

Extended reading notes

Core claim

The central claim is that a translation-based pipeline, augmented with attention layers and large language models, closes most of the gap between high-resource and low-resource hate speech detection. Concretely, the paper reports macro-F1 scores (the average of the hateful and not-hateful class F1s) of 0.87 for English and 0.85 for Spanish with GPT-3.5 Turbo, and 0.81 for Urdu and 0.88 for the joint trilingual model with Qwen 2.5 72B, against SVM baselines of 0.80, 0.78, 0.77, and 0.82. The authors interpret the joint model's outperformance of every monolingual model as evidence that cross-lingual training strengthens detection, and they trace the residual Urdu weakness to code-mixing, Roman-script variants, and translation errors that soften slang insults. Their error analysis attributes part of the false negatives to translations such as rendering 'hijo de puta' as 'jerk,' which motivates their proposed next step of slang-aware machine translation.

Load-bearing premise

The headline scores assume the LLMs were evaluated on test data they had never seen, because Section 5.5 says the LLMs were “trained on the entire dataset” while Section 7.4 says they were “evaluated on the test set using a few-shot learning approach,” and the paper never reconciles these two statements, so the reported 0.81–0.88 F1 results may rest on test-set contamination.

Editorial extensions

If this is right

  • If the reported numbers are clean, a single translation-based pipeline brings Urdu hate speech detection to macro F1 0.81, a 5.19 percent gain over the SVM baseline and far above the best monolingual transformer in the paper (BERT at 0.50).
  • The joint trilingual model (0.88) beating every monolingual model implies that pooling translated languages helps rather than hurts, so folding more languages into the same translated pool is a direct corollary the paper itself proposes.
  • Model choice matters per language: GPT-3.5 Turbo leads in English and Spanish while Qwen 2.5 72B leads in Urdu and the joint set, pointing to open-weight models as the stronger route for low-resource deployment.
  • The error analysis's finding that translation softens insults (for example, 'hijo de puta' rendered as 'jerk') identifies slang-aware machine translation as the concrete bottleneck to fix next.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the method is model-agnostic at its core, so if translation is the active ingredient, future or cheaper LLMs can be swapped in without new annotation, and any improvement in slang-aware machine translation should improve Urdu detection directly.
  • Editorial inference: the paper never runs the key ablation of querying the same LLMs on untranslated Urdu text versus translated text, so it does not isolate how much of the 0.81 F1 comes from translation rather than the models' own multilingual competence.
  • Editorial inference: the joint-model advantage hints at a scaling effect — adding more low-resource languages to the translated pool might keep raising per-language F1 — but three languages are too few to establish that trend.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper presents a trilingual (English, Urdu, Spanish) hate speech detection dataset of 10,193 tweets and benchmarks a wide range of models, including SVM, BiLSTM, transformers, and LLMs (GPT-3.5 Turbo, Qwen 2.5 72B), claiming that attention-layer-integrated LLMs achieve macro F1 scores of 0.87 (English), 0.85 (Spanish), 0.81 (Urdu), and 0.88 (joint). The main claimed contributions are the novel dataset with high inter-annotator agreement (Fleiss' Kappa 0.821), a translation-based pipeline, and an attention-augmented multilingual framework. The paper reports consistent-sounding aggregate metrics, but the supporting numerical evidence contains internal contradictions that preclude verification of the central claims.

Significance. If the empirical claims were substantiated, this work would contribute a useful trilingual hate speech benchmark and a practical comparison of proprietary and open-weight LLMs against classical baselines, including a low-resource language (Urdu) that is genuinely underexplored. The dataset construction with three native-speaker annotators and reported agreement of 0.821 is a potentially valuable resource, and the explicit reporting of confusion matrices is a good-faith effort at transparency. However, the current manuscript does not provide the evidence needed to support its headline results: the confusion matrices contradict the stated experimental design, the LLM evaluation protocol is ambiguous on a point that determines validity, and the central attention-layer contribution is asserted without any ablation. As such, the scientific significance of the paper, as written, is not established.

major comments (5)
  1. [Section 8, Tables 10-13; Section 5.5] The confusion matrices for the best English, Spanish, and Urdu models each sum to 2,039 instances (e.g., English: 920+854+99+166 = 2,039; Urdu: 860+792+159+228 = 2,039), which is exactly the size of the joint test set described in Section 5.5. Given the stated 80/20 split per language, the expected monolingual test-set sizes are approximately 767 (English), 633 (Spanish), and 640 (Urdu). This discrepancy means either the monolingual models were evaluated on the combined 2,039-instance test set, in which case the per-language F1 scores in Tables 8-9 do not measure language-specific generalization, or the confusion matrices are misreported. In either case, the central empirical claim of per-language macro F1 values 0.87, 0.85, and 0.81 is not supported by the paper's own numerical evidence.
  2. [Section 5.5 vs Section 7.4] Section 5.5 states that 'LLMs were trained on the entire dataset' to maximize contextual understanding, while Section 7.4 states that LLMs were 'evaluated on the test set using a few-shot learning approach.' These are incompatible evaluation protocols, and the paper never clarifies whether the reported F1 scores come from fine-tuning on all data, few-shot prompting with the training set, or some other procedure. If any reported LLM result was obtained from a model that saw the test instances during training or prompt construction, the result is invalid as a held-out estimate. This ambiguity is load-bearing and must be resolved with an explicit protocol description and, ideally, released code or data.
  3. [Section 1 vs Table 14] The Introduction's contributions list claims 'up to 10.17% F1-score improvements over baselines' and, specifically for Urdu, a '10.17% F1-score increase over baseline models.' However, Table 14 reports Urdu improvement over SVM as 5.19%, English as 8.75%, Spanish as 8.97%, and Joint as 7.32%. No reported value in Table 14 equals 10.17%. The headline improvement claim is therefore contradicted by the paper's own summary table.
  4. [Section 5.4.4] The attention-augmented architecture is described as 'central to our model's success' and is presented as a key contribution, but there is no ablation comparing the same LLM or transformer with and without the inserted attention layer. Moreover, the paper does not explain how an external attention layer can be applied to API-based GPT-3.5 Turbo, which accepts text prompts rather than custom encoder blocks. Without an ablation or a concrete implementation description, the claim that attention layers drive the reported gains is unsupported.
  5. [Section 7.4 error analysis; Tables 10-13] The error-analysis counts are internally inconsistent with the reported confusion matrices. For English, the text says '5% of GPT-3.5 Turbo's misclassifications (102 instances)' and '7% of roberta-base's errors (143 instances),' but 5% and 7% of the respective misclassification counts (265 for GPT-3.5 Turbo in Table 10) do not equal 102 or 143; these numbers instead appear to be 5% and 7% of the full 2,039-instance test set. Similar discrepancies appear for Spanish (61 vs 306 errors in Table 11) and Urdu (143 vs 387 errors in Table 12). This makes the qualitative error analysis unreliable as a description of the models' mistakes.
minor comments (6)
  1. [Section 7.1, Table 5] The table displayed under Section 7.1 is captioned 'Macro F1-scores for traditional machine learning models' but contains BERT, ELECTRA, RoBERTa, and XLM-RoBERTa results, while the SVM, Random Forest, Decision Tree, and XGBoost results described in the text are not shown in that table. The table numbering and captions appear misaligned throughout Section 7.
  2. [Section 7.4] The heading '6.5. Best Performance' appears in the middle of Section 7.4, after the LLM results are described. This is a section-numbering artifact that should be corrected; the content should be a subsection of Section 7.
  3. [Keywords] The author-supplied keywords contain a double comma and an empty entry: 'Social Media, , Multilingual NLP,'. This should be cleaned up.
  4. [Data Availability] The Data Availability statement reads 'Data will be mad upon request.' This is likely a typo for 'made available'; please correct it.
  5. [Section 5.5 and Table 8] The model name is given as 'LLaMA 3.1 70B' in Section 5.5 but as 'LLaMA 3.3 70B' in Table 8 and elsewhere. The version number should be consistent.
  6. [References] Several in-text citation-name pairings do not match the reference list entries: for example, 'Ousidhoum et al.' is cited as (Siddiqui et al., 2024), 'Bahdanau et al.' and 'Vig et al.' are both cited as (Gillioz et al., 2020), and the FlashAttention reference is attributed to (Fu et al., 2022), which in the reference list is titled 'Hungry hungry hippos.' These citation mismatches need systematic correction.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: empirical benchmark results are not derived from their own inputs; self-citations are contextual and not load-bearing.

full rationale

This is an empirical benchmarking paper rather than a derivation chain, so the standard circularity patterns do not apply. The central claims are reported macro F1 scores for models evaluated on held-out test sets; these scores are empirical outputs, not quantities derived from fitted parameters that were defined in terms of the same scores. The attention-layer equations (Section 5.4.4) are standard textbook formulations and are not shown to be fitted to the test data, so the 'attention-enhanced' framing is an overclaim about architecture rather than a circular step. The numerous self-citations in the Introduction and Literature Review (e.g., Shahiki Tash et al. 2024, Tash et al. 2024, Tash et al. 2025, Ahmad et al. 2024, 2025) are contextual references to the authors' other work and are not used to justify the reported F1 values or to forbid alternative methods. The paper does contain serious internal inconsistencies, notably that the per-language confusion matrices in Section 8 each sum to 2,039 (the joint test-set size) despite the stated 80/20 per-language splits from Section 5.5, and the ambiguous claim that LLMs were 'trained on the entire dataset' versus 'evaluated on the test set using a few-shot learning approach.' These are correctness and reproducibility concerns, not circularity: they do not make any prediction equivalent to its own input by construction. No load-bearing step reduces to a self-citation or to the target result, so the appropriate finding is no significant circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on dataset quality, translation fidelity, and the evaluation protocol. The dataset is not released, the translation assumption is contradicted by the paper's own error analysis, and the LLM training/test description is ambiguous, so the reported numbers cannot be independently verified.

free parameters (3)
  • Best LLM selection per dataset = GPT-3.5 for English/Spanish, Qwen for Urdu/Joint
    Reported F1 is the maximum over multiple LLMs per dataset, chosen based on test-set performance with no fixed validation protocol or multiple-run variance.
  • Sparse attention rank (k) = top 10% of token dependencies
    Hyperparameter chosen by hand in Section 5.4.4 with no ablation to justify the value or its contribution.
  • Keyword list for data collection = "fuck", "cunt", "shithead", "kutta", "bhenchod", "hijo de puta", "mierda", "gracias", "support", "war", "sorrow"
    The dataset composition depends on manually chosen English, Urdu, and Spanish keywords, which shapes label balance and content toward explicit profanity.
assumptions (4)
  • domain assumption Google Translate preserves hate speech meaning and intensity across English, Urdu, and Spanish
    The entire translation-based pipeline in Section 5.2 assumes translation fidelity, but the paper's own error analysis finds softened translations of insults like "hijo de puta" and "bhenchod".
  • domain assumption Majority vote of three annotators per language yields correct ground truth
    Fleiss Kappa 0.821 shows consistency but not validity; annotators are postgraduate CS students, not experts, and the guidelines may not capture all hate speech forms.
  • domain assumption Keyword-filtered tweets are representative of hate speech in each language
    Selection by profanity keywords biases toward explicit hate speech and may miss implicit or sarcastic hate speech, as the paper's own error analysis acknowledges.
  • ad hoc to paper LLM evaluation is leakage-free despite being 'trained on the entire dataset'
    Section 5.5 states LLMs were trained on the entire dataset, while Section 7.4 reports test-set evaluation; the paper does not clarify whether fine-tuning included test labels.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multilingual Hate Speech Detection in Social Media Using Translation-Based Approaches with Large Language Models." pith.science (2026). https://pith.science/paper/5VTHSAMA

@misc{pith2026250608147,
  author       = {Pith},
  title        = {Pith review of: Multilingual Hate Speech Detection in Social Media Using Translation-Based Approaches with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5VTHSAMA}},
  note         = {Machine review of arXiv:2506.08147}
}
read the original abstract

Social media platforms are critical spaces for public discourse, shaping opinions and community dynamics, yet their widespread use has amplified harmful content, particularly hate speech, threatening online safety and inclusivity. While hate speech detection has been extensively studied in languages like English and Spanish, Urdu remains underexplored, especially using translation-based approaches. To address this gap, we introduce a trilingual dataset of 10,193 tweets in English (3,834 samples), Urdu (3,197 samples), and Spanish (3,162 samples), collected via keyword filtering, with a balanced distribution of 4,849 Hateful and 5,344 Not-Hateful labels. Our methodology leverages attention layers as a precursor to transformer-based models and large language models (LLMs), enhancing feature extraction for multilingual hate speech detection. For non-transformer models, we use TF-IDF for feature extraction. The dataset is benchmarked using state-of-the-art models, including GPT-3.5 Turbo and Qwen 2.5 72B, alongside traditional machine learning models like SVM and other transformers (e.g., BERT, RoBERTa). Three annotators, following rigorous guidelines, ensured high dataset quality, achieving a Fleiss' Kappa of 0.821. Our approach, integrating attention layers with GPT-3.5 Turbo and Qwen 2.5 72B, achieves strong performance, with macro F1 scores of 0.87 for English (GPT-3.5 Turbo), 0.85 for Spanish (GPT-3.5 Turbo), 0.81 for Urdu (Qwen 2.5 72B), and 0.88 for the joint multilingual model (Qwen 2.5 72B). These results reflect improvements of 8.75% in English (over SVM baseline 0.80), 8.97% in Spanish (over SVM baseline 0.78), 5.19% in Urdu (over SVM baseline 0.77), and 7.32% in the joint multilingual model (over SVM baseline 0.82). Our framework offers a robust solution for multilingual hate speech detection, fostering safer digital communities worldwide.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

3 extracted references · 2 canonical work pages

  1. [3]

    Online Social Support Detection in Spanish Social Media Texts

    Enhancing hate speech detection in the digital age: A novel model fusion approach leveraging a comprehensive dataset . IEEE Access . Sharjeel, M., Nawab, R.M.A., Rayson, P., 2017. Counter: corpus of urdu news text reuse. Language resources and evaluation 51, 777–803. Sharma, A., Dubey, G.P., Shrivastava, A., Sharma, M., Likhar, A., Sharma, L., 2024. Onlin...

    work page Pith review arXiv 2017

  2. [2022]

    arXiv preprint arXiv:2212.14052

    Hungry hungry hippos: Towards language modeling with state space models. arXiv preprint arXiv:2212.14052 . Ghulam, H., Zeng, F., Li, W., Xiao, Y., 2019. Deep learning -based sentiment analysis for roman urdu text. Procedia computer science 147, 131–135. Gillioz, A., Casas, J., Mugellini, E., Abou Khaled, O., 2020. Overview of the transformer -based models...

    arXiv 2019

  3. [2024]

    Cognitive Computation , 1–18

    Analyzing emotional trends from x platform using sentic - net: A comparative analysis with cryptocurrency price. Cognitive Computation , 1–18. Sharif, W., Abdullah, S., Iftikhar, S., Al-Madani, D., Mumtaz, S.,

    work page

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.