Pith. sign in

REVIEW 5 major objections 4 minor 26 references

An Annotated Corpus of Arabic Tweets for Hate Speech Analysis

T0 review · 5 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper establishes a 10,000-tweet Arabic hate speech corpus whose distinctive contribution is that every offensive tweet is annotated with all hate speech targets it contains, not just a single category.

desk verdict The released corpus is real, but the 'first-time multilabel' claim contradicts the authors' own 2024 dataset, and the IAA and percentage reporting are inconsistent. read the letter →

arxiv 2505.11969 v2 pith:WNXIBVWO submitted 2025-05-17 cs.CL

Wajdi Zaghouani , Md. Rafiul Biswas This is my paper · ORCID
classification cs.CL
keywords ArabichatespeechmultilabelannotationTwittercorpusoffensivelanguagedetectioninter-annotatoragreementAraBERTdialectaltargets
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's contribution is a resource: 10,000 Arabic tweets annotated for whether each tweet is offensive and, when it is, for every hate speech target it hits among seven categories: political ideology, origin/ethnicity/country, religion/sect, social class/profession, physical appearance/disability, gender, and other. The authors argue that this multilabel scheme, in which all target labels proposed by annotators are merged rather than forcing one label, is the first of its kind for Arabic hate speech. They report annotator agreement for the offensive/clean decision and for targets, and they show that a fine-tuned AraBERT model outperforms two other transformer baselines. If the corpus is reliable, it gives researchers a reusable benchmark for detecting not just whether an Arabic tweet is hateful but whom it targets.

What carries the argument

The dataset itself is the central object: a 10,000-tweet Arabic Twitter corpus. The load-bearing design decision is multilabel annotation: instead of choosing one target per offensive tweet, the guidelines ask annotators to identify all applicable groups among seven categories, and the final label set for a tweet is the union of the labels assigned by annotators. The other key mechanism is the pre-filtering pipeline, in which offensive content is selected with ASAD's confidence scores across high, medium, and low bands and balanced by sentiment, so the final sample is intended to be diverse rather than a random draw from Twitter. Fine-tuned transformer models, primarily AraBERTv2, serve as a sanity check on label quality.

What would settle it

Re-annotate a random sample of the released 10,000 tweets under the published guidelines with fresh annotators and compare their offensive/clean labels and target sets with the released labels; a match far below the paper's reported agreement levels would refute the corpus's reliability. The check should also reconcile the two agreement values stated in the paper, 0.86 in the abstract and 0.8143 in the body.

Watch

Extended reading notes

Core claim

The central claim is that hateful Arabic tweets often hit several groups at once, and the paper provides what it says is the first Arabic dataset designed to capture that multiplicity by instructing annotators to mark all applicable target labels and then merging disagreeing labels into one set per tweet. The construction pipeline starts from 60 million tweets collected over roughly two months, downsamples through ASAD's offensive-language confidence tiers and sentiment strata, and yields 10,000 tweets. In the resulting annotations, 60.36% of tweets are labelled hateful or offensive; among those, political ideology/sports and other targets are the most frequent (28.76% and 28.51%), followed by origin/ethnicity/country (26.20%) and religion/sect (13.51%). On the annotation-quality check, the paper reports a Fleiss' kappa of 0.8143 for offensive versus clean annotation and 0.71 for hate speech target annotation, and AraBERTv2 reaches micro-F1 scores of 0.7865 for offensive/clean and 0.6889 for target classification.

Load-bearing premise

The load-bearing premise is that ASAD's automatic offensive-content confidence scores are accurate enough to pick out genuinely offensive and genuinely clean Arabic tweets; if those scores are noisy or biased by dialect, the 10,000-tweet benchmark will not be a reliable sample for training or evaluating hate speech systems.

Editorial extensions

If this is right

  • A model trained on this corpus can predict multiple targets for a single hateful tweet, matching the way real posts attack more than one identity at once.
  • The reported baselines set a concrete reference point for Arabic multilabel hate speech detection: 0.7865 micro-F1 for offensive versus clean and 0.6889 for target classification.
  • The label distribution shows strong imbalance, with political/ideological and 'other' targets dominating while gender and physical-appearance targets are rare, so practical systems must handle low-frequency target classes.
  • Releasing the annotations and guidelines under an open license gives other researchers a documented gold standard for dialectally diverse Arabic social media content.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the sample is stratified by an automatic classifier rather than randomly sampled from Twitter, the corpus is best read as a benchmark for offensive speech that an automatic filter would flag, not as a neutral estimate of hate speech prevalence in Arabic Twitter.
  • The same 'mark all applicable targets and merge disagreeing labels' guideline could transfer to other low-resource languages; if adopted, it would make cross-language comparisons of hate speech targets feasible.
  • A testable extension would check whether target-label agreement improves when annotators share the same dialect region, since the paper motivates its annotator mix precisely by dialect-driven differences in interpretation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper introduces an annotated Arabic Twitter corpus of 10,000 tweets for hate speech analysis. Each tweet is labeled as offensive/clean, and offensive tweets are further labeled with one or more of seven hate speech target categories. The authors describe a pipeline using the ASAD tool to preselect tweets, a multi-annotator annotation procedure with a manager resolving conflicts, and inter-annotator agreement measured with Fleiss' kappa. They also present fine-tuning experiments with AraBERTv2, CamelBERT, and XLM-RoBERTa for both offensive/clean classification and hate speech target classification, reporting that AraBERTv2 performs best.

Significance. If the corpus were reliable and genuinely novel, it would be a useful resource for Arabic hate speech research, especially given the public Zenodo release, the CC BY 4.0 license, the code repository, and the inclusion of multilingual annotators from different Arab countries. The paper also provides a practical evaluation of three transformer models. However, the central claims of uniqueness and dataset reliability are undercut by internal inconsistencies: the novelty claim contradicts the paper's own related-work table, the abstract reports agreement values that the body does not support, and the dataset description contains numerical inconsistencies. These issues affect the main contribution, so the significance of the work as presented cannot be accepted without major revision or re-framing.

major comments (5)
  1. [Related Works and Novelty] The claim in the Novelty section that 'Annotating each tweet to all the possible labels has been proposed for the first time in our dataset' is directly contradicted by Table 1, which lists Zaghouani, Mubarak, and Biswas 2024 with 'Multilabel: Yes' and a size of 15,965 tweets, and by the same group's LREC-COLING 2024 paper 'So hateful! Building a multi-label hate speech annotated Arabic dataset.' Because the cited prior work is in the same reference list and shares two authors, this is an internal inconsistency rather than a disagreement with outside consensus. The authors must either withdraw the first-time claim or position the new corpus as an extension or re-annotation with a clearly documented delta in size, annotation scheme, and content.
  2. [Abstract and Inter-Annotator Agreement (IAA)] The abstract reports inter-annotator agreement of 0.86 for offensive content and 0.71 for multiple hate speech targets, but the IAA section reports a Fleiss' kappa of 0.8143 for the binary offensive/not-offensive task and explicitly states 'we didn't measure for hate speech target as we combine all possible annotation label to maximize the target group.' The 0.71 value in the abstract is therefore unsupported by any reported measurement, and the 0.86 value does not match the 0.8143 reported in the body. These numbers must be reconciled or removed.
  3. [Table 2] The counts in Table 2 do not add to the stated dataset size: 6036 + 3719 + 63 + 26 + 20 + 184 = 10,048, not 10,000. Additionally, the percentages for the small categories are off by a factor of 100 (e.g., 63/10,000 = 0.63%, not 0.006%). This undermines the reliability of the reported class distribution and must be corrected with the actual counts and denominator.
  4. [Data Collection] The sampling procedure is described as choosing 4,000 tweets from the highest ASAD confidence band (80–100%), 4,000 from the average band (60–79%), and 2,000 from the low band (1–39%), followed by sentiment-based selection of 4,000 positive, 4,000 negative, and 2,000 neutral tweets. This is not a random sample of Arabic tweets, and the paper does not validate ASAD's confidence scores against human judgments. As a result, the offensiveness distribution in Table 2 cannot be interpreted as representative of Arabic Twitter, and models trained on this corpus may not generalize. The authors should either provide a validation of the ASAD confidence bands or explicitly describe the sampling design as purposive and discuss its limits.
  5. [Annotation Procedure] The annotation procedure states that when annotators disagree on the target group, 'we combine all the target group' rather than adjudicating to a single gold label. This makes the target labels a union of annotator choices, not a resolved consensus. Combined with the explicit statement that target-level IAA was not measured, there is no evidence that the seven target categories are reliable. The target distribution in Table 3 is therefore difficult to interpret as ground truth, and the downstream target classification results rest on label definitions that have not been validated.
minor comments (4)
  1. [Figure 2] The caption of Figure 2 says 'Data Collection' but the figure is described in the text as a glimpse of the annotation guidelines; the caption should be corrected.
  2. [Related Works] In the paragraph on Arabic hate speech datasets, the text attributes a 3,075-tweet collection to Alshaalan and Al-Khalifa, but Table 1 assigns the same size and 'Gulf countries' dialect to Alsafari, Sadaoui, and Mouhoub; this attribution mismatch should be fixed.
  3. [References] The reference for Antoun, Baly, and Hajj has '????' as the publication year; the year should be replaced with the correct LREC 2020 date.
  4. [Overall] There are several typos and grammatical errors, including 'Methodolgy' (should be 'Methodology'), 'we didn't distuinsh the data' (should be 'distinguish'), 'between between' in Related Works, 'techiniques', 'wheather', 'techqniue', 'transforemr', and 'mahine'. A careful proofreading pass is recommended.

Circularity Check

1 steps flagged · score 2.0 of 10

No construction-level circularity; the multilabel novelty claim is undermined by the paper's own cited prior work.

  1. other [Related Works, final paragraph after Table 1; Table 1 row for Zaghouani, Mubarak, and Biswas 2024]
    "To extend the existing Arabic corpus, our dataset provides multilabel hate speech which is unique. Annotating each tweet to all the possible labels has been proposed for the first time in our dataset."

    The claimed 'first time' multilabel annotation is made in the same section that cites the authors' own LREC-COLING 2024 paper (Zaghouani, Mubarak, and Biswas 2024), whose Table 1 row lists 'Multilabel: Yes' with 15,965 tweets and the same collection period (Aug 12-Oct 4, 2020). The novelty claim therefore does not rest on a new derivation; it is asserted against the paper's own cited record. At most the current 10,000-tweet set is an extension or re-annotation of the earlier multilabel resource, yet no delta is documented. This is a load-bearing self-citation in the contribution claim, although it is a factual inconsistency rather than an algebraic reduction.

full rationale

This is primarily a dataset resource paper with no first-principles derivation chain. The data collection uses ASAD only to select candidate tweets; the final labels come from human annotators, and the transformer evaluations are standard supervised benchmarks on a train/test/validation split, not predictions forced by fitted constants. The inter-annotator agreement figures are measurements, not fitted inputs. The only circularity-adjacent issue is the uniqueness claim in Related Works, which is internally contradicted by Table 1's listing of the authors' own 2024 multilabel Arabic hate speech dataset. That is a serious novelty and scholarly-consistency problem, but it is not an equivalence-by-construction or a fitted-parameter-renamed-as-prediction, so it does not constitute derivation-level circularity. Accordingly, the paper is assigned a low score of 2, reflecting the self-citation concern while recognizing that the corpus construction and evaluation are otherwise self-contained.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The corpus construction depends on ASAD's automatic scores, the union-of-annotators labeling rule, and a standard agreement formula. These are the main premises the reader must accept beyond the raw annotation effort.

free parameters (2)
  • ASAD confidence sampling fractions = 4000 from 80-100%, 4000 from 60-79%, 2000 from 1-39%
    The proportions of tweets selected from each ASAD confidence band are chosen by hand and directly determine the label distribution in the corpus.
  • Sentiment and adult content quotas = 4000 positive, 4000 negative, 2000 neutral; plus adult/spam inclusion
    The quotas for sentiment categories and the inclusion of adult/spam labelled tweets are arbitrary choices that further shape the sample composition.
assumptions (3)
  • domain assumption ASAD tool's offensive language and hate speech scores are sufficiently accurate to preselect a corpus for human annotation.
    The entire data collection pipeline in the Data Collection section relies on ASAD confidence scores to choose which tweets enter the dataset, without any evaluation of ASAD's accuracy on this sample.
  • standard math Fleiss' Kappa formula is correctly applied to the 500-tweet sample with more than two annotators.
    The paper provides the formula but does not describe how categories were aggregated or how missing labels were handled.
  • ad hoc to paper The union of all annotator target labels is a valid representation of a tweet's hate speech targets.
    In the Annotation Procedure section the authors state that if annotators disagree on target groups, they combine all target groups, which can create labels that no single annotator assigned.

how reviews work

0 comments
Cite this review

Pith. "Pith review of An Annotated Corpus of Arabic Tweets for Hate Speech Analysis." pith.science (2026). https://pith.science/paper/WNXIBVWO

@misc{pith2026250511969,
  author       = {Pith},
  title        = {Pith review of: An Annotated Corpus of Arabic Tweets for Hate Speech Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WNXIBVWO}},
  note         = {Machine review of arXiv:2505.11969}
}
read the original abstract

Identifying hate speech content in the Arabic language is challenging due to the rich quality of dialectal variations. This study introduces a multilabel hate speech dataset in the Arabic language. We have collected 10000 Arabic tweets and annotated each tweet, whether it contains offensive content or not. If a text contains offensive content, we further classify it into different hate speech targets such as religion, gender, politics, ethnicity, origin, and others. A text can contain either single or multiple targets. Multiple annotators are involved in the data annotation task. We calculated the inter-annotator agreement, which was reported to be 0.86 for offensive content and 0.71 for multiple hate speech targets. Finally, we evaluated the data annotation task by employing a different transformers-based model in which AraBERTv2 outperformed with a micro-F1 score of 0.7865 and an accuracy of 0.786.

Figures

Figures reproduced from arXiv: 2505.11969 by the authors.

Figure 1
Figure 1. Data Collection [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Data Collection Offensive: Offensive content refers to cause harm, insult, or discomfort to individuals or groups. Example: ”This is a lie from you and has no source except from a seller among you”. Socially acceptable\Clean: Tweet content does not carry offensive meaning and is socially acceptable to all. Example: ”And old customers, there is nothing left for them.. A failed marketing method. Your customer who has … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 20 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    M.; Azab, S

    Abdelsamie, M. M.; Azab, S. S.; and Hefny, H. A. 2024. A comprehensive review on Arabic offensive language and hate speech detection on social media: methods, challenges and solutions. Social Network Analysis and Mining, 14(1): 1--49

    work page 2024

  4. [4]

    Abdul-Nabi, R.; Obeidat, R.; and Bsoul, A. 2024. A Survey on Machine Translation of Low-Resource Arabic Dialects. In 2024 15th International Conference on Information and Communication Systems (ICICS), 1--6. IEEE

    work page 2024

  5. [5]

    M.; Ali, M

    Al-Ibrahim, R. M.; Ali, M. Z.; and Najadat, H. M. 2023. Detection of hateful social media content for arabic language. ACM Transactions on Asian and Low-Resource Language Information Processing, 22(9): 1--26

    work page 2023

  6. [6]

    Albadi, N.; Kurdi, M.; and Mishra, S. 2018. Are They Our Brothers? Analysis and Detection of Religious Hate Speech in the Arabic Twittersphere. In Proceedings of the 2018 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining, 69--76. ACM

    work page 2018

  7. [7]

    Alhazmi, A.; Mahmud, R.; Idris, N.; Abo, M. E. M.; and Eke, C. 2024 a . A systematic literature review of hate speech identification on Arabic Twitter data: research challenges and future directions. PeerJ Computer Science, 10: e1966

    work page 2024

  8. [8]

    E.; and Eke, C

    Alhazmi, A.; Mahmud, R.; Idris, N.; Mohamed Abo, M. E.; and Eke, C. I. 2024 b . Code-mixing unveiled: Enhancing the hate speech detection in Arabic dialect tweets using machine learning models. Plos one, 19(7): e0305657

    work page 2024

Show all 26 references
  1. [9]

    Alkomah, F.; and Ma, X. 2022. A literature review of textual hate speech detection methods and datasets. Information, 13(6): 273

    2022

  2. [10]

    Alsafari, S.; Sadaoui, S.; and Mouhoub, M. 2020. Hate and offensive speech detection on Arabic social media. Online Social Networks and Media, 19: 100096

    2020

  3. [11]

    Alshaalan, R.; and Al-Khalifa, H. 2020. Hate speech detection in saudi twittersphere: A deep learning approach. In Proceedings of the fifth Arabic natural language processing workshop, 12--23

    2020

  4. [12]

    ???? AraBERT: Transformer-based Model for Arabic Language Understanding

    Antoun, W.; Baly, F.; and Hajj, H. ???? AraBERT: Transformer-based Model for Arabic Language Understanding. In LREC 2020 Workshop Language Resources and Evaluation Conference 11--16 May 2020, 9

    2020

  5. [13]

    Conneau, A. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116

    2019 arXiv

  6. [14]

    Costello, M.; and Hawdon, J. 2020. Hate speech in online spaces. The Palgrave handbook of international cybercrime and cyberdeviance, 1397--1416

    2020

  7. [15]

    Gaind, B.; Syal, V.; and Padgalwar, S. 2019. Emotion detection and analysis on social media. arXiv preprint arXiv:1901.08458

    2019 arXiv

  8. [16]

    Hassan, S.; Mubarak, H.; Abdelali, A.; and Darwish, K. 2021. Asad: Arabic social media analytics and understanding. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, 113--118

    2021

  9. [17]

    Inoue, G.; Alhafni, B.; Baimukan, N.; Bouamor, H.; and Habash, N. 2021. The Interplay of Variant, Size, and Task Type in A rabic Pre-trained Language Models. In Proceedings of the Sixth Arabic Natural Language Processing Workshop. Kyiv, Ukraine (Online): Association for Comput...

    2021

  10. [18]

    M.; and Paramasivam, A

    Kakulapati, V.; Reddy, S. M.; and Paramasivam, A. 2023. Managing postpandemic effects using artificial intelligence with human-computer interaction. In Innovations in Artificial Intelligence and Human-Computer Interaction in the Digital Era, 207--232. Elsevier

    2023

  11. [19]

    Kansok-Dusche, J.; Ballaschk, C.; Krause, N.; Zei ig, A.; Seemann-Herz, L.; Wachs, S.; and Bilz, L. 2023. A systematic review on hate speech among children and adolescents: Definitions, prevalence, and overlap with related phenomena. Trauma, violence, & abuse, 24(4): 2598--2615

    2023

  12. [20]

    Khezzar, R.; Moursi, A.; and Al Aghbari, Z. 2023. arHateDetector: detection of hate speech from standard and dialectal Arabic Tweets. Discover Internet of Things, 3(1): 1

    2023

  13. [21]

    Mulki, H.; Haddad, H.; Bechikh Ali, C.; and Alshabani, H. 2019. L - HSAB : A L evantine T witter Dataset for Hate Speech and Abusive Language. In Roberts, S. T.; Tetreault, J.; Prabhakaran, V.; and Waseem, Z., eds., Proceedings of the Third Workshop on Abusive Language Online,...

    2019

  14. [22]

    C.; Quijano-S \'a nchez, L.; Liberatore, F.; and Camacho-Collados, M

    Pereira-Kohatsu, J. C.; Quijano-S \'a nchez, L.; Liberatore, F.; and Camacho-Collados, M. 2019. Detecting and monitoring hate speech in Twitter. Sensors, 19(21): 4654

    2019

  15. [23]

    R \"o ttger, P.; Vidgen, B.; Hovy, D.; and Pierrehumbert, J. B. 2021. Two contrasting data annotation paradigms for subjective NLP tasks. arXiv preprint arXiv:2112.07475

    2021 arXiv

  16. [24]

    Xian-Yi, C.; Ling-ling, Z.; Qian, Z.; and Jin, W. 2010. The framework of network public opinion monitoring and analyzing system based on semantic content identification. Journal of Convergence Information Technology, 5(10): 1--5

    2010

  17. [25]

    Zaghouani, W.; Mohit, B.; Habash, N.; Obeid, O.; Tomeh, N.; Rozovskaya, A.; Farra, N.; Alkuhlani, S.; and Oflazer, K. 2014. Large scale arabic error annotation: Guidelines and framework

    2014

  18. [26]

    Zaghouani, W.; Mubarak, H.; and Biswas, M. R. 2024. So hateful! Building a multi-label hate speech annotated Arabic dataset. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 15044--15055

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.