Pith. sign in

REVIEW 3 major objections 5 minor 3 cited by

HateDay: Insights from a Global Hate Speech Dataset Representative of a Day on Twitter

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read HATEDAY, a random sample of 240,000 tweets from one day on Twitter, shows public hate speech detectors reach only 9.4% average precision in real-world data versus 40% on academic datasets.

desk verdict HateDay is a genuinely useful representative dataset, but its headline claim that academic evaluations overestimate real-world performance rests on an average-precision comparison that conflates base rates with model skill. read the letter →

arxiv 2411.15462 v3 pith:B66WKEKK submitted 2024-11-23 cs.CL

Manuel Tonneau , Diyi Liu , Niyati Malhotra , Scott A. Hale , Samuel P. Fraiberger , Victor Orozco-Olvera , Paul Röttger This is my paper · ORCID
classification cs.CL
keywords hatespeechdetectionrepresentativedatasetTwitteraverageprecisioncross-lingualevaluationcontentmoderationhuman-in-the-loopoffensivelanguage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

HATEDAY is a dataset of 240,000 tweets randomly sampled from everything posted on Twitter on September 21, 2022, covering eight languages and four English-speaking countries. The paper's central claim is that this sample represents what a real moderation system would encounter, and that on such data publicly available hate speech detectors achieve only 9.4% average precision, versus 40% on academic datasets and 87.2% on functional tests. If correct, the field's standard evaluations substantially overstate real-world detection ability, especially for Arabic, Indonesian, and Turkish. The paper also argues that fully automatic moderation is therefore unsafe, while human-in-the-loop review can work only with substantial human effort.

What carries the argument

The load-bearing mechanism is HATEDAY itself: 240,000 tweets randomly sampled from a full 24-hour corpus of Twitter posts, with 20,000 tweets per language or country, labeled by three annotators using a prescriptive hateful, offensive, or neutral taxonomy with majority-vote labels. This random sampling is what makes the evaluation representative of real-world conditions, and it is paired with average precision as the evaluation metric because it handles severe class imbalance. The second mechanism is the target-alignment measure, a cosine similarity between the target focus of academic datasets and the targets observed in HATEDAY, which the paper uses to explain cross-language and cross-country performance differences.

What would settle it

Evaluate the same models on a HATEDAY subsample where hateful tweets are oversampled to match the prevalence of academic datasets; if average precision climbs to near 40%, the reported overestimation is largely a prevalence artifact, and if it stays near 9.4%, the paper's conclusion stands.

Watch

Extended reading notes

Core claim

The central discovery is that real-world hate speech detection performance is far below what academic benchmarks report, and that the gap is systematic rather than random. Using HATEDAY, the paper measures the prevalence of hate across languages and countries at about 0.7% of posts on average, then evaluates publicly available supervised and zero-shot models, finding average precision of 9.4% on HATEDAY versus 40% on academic datasets and 87.2% on HateCheck functional tests. The paper traces the gap to two main drivers: offensive tweets crowding the top of the hate score distribution, and a mismatch between targets emphasized in academic datasets such as religion and race and the political and gender hate that dominates real-world data; target alignment correlates with performance (Pearson's r = 0.76) while dataset size does not. The paper concludes that fully automatic moderation is not viable and that human-in-the-loop moderation requires reviewing at least 10% of daily tweets to catch more than 80% of hate.

Load-bearing premise

The load-bearing premise is that the gap in average precision, the metric that rewards ranking hateful tweets above non-hateful ones, measures model skill rather than the rarity of hate in real-world data, since a random classifier's average precision equals the base rate.

Editorial extensions

If this is right

  • Removing retweets and sampling uniformly from a full day's output makes HATEDAY the first evaluation set whose class balance and topic mix match what a deployed moderator would actually see.
  • Because average precision on HATEDAY is 9.4% versus 40% on academic sets, models that look strong in the lab would flood a real moderation queue with false positives and still miss most hate.
  • Human-in-the-loop moderation can catch 70 to 90% of hate in most languages, but only if human reviewers examine at least 10% of all daily tweets.
  • Performance gaps across languages track how well academic datasets match real-world hate targets, not how much annotated data exists for a language.
  • Public models are currently ill-suited for fully automatic hate speech moderation, so deployments should assume a substantial human review component.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Our inference: a prevalence-matched control, such as evaluating the same models on a HATEDAY subsample oversampled to academic base rates, would isolate how much of the 40 to 9.4 percentage-point drop is a pure class-imbalance artifact; part of the gap would likely persist but shrink.
  • Our inference: the target-alignment correlation suggests a concrete fix, namely rebalancing training data toward political hate speech, especially for English and US contexts, and that this fix should transfer to other harm-detection tasks.
  • Our inference: the same random-day sampling design could be applied to misinformation or toxicity detection, where non-representative benchmarks are also the norm and where real-world prevalence may similarly be much lower than benchmark prevalence.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper introduces HATEDAY, a dataset of 240,000 tweets sampled uniformly at random from all tweets posted on September 21, 2022, covering eight languages and four English-majority countries, with 20,000 tweets per stratum. Three annotators labeled each tweet for hateful, offensive, or neutral content following a prescriptive guideline, with majority voting and high agreement. The authors estimate hate speech prevalence (average 0.7%), analyze target composition, and evaluate public hate speech detection models on HATEDAY, academic supersets, and HateCheck functional tests using average precision. They report that average precision is 9.4% on HATEDAY versus 40% on academic datasets and 87.2% on HateCheck, and draw conclusions about performance overestimation, cross-lingual gaps, and the infeasibility of fully automated moderation.

Significance. The dataset is a valuable resource: it is publicly released, uses a rigorous random sampling design, documents annotator demographics and guidelines, and provides bootstrapped confidence intervals. The cross-geographic scope (eight languages and four countries from one platform day) is substantially broader than prior representative-sample evaluations such as NaijaHate. If the main result survives reanalysis with a prevalence-appropriate metric, it would be an important corrective to benchmark-based evaluations of hate speech detection and would strengthen the case for human-in-the-loop moderation. The paper also contributes a concrete analysis of target-alignment and offensive false positives. However, the headline quantitative claim is currently under-supported because the chosen metric is not comparable across datasets with very different class prevalences.

major comments (3)
  1. [Section 4.1, Tables 2 and 3; abstract and Section 5] The central comparison of average precision across HATEDAY, academic supersets, and HateCheck is not interpretable as a statement about model skill, because AP is prevalence-dependent. For a random scoring model, the expected AP equals the positive-class base rate. HATEDAY has roughly 0.7% hate prevalence, so a random model has AP near 0.7%; the academic supersets are enriched and HateCheck is approximately balanced, so random AP there is much higher. Consequently, the report that AP is 9.4% on HATEDAY versus 40% on academic datasets and 87.2% on HateCheck conflates base rates with ranking quality. In fact, an AP of 9.4% at 0.7% prevalence is about 13 times the random baseline, while an AP of 40% on a 50%-prevalence benchmark is no better than chance. The paper should report a prevalence-normalized metric (for example, AP lift over the random baseline, ROC-AUC, or precision at a fixed recall level) and include random baselines in Tables 2 and 3. Without this, the abstract's claim that 'evaluations on academic datasets greatly overestimate real-world detection performance' is not supported as stated.
  2. [Section 4.1, Tables 2 and 3] The aggregate figures 9.4%, 40%, and 87.2% are not defined in the text, and the 'Best OS' rows select different models per language/country based on their HATEDAY AP. This makes the cross-dataset comparison sensitive to model selection. For example, if the best model on HATEDAY happens to be weaker on academic data than the best model on academic data, the gap is inflated. Please specify exactly how the aggregates are computed (which models, which strata, weighting), and verify that the main overestimation conclusion holds when using a fixed model per language or paired per-model comparisons across evaluation sets.
  3. [Section 2.3, Section 4.3, Limitations] The number of hate positives in several strata is very small (e.g., 31 in Kenya), and the paper itself notes the resulting uncertainty in Tables 5 and 6. This is particularly consequential for the moderation feasibility curves in Figure 4, where recall targets of 80-90% are extrapolated from very few positives, and for the cross-country ranking (e.g., Kenya's AP of 9.1% with a 95% interval of ±6.1). The paper should provide bootstrap or other uncertainty estimates for the moderation curves and state which cross-language/country performance differences are statistically significant, rather than relying on the noisy point estimates in the headline.
minor comments (5)
  1. [Section 2.2 and Appendix B.1] The main text says 36 annotators (three per 12 strata), but Appendix B.1 opens with 'a team of 30 annotators'; please correct the inconsistency in B.1.
  2. [Figure 1 caption] The caption contains 'T witter' with an unintended space; it should be 'Twitter'.
  3. [Section 4.1] The name 'Perpective API' is misspelled twice in the model-type paragraph; it should be 'Perspective API'.
  4. [Abstract and Section 5] The abstract describes HATEDAY as 'representative of social media settings', but the data reflects one platform and one 24-hour period. The Limitations section already qualifies this; the abstract and conclusion should include the same qualification or use wording such as 'representative of a day on Twitter'.
  5. [Tables 2 and 5] For the HateCheck columns, the paper should remind readers that HateCheck is a curated challenge suite rather than a random sample, so its AP values are not directly interpretable as real-world performance estimates; the 'overestimation' interpretation applies primarily to the academic dataset comparison.

Circularity Check

2 steps flagged · score 4.0 of 10

HateDay's sample is independent, but the headline AP comparison is confounded by prevalence and the Nigeria best-model result is alignment-driven.

  1. other [Section 3.2 (Evaluation metric) and Section 4.1 (Performance across datasets)]
    "We evaluate model performance using average precision, which corresponds to the area under the precision-recall curve, and is well suited when class imbalance is high. ... Indeed, average precision is just 9.4% on HD, compared to 40% on AD and 87.2% on HC."

    Average precision is prevalence-dependent: the AP of a random ranking equals the positive-class base rate. HateDay is a random sample with roughly 0.7% hate prevalence, the academic supersets are enriched by construction, and HateCheck is a balanced functional test. The reported gap is therefore partly guaranteed by the differing label prevalences of the three evaluation settings rather than by model skill alone. The paper does not report a prevalence-normalized metric or a random baseline in this main comparison, so the central 'overestimation' conclusion is statistically forced by the construction of the evaluation sets.

  2. self definitional [Section 5, 'Dominance of Perspective API' paragraph]
    "Also, we observe that Perspective’s dominance does not always hold at the country level, with the open-source hate speech classifier outperforming Perspective in the case of Nigeria. This may be due to conceptual alignment: the HATEDAY annotation guidelines were partly based on the same definition and instructions used to train the Nigerian classifier (Tonneau et al., 2024b)."

    The 'open-source hate speech classifier' for Nigeria is worldbank/naija-xlm-twitter-base-hate, trained by the same authors on NaijaHate. Because HateDay's annotation guidelines are partly based on the same definition and instructions, the model's relatively high HateDay AP (30.9%) is partially guaranteed by construct alignment rather than by independent generalization. The paper acknowledges this circularity but still uses the Nigeria result in the country-level comparison and in the 'best OS' row, making that particular cell partially self-definitional.

full rationale

The dataset itself is genuinely independent: HateDay is a random sample from Twitter Day, annotated with a prescriptive scheme, and no model parameter is fitted to the HateDay labels. The absolute claim that real-world performance is low rests on external model evaluations on this independently collected sample, so the core empirical contribution is not circular. However, the comparative claim that academic evaluations 'greatly overestimate' real-world performance is partly an artifact of comparing average precision across evaluation sets with very different positive-class prevalences; the paper's own metric is prevalence-sensitive and no normalization or random baseline is provided in the main comparison. Separately, the Nigeria 'best open-source model' result is partially circular because the model and HateDay share the same annotation definition/instructions and the model comes from the authors' prior work, a point the paper openly acknowledges. Other self-citations, such as the academic supersets and HateCheck, are external open resources and do not by themselves constitute circularity. Overall, the central claim retains independent content, but two load-bearing comparisons are partially built into the evaluation design, warranting a score of 4.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entities; HATEDAY is a dataset resource, not a postulated mechanism. The main assumptions are about the representativeness of the Twitter Day capture, the accuracy of stratification tools, the validity of annotation, and the comparability of average precision across datasets with different base rates.

assumptions (4)
  • domain assumption Representativeness of Twitter Day: the 375 million posts in Twitter Day are a complete capture of one day on Twitter, and the 10-minute collection delay means minimal moderation occurred before capture.
    Used in Section 2.1 to justify that samples are representative of real-world Twitter and that prevalence estimates are not heavily censored; the paper acknowledges moderation before collection as a limitation.
  • domain assumption Stratification accuracy: language detection and Google Geocoding of self-reported profile locations correctly assign tweets to languages and countries.
    Invoked in Section 2.1 to build the eight language and four country samples; the paper acknowledges imperfect stratification in the Limitations.
  • domain assumption Annotation validity: majority vote of three annotators following prescriptive guidelines yields a valid ground-truth label for hate speech across all languages and countries.
    Established in Section 2.2 and Appendix B.1; cross-cultural consistency of the hate speech definition is assumed rather than demonstrated.
  • standard math Average precision comparability: AP values are comparable across datasets with very different class prevalence.
    Used in Section 3.2 and Section 4.1 to compare performance on HD, AD, and HC; a random ranker's AP equals the base rate, so this assumption is questionable.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HateDay: Insights from a Global Hate Speech Dataset Representative of a Day on Twitter." pith.science (2026). https://pith.science/paper/B66WKEKK

@misc{pith2026241115462,
  author       = {Pith},
  title        = {Pith review of: HateDay: Insights from a Global Hate Speech Dataset Representative of a Day on Twitter},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B66WKEKK}},
  note         = {Machine review of arXiv:2411.15462}
}
read the original abstract

To address the global challenge of online hate speech, prior research has developed detection models to flag such content on social media. However, due to systematic biases in evaluation datasets, the real-world effectiveness of these models remains unclear, particularly across geographies. We introduce HateDay, the first global hate speech dataset representative of social media settings, constructed from a random sample of all tweets posted on September 21, 2022 and covering eight languages and four English-speaking countries. Using HateDay, we uncover substantial variation in the prevalence and composition of hate speech across languages and regions. We show that evaluations on academic datasets greatly overestimate real-world detection performance, which we find is very low, especially for non-European languages. Our analysis identifies key drivers of this gap, including models' difficulty to distinguish hate from offensive speech and a mismatch between the target groups emphasized in academic datasets and those most frequently targeted in real-world settings. We argue that poor model performance makes public models ill-suited for automatic hate speech moderation and find that high moderation rates are only achievable with substantial human oversight. Our results underscore the need to evaluate detection systems on data that reflects the complexity and diversity of real-world social media.

Figures

Figures reproduced from arXiv: 2411.15462 by the authors.

Figure 1
Figure 1. HATEDAY consists of twelve annotated repre￾sentative sets (N=20K each) randomly sampled from all tweets posted on September 21, 2022. The dataset cov￾ers eight languages (Arabic, English, French, German, Indonesian, Portuguese, Spanish, and Turkish) and four countries where English is the main language on Twitter (India, Kenya, Nigeria, and the United States). First, the performance of hate speech detection systems … view at source ↗
Figure 2
Figure 2. Prevalence of harmful content (left) and targets of hate speech (right) in HATEDAY. The target category “Other” contains rare target labels such as “Caste”, “Age”, “Occupation”, “Disability” and “Social Class”. 2.2 Data Annotation We recruit a team of 36 annotators, three per lan￾guage or country. For languages that are spoken widely around the globe, such as English, Arabic or French, we maximize the diversity of a… view at source ↗
Figure 3
Figure 3. Share of hateful , offensive and neutral content in the top 5% scored tweets (N=1,000) in HATE￾DAY for each language and country. We use the hateful￾ness score distribution of the best performing model on HATEDAY (Tables 2 and 3). The x-axis corresponds to the descending rank and each bar shows the distribution of content in a window of 50 tweets. Qualitative analysis of false positives We fur￾ther examine top-score… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Cost-recall tradeoff in human-in-the-loop moderation. Share of all HATEDAY tweets flagged as hateful and reviewed by moderators (%) versus share of all moderated hate in HATEDAY (%). We use the best model on HATEDAY for each language and country (Tables 2 and 3). The d…
Figure 5
Figure 5. Figure 5: Precision-recall curves for each language and country [PITH_FULL_IMAGE:figures/full_fig_p021_5.png]
Figure 6
Figure 6. Figure 6: Comparison between the target-level share of all hate in [PITH_FULL_IMAGE:figures/full_fig_p024_6.png]
Figure 7
Figure 7. Figure 7: Offensive-to-hateful count ratio versus the share of offensive content in top tweets (in %). Top tweets are [PITH_FULL_IMAGE:figures/full_fig_p025_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Compositional Generalisation for Explainable Hate Speech Detection

    cs.CL 2025-06 conditional novelty 7.0 of 10

    A grammar-generated balanced synthetic dataset, U-PLEAD, plus a new TARGET benchmark shows that mixing synthetic examples into training improves hate speech models' recognition of unseen target-expression combinations...

  2. Can Prompting LLMs Unlock Hate Speech Detection across Languages? A Zero-shot and Few-shot Study

    cs.CL 2025-05 conditional novelty 5.0 of 10

    Prompted LLMs beat fine-tuned encoders on HateCheck functional tests but trail them on real-world test sets across eight non-English languages.

  3. Community Moderation and the New Epistemology of Fact Checking on Social Media

    cs.SI 2025-05 conditional novelty 3.0 of 10

    Community-driven fact-checking is promising and useful, but it cannot fully replace professional fact-checkers; hybrid collaboration is the recommended path.

Reference graph

Works this paper leans on

27 extracted references · 22 canonical work pages · cited by 3 Pith papers

  1. [1]

    The dataset is documented, meaning it is at- tached to a research paper or a README file describing its construction

    work page

  2. [2]

    The dataset is either publicly available or could be retrieved after contacting the authors

    work page

  3. [3]

    C.2.2 Target categorization We categorize each surveyed dataset in terms of the target categories they focus on

    The dataset focuses on hate speech, defined broadly as “any kind of communication in speech, writing or behavior, that attacks or uses pejorative or discriminatory language with reference to a person or a group on the basis of who they are, in other words, based on their religion, ethnicity, nationality, race, color, descent, gender or other identity fact...

    work page 2019

  4. [4]

    Disentangling Hate Across Target Identities

    Disentangling hate across target identities. arXiv preprint arXiv:2410.10332. David Jurgens, Yulia Tsvetkov, and Dan Jurafsky. 2017. Incorporating dialectal variability for socially equi- table language identification. In Proceedings of the 55th Annual Meeting of the Association for Compu- tational Linguistics (Volume 2: Short Papers), pages 51–57, Vancou...

    work page Pith review arXiv 2017

  5. [5]

    In International Conference on Informa- tion Processing and Management of Uncertainty in Knowledge-Based Systems, pages 681–695

    Handling disagreement in hate speech mod- elling. In International Conference on Informa- tion Processing and Management of Uncertainty in Knowledge-Based Systems, pages 681–695. Springer. Alyssa Lees, Vinh Q Tran, Yi Tay, Jeffrey Sorensen, Jai Gupta, Donald Metzler, and Lucy Vasserman. 2022. A new generation of perspective api: Efficient multi- lingual c...

    arXiv 2022

  6. [6]

    In Find- ings of the Association for Computational Linguis- tics: ACL 2022, pages 1089–1104, Dublin, Ireland

    Listening to affected communities to define extreme speech: Dataset and experiments. In Find- ings of the Association for Computational Linguis- tics: ACL 2022, pages 1089–1104, Dublin, Ireland. Association for Computational Linguistics. Binny Mathew, Punyajoy Saha, Seid Muhie Yimam, Chris Biemann, Pawan Goyal, and Animesh Mukher- jee. 2021. HateXplain: A...

    work page 2022

  7. [7]

    arXiv preprint arXiv:2312.12651

    Toxic bias: Perspective api misreads german as more toxic. arXiv preprint arXiv:2312.12651. Debora Nozza. 2021. Exposing the limits of zero-shot cross-lingual hate speech detection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2...

    arXiv 2021

  8. [8]

    In Proceedings of the inter- national AAAI conference on web and social media, volume 17, pages 1073–1081

    Just another day on twitter: a complete 24 hours of twitter data. In Proceedings of the inter- national AAAI conference on web and social media, volume 17, pages 1073–1081. Flor Miriam Plaza-del arco, Debora Nozza, and Dirk Hovy. 2023. Respectful or toxic? using zero-shot learning with language models to detect hate speech. In The 7th Workshop on Online A...

    work page arXiv 2023

Show all 27 references
  1. [10]

    In Proceedings of the Thirteenth Language Resources and Evaluation Conference , pages 2215–2225, Marseille, France

    Large-scale hate speech detection with cross- domain transfer. In Proceedings of the Thirteenth Language Resources and Evaluation Conference , pages 2215–2225, Marseille, France. European Lan- guage Resources Association. UN. 2019. Plan of action on hate speech.(2019). Tech- n...

    2019

  2. [17]

    Target focus: race/ethnicity/national origin, gender, caste, religion

    A Dataset of Hindi-English Code-Mixed So- cial Media Text for Hate Speech Detec- tion (Bohra et al., 2018). Target focus: race/ethnicity/national origin, gender, caste, religion

    2018

  3. [18]

    Target focus: religion, caste, gender, politics 3https://hatespeechdata.com/

    Overview of the HASOC track at FIRE 2019: Hate Speech and Offensive Content Identifi- cation in Indo-European Languages (Mandl et al., 2019). Target focus: religion, caste, gender, politics 3https://hatespeechdata.com/

    2019

  4. [19]

    Target focus: race/ethnicity/national origin, religion

    Hostility Detection Dataset in Hindi (Bhardwaj et al., 2020). Target focus: race/ethnicity/national origin, religion

    2020

  5. [20]

    Target focus: religion, caste

    Listening to Affected Communities to Define Extreme Speech: Dataset and Experiments (Maronikolakis et al., 2022). Target focus: religion, caste

    2022

  6. [21]

    Target focus: politics We identify two additional datasets (Mathur et al., 2018; Saroj and Pal, 2020) that we leave out of the analysis as we could not retrieve them

    Uncovering Political Hate Speech During Indian Election Campaign: A New Low- Resource Dataset and Baselines (Jafri et al., 2023). Target focus: politics We identify two additional datasets (Mathur et al., 2018; Saroj and Pal, 2020) that we leave out of the analysis as we could...

    2023

  7. [22]

    Target focus: race/ethnicity/national origin

    HERDPhobia: A Dataset for Hate Speech against Fulani in Nigeria (Aliyu et al., 2022). Target focus: race/ethnicity/national origin

    2022

  8. [23]

    Target focus: race/ethnicity/national origin, religion, poli- tics

    Detection of Hate Speech Code Mix In- volving English and Other Nigerian Lan- guages (Ndabula et al., 2023). Target focus: race/ethnicity/national origin, religion, poli- tics

    2023

  9. [24]

    Target focus: politics

    EkoHate: Abusive Language and Hate Speech Detection for Code-switched Political Dis- cussions on Nigerian Twitter (Ilevbare et al., 2024). Target focus: politics

    2024

  10. [25]

    Target focus: race/ethnicity/national origin, gender, sexual orientation, religion Kenya

    NaijaHate: Evaluating Hate Speech Detec- tion on Nigerian Twitter Using Representative Data (Tonneau et al., 2024b). Target focus: race/ethnicity/national origin, gender, sexual orientation, religion Kenya

  11. [26]

    Target focus: race/ethnicity/national origin, politics

    Building and annotating a codeswitched hate speech corpora (Ombui et al., 2021). Target focus: race/ethnicity/national origin, politics

    2021

  12. [27]

    Here is a social media post: [POST]. Classify this post as “hateful

    Listening to Affected Communities to Define Extreme Speech: Dataset and Experiments (Maronikolakis et al., 2022). Target focus: race/ethnicity/national origin, religion, poli- tics D Models D.1 Hugging Face models We list the Hugging Face models used as bench- mark below for e...

    2024

  13. [39]

    Two annotators are educated to undergraduate level and the last one to research degree (i.e. PhD). All three are native Indonesian speakers, all from Indonesia. Portuguese We recruit two Portuguese-speaking female annotators and one male. All annotators are 30-39 and are educa...

    2024

  14. [2019]

    Predicting the type and target of offensive posts in social media. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1415–1420, Minneapolis,...

    2019

  15. [2020]

    Pew Research Center

    Most Americans Think Social Media Sites Censor Political Viewpoints. Pew Research Center. Maximilian Wich, Adrian Gorniak, Tobias Eder, Daniel Bartmann, Burak Enes Cakici, and Georg Groh. 2022. Introducing an abusive language classification frame- work for telegram to investig...

    2022

  16. [2021]

    HateCheck: Functional tests for hate speech detection models. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 41–58, Online....

    2019

  17. [2022]

    arXiv preprint arXiv:2211.15262

    Herdphobia: A dataset for hate speech against fulani in nigeria. arXiv preprint arXiv:2211.15262. Sai Saket Aluru, Binny Mathew, Punyajoy Saha, and Animesh Mukherjee. 2020. Deep learning models for multilingual hate speech detection. arXiv preprint arXiv:2004.06465. Aymé Arang...

    2020 arXiv

  18. [2023]

    In Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023), pages 1–13, Dubrovnik, Croatia

    Analyzing zero-shot transfer scenarios across Spanish variants for hate speech detection. In Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023), pages 1–13, Dubrovnik, Croatia. Association for Computational Linguistics. Mithun Das, Somnath Baner...

    2023 arXiv

  19. [2024]

    Improving adversarial data collection by sup- porting annotators: Lessons from GAHD, a German hate speech dataset. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long ...

    2024 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.