Pith. sign in

REVIEW 4 cited by

HateCheck: Functional Tests for Hate Speech Detection Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2012.15606 v2 pith:KJJHGNM2 submitted 2020-12-31 cs.CL

Paul Röttger , Bertram Vidgen , Dong Nguyen , Zeerak Waseem , Helen Margetts , Janet B. Pierrehumbert This is my paper · ORCID
classification cs.CL
keywords modelshatemodelspeechdetectionhatechecktestdifficult
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Detecting online hate is a difficult task that even state-of-the-art models struggle with. Typically, hate speech detection models are evaluated by measuring their performance on held-out test data using metrics such as accuracy and F1 score. However, this approach makes it difficult to identify specific model weak points. It also risks overestimating generalisable model performance due to increasingly well-evidenced systematic gaps and biases in hate speech datasets. To enable more targeted diagnostic insights, we introduce HateCheck, a suite of functional tests for hate speech detection models. We specify 29 model functionalities motivated by a review of previous research and a series of interviews with civil society stakeholders. We craft test cases for each functionality and validate their quality through a structured annotation process. To illustrate HateCheck's utility, we test near-state-of-the-art transformer models as well as two popular commercial models, revealing critical model weaknesses.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Echoes of Discord: Forecasting Hater Reactions to Counterspeech

    cs.CL 2025-01 conditional novelty 6.0 of 10

    A three-way classifier trained on Reddit hate speech/counterspeech pairs predicts hater reentry and reentry type more accurately than a two-stage predictor, with linguistic features of counterspeech signaling differen...

  2. Collaborative Content Moderation in the Fediverse

    cs.SI 2025-01 conditional novelty 6.0 of 10

    A federated-learning system for Mastodon lets instances share model weights, improving average macro-F1 over local-only moderation on three content moderation tasks.

  3. Agentic Web: Weaving the Next Web with AI Agents

    cs.AI 2025-07 conditional novelty 3.0 of 10

    A position paper defines the Agentic Web as the next web era and proposes a three-dimensional conceptual framework for understanding and building it.

  4. A Survey on Automatic Online Hate Speech Detection in Low-Resource Languages

    cs.CL 2024-11 conditional novelty 3.0 of 10

    A survey cataloging datasets, features, and machine-learning methods for automatic hate speech detection in low-resource languages, organized by world region, with an overview of open challenges.

Pith tools