REVIEW 3 major objections 5 minor 2 cited by
STATE ToxiCN: A Benchmark for Span-level Target-Aware Toxicity Extraction in Chinese Hate Speech Detection
T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper introduces STATE ToxiCN, the first span-level Chinese hate speech dataset, with 8,029 posts annotated as Target-Argument-Hateful-Group quadruples, alongside the first annotated Chinese hateful slang lexicon of 830 terms.
desk verdict Genuinely useful first span-level Chinese hate speech dataset and slang lexicon, but the unquantified filtering step needs to be addressed before the benchmark scope claim is accepted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the Target-Argument-Hateful-Group quadruple, a labeled tuple (Target, Argument, Hateful, Group) extracted from a single post. Target is the attacked span, Argument is the claim or characterization made about it, Hateful marks whether the pair constitutes hate speech, and Group names the targeted category (sexism, racism, region, LGBTQ, others; multiple groups are allowed). This schema converts post-level classification into a structured extraction task, forcing models to say exactly which spans carry hate and toward whom. It is paired with hard- and soft-matching metrics, where hard matching demands exact boundary agreement and soft matching credits overlapping predictions, and with the 830-term annotated Chinese hateful slang lexicon that supplies the cultural background needed to interpret disguised hate.
What would settle it
Take a random sample of the TOXICN posts that were excluded from STATE ToxiCN and annotate them with the same quadruple scheme; if a substantial fraction turn out to express identifiable hate toward a clear target, the filtering has biased the benchmark. A simpler version: compare the proportion of implicit-hate labels in the excluded set versus the included set, and look for a large imbalance.
Extended reading notes
Core claim
The paper's central claim is that span-level Chinese hate speech detection is a tractable and necessary next step, and that the missing piece has been fine-grained annotation resources. Its discovery is the dataset itself: by annotating Target-Argument-Hateful-Group quadruples, the authors show that the target and argument of Chinese hate speech can be extracted at span level even though Chinese lacks word delimiters and permits flexible word order. In their evaluation, no model solves the task: the best fine-tuned models reach hard-match F1 scores below 30% on full quadruples, and API-only LLMs fall below 12%, identifying precise span identification as the main bottleneck. For hateful slang, LLM APIs display better background knowledge than fine-tuned models but still miss culturally specific terms such as merged-character insults. The paper reads these results as evidence that the benchmark exposes actionable gaps for future Chinese hate speech detection.
Load-bearing premise
The benchmark's representativeness rests on the assumption that removing posts without a clear Target-Argument structure removes only unusable noise, not a meaningful share of real-world Chinese hate speech; if that assumption fails, the dataset oversamples explicit, well-formed hate and under-represents the implicit cases that are hardest to detect.
Editorial extensions
If this is right
- Fine-tuned open models such as LLaMA3-8B and Qwen2.5-7B can identify target and argument spans with soft-match F1 scores near 70% or higher, but full quadruple hard-match scores stay below 30%, so the benchmark resets expectations for joint extraction.
- API-only LLMs without task-specific fine-tuning lag far behind, with quadruple hard-match F1 below 12%, showing that few-shot prompting alone is not enough for span-level Chinese hate extraction.
- Hateful slang degrades fine-tuned models' target and argument extraction while slightly improving hatefulness classification, indicating that the main slang problem is span and group identification, not hate detection.
- LLM APIs outperform fine-tuned models at explaining culturally grounded slang such as '冉闵', which points to background-knowledge infusion as a concrete path for improving smaller models.
- The hard- and soft-matching evaluation protocol provides a reusable way to compare future models despite the ambiguity of Chinese span boundaries.
Reading between the lines
- A testable extension the paper leaves open: annotate the TOXICN posts that were filtered out for lacking a clear Target-Argument structure and compare their implicit-hate rate with the included posts; if the excluded posts are systematically harder, the benchmark overstates how well models handle indirect Chinese hate speech.
- The paper's own limitation section concedes that flexible grammar and ambiguous boundaries make precise span annotations imperfect; since hard matching depends entirely on boundary precision, some of the reported model shortfall may reflect annotation boundary noise rather than pure model failure.
- The limitation section also warns that the 830-term lexicon will miss rapidly evolving internet slang, so the difficulty of the slang portion will drift and will need periodic lexicon refresh to stay representative.
- Because the dataset includes non-hate Target-Argument pairs, it supports contrastive learning between hateful and non-hateful statements about the same target, an avenue the paper does not exploit.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces STATE ToxiCN, a span-level Chinese hate speech dataset of 8,029 posts derived from the post-level TOXICN dataset, annotated with 9,533 Target-Argument-Hateful-Group quadruples. It also presents a lexicon of 830 Chinese hateful slang terms with group labels and explanatory definitions. The authors evaluate twelve open-source, safety-domain, and closed-source LLMs on target/argument span extraction, hatefulness classification, and group classification under hard and soft matching, and conduct case studies of LLM understanding of hateful slang. The central claims are that STATE ToxiCN is the first span-level Chinese hate speech dataset and that the lexicon is the first annotated Chinese hateful slang lexicon with interpretive annotations.
Significance. If the resource is representative, it fills a genuine gap: Chinese hate speech research lacks fine-grained span-level annotations, and the slang lexicon addresses a real evasion phenomenon. The paper ships public code and data, reports inter-annotator agreement on all label types, and provides detailed annotation guidelines and quality-control procedures. The evaluation across twelve models, including safety-specific models, gives a useful baseline. The significance of the resource claims, however, depends on how the filtering from TOXICN affects representativeness and on whether the reported model comparisons are unbiased, so I treat those as load-bearing.
major comments (3)
- [§3.2 (Data Source and Filtering)] The exclusion of posts 'lacking a clear Target-Argument structure' is not quantified. The paper reports no counts, per-rule statistics, or comparison of the retained posts with the source TOXICN corpus. Given that TOXICN includes explicit and implicit toxic language, and Section 5.3.1 states that the removed posts include 'implicit hate expressions,' the filtering may systematically drop implicit or vaguely targeted posts. If so, STATE ToxiCN is a benchmark for explicitly targeted posts only, and the headline 'first span-level Chinese hate speech dataset' overstates its scope. Please report the number of posts removed at each filtering step, annotate a sample of excluded posts to characterize their target-argument structure, and discuss the resulting coverage of implicit hate speech.
- [Appendix F (Detailed Information of the Fine-tuning)] Learning rate selection is performed on the test set: the authors train models for each learning rate, select the result with the highest F1 on the test set, and then calculate the final performance by weighted averaging. This makes the reported numbers in Tables 6 and 7 optimistic and not unbiased estimates of generalization. The benchmark comparison cannot support conclusions about model ranking without a proper validation split or nested cross-validation. Please introduce a development set for hyperparameter selection, report the chosen learning rates, and clarify the 'weighted averaging' procedure.
- [§5.3.1 (Impact of Hateful Slang on Chinese Hate Speech Detection)] The procedure for identifying the 502-post hateful-slang subset is not specified. It is unclear whether posts were selected by lexicon matching, manual annotation, or some other rule, and what coverage of the lexicon this subset represents. Since RQ3 compares model performance on this subset with the full test set, a confounded selection method could drive the observed differences. Please specify the selection algorithm, its validation, and report how many of the 830 lexicon terms appear in the subset.
minor comments (5)
- [§3.4 and Table 4] The text states 6,034 hateful quadruples (63.60%), while Table 4 reports 6,063 hateful quadruples (63.60%); the counts are inconsistent and the percentage does not match the stated total. The text also cites 854 multi-group instances while Table 4 lists 866. Please reconcile these numbers.
- [Table 8] The caption reads 'Annotators Demographics' but the table lists training hyperparameters; the captions for Tables 8 and 9 appear to be swapped.
- [Table 1] The word 'Slience' should be 'Silence' in the English gloss for '默'.
- [Table 7] The difference values are printed as plain numbers immediately after the F1 scores (e.g., '56.83 2.32'), but the text refers to superscripts; using explicit '±' or bracketed differences would improve readability.
- [Appendix F] The phrase 'To reduce hyperparameter sensitivity' is misleading because the described test-set selection actually increases sensitivity to the test set; please rephrase once the methodology is fixed.
Circularity Check
No significant circularity: the paper's contributions are new empirical annotations and benchmark evaluations, not predictions derived from fitted inputs.
full rationale
The paper's derivation chain is a dataset-construction pipeline: it starts from the existing post-level TOXICN corpus, filters it, adds novel span-level Target-Argument-Hateful-Group annotations, builds a slang lexicon, and then evaluates models on the resulting benchmark. No claimed result is defined in terms of another claimed result. The span-level annotations and the 830-term lexicon are new human-produced resources, and the model evaluations in Tables 6 and 7 are independent measurements against those annotations. The use of TOXICN (Lu et al., 2023), which shares authors with the present paper, is a source dependency rather than a circular step: the paper's contribution is the additional annotation layer, and the benchmark results are falsifiable by re-annotation or external evaluation. The Section 3.2 exclusion of posts 'lacking a clear Target-Argument structure' is a legitimate scope and representativeness concern, and it is even acknowledged in Section 5.3.1 that removed posts include 'implicit hate expressions,' but this affects dataset composition rather than creating a logical dependence of any prediction on its inputs. There are no fitted parameters renamed as predictions, no uniqueness theorem imported from prior work, and no ansatz smuggled in through self-citation. Accordingly, the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (2)
- learning_rate =
1e-5 to 5e-5, selected by test-set F1
- post_length_filter =
5 to 500 characters
assumptions (4)
- domain assumption TOXICN provides a reliable source of post-level toxic language labels.
- domain assumption Human annotations of hatefulness and target groups are treated as ground truth.
- domain assumption The set of 830 slang terms is representative of Chinese hateful slang.
- domain assumption The soft-matching algorithm of Han et al. (2023) transfers to Chinese span evaluation.
Cite this review
Pith. "Pith review of STATE ToxiCN: A Benchmark for Span-level Target-Aware Toxicity Extraction in Chinese Hate Speech Detection." pith.science (2026). https://pith.science/paper/NCFMMBWV
@misc{pith2026250115451,
author = {Pith},
title = {Pith review of: STATE ToxiCN: A Benchmark for Span-level Target-Aware Toxicity Extraction in Chinese Hate Speech Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/NCFMMBWV}},
note = {Machine review of arXiv:2501.15451}
}
read the original abstract
The proliferation of hate speech has caused significant harm to society. The intensity and directionality of hate are closely tied to the target and argument it is associated with. However, research on hate speech detection in Chinese has lagged behind, and existing datasets lack span-level fine-grained annotations. Furthermore, the lack of research on Chinese hateful slang poses a significant challenge. In this paper, we provide a solution for fine-grained detection of Chinese hate speech. First, we construct a dataset containing Target-Argument-Hateful-Group quadruples (STATE ToxiCN), which is the first span-level Chinese hate speech dataset. Secondly, we evaluate the span-level hate speech detection performance of existing models using STATE ToxiCN. Finally, we conduct the first study on Chinese hateful slang and evaluate the ability of LLMs to detect such expressions. Our work contributes valuable resources and insights to advance span-level hate speech detection in Chinese.
Figures
Forward citations
Cited by 2 Pith papers
-
Lost in Pronunciation: Detecting Chinese Offensive Language Disguised by Phonetic Cloaking Replacement
A new 500-post benchmark of naturally occurring phonetic cloaking shows LLMs detect such Chinese offensive language with F1 at most 0.672, and Pinyin-augmented prompting partially repairs the gap.
-
System Report for CCL25-Eval Task 10: SRAG-MAV for Fine-Grained Chinese Hate Speech Recognition
A Qwen2.5-7B system with task reformulation, self-retrieval augmentation, and multi-round voting scores 37.505 average on STATE ToxiCN, outperforming cited baselines.
Reference graph
Works this paper leans on
-
[3]
The score for argument span is the lowest, mainly due to the ambiguity of span boundaries in Chinese and the added complexity of argument spans. Despite having established relevant guidelines, it is not pos- sible to fully standardize annotators’ labeling prac- tices. The Target span boundaries are clearer, re- sulting in a higher score. The Kappa score f...
work page 2023
-
[5]
arXiv preprint arXiv:2201.06025
Cold: A benchmark for chinese offensive language detec- tion. arXiv preprint arXiv:2201.06025. David M. Eberhard, Gary F. Simons, and Charles D. Fennig
-
[7]
arXiv preprint arXiv:2203.09509
Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. arXiv preprint arXiv:2203.09509. Sanne Hoeken, Sophie Spliethoff, Silke Schwandt, Sina Zarrieß, and Özge Alacam
-
[8]
arXiv preprint arXiv:2310.06825
Mistral 7b. arXiv preprint arXiv:2310.06825. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al
-
[9]
arXiv preprint arXiv:2412.19437
Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. Junyu Lu, Bo Xu, Xiaokun Zhang, Changrong Min, Liang Yang, and Hongfei Lin
-
[10]
In Proceedings of the 15th international workshop on semantic evaluation (SemEval-2021), pages 59–69
Semeval-2021 task 5: Toxic spans detection. In Proceedings of the 15th international workshop on semantic evaluation (SemEval-2021), pages 59–69. Leandro Silva, Mainack Mondal, Denzil Correa, Fabrí- cio Benevenuto, and Ingmar Weber
work page 2021
-
[11]
arXiv preprint arXiv:2312.11805
Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. Qwen Team
-
[12]
ToxiCloakCN: Evaluating robust- ness of offensive language detection in Chinese with cloaking perturbations. In Proceedings of the 2024 Conference on Empirical Methods in Natural Lan- guage Processing, pages 6012–6025, Miami, Florida, USA. Association for Computational Linguistics. L Xue
work page 2024
Show all 16 references
-
[13]
arXiv preprint arXiv:2010.11934
mt5: A massively multilingual pre- trained text-to-text transformer. arXiv preprint arXiv:2010.11934. Marcos Zampieri, Skye Morgan, Kai North, Tharindu Ranasinghe, Austin Simmons, Paridhi Khandelwal, Sara Rosenthal, and Preslav Nakov
2010 arXiv
-
[14]
arXiv preprint arXiv:2402.16444
Shieldlm: Empowering llms as aligned, customizable and explainable safety detec- tors. arXiv preprint arXiv:2402.16444. Jingyan Zhou, Jiawen Deng, Fei Mi, Yitong Li, Yasheng Wang, Minlie Huang, Xin Jiang, Qun Liu, and Helen Meng
-
[15]
In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3576–3591
Towards identifying social bias in di- alog systems: Framework, dataset, and benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3576–3591. Xianbing Zhou, Yang Yong, Xiaochao Fan, Ge Ren, Yunfeng Song, Yufeng Diao, Liang Yang, and Hongfei Lin
2022
-
[2020]
arXiv preprint arXiv:2010.12472
Hatebert: Retraining bert for abusive language detection in english. arXiv preprint arXiv:2010.12472. I Chung and Chuan-Jie Lin
2010 arXiv
-
[2021]
In 2021 IEEE 22nd International Conference on Information Reuse and Integration for Data Science (IRI), pages 445–452
Tocab: A dataset for chinese abusive language processing. In 2021 IEEE 22nd International Conference on Information Reuse and Integration for Data Science (IRI), pages 445–452. IEEE. Gloria Cowan and Cyndi Hodge
2021
-
[2022]
Com- puter Speech & Language, 74:101365
Hate speech detection on twitter using transfer learning. Com- puter Speech & Language, 74:101365. Badr AlKhamissi, Faisal Ladhak, Srini Iyer, Ves Stoy- anov, Zornitsa Kozareva, Xian Li, Pascale Fung, Lambert Mathias, Asli Celikyilmaz, and Mona Diab. 2022a. Token: Task decompo...
2022 arXiv
-
[2023]
arXiv preprint arXiv:2305.14450
Is information extrac- tion solved by chatgpt? an analysis of performance, evaluation criteria, robustness and errors. arXiv preprint arXiv:2305.14450. Laura Hanu and Unitary team
-
[2024]
In Findings of the Association for Computational Linguistics ACL 2024, pages 10444–10455
Sharedcon: Implicit hate speech detection using shared semantics. In Findings of the Association for Computational Linguistics ACL 2024, pages 10444–10455. AI@Meta
2024
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.