REVIEW 4 major objections 5 minor 1 cited by
Breaking the Cloak! Unveiling Chinese Cloaked Toxicity with Homophone Graph and Toxic Lexicon
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read A training-free, prompt-free method, C2TU, matches Chinese homophone-disguised words against a homo-graph and toxic lexicon, then uses BERT or LLM probabilities to filter and restore the original toxic words, reporting up to 71% higher F1…
desk verdict A clever, reproducible training-free pipeline for Chinese cloaked toxicity, undercut by an evaluation built on test cloaks generated from the method's own resources. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Chinese homo-graph: a graph whose nodes are Chinese characters and whose edges link characters with the same tone-ignored pinyin, extended with polyphonic matches, five dialect confusions (n/l, zh/z, ch/c, sh/s, and *ng/*n), and self-loops. Together with a deduplicated toxic lexicon of proto words, it defines which substrings can be candidates. Filtering is carried by the probability-difference rule: for each $(w,l)$ pair, compute $\log(P_w/P_l)$ and replace the pair with the most negative value that is below zero, then repeat. For LLMs, Theorem 1 converts the word-probability difference into the sentence-probability difference, so the full sentence context is available despite the autoregressive left-only word probabilities.
What would settle it
Collect naturally obfuscated Chinese posts that were not produced by a homophone-substitution tool and whose gold toxic words come from a fresh annotation, then run C2TU; if sentence-level F1 falls substantially below the 74.67% reported on the constructed benchmark, the method's success is tied to the test data sharing its own homophone and lexicon resources.
Extended reading notes
Core claim
At the core is the claim that cloaked-toxicity unveiling reduces to candidate generation plus probability filtering, with no parameter updates and no prompts. On a text $X$, C2TU enumerates substrings $w$ that are homophonically aligned character-by-character with a lexicon entry $l$, producing pairs $(w,l)$. It then compares the probability of the surface form $w$ against the probability of the restored toxic form $l$ in the same context, and iteratively replaces the candidate pair with the largest log-probability gap that favors $l$. For the LLM variant, the paper proves that the word-level probability difference equals the sentence-level log-probability difference $\log P(X) - \log P(X')$, where $X'$ is the sentence with $w$ replaced by $l$; that identity is what lets an autoregressive LLM use both left and right context without changing its architecture. The BERT variant instead masks each character and reads the masked-token probabilities directly.
Load-bearing premise
The method's recall rests on the assumption that real cloaked toxic words are homophonic substitutions of words already present in the toxic lexicon and that the homo-graph contains the corresponding pronunciations; if a cloak uses a pronunciation outside the graph or a toxic word outside the lexicon, the method will never see it.
Editorial extensions
If this is right
- A toxicity detector can run C2TU as a preprocessing normalization step, restoring cloaked words to their proto forms before classification, with no fine-tuning of the detector.
- The same matching-and-filtering skeleton transfers to a new domain by swapping in a new toxic lexicon and homo-graph, so coverage is controlled by resources rather than by training data.
- The LLM variant's sentence-level probability identity means uncloaking can be done with a 7B-8B model and still beat much larger generation-based baselines, reducing deployment cost.
- The iterative replacement of the pair with the largest probability gap accounts for context changes after each correction, which the single-round ablation shows is needed for accuracy.
Reading between the lines
- If C2TU is placed before an existing classifier, its benefit should grow with the cloak rate; on mildly obfuscated text the classifier may already cope, but the size of the gain is a testable prediction about the distribution of homophonic substitutions.
- The manually corrected, deduplicated lexicon is the main bottleneck for recall, so an automatic extension would generate candidate proto words from the homo-graph's connected components rather than from a fixed word list.
- The Bayes-based probability-difference identity could apply to other character-level text normalization tasks, such as restoring social-media abbreviations, where autoregressive LMs otherwise see only left context.
- Because the reported cloaked test sets were constructed with the same homophone substitution tool and lexicon family, the method's real-world edge remains an open empirical question that a field deployment with naturally occurring obfuscation would answer.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes C2TU, a training-free and prompt-free method for unveiling Chinese cloaked toxicity. It builds a homophone graph and a toxic lexicon, matches candidate toxic substrings, and then filters false positives using either BERT or an LLM. For the LLM variant, the authors prove a Bayes' theorem identity that converts word-level probability differences into sentence-level probability differences, thereby using full context despite autoregressive decoding. Experiments on ToxicloakCN and COLDataset report large gains over CSC and prompt-based baselines, with the best variant outperforming the strongest competitor by up to 71% in F1 and 35% in accuracy.
Significance. If the empirical results hold, C2TU would be a practical, parameter-free contribution to an underexplored task, and the Bayes' reformulation for LLM filtering is a clean theoretical insight that could transfer to other word-replacement tasks. The authors release code and data, and the method avoids both fine-tuning and prompt engineering. However, the significance is tempered by the evaluation's close alignment between the test cloaking process and the method's own lexicon and homo-graph; the current experiments do not demonstrate that the gains transfer to independently occurring cloaking distributions.
major comments (4)
- [Appendix E and Table 1] The COLDataset cloaked test set is constructed by the authors themselves: they merge the validation and test sets, apply JioNLP homophone substitution, and use the crawler keywords from COLDataset construction as the toxic lexicon. That lexicon is the same one C2TU is given (Table 9), and the homo-graph is built with tone-ignored pinyin from pypinyin, matching JioNLP's substitution rule. Likewise, ToxicloakCN's cloaks were generated by JioNLP from the ToxiCN lexicon, and C2TU's ToxicloakCN lexicon is a manually cleaned version of that same ToxiCN lexicon (Appendix E). Consequently, the matching stage is aligned with the exact noise process that produced the ground-truth corrections. The headline gains in Table 1 are therefore not established for independent cloaking distributions. Please re-evaluate on test data whose cloaking process is not derived from the method's own lexicons (e.g., human-generated cloaks or a separate substitution model), or clearly scope the claims to this aligned setting.
- [Table 9 and Section 4.2] The COLDataset lexicon in Table 9 contains many non-toxic crawler keywords, including province names, '女性', '男人', and other neutral terms. The F1 and accuracy scores on COLDataset therefore partly measure the restoration of crawled keywords rather than the unveiling of toxicity, and the claim that C2TU 'unveils cloaked toxicity' is conflated with general homophone matching. Please report results using a toxic-only curated lexicon, or provide a breakdown that separates toxic from non-toxic matched candidates.
- [Appendix E and Appendix B] Appendix E states that ToxicloakCN includes both homophone substitution and semantically similar emoji substitution, while Appendix B states that C2TU does not address emoji replacements. The main results in Table 1 do not indicate whether emoji-cloaked instances were excluded from the evaluation. If they were not excluded, the comparison is unfair to C2TU; if they were, please state the filtering criterion and the resulting dataset sizes, since this materially affects the interpretation of the reported F1 and accuracy numbers.
- [NeurIPS Checklist Item 7 and Table 1] The paper reports no error bars, confidence intervals, or significance tests, and the checklist acknowledges that API evaluations used single runs. Given that the main contribution is empirical and the reported gaps are large, please report variance across at least a few runs (or bootstrap estimates) for the deterministic local models, and clearly state which numbers are based on a single API call.
minor comments (5)
- [Figures 3 and 5] Section 4.4 and Appendix I: Figures 3 and 5 contain rendering artifacts (sequences such as '/uni00000013/uni00000014/...') and are not legible; please replace them with correctly rendered figures.
- [Section 4.1] The sentence 'None of the models’ parameters have been modified, nor have any prompts been used' is stated in the setup for all methods, but prompt-based baselines are used later. Please clarify that this property holds for C2TU, not for the baselines.
- [Algorithms 1 and 2] Algorithm 2 computes d_i as log(P_w/P_l) and then selects argmin_i d_i; since d_i is negative when P_w < P_l, the argmin selects the pair with the most negative log-ratio. Please add a brief note in the pseudocode or text to make this equivalence explicit for readers.
- [Abstract and Introduction] The claim of being 'the first to solve the problem under Chinese' is strong; please qualify it with respect to the exact task definition (correction of cloaked toxic words) and note that prior work [24] introduced cloaked toxicity evaluation but not a correction solution.
- [Section 3.1.1 and 4.1] The text refers to 'pypinyin2' in Section 3.1.1 but 'pypinyin' in Section 4.1; please unify the library name and version consistently.
Circularity Check
Empirical headline is partly circular: the COLDataset test cloaks are authored by the paper from the same lexicon and JioNLP homophone substitution that C2TU consumes, while the ToxicloakCN lexicon is a cleaned version of the same ToxiCN keyword list that generated its test cloaks; the Bayes-based derivation itself is independent.
-
fitted input called prediction
[Appendix E (Dataset details, COLDataset paragraph); cross-ref Table 9 and Section 3.2]
"Since no existing work has compiled a toxic lexicon for COLDataset, we instead utilize the crawler keywords used during dataset construction[7] as toxic lexicon. Based on the lexicon, we merge the validation set with the test set and apply JioNLP [5] to perform homophone substitution and finally get the cloaked dataset, following the noise injection procedure of ToxicloakCN[24]."
The COLDataset benchmark is a function of C2TU's own input: the same crawler-keyword list is (i) given to C2TU as its toxic lexicon and (ii) used to generate the test cloaks via JioNLP, whose tone-ignored pinyin substitution is exactly the relation encoded by C2TU's homo-graph (Section 3.1.1). The reported COLDataset F1/accuracy therefore measures restoration of cloaks generated from the method's own matching resources, not performance on an independently cloaked distribution. Many entries in Table 9 (e.g., '女性', '男人', province names) are non-toxic crawler keywords, so part of the score is keyword restoration rather than toxicity unveiling by construction.
-
other
[Appendix E (Toxic Lexicon, ToxicloakCN paragraph); cross-ref Section 4.1]
"For keyword-based substitution, the toxic lexicon from ToxiCN is used as keywords. For cleaner lexicon without cloak, we further correct and deduplicate the ToxiCN's lexicon. We first manually correct all toxic words in the original lexicon back to their protowords, and then retain each protoword."
ToxicloakCN's cloaks were generated by homophone-substituting ToxiCN's lexicon, and C2TU's ToxicloakCN lexicon is a manually corrected and deduplicated version of that same ToxiCN lexicon, with cloaked entries restored to protowords. The matching stage is thus pre-aligned to the exact word list that was corrupted to make the test set. The headline 'up to 71% on the F1 score and 35% on accuracy' is therefore largely a measure of how well the pipeline inverts the specific JioNLP corruption of a known keyword list, rather than evidence about arbitrary real-world Chinese cloaking, whose substitution set and lexicon are not provided by the paper.
full rationale
The core algorithmic derivation is not circular: Equation (5) is a direct Bayes' rule identity (Appendix C), and the filtering step computes probabilities from a frozen language model with no fitted parameters or labels, so no prediction is logically forced by the method's equations. There is also no load-bearing self-citation chain; the external benchmarks and public models carry the argument. However, the empirical evidence is substantially self-referential. For COLDataset, there is no pre-existing cloaked test set: the authors construct one by applying JioNLP homophone substitution to the validation+test sentences using the same crawler-keyword lexicon that C2TU is given (Appendix E, Table 9). Because the homo-graph is built with the same tone-ignored pypinyin relation that JioNLP uses, the matching stage is guaranteed to cover the exact corruption process that generates the ground truth; the remaining work is only LM filtering. For ToxicloakCN, the test cloaks were generated from the ToxiCN keyword list, and C2TU's lexicon is a manually cleaned version of that same list. The method is therefore never tested against an independently assembled cloaking distribution or a different homophone rule. Section 3.1.1 further builds the homo-graph from the characters of the evaluation dataset itself, compounding the self-reference. This does not make the internal pipeline circular, but it makes the headline performance numbers partially an evaluation of the method against its own input resources. The paper's own Appendix B concedes that emoji, character-splitting, and cross-lingual cloak types are out of scope, yet ToxicloakCN contains emoji substitutions without an explicit exclusion from Table 1, which widens the external-validity gap. Overall score 5: partial circularity in the experimental claim, with an independent derivation.
Assumptions & free parameters
free parameters (3)
- Dialectal confusion edge types in homo-graph =
n-l, zh-z, ch-c, sh-s, *ng-*n
- ToxicloakCN toxic lexicon =
387 words after manual correction and deduplication
- COLDataset toxic lexicon =
115 crawler keywords
assumptions (5)
- domain assumption Homophonic character substitution is the dominant cloak type.
- domain assumption Tone-ignored pinyin equality plus five dialectal confusions captures real cloaking.
- domain assumption The curated toxic lexicon covers the cloaks present in the test data.
- domain assumption LM probability is a valid naturalness score for judging whether a replacement is a cloak.
- standard math Bayes' rule applies to auto-regressive sequence probabilities.
Cite this review
Pith. "Pith review of Breaking the Cloak! Unveiling Chinese Cloaked Toxicity with Homophone Graph and Toxic Lexicon." pith.science (2026). https://pith.science/paper/NLWK4TJV
@misc{pith2026250522184,
author = {Pith},
title = {Pith review of: Breaking the Cloak! Unveiling Chinese Cloaked Toxicity with Homophone Graph and Toxic Lexicon},
year = {2026},
howpublished = {\url{https://pith.science/paper/NLWK4TJV}},
note = {Machine review of arXiv:2505.22184}
}
abstract
Social media platforms have experienced a significant rise in toxic content, including abusive language and discriminatory remarks, presenting growing challenges for content moderation. Some users evade censorship by deliberately disguising toxic words through homophonic cloak, which necessitates the task of unveiling cloaked toxicity. Existing methods are mostly designed for English texts, while Chinese cloaked toxicity unveiling has not been solved yet. To tackle the issue, we propose C$^2$TU, a novel training-free and prompt-free method for Chinese cloaked toxic content unveiling. It first employs substring matching to identify candidate toxic words based on Chinese homo-graph and toxic lexicon. Then it filters those candidates that are non-toxic and corrects cloaks to be their corresponding toxicities. Specifically, we develop two model variants for filtering, which are based on BERT and LLMs, respectively. For LLMs, we address the auto-regressive limitation in computing word occurrence probability and utilize the full semantic contexts of a text sequence to reveal cloaked toxic words. Extensive experiments demonstrate that C$^2$TU can achieve superior performance on two Chinese toxic datasets. In particular, our method outperforms the best competitor by up to 71% on the F1 score and 35% on accuracy, respectively. Our code and data are available at https://github.com/XDxc-cuber/C2TU-Chinese-cloaked-toxicity-unveiling.
Figures
Forward citations
Cited by 1 Pith paper
-
Lost in Pronunciation: Detecting Chinese Offensive Language Disguised by Phonetic Cloaking Replacement
A new 500-post benchmark of naturally occurring phonetic cloaking shows LLMs detect such Chinese offensive language with F1 at most 0.672, and Pinyin-augmented prompting partially repairs the gap.
Reference graph
Works this paper leans on
-
[1]
Llama 3 model card
AI@Meta. Llama 3 model card. 2024
2024
- [2]
-
[3]
T. Bayes. An essay towards solving a problem in the doctrine of chances.Biometrika, 45(3- 4):296–315, 1958
work page 1958
- [4]
-
[5]
C. Cui. Jionlp, 2020
work page 2020
-
[6]
Davidson, D
T. Davidson, D. Warmsley, M. Macy, and I. Weber. Automated hate speech detection and the problem of offensive language. InProceedings of the international AAAI conference on web and social media, volume 11, pages 512–515, 2017
2017
-
[7]
J. Deng, J. Zhou, H. Sun, C. Zheng, F. Mi, H. Meng, and M. Huang. Cold: A benchmark for chinese offensive language detection.arXiv preprint arXiv:2201.06025, 2022
arXiv 2022
-
[8]
Devlin, M.-W
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technolo- gies, volume 1 (long and short papers), pages 4171–4186, 2019
2019
Show all 33 references
-
[9]
G. Dong, J. Zhao, T. Hui, D. Guo, W. Wang, B. Feng, Y . Qiu, Z. Gongque, K. He, Z. Wang, et al. Revisit input perturbation problems for llms: A unified robustness evaluation framework for noisy slot filling task. InCCF International Conference on Natural Language Processing an...
2023
-
[10]
M. Dong, Y . Chen, M. Zhang, H. Sun, and T. He. Rich semantic knowledge enhanced large language models for few-shot chinese spell checking.arXiv preprint arXiv:2403.08492, 2024
2024 arXiv
-
[11]
Ghosh, P
S. Ghosh, P. Varshney, E. Galinkin, and C. Parisien. Aegis: Online adaptive ai content safety moderation with ensemble of llm experts.arXiv preprint arXiv:2404.05993, 2024
2024 arXiv
-
[12]
H. Gong, Y . Li, S. Bhat, and P. Viswanath. Context-sensitive malicious spelling error correction. InThe World Wide Web Conference, pages 2771–2777, 2019
2019
-
[13]
Jafari, J
N. Jafari, J. Allan, and S. M. Sarwar. Target span detection for implicit harmful content. InProceedings of the 2024 ACM SIGIR International Conference on Theory of Information Retrieval, pages 117–122, 2024
2024
-
[14]
Jiang, X
A. Jiang, X. Yang, Y . Liu, and A. Zubiaga. Swsr: A chinese dataset and lexicon for online sexism detection.Online Social Networks and Media, 27:100182, 2022
2022
-
[15]
H. Kirk, B. Vidgen, P. Rottger, T. Thrush, and S. A. Hale. Hatemoji: A test suite and adversarially-generated dataset for benchmarking and detecting emoji-based hate. In M. Carpuat, M.-C. de Marneffe, and I. V . Meza Ruiz, editors,Proceedings of the 2022 Conference of the Nort...
2022
-
[16]
J. Li, Q. Wang, Z. Mao, J. Guo, Y . Yang, and Y . Zhang. Improving chinese spelling check by character pronunciation prediction: The effects of adaptivity and granularity.arXiv preprint arXiv:2210.10996, 2022
2022 arXiv
-
[17]
K. Li, Y . Hu, L. He, F. Meng, and J. Zhou. C-llm: Learn to check chinese spelling errors character by character.arXiv preprint arXiv:2406.16536, 2024. 11
2024 arXiv
-
[18]
A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024
2024 arXiv
-
[19]
J. Lu, B. Xu, X. Zhang, C. Min, L. Yang, and H. Lin. Facilitating fine-grained detection of chinese toxic language: Hierarchical taxonomy, resources, and benchmarks.arXiv preprint arXiv:2305.04446, 2023
2023 arXiv
-
[20]
Phute, A
M. Phute, A. Helbling, M. Hull, S. Peng, S. Szyller, C. Cornelius, and D. H. Chau. Llm self de- fense: By self examination, llms know they are being tricked.arXiv preprint arXiv:2308.07308, 2023
2023 arXiv
-
[21]
X. Rao, Y . Zhang, S. Peng, Q. Jia, and X. Liu. Chinese hate speech detection method based on RoBERTa-WWM. In M. Sun, B. Qin, X. Qiu, J. Jiang, and X. Han, editors,Proceedings of the 22nd Chinese National Conference on Computational Linguistics, pages 501–511, Harbin, China, A...
2023
-
[22]
Sheth, V
A. Sheth, V . L. Shalin, and U. Kursuncu. Defining and detecting toxicity on social media: context and knowledge are key.Neurocomputing, 490:312–318, 2022
2022
-
[23]
Q. Team. Qwen2.5: A party of foundation models, September 2024
2024
-
[24]
Y . Xiao, Y . Hu, K. T. W. Choo, and R. K.-w. Lee. Toxicloakcn: Evaluating robustness of offen- sive language detection in chinese with cloaking perturbations.arXiv preprint arXiv:2406.12223, 2024
2024 arXiv
-
[25]
A. Yang, B. Xiao, B. Wang, B. Zhang, C. Bian, C. Yin, C. Lv, D. Pan, D. Wang, D. Yan, et al. Baichuan 2: Open large-scale language models.arXiv preprint arXiv:2309.10305, 2023
2023 arXiv
-
[26]
Zhang, H
H. Zhang, H. Gao, Q. Hu, G. Chen, L. Yang, B. Jing, H. Wei, B. Wang, H. Bai, and L. Yang. Chinesesafe: A chinese benchmark for evaluating safety in large language models.arXiv preprint arXiv:2410.18491, 2024
2024 arXiv
-
[27]
Zhang, H
S. Zhang, H. Huang, J. Liu, and H. Li. Spelling error correction with soft-masked bert.arXiv preprint arXiv:2005.07421, 2020
2005 arXiv
-
[28]
Y . Zhao, J. Zhu, C. Xu, and X. Li. Enhancing llm-based hatred and toxicity detection with meta-toxic knowledge graph.arXiv preprint arXiv:2412.15268, 2024
2024 arXiv
-
[29]
H. Zhou, Z. Li, B. Zhang, C. Li, S. Lai, J. Zhang, F. Huang, and M. Zhang. A simple yet effective training-free prompt-free approach to chinese spelling correction based on large language models.arXiv preprint arXiv:2410.04027, 2024
-
[30]
J. Zhou, J. Deng, F. Mi, Y . Li, Y . Wang, M. Huang, X. Jiang, Q. Liu, and H. Meng. Towards identifying social bias in dialog systems: Framework, dataset, and benchmark. In Y . Goldberg, Z. Kozareva, and Y . Zhang, editors,Findings of the Association for Computational Linguist...
2022
-
[31]
Input-output length consistency is mandatory
-
[32]
魄”, “蠢”, “鸡
Return only the corrected sentence without any additional output. - User: <Input sentence> - Assistant: <Output sentence> H Toxic Lexicon We present toxic lexicons of ToxicloakCN and COLDataset here, splitting each lexicon by toxic words’ length. Specifically, the toxic lexico...
-
[33]
Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects
Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...
2025
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.