{"id":"f811d762-678d-4ed9-b739-ef92f12699e4","arxiv_id":"2506.04693","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Adding six rhetorical codetype descriptions to frozen LLM embeddings improves implicit hate speech detection on Chinese and English benchmarks, though prompt-based gains are inconsistent.","lead":"This paper introduces six 'codetype' categories, such as metaphor, pun, and abbreviation, that capture how implicit hate speech hides its meaning, and tests feeding these categories into large language models. The method improves detection of hidden hate speech on Chinese and English social media, especially when the categories are folded into the model's internal sentence representations.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 3 contradicts the 'consistently improves' claim: for prompt-based method on ToxiCN, only 1 of 21 codetype configurations beats the no-codetype baseline, so the central claim is not supported as stated.","rationale":"The reader's verdict is CONDITIONAL and their rationale already notes that 'the abstract's blanket improvement claim is not supported by the prompt-based results,' so we partially agree. However, their formal weakest_assumption focuses on Samp leakage from evaluation datasets. That is a valid secondary risk, but it only affects configurations containing Samp and could be resolved by deduplication; it does not address the fact that Table 3, taken at face value, contradicts the central claim for the prompt-based method on ToxiCN. The most load-bearing condition for the central claim is that codetype information reliably improves detection; the paper's own data falsify that condition for 20 of 21 prompt-based ToxiCN cells. This is an internal inconsistency, not a matter of external consensus. I therefore recommend keeping the CONDITIONAL verdict: the paper has a plausible embedding-based contribution and a testable taxonomy, but the headline claim must be narrowed and the evaluation needs the reader's proposed deduplication and statistical rigor before acceptance.","tokens_in":19293,"tokens_out":10065,"duration_ms":108628,"concrete_test":"Run a sign test on Table 3: for each of the 9 model-dataset pairs and each of the two methods, count how many of the 7 codetype configurations (Name, Expl, Samp, and their combinations) yield F1 strictly above the no-codetype baseline. If prompt-based ToxiCN yields 1/21, the 'consistently improves' formulation is false; the claim must be restricted to embedding-based methods or to specific combinations, and the abstract and conclusion must be revised accordingly. This can be done by hand from the published table in under an hour and requires no additional experiments.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that adding codetype information improves im-HS detection. Table 3 shows this is false for the prompt-based method on ToxiCN: across 3 models x 7 codetype combinations, exactly one entry (Baichuan2-13B with Samp, 0.2988 vs 0.2556 baseline) exceeds the baseline; all 7 Llama2-7B and all 7 Llama2-13B prompt-based configurations fall below baseline. Section 5.4 concedes 'the average F1 score of the prompt-based method is generally lower in ToxiCN comparing to its baseline.' The abstract and Section 1 nonetheless state that codetypes 'consistently improves' detection, and the conclusion claims the experiments 'establish the efficacy.' Because the paper's own numbers contradict the unqualified claim, the central claim is internally inconsistent. This is more load-bearing than the potential Samp leakage: even if every codetype sample were excluded from the test splits, the prompt-based ToxiCN results would still fail to support the stated claim. The paper could be repaired by narrowing the claim to embedding-based methods and to configurations that actually improve, but as written the headline result is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a six-category taxonomy of \"codetypes\" (Irony, Metaphor, Pun, Argot, Abbreviation, Idiom) claimed to characterize implicit hate speech, and it tests two ways of integrating this taxonomy into LLM-based detection: a prompt-based method that adds codetype names, explanations, and sample sentences to the instruction, and an embedding-based method that appends codetype information to the input of frozen LLMs and feeds the resulting representations to a logistic regression classifier. Experiments are run on three datasets (ToxiCN in Chinese; Latent-hatred and ISHate in English) with three models (Baichuan2-13B-Chat, Llama2-Chinese-7B-Chat, Llama2-Chinese-13B-Chat). The paper claims that codetypes consistently improve implicit hate speech detection across languages and methods.","tokens_in":19590,"tokens_out":3863,"duration_ms":46652,"significance":"The codetype taxonomy is a genuine descriptive contribution that organizes implicit hate speech into interpretable linguistic strategies, and the embedding-based methods show substantial and often consistent F1 gains over the no-codetype baseline, especially for the EII and EIII variants; the best result reaches 0.8091 F1 on ToxiCN. The inclusion of two English datasets provides some cross-linguistic validation, which is a strength. The paper also usefully reports negative prompt-based results in Section 5.4, although the abstract and conclusion do not reflect this nuance. The main limitation is that the headline claim of consistent improvement is contradicted by the paper's own data for the prompt-based method on ToxiCN, and the evaluation lacks statistical validation and does not document whether codetype example sentences are excluded from the test splits.","major_comments":[{"comment":"The central claim that codetypes \"consistently improve\" implicit hate speech detection is not supported by Table 3. For the prompt-based method on ToxiCN, only 1 of 21 codetype configurations (Baichuan2-13B with Samp, 0.2988 vs 0.2556 baseline) exceeds the no-codetype baseline; all seven Llama2-7B and all seven Llama2-13B prompt-based configurations fall below baseline. Section 5.4 concedes this point, but the abstract and Section 6 still assert general efficacy. The claim must be narrowed to the embedding-based methods and to the specific configurations that actually improve, and the abstract/conclusion should be revised accordingly.","section":"§5.4, Table 3"},{"comment":"The taxonomy is built from a 200-sample subset of ToxiCN, and the Samp examples used in prompts and embedding inputs are explicitly drawn from ToxiCN (e.g., Table 1's 'txl是社会问题...' and Figure 1). The paper never states whether these example sentences are excluded from the ToxiCN test split. If any Samp example overlaps with a test sentence, the gains on ToxiCN could be inflated by memorization rather than by taxonomy understanding. The authors must either confirm the exclusion of all codetype examples from test splits or re-run the ToxiCN evaluation with held-out examples before the Chinese results can be trusted.","section":"§3.1, Table 1, §5.3"},{"comment":"All experimental results come from a single 8:1:1 split with no error bars, repeated runs, or significance tests. Many critical comparisons in Table 3 differ by less than 0.02 F1 (e.g., Llama2-7B on ISHate: 0.6625 baseline vs 0.6667 Name+Expl in EIII), so the reported improvements are not established as reliable. The authors should provide variance estimates (e.g., bootstrap confidence intervals) and significance tests for the headline comparisons, or at least discuss the magnitude of the differences relative to the lack of variance information.","section":"§5.3"}],"minor_comments":[{"comment":"The word 'Argot' is misspelled as 'Agort' in several places, including Figure 2 and Figure 5; this should be corrected throughout.","section":"Figure 2, Figure 5, Table 5"},{"comment":"The section heading reads 'Conlusions' instead of 'Conclusions'.","section":"§6"},{"comment":"The text contains corrupted glyph sequences (e.g., 'uni0000002c/uni00000055...' in Section 3.1 and Appendix A.2) that appear to be rendering errors for Chinese characters; these must be fixed to the actual characters.","section":"§3.1, Appendix A.2"},{"comment":"The composition of the 'Samp' component is ambiguous: the paper should specify whether the selected samples are drawn from the training split, whether they are identical across datasets, and whether they are the same for all models and prompt variants.","section":"§5.3"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid core in the embedding-based results and the taxonomy is a useful contribution, but the headline claim is overstated and the contamination question is unresolved. If the authors can confirm test-set separation for the codetype examples and revise the abstract/conclusion to reflect the scope of the improvements, I would support acceptance. As written, the internal inconsistency between the stated claim and Table 3 warrants major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the six-codetype taxonomy is a real attempt to give im-HS detection a usable linguistic handle, and the embedding-based results are genuinely better across all three datasets. But the abstract says codetypes 'consistently improve' detection, and that is not what Table 3 shows. On ToxiCN with the prompt-based method, 19 of 21 codetype configurations land below the no-codetype baseline; on Llama2-7B and Llama2-13B every single one does. The stress-test note said only one configuration beats baseline; it's actually two (Samp and Name+Samp on Baichuan2-13B, both marginal), but the point stands. Section 5.4 concedes this in a single sentence, then the conclusion quietly claims the experiments 'establish the efficacy.' That's an internal contradiction the authors need to fix, not a stylistic choice.\n\nWhat's actually new: the taxonomy construction pipeline (Chinese rhetoric sources, Wikipedia filtering, word2vec similarity pruning, 200-sample annotation with Fleiss' kappa) is systematic and I haven't seen it in the cited prior work. The three embedding fusion strategies are simple but they work: the gains on Latent-hatred and ISHate, plus ToxiCN, are consistent and often large. Testing on English datasets gives the taxonomy some independent grounding, even if the Chinese origin and ToxiCN-based category selection make the circularity only partial, not absent.\n\nSoft spots, in order. First, the claim/report mismatch above is the main issue. Second, the evaluation rests on a single 8:1:1 split with no error bars, no significance tests, and no code or data release. Some differences are large enough to be believable, but we can't rule out split luck. Third, leakage: Table 1 and Figure 1 use ToxiCN sentences as codetype examples, and the paper never states that these sentences were excluded from test splits. If they were not, the Samp-based gains could be inflated. The paper does not address this. Fourth, the taxonomy's conceptual novelty is modest—irony, metaphor, pun, and so on are ancient categories—but the contribution is the empirical packaging, not the names.\n\nWho this is for: people working on content moderation, especially multilingual or Chinese im-HS. The overclaim makes it hard to cite as-is. Recommend: send to peer review with a request for major revision—narrow the claims to embedding-based methods and to specific configurations, add error bars or significance tests, release data/code with deduplicated splits, and state whether illustration examples overlap with test data. A serious referee can get this to a publishable state.","headline":"The taxonomy construction and embedding-based gains are real, but the 'consistently improves' claim is contradicted by their own prompt-based ToxiCN results; the paper needs claim-narrowing and leakage checks.","tokens_in":20087,"tokens_out":4390,"would_cite":false,"duration_ms":45527,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding a six-codetype taxonomy of rhetorical encoding tricks to prompts or to frozen-LLM embeddings improves implicit hate speech detection across Chinese and English benchmarks.","keywords":["implicit hate speech","hate speech detection","codetype taxonomy","rhetorical strategies","large language models","embedding-based classification","prompt-based classification","cross-lingual detection"],"falsifier":"Check every test example in ToxiCN, Latent-hatred, and ISHate for exact or near-verbatim overlap with the codetype sample sentences used in the paper, such as the Abbreviation sample 'txl是社会问题，应该有人管，而不是当透明人' or 'WPWW', then rerun the prompt and embedding baselines with any overlapping samples withdrawn from the codetype information; the central claim is falsified if the F1 advantage over the no-codetype baseline disappears.","tokens_in":19114,"feed_emoji":"🛡️","tokens_out":8382,"duration_ms":87295,"temperature":0.7,"pith_summary":"Implicit hate speech hides its target behind indirect devices: a slur compressed into an abbreviation, a group compared to an animal, a homophone swapped in to evade filters. This paper claims that large language models detect such speech more accurately when they are first shown what these encoding tricks look like. The authors define six 'codetypes' — abbreviation, metaphor, irony, pun, idiom, and argot — and supply the names, definitions, and example sentences to LLMs either in the prompt or as extra context in the embedding layer. Across one Chinese and two English datasets, the codetype-augmented setups achieve higher F1 scores than the same models without codetypes, with the largest and most consistent gains coming from the embedding-based methods. If the claim holds, content moderators get a cheap, language-agnostic way to sharpen veiled-hate detection without retraining a model.","feed_headline":"Six codetype labels lift LLMs' detection of veiled hate","feed_subtitle":"Feeding LLMs the names, definitions, and examples of six hate-encoding tricks raises F1 across three datasets","key_machinery":"The central object is the codetype taxonomy: a set of six rhetorical encoding strategies — Abbreviation, Metaphor, Irony, Pun, Idiom, and Argot — that the paper claims cover roughly 80% of implicit hate speech in its annotated sample. The load-bearing mechanism is feeding these codetype labels, Wikipedia-derived explanations, and sample sentences to LLMs. In the prompt-based method, the codetype block is prepended to the classification instruction; in the embedding-based methods, the taxonomy is concatenated with the sentence before frozen LLMs are used as encoders, either all at once, per codetype with concatenated embeddings, or per codetype with averaged embeddings, and the multi-head attention outputs from all transformer layers are fed to a logistic regression classifier.","core_discovery":"The paper's central discovery is that implicit hate speech detection is improved by making explicit the linguistic encoding strategies that make hate implicit. Concretely, the authors propose a taxonomy of six codetypes — Abbreviation, Metaphor, Irony, Pun, Idiom, and Argot, with an Other category — built by filtering Chinese rhetorical categories through Wikipedia and annotating 200 samples from ToxiCN. They then use this taxonomy in two ways: a prompt-based method that prepends codetype names, definitions, and samples to the classification instruction, and an embedding-based method that treats LLMs as frozen encoders and supplies codetype information through three constructions — concatenating all codetypes with the sentence, concatenating per-codetype embeddings, and averaging per-codetype embeddings. On ToxiCN (Chinese), Latent-hatred (English), and ISHate (English), the embedding-augmented variants improve F1 over the no-codetype baseline, with the per-codetype matching strategies being more robust than the single-concatenation strategy. The prompt-based gains are smaller and dataset-dependent, and the paper attributes some failures to overcorrection when too much codetype information is provided.","pith_inferences":["Beyond the paper, the codetype idea should transfer to other indirect harmful language, such as masked slurs in cyberbullying or dog-whistle politics; a quick test would be annotating one such corpus with the six codetypes and checking whether the same embedding boost appears.","A natural extension is an automatic codetype tagger that predicts which of the six strategies actually appears in a sentence and feeds only those labels to the model, which could reduce the overcorrection errors the paper observes with full-taxonomy inputs.","The sample-based gains suggest the taxonomy may be working partly as a retrieval scaffold — the model sees a canonical example and pattern-matches — rather than as a deep understanding of the definitions; an ablation with the sample examples removed but definitions kept would separate these two mechanisms.","If the cross-linguistic result holds, the same six codetypes could be used as an annotation schema for detecting implicit hate in third languages, testing whether the taxonomy is universal or only shared between Chinese and English."],"forward_implications":["If the codetype claim is right, content moderators can improve veiled-hate detection simply by adding a six-line taxonomy to their existing prompt, without fine-tuning a model.","The embedding-based results imply that even a frozen, instruction-tuned LLM contains enough linguistic knowledge for the taxonomy to redirect its attention; no parameter updates are needed for the gain to appear.","Per-codetype matching outperforms feeding all codetypes at once, suggesting that letting the model check each encoding strategy against the sentence individually is the more reliable way to use the taxonomy.","Codetype examples act as few-shot demonstrations in prompt-based settings, but combining them with names and explanations can overcorrect the model into flagging neutral statements about sensitive groups as hateful.","The taxonomy transfers across Chinese and English, supporting the claim of cross-linguistic commonalities in how hate is encoded and pointing to a single taxonomy usable in both languages."],"supporting_citations":[{"why":"Supplies the ToxiCN Chinese dataset, the taxonomy examples in Figure 1, and the primary Chinese evaluation benchmark.","marker":"(Lu et al., 2023)"},{"why":"Supplies the Latent-hatred English dataset and the seven-category implicit hate taxonomy the paper contrasts with its codetype taxonomy.","marker":"(ElSherief et al., 2021)"},{"why":"Supplies the ISHate English dataset with implicit and subtle hate categories used as the third evaluation benchmark.","marker":"(Ocampo et al., 2023)"},{"why":"Provides the Chinese rhetorical-styles inventory from which the codetype candidates are initially drawn.","marker":"(Jiang, 2019)"},{"why":"Provides the word2vec embeddings used to measure codetype similarity and filter the 43 candidates down to 10.","marker":"(Mikolov, 2013)"},{"why":"Provides the kappa statistic used to measure inter-annotator agreement during codetype taxonomy validation.","marker":"(Fleiss, 1971)"},{"why":"Motivates the embedding-based method by showing that LLM hidden layers contain task-relevant latent information.","marker":"(Burns et al., 2022)"},{"why":"Supplies the practice of using multi-head attention outputs from all transformer layers as features for the frozen-encoder classifier.","marker":"(Li et al., 2024)"}],"fun_headline_variants":["Coding hate: Taxonomy of six tricks sharpens LLM detection","Implicit hate unmasked: Codetypes improve LLM F1 scores","LLMs spot veiled hate with six linguistic codetypes","Codetype taxonomy helps LLMs catch subtle hate speech","Six coding strategies make hate detection more implicit-aware"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the illustrative example sentences shown for each codetype are not drawn from the test portions of the evaluation datasets; if any test sentence or near-duplicate appears among those examples, the measured gains could come from memorization instead of taxonomy understanding.","fun_headline_variants_meta":{"raw":{"variants":["Coding hate: Taxonomy of six tricks sharpens LLM detection","Implicit hate unmasked: Codetypes improve LLM F1 scores","LLMs spot veiled hate with six linguistic codetypes","Codetype taxonomy helps LLMs catch subtle hate speech","Six coding strategies make hate detection more implicit-aware"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000416,"raw_usage":{"total_tokens":2137,"prompt_tokens":925,"completion_tokens":1212,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":1127}},"tokens_in":541,"tokens_out":1212,"duration_ms":11852,"temperature":1.0,"reasoning_tokens":1127,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:35:28.432300+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check every test example in ToxiCN, Latent-hatred, and ISHate for exact or near-verbatim overlap with the codetype sample sentences used in the paper, such as the Abbreviation sample 'txl是社会问题，应该有人管，而不是当透明人' or 'WPWW', then rerun the prompt and embedding baselines with any overlapping samples withdrawn from the codetype information; the central claim is falsified if the F1 advantage over the no-codetype baseline disappears.","supporting_citations":[],"review_version":1}