{"id":"e48334f9-c2a1-4b4a-b409-bcc656ed8f90","arxiv_id":"2507.11292","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"The paper creates a span-level Chinese hate speech dataset and a 830-term coded hate lexicon, but its two-stage training method's reported superiority is contradicted by the paper's own COLD results.","lead":"This paper builds a new Chinese-language dataset that labels hate speech at the level of individual phrases, including targets and arguments, plus a lexicon of 830 coded slur terms. It also tests whether large language models can spot and explain these terms, and whether adding the lexicon to a detection model improves performance.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central method claim is contradicted by Table X: on COLD, Only Lexicon F1 (71.19) exceeds Two Stage F1 (70.57), so the two-stage framework does not achieve the highest F1 on both datasets; single-run results with test-set learning-rate selection leave the claimed advantage statistically…","rationale":"The strongest claim of the paper is contribution (iii), explicitly stated in the abstract and in Section VIII. The only supporting experiment is Table X. The paper's own numbers show the claimed pattern is not monotone: Only Lexicon beats Two Stage on COLD F1. The text in Section VIII-b ('consistently outperforms ... highest F1 scores on both') is factually inconsistent with the table. Even setting that aside, the experimental protocol cannot rule out chance: Section V-C says the learning rate was chosen by test F1, and no seeds or error bars are reported. Differences of about 1.8 points on test sets are not interpretable without variance estimates. Therefore the central methodological contribution is unsupported. I do not object to the dataset and lexicon, which are documented with annotation statistics and IAA, and the probing results may be useful; but the paper's own framing makes the two-stage method the headline. The verdict should remain REJECT, or at best CONDITIONAL on a corrected statistical analysis. The reader's weakest-assumption analysis identifies the same experimental fragility; I would additionally foreground the direct Table X contradiction.","tokens_in":20254,"tokens_out":3512,"duration_ms":39419,"concrete_test":"Re-run the COLD and Cdial-Bias comparisons from Section VIII with at least 5 random seeds per condition under the Table XI hyperparameters; report mean and standard deviation of F1 and apply a paired significance test (e.g., McNemar or bootstrap) between Two Stage, Only LoRA, and Only Lexicon. In addition, independently recompute the Table X COLD entries to confirm whether 71.19 versus 70.57 is correct. The two-stage claim stands only if it is significantly better than Only LoRA on both datasets and not worse than Only Lexicon on COLD.","verdict_should_be":"REJECT","load_bearing_attack":"The third contribution, that lexicon integration 'significantly enhances hate speech detection performance' (Abstract), is validated only by Table X in Section VIII. Section VIII-b states that the two-stage framework 'consistently outperforms both Only LoRA and Only Lexicon by achieving the highest F1 scores on both COLD (70.57) and Cdial-Bias (60.04).' The table itself contradicts this: on COLD, Only Lexicon achieves F1 71.19, which is higher than Two Stage's 70.57. The superiority claim is therefore false on one of the two datasets. Additionally, Section V-C reports that the learning rate was selected by highest test-set F1, and Tables VII, VIII, and X show single-run metrics with no standard deviations, confidence intervals, or significance tests; the remaining COLD advantage over Only LoRA is only 1.89 F1 points and the Cdial-Bias advantage is 1.77 points. Under these conditions, the observed ordering may reflect test-set selection and seed noise rather than a genuine effect of the lexicon-plus-sequential-fine-tuning method. Because this experiment is the sole support for the paper's headline method, the central claim is not currently established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents STATE ToxiCN, a span-level Chinese hate speech dataset with 8,029 posts and 9,533 Target-Argument-Hateful-Group quadruples derived from TOXICN; a Chinese coded hate term lexicon of 830 terms with group and explanation annotations; and an evaluation protocol with three tasks (quadruple identification, coded term identification, coded term explanation) applied to twelve LLMs. The final contribution is a two-stage fine-tuning framework that first trains on the lexicon and then on the target detection dataset, evaluated on COLD and CDial-Bias. The paper claims this framework consistently achieves the highest F1 on both external datasets.","tokens_in":20534,"tokens_out":6937,"duration_ms":83267,"significance":"The resource contributions are potentially significant if released: STATE ToxiCN appears to be the first span-level Chinese hate speech dataset with quadruple annotations, and the 830-term coded hate lexicon with explanations is a novel resource for studying implicit hate in Chinese. The evaluation of LLMs, including open, safety-domain, and closed models, documents a clear performance gap between fine-tuned models and API models on span-level tasks and identifies DeepSeek-v3 as notably stronger on coded term tasks. The strength of the resource construction is undercut by the unsupported third contribution: the central claim that the two-stage framework 'significantly enhances' detection is contradicted by the paper's own Table X and rests on test-set hyperparameter selection with single runs and no significance testing. I do not see a circularity problem in the evaluation: the two-stage experiment is run on external COLD and CDial-Bias, and the lexicon's provenance from TOXICN does not leak those labels. The paper ships no machine-checked proofs, but the empirical claims are testable; the main issue is that they are not established as stated.","major_comments":[{"comment":"The claim in Section VIII(b) that the two-stage framework 'consistently outperforms both Only LoRA and Only Lexicon by achieving the highest F1 scores on both COLD (70.57) and Cdial-Bias (60.04)' is contradicted by Table X: on COLD, Only Lexicon achieves F1 71.19, which is higher than Two Stage's 70.57. Consequently, the abstract's assertion that the proposed method 'significantly enhanc[es] hate speech detection performance' and the conclusion's 'demonstrably enhancing' are not supported by the reported data on one of the two datasets.","section":"Section VIII, Table X"},{"comment":"The learning rate is selected by highest test-set F1 ('ultimately selecting the one that yielded the highest F1 score on the test set'), and all reported metrics are single runs without standard deviations, confidence intervals, or significance tests. The advantages of Two Stage over Only LoRA are 1.89 F1 points on COLD and 1.77 F1 points on Cdial-Bias; under test-set selection and single-run noise, this ordering cannot be reliably attributed to the method. This concern applies not only to the two-stage claim but also to the fine-tuned-versus-API comparisons in Section VI.","section":"Section V-C, Tables VII, VIII, X"},{"comment":"The training setup for 'Only Lexicon' and the first stage of 'Two Stage' is not specified precisely enough to be reproducible: a lexicon of (Term, Groups, Definition) entries is not a labeled binary classification corpus, and the manuscript does not state how these entries are converted into training examples, what labels are used, or how many examples result. Without this detail, the comparison between Only Lexicon and the other conditions cannot be interpreted.","section":"Section VIII, Table XI"}],"minor_comments":[{"comment":"Task 2 is described as using recall as the primary metric, but Table IX reports F1 scores and the text says 'hard and soft F1 scores'; the metric should be stated consistently.","section":"Sections V-B and VII-A, Table IX"},{"comment":"Figure 1 is not legible in the supplied manuscript: it appears as a long run of encoded tokens rather than a bar chart, so the win/tie/loss rates cannot be checked.","section":"Figure 1"},{"comment":"There are inconsistencies in naming and formatting: 'STATE TOXICN' and 'STATE ToxiCN' are used interchangeably, and reference [16] appears as '[16]Zhou et al.' without a space.","section":"Throughout"},{"comment":"The term 'weighted averaging' is unexplained; if multiple runs or learning rates are averaged, the procedure should be described.","section":"Section V-C"}],"recommendation":"reject","confidential_remarks":"For the editor: the dataset and lexicon could form the basis of a useful resource paper if the method claim is removed or thoroughly rewritten; in its current form, the headline contribution is contradicted by the paper's own results. The novelty of the dataset relative to TOXICN should also be checked at revision, since STATE ToxiCN is derived from the authors' earlier TOXICN."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The dataset and lexicon are genuinely useful; the headline method claim is not supported by the paper's own numbers. On COLD in Table X, Only Lexicon F1 is 71.19, Two Stage is 70.57, yet Section VIII-B says Two Stage achieves the highest F1 on both datasets. That is a direct internal contradiction, and it is the third contribution's only evidence.\n\nThe first two contributions deserve credit. STATE ToxiCN is the first span-level Chinese hate speech dataset with target-argument-group-hateful quadruples, built on TOXICN with a sensible two-stage filtering process, detailed annotation guidelines, Fleiss kappa between 0.61 and 0.75, and a diverse annotator pool. The 830-term coded-hate lexicon with definitions and group labels fills a real gap, and the LLM evaluation across twelve models is broad. The analysis of why coded terms hurt fine-tuned span extraction but help API models is plausible.\n\nThe soft spots are the method section and its evaluation. The two-stage framework is sequential fine-tuning, which is not a new idea. More importantly, the experiment that validates it is underpowered: learning rates were chosen by best test-set F1, results are single runs with no standard deviations or significance tests, and the claimed advantages over Only LoRA are 1.89 F1 points on COLD and 1.77 on CDial-Bias. Those gaps could easily be seed noise. And the contradiction with Only Lexicon on COLD means the statement 'consistently outperforms both' is simply false.\n\nThere are also smaller issues: the abstract says 'significantly enhances' but no significance test is run; Table VIII's color coding is not visible in the text; and Task 3's 100-sample human evaluation uses GPT-4o as judge without reporting agreement with humans. None of these are fatal on their own.\n\nOverall, the resource contributions are solid and the paper should not be rejected on that basis. But the method claim needs to be either corrected or removed. I would send this to peer review with a request for major revision: rerun with multiple seeds, select hyperparameters on a validation split, report error bars, and either fix the contradiction or soften the claim. The paper is worth engaging with.","headline":"Valuable new Chinese hate-speech resources wrapped around a method claim that its own Table X contradicts.","tokens_in":21063,"tokens_out":2296,"would_cite":true,"duration_ms":25814,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that two-stage fine-tuning—first on a coded-hate-term lexicon, then on task data—improves Chinese hate speech detection beyond either stage alone.","keywords":["hate speech detection","span-level annotation","Chinese coded hate terms","hate lexicon","two-stage fine-tuning","large language models","target-argument extraction","offensive language"],"falsifier":"Rerun Only LoRA, Only Lexicon, and Two Stage with multiple random seeds and a validation split for hyperparameter choice; if the F1 gap on COLD (70.57 versus 68.68) and on CDial-Bias (60.04 versus 58.27) falls within one standard deviation, the method's advantage is not established.","tokens_in":20053,"feed_emoji":"🚫","tokens_out":7314,"duration_ms":77359,"temperature":0.7,"pith_summary":"Chinese hate speech is under-served by fine-grained resources, so this paper builds a span-level dataset, a coded-hate-term lexicon, and a training recipe that combines them. The dataset, STATE ToxiCN, annotates 8,029 posts with 9,533 target–argument–hateful–group quadruples to expose multi-target and implicit hate that sentence-level labels miss. The lexicon records 830 Chinese coded hate terms with target-group and explanation annotations. On the COLD and CDial-Bias benchmarks, the proposed two-stage fine-tuning—lexicon first, then task data—achieves the highest F1 scores (70.57 and 60.04), above both lexicon-only and task-only fine-tuning. The intended contribution is evidence that explicit coded-hate knowledge, injected before task fine-tuning, makes detection more sensitive and more precise.","feed_headline":"Lexicon-first training lifts Chinese hate speech F1","feed_subtitle":"A two-stage fine-tuning recipe—coded-term lexicon, then task data—tops both single-stage baselines on COLD and CDial-Bias.","key_machinery":"The load-bearing machinery is a two-stage fine-tuning pipeline: a LoRA adapter is trained first on the annotated lexicon of 830 Chinese coded hate terms, then fine-tuned on the target hate-speech dataset, so the first stage supplies high-recall term knowledge and the second stage recontextualizes it. The lexicon is organized into phono-graphemic variation terms (homophones, acronyms, character splitting or combining) and hateful semantic terms (slang, cultural allusions, subculture usages), each carrying targeted-group labels and a plain-language explanation. The evaluation scaffold around the pipeline is the Target-Argument-Hateful-Group quadruple schema of STATE ToxiCN, which provides span-level ground truth for measuring whether models locate exactly who is attacked and why.","core_discovery":"The paper's central claim is that fine-grained Chinese hate speech understanding improves when span-level supervision and explicit coded-term knowledge are combined. STATE ToxiCN is presented as the first span-level Chinese hate speech dataset, using a Target-Argument-Hateful-Group quadruple schema to capture which group is attacked, with what argument, and whether the pair is hateful. The Chinese coded hate lexicon is presented as the first interpretable resource of its kind, dividing 830 terms into phono-graphemic variation terms (sound or character disguises) and hateful semantic terms (culturally loaded slurs), each with target-group and explanation annotations. The culminating method result is that two-stage fine-tuning—lexicon pre-tuning followed by dataset-specific LoRA fine-tuning—outperforms both Only LoRA and Only Lexicon on COLD and CDial-Bias, with F1 of 70.57 versus 68.68 on COLD and 60.04 versus 58.27 on CDial-Bias. Along the way, the paper reports that fine-tuned open models beat closed LLM APIs on span extraction, and that DeepSeek-v3 leads in coded-term identification and explanation.","pith_inferences":["Editorial inference: the two-stage recipe likely transfers to other low-resource abusive-language settings, since a small annotated lexicon is inexpensive to build and the pipeline only needs a modest task corpus for the second stage.","Editorial inference: the observed asymmetry—fine-tuned models lose span F1 on coded-term posts while LLM APIs gain—suggests a hybrid that feeds lexicon explanations as context to a fine-tuned extraction model could beat both; the paper does not test this.","Editorial inference: because the lexicon is stored as JSON terms with groups and definitions, it could be updated continuously as new coded terms appear, and the two-stage framework re-run on each update; the paper lists adaptive lexicons only as future work.","Editorial inference: the language bias in the evaluation (Chinese-specific homophones and character forms) means the coded-term taxonomy is probably not portable as-is, but the two-category scheme likely is."],"forward_implications":["The two-stage lexicon-then-fine-tuning recipe lifts F1 on COLD from 68.68 to 70.57 and on CDial-Bias from 58.27 to 60.04 relative to LoRA alone, so it is the recommended configuration among those tested.","Lexicon-only fine-tuning alone reaches 87.83 recall on COLD, meaning the lexicon can serve as a fast high-recall screening stage even without the second fine-tuning step.","Span-level supervision is the main differentiator in hard tasks: fine-tuned models stay near 50% soft F1 on quadruple extraction while API-only LLMs stay between 11% and 27%, identifying span-boundary precision as the bottleneck.","STATE ToxiCN can be used as a benchmark for semantic understanding of Chinese hate speech, not just binary label accuracy.","Coded hate term explanations are hardest for phono-graphemic variation terms across all models, so future interpretability work should target that category."],"supporting_citations":[{"why":"Supplies the TOXICN post corpus from which STATE ToxiCN is filtered and annotated.","marker":"[11]"},{"why":"Provides COLD, one of the two external Chinese hate speech benchmarks used to validate the two-stage framework.","marker":"[14]"},{"why":"Provides CDial-Bias, the second external benchmark used to validate the two-stage framework.","marker":"[49]"},{"why":"Supplies the LoRA fine-tuning machinery used in all training conditions of the two-stage comparison.","marker":"[48]"},{"why":"Provides the soft-matching span evaluation algorithm used to score predicted quadruples.","marker":"[44]"},{"why":"Establishes the target-argument-harmful span extraction paradigm that the quadruple schema extends.","marker":"[7]"},{"why":"Provides the Fleiss' kappa measure used to report inter-annotator agreement on the span annotations.","marker":"[33]"}],"fun_headline_variants":["Lexicon-first training boosts Chinese hate speech F1","Two-stage fine-tuning with coded terms beats baselines on hate speech","Span-level Chinese hate dataset and coded lexicon improve detection","Coded-term lexicon and span-level data sharpen hate speech models","New dataset and lexicon push Chinese hate speech detection forward"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim collapses if the two-stage improvement over the baselines is noise, because the paper chooses learning rates by highest test F1 and reports single runs with no variance estimate.","fun_headline_variants_meta":{"raw":{"variants":["Lexicon-first training boosts Chinese hate speech F1","Two-stage fine-tuning with coded terms beats baselines on hate speech","Span-level Chinese hate dataset and coded lexicon improve detection","Coded-term lexicon and span-level data sharpen hate speech models","New dataset and lexicon push Chinese hate speech detection forward"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00071,"raw_usage":{"total_tokens":3225,"prompt_tokens":999,"completion_tokens":2226,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":2145}},"tokens_in":615,"tokens_out":2226,"duration_ms":19216,"temperature":1.0,"reasoning_tokens":2145,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:11:35.099076+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun Only LoRA, Only Lexicon, and Two Stage with multiple random seeds and a validation split for hyperparameter choice; if the F1 gap on COLD (70.57 versus 68.68) and on CDial-Bias (60.04 versus 58.27) falls within one standard deviation, the method's advantage is not established.","supporting_citations":[{"cited_title":"Facilitating fine-grained detection of Chinese toxic language: Hierarchical taxonomy, resources, and benchmarks,","cited_arxiv_id":null,"evidence_quote":"Supplies the TOXICN post corpus from which STATE ToxiCN is filtered and annotated."},{"cited_title":"COLD: A Benchmark for Chinese Offensive Language Detection","cited_arxiv_id":"2201.06025","evidence_quote":"Provides COLD, one of the two external Chinese hate speech benchmarks used to validate the two-stage framework."},{"cited_title":"Overview of nlpcc 2022 shared task 7: Fine-grained dialogue social bias measurement,","cited_arxiv_id":null,"evidence_quote":"Provides CDial-Bias, the second external benchmark used to validate the two-stage framework."},{"cited_title":"Target-based offensive language identification,","cited_arxiv_id":null,"evidence_quote":"Establishes the target-argument-harmful span extraction paradigm that the quadruple schema extends."}],"review_version":1}