{"id":"7d4594f5-366c-4238-a482-b687118a16bd","arxiv_id":"2505.15297","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new Chinese detoxification dataset and 17-model benchmark show that LLMs can remove toxic words but often distort emotional tone, especially for emoji, homophone, and dialogue-based toxicity.","lead":"The authors built a Chinese-language dataset of 1,556 toxic sentences paired with rewrites that remove insults while keeping the speaker's anger or frustration. They tested 17 language models on rewriting, finding that even the best models fail on subtle cases with emojis, homophones, or multi-turn conversations.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The sentiment polarity metric is circular: the Qwen3-32B classifier used to evaluate 'neutral' rewrites is fine-tuned on ToxiRewriteCN's own human rewrites, so model rankings on emotional fidelity may reflect classifier bias rather than true sentiment preservation.","rationale":"The paper's core contribution is a sentiment-polarity-preserving detoxification dataset and a benchmark showing that LLMs fail to preserve emotional tone. The evaluation of emotional tone depends on a classifier trained on the same human rewrites that define the gold standard, creating a circular measure. The reader flagged this as a conditional issue; I agree and sharpen the focus on the sentiment classifier, since that metric is the unique part of the benchmark. A concrete validation study with fresh human labels would settle whether the polarity rankings are real. If the classifier agrees well with humans, the concern is resolved; if not, the comparative claims should be downgraded. The dataset itself may still be valuable, but the headline findings about sentiment polarity need this fix. Therefore the conditional verdict stands.","tokens_in":17377,"tokens_out":3489,"duration_ms":29654,"concrete_test":"Run a validation study: sample 200 model outputs stratified across the 17 models (or all test-set outputs), and have three new native-Chinese annotators, blind to the study's purpose, independently label each output as toxic, neutral, or polite. Compute agreement between the fine-tuned Qwen3-32B classifier and the majority human label (Cohen's kappa or percent agreement). Then recompute the 'Neutral ↑' and 'Polite ↓' columns of Table 1 using human labels instead of classifier labels. If kappa is below 0.6, or if the relative ranking of models on the neutral rate changes by more than 3 positions, the circularity concern lands and the conclusions about sentiment-polarity preservation are not reliable. Report the kappa and the recomputed table.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim—that LLMs struggle to preserve sentiment polarity during detoxification—rests entirely on the 'Sentiment Polarity' columns of Tables 1–6, which report the fraction of model outputs classified as 'neutral' by a fine-tuned Qwen3-32B three-class classifier. Appendix D states that this classifier's training data was constructed entirely from the ToxiRewriteCN dataset, with the neutral class consisting of the dataset's human-written rewrites (duplicated to a 1:2:1 ratio). Thus a high 'neutral' rate means only that a model's output resembles the human rewrites in the eyes of a classifier that learned to call those very rewrites 'neutral.' This is circular: the metric does not independently measure emotional fidelity; it measures stylistic proximity to the gold rewrites. No held-out evaluation of the classifier against fresh human judgments is reported, and the paper provides no inter-annotator agreement for the original rewrite labels. Consequently, the reported differences between models (e.g., GPT-4o's 67.16% neutral vs. Qwen-Max's 64.46%, or the claim that dense models over-sanitize) could shift or disappear under a non-circular sentiment measure. This concern is load-bearing because the dataset's defining innovation is sentiment-polarity-consistent rewriting; if the polarity metric is unreliable, the main comparative conclusions are unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ToxiRewriteCN, a Chinese dataset of 1,556 triplets consisting of a toxic sentence, a sentiment-aligned non-toxic rewrite, and labeled toxic spans, covering standard, emoji-induced, homophonic, single-turn, and multi-turn dialogue scenarios. The dataset is constructed through a six-step human-in-the-loop pipeline with LLM-assisted coarse rewriting followed by human post-editing and cross-verification. The authors evaluate 17 commercial and open-source LLMs on four dimensions: detoxification accuracy, fluency, content preservation, and sentiment polarity. They report that commercial and MoE models perform best overall but that all models struggle with sentiment-preserving detoxification, especially for emoji, homophone, and multi-turn dialogue inputs, where over-sanitization and under-neutralization are common.","tokens_in":17826,"tokens_out":3580,"duration_ms":31062,"significance":"If the dataset and evaluation are sound, ToxiRewriteCN would be a valuable resource: it is the first Chinese detoxification dataset explicitly designed to preserve sentiment polarity, and it addresses a real gap given existing Chinese toxicity datasets focus on classification rather than rewriting. The human post-editing and cross-verification pipeline is a strength, as is the decision to release the data and code. The paper also provides a broad benchmark across 17 models and fine-grained scenario analyses. However, the evaluation's reliability is weakened by the circular sentiment-polarity metric, the absence of inter-annotator agreement, and the very small scenario subsets. These issues directly bear on the paper's central claim about LLMs' inability to preserve sentiment polarity during detoxification.","major_comments":[{"comment":"The Sentiment Polarity metric is circular. The three-class Qwen3-32B sentiment classifier is fine-tuned with the neutral class consisting of ToxiRewriteCN's own human-written rewrites, duplicated to a 1:2:1 ratio. Therefore, the 'Neutral↑' percentages in Tables 1–6 measure how closely model outputs resemble the gold rewrites in the eyes of a classifier trained to call those very rewrites 'neutral,' not an independent measure of emotional fidelity. The main comparative claims—for example, that GPT-4o achieves 67.16% neutral while Llama3-8B falls to 43.44%, or that dense models over-sanitize—rest on this metric. Please validate the classifier on held-out human judgments or replace it with an independently trained sentiment classifier, and report the classifier's accuracy and agreement on a held-out test set.","section":"§4.1 and Appendix D"},{"comment":"The scenario-level subsets are very small (emoji n=49, homophone n=39, multi-turn n=34) and no confidence intervals or significance tests are reported. The scenario-level conclusions, such as QwQ-32B's sentence-level detoxification accuracy dropping from 85.71% overall to 24.49% on emoji inputs, or the claim that multi-turn dialogue is the hardest setting, rely on percentages that shift by several points with just a few instances. Please provide exact binomial confidence intervals or bootstrap estimates, and perform significance tests when comparing scenarios or models.","section":"Tables 3, 4, and 6"},{"comment":"The paper reports cross-verification with average Likert scores but provides no inter-annotator agreement statistic, such as Cohen's kappa or Krippendorff's alpha. Without a measure of agreement, the reliability of the human rewrites and toxic span labels—the gold standard for the entire benchmark—is not established. Please report agreement on a shared subset of the data.","section":"§3.4"},{"comment":"Qwen-Max generated the initial coarse rewrites, 482 of which were accepted unchanged into the final reference set, and Qwen-Max is also one of the 17 benchmarked models. This creates a contamination path: Qwen-Max's outputs are, by construction, closer to a portion of the references, which may inflate its fluency and sentiment-polarity scores. The paper should either exclude Qwen-Max from the benchmark or provide an analysis of how its scores differ on accepted versus edited reference items.","section":"§3.3 and Table 1"}],"minor_comments":[{"comment":"The dataset name is rendered inconsistently as both TOXIREWRITE CN and ToxiRewriteCN; please unify the nomenclature.","section":"Throughout"},{"comment":"The sentence 'Reasoning models such as Deepseek-R1 and QwQ-32B reveals an interesting trade-off' contains a subject-verb agreement error; 'reveals' should be 'reveal.'","section":"§4.3"},{"comment":"The reference 'X. X (formerly twitter)' contains a placeholder author name and an incomplete citation; this entry appears unfinished.","section":"References"},{"comment":"The captions state 'Box highlights' the best scores, but the boxes are not visible in the manuscript text; please ensure the final PDF displays them, or describe the best scores in the caption text.","section":"Tables 2–6"}],"recommendation":"major_revision","confidential_remarks":"The central dataset contribution is credible and the human-in-the-loop pipeline is a genuine strength. However, the sentiment-polarity evaluation is circular in a way that directly undermines the paper's headline finding, and the scenario analyses rest on very small samples without statistical support. These issues are fixable with additional validation and reporting, so I recommend major revision rather than rejection. I would also encourage the editor to ensure that the authors address the Qwen-Max contamination point explicitly, as it affects the interpretation of the model benchmark."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know about arXiv:2505.15297: it's a resource paper, and the resource is the contribution. ToxiRewriteCN gives Chinese NLP 1,556 toxic sentences with sentiment-preserving non-toxic rewrites, toxic span labels, and five scenario types (standard, emoji, homophone, single-turn, multi-turn). The six-step human-in-the-loop pipeline is described in detail, and human post-editing plus cross-verification at an average threshold of 4.0/5 is a real strength. This is genuinely new: no prior Chinese toxicity dataset provides sentiment-aligned rewrites.\n\nThe problem is the evaluation. The headline empirical claim—that LLMs struggle to preserve sentiment polarity during detoxification, with specific model rankings—rests on a sentiment classifier that is circular. As Appendix D makes transparent, the Qwen3-32B three-way classifier was trained entirely on ToxiRewriteCN, with the neutral class being the dataset's human rewrites (duplicated 1:2:1). A model output labeled 'neutral' is therefore just 'close to the human rewrites according to a classifier trained on those rewrites.' That is not an independent measure of emotional fidelity. And Qwen-Max generated the initial coarse rewrites, was edited into the references, and is one of the 17 evaluated models. Fine, but you should treat the neutral/polite percentages and the rankings they generate as suggestive, not definitive.\n\nOther soft spots: no inter-annotator agreement is reported, which matters for a dataset claiming careful annotation. The scenario subsets are tiny (emoji n=49, homophone n=39, multi-turn n=34) and percentages swing wildly without confidence intervals. The toxicity classifier is also partly trained on the target dataset, though it at least adds ToxiCN data.\n\nWhat holds up: the dataset itself. The human preference study in Appendix C (79% preference for neutral over polite rewrites) is independent of the circular classifier and supports the core motivation. The paper is honest about its limitations. If the authors add a held-out human evaluation of the sentiment classifier and report agreement statistics, the benchmark claims would be much stronger.\n\nWho benefits: anyone working on Chinese detoxification, controllable rewriting, or moderation systems. It deserves a serious referee. I'd send it to review, ask for the circularity fix and agreement stats, and the resource will likely be reusable regardless.\n\nRecommendation: engage with it, but don't take the model rankings at face value.","headline":"The resource is real and useful; the sentiment-polarity rankings are built on a circular classifier and should not be trusted yet.","tokens_in":18203,"tokens_out":2400,"would_cite":true,"duration_ms":20161,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"New Chinese dataset measures whether LLMs can remove toxicity without flattening the speaker's emotion.","keywords":["Chinese toxic language","detoxification","sentiment polarity","text rewriting","emoji-induced toxicity","homophonic toxicity","dialogue detoxification","LLM evaluation"],"falsifier":"Take a random sample of, say, 200 of the 1,556 toxic inputs, have independent native Chinese speakers write their own sentiment-preserving rewrites and label toxic spans, and then compare the original dataset's labels and the Qwen3-32B classifier outputs against those independent judgments; if agreement is low or the models' relative ranking changes under the new labels, the paper's central performance claims are not stable.","tokens_in":17218,"feed_emoji":"🧹","tokens_out":6366,"duration_ms":50576,"temperature":0.7,"pith_summary":"This paper introduces ToxiRewriteCN, a Chinese-language dataset of 1,556 triplets in which every toxic sentence has a non-toxic rewrite that keeps the original sentiment polarity, plus labeled toxic spans. The dataset covers five scenarios: ordinary offensive sentences, toxicity carried by emojis, toxicity hidden in homophones, single-turn dialogues, and multi-turn dialogues. The authors use it to evaluate 17 large language models on detoxification accuracy, fluency, content preservation, and sentiment polarity. They claim that current models, including commercial and mixture-of-experts systems, can remove obvious vulgar words but frequently over-sanitize emotional tone or fail to resolve covert toxicity, especially in multi-turn conversations. The paper also reports a human preference study in which 79 of 100 annotators preferred sentiment-preserving rewrites over polite rewrites.","feed_headline":"ToxiRewriteCN: LLMs struggle to keep tone while detoxifying Chinese","feed_subtitle":"Dataset of 1,556 toxic Chinese triples shows models over-polish or miss hidden slurs in emoji and dialogue cases.","key_machinery":"The load-bearing object is the ToxiRewriteCN triplet: an original toxic Chinese sentence, a sentiment-aligned non-toxic rewrite, and labeled toxic spans. These triplets are produced by a six-task, human-in-the-loop pipeline: filtering by toxicity category and rewrite suitability, coarse rewriting by an LLM under instructions to preserve emotional tone while changing only toxic components, human correction, and cross-verification by a second annotator on a 5-point scale. The sentiment polarity axis is what makes the dataset novel: rewrites are required to stay in the same emotional register rather than drift into customer-service politeness. The dataset's scenario split, covering standard, emoji-induced, homophonic, single-turn, and multi-turn cases, is the mechanism that exposes where models fail.","core_discovery":"The central claim is that sentiment polarity is not decoration but part of the meaning of toxic language, and that Chinese detoxification can and should be evaluated with reference rewrites that preserve it. Concretely, the paper claims that a dataset built by filtering general offensive language, rewriting with emotional guidance, and human post-editing can serve as a benchmark for this task. On this benchmark, the best closed-source generation models reach roughly 88 percent sentence-level detoxification on standard sentences, but all evaluated models degrade sharply on emoji, homophone, and multi-turn settings, where top models' sentence-level detoxification scores fall below 56 percent. The paper thus positions sentiment-preserving rewriting as a distinct capability that current LLMs have not mastered.","pith_inferences":["Editorial inference: if ToxiRewriteCN is adopted as a benchmark, the emoji, homophone, and multi-turn subsets will likely become the informative axes, because standard sentences already sit near saturation for top models.","Editorial inference: the same polarity-preserving rewrite protocol could be adapted to code-mixed Chinese and to other high-context languages, and the sharp performance drop in multi-turn dialogues suggests discourse-level detoxification needs a separate modeling treatment.","Editorial inference: the observed tension between residual toxicity and over-politeness points toward a controllable emotional-intensity parameter as a natural next system component, though the paper itself does not propose one."],"forward_implications":["A reusable Chinese benchmark now exists for measuring whether a model detoxifies without flattening emotion, with 1,556 reference rewrites and toxic span labels.","On this benchmark, closed-source and large MoE models lead overall, but no evaluated model keeps sentiment polarity consistently; reasoning models reduce polite over-sanitization at the cost of higher residual toxicity.","Emoji-induced and homophone-based toxicity are substantially harder than standard sentences, and multi-turn dialogue is the hardest setting, with top models' sentence-level detoxification dropping below 56 percent.","Human preference data indicate that neutral, sentiment-preserving rewrites are preferred to polite rewrites in 79 of 100 comparisons, supporting the task definition.","The main bottleneck is not producing fluent rewrites but understanding subtle toxic expressions, interpreting context, and keeping emotional tone intact."],"supporting_citations":[{"why":"Supplies the direct toxic sentence candidates that form the largest share of the dataset.","marker":"Lu et al. 2023"},{"why":"Provides COLD, a second source of general toxic sentences used for sentence-level candidates.","marker":"Deng et al. 2022"},{"why":"Provides ToxiCloakCN, the source of emoji-induced and homophonic toxicity examples.","marker":"Xiao et al. 2024"},{"why":"Provides Cdial-bias, a source of single-turn dialogue data.","marker":"Zhou et al. 2022"},{"why":"Provides SWSR, a source of single-turn dialogue data focused on sexism.","marker":"Jiang et al. 2022"},{"why":"Provides SCCD, the source of both single-turn and multi-turn dialogue candidates.","marker":"Yang et al. 2025b"},{"why":"Qwen-Max is used for toxicity revalidation and for the initial coarse rewrites that annotators then corrected.","marker":"Team 2024"},{"why":"Multilingual detoxification study that motivates the Chinese-specific difficulty and the need to filter hate speech before rewriting.","marker":"Dementieva et al. 2025"}],"fun_headline_variants":["LLMs can't detoxify Chinese without losing the tone","Sentiment-preserving detox of Chinese stumps top LLMs","ToxiRewriteCN benchmark shows LLMs miss hidden slurs","Chinese emoji and homophones break LLM detoxifiers","Detoxifying Chinese while keeping sentiment: LLMs fail"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark treats the human-written rewrites and the Qwen3-32B classifier labels as gold standard, but annotator agreement is not reported and the same model family that generated the initial reference rewrites is among the evaluated systems.","fun_headline_variants_meta":{"raw":{"variants":["LLMs can't detoxify Chinese without losing the tone","Sentiment-preserving detox of Chinese stumps top LLMs","ToxiRewriteCN benchmark shows LLMs miss hidden slurs","Chinese emoji and homophones break LLM detoxifiers","Detoxifying Chinese while keeping sentiment: LLMs fail"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000582,"raw_usage":{"total_tokens":2731,"prompt_tokens":930,"completion_tokens":1801,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":1731}},"tokens_in":546,"tokens_out":1801,"duration_ms":11688,"temperature":1.0,"reasoning_tokens":1731,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:19:56.344386+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of, say, 200 of the 1,556 toxic inputs, have independent native Chinese speakers write their own sentiment-preserving rewrites and label toxic spans, and then compare the original dataset's labels and the Qwen3-32B classifier outputs against those independent judgments; if agreement is low or the models' relative ranking changes under the new labels, the paper's central performance claims are not stable.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Multilingual detoxification study that motivates the Chinese-specific difficulty and the need to filter hate speech before rewriting."}],"review_version":1}