Pith. sign in

REVIEW 3 major objections 5 minor 20 references

Span-guided detoxification is not uniformly better: humans prefer it mainly when broad rewrites over-modify, and prefer unguided rewrites when local edits leave residual harm.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-30 20:50 UTC pith:NBKWUINT

load-bearing objection Dense human evidence of a real residual-harm vs over-modification trade-off under one frozen generator; strata are heuristic but the paper already disclaims routing claims. the 3 major comments →

arxiv 2607.26795 v1 pith:NBKWUINT submitted 2026-07-29 cs.CL

When Does Span-Guided Detoxification Help? Human Preferences and Evaluator Diagnostics in a Controlled Comparison

Kyungwon Park This is my paper
classification cs.CL
keywords text detoxificationspan-guided rewritinghuman preference evaluationtoxicity reductionmeaning preservationLLM judgesHateXplainevaluation diagnostics
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks when localizing detoxification edits to harmful spans actually helps, versus letting a model rewrite the whole sentence. In a blinded human study of 60 English inputs with one fixed generator and 1,860 judgments, neither strategy wins overall. Span-guided outputs are preferred when they keep the original stance and avoid unnecessary change; unguided outputs are preferred when broader rewriting removes harmful intent more completely. That split is sharp across the study’s own strong and mild strata: the two strategies are nearly tied on strong items, while unguided dominates on mild items. Annotator rationales name the complementary failures—residual harm after narrow edits, over-modification after broad ones. Automatic toxicity–similarity scores and two general-purpose LLM judges recover only the aggregate lean toward unguided rewriting, not the same stratified pattern. The paper therefore argues for evaluating mitigation sufficiency and meaning preservation separately, and for reporting residual harm and over-modification alongside aggregate scores, rather than treating a single scalar or judge as a substitute for human preference.

Core claim

Under a fixed single-generator, dense blinded human evaluation, span-guided and unguided detoxification trade off rather than one dominating: item-level pluralities are nearly balanced in the study-defined strong stratum but strongly favor unguided rewriting in the mild stratum, with rationales attributing the difference to residual harm versus over-modification. Automatic scalarizations and two LLM judges recover parts of the overall unguided lean but not an analogous stratified contrast.

What carries the argument

A controlled bundled contrast between two prompting strategies—span-guided localized editing versus unguided whole-sentence rewriting—read out through item-level human plurality preference and primary rationales, stratified by the study’s operational strong/mild groupings, with automatic toxicity–similarity scalarizations and LLM judges treated only as diagnostics.

Load-bearing premise

The study’s strong and mild labels are treated as analytically meaningful groupings for strategy preference even though they are heuristic operational assignments, not an independently validated severity scale.

What would settle it

Replicate the same blinded human protocol on a larger, independently stratified set (or with predicted rather than provided spans and multiple generators): if guided and unguided pluralities no longer split by stratum, or if residual-harm versus over-modification rationales no longer track that split, the central trade-off claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Detoxification evaluation should report residual harm and over-modification as separate failure modes, not only aggregate toxicity and similarity.
  • A single weighted toxicity–similarity score can flip strategy rankings with the weight and should not be treated as a human preference proxy.
  • General-purpose LLM judges that match overall human lean may still miss stratified preference structure on the same pairs.
  • Adaptive routing between local and global rewriting is motivated as a hypothesis but is not justified from these strata or labels alone.
  • Oracle span-guided results describe behavior when harmful spans are given; systems that predict spans may shift the residual-harm risk.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Moderation and authoring tools that always force local edits may systematically under-mitigate milder or less explicit harm, while always-global rewrites may quietly rewrite speaker stance.
  • If residual harm and over-modification are logged as first-class outcomes, product dashboards could surface when a detox pipeline is failing by under-editing versus over-sanitizing.
  • The gap between human stratified preference and LLM-judge flatness suggests judge prompts may need explicit residual-harm and over-modification checks, not only overall appropriateness.
  • A natural next measurement is whether predicted-span guided rewriting closes or widens the mild-stratum preference gap relative to gold spans.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper reports a controlled, exploratory comparison of span-guided versus unguided detoxification under a fixed generator (Qwen2.5-7B-Instruct), using a 60-item mixed-source English set (30 manually curated + 30 HateXplain) and dense blinded human evaluation (31 annotators × 60 pairs = 1,860 judgments). Item-level pluralities favor unguided overall (39/60) but are nearly balanced in the study-defined strong stratum (14 guided vs 13 unguided) and dominated by unguided in the mild stratum (26 vs 1), with a large association (Fisher p = 1.29×10⁻⁴, Cohen’s h = 1.22). Rationales attribute the split to complementary failures—residual harm after localized edits versus over-modification after broader rewrites. Toxicity–similarity scalarizations, a five-generator proxy study, and two LLM judges recover aggregate unguided lean but not the human stratified pattern. The authors explicitly disclaim universal severity taxonomies and routing rules, and argue for evaluation protocols that separate mitigation sufficiency from meaning preservation and report both residual harm and over-modification.

Significance. If the scoped result holds, the paper makes a useful methodological contribution to detoxification evaluation: it shows that editing-scope strategies expose distinct, human-visible failure modes that aggregate toxicity–similarity scores and off-the-shelf LLM judges do not recover in stratified form. Strengths include dense full-panel annotation, blinded A/B presentation, explicit non-decisive handling, association statistics with bootstrap CI, judgment-level rationale distributions, and a clear separation of primary human evidence from secondary diagnostics. The work is carefully hedged and does not overclaim a routing policy. That combination—trade-off evidence plus evaluator diagnostics—is of practical interest for moderation and safety-editing pipelines even without a new model or detector.

major comments (3)
  1. [§3.6, §4.1] §3.6 and §4.1 report item-level pluralities and a 2×2 Fisher test but no inter-annotator agreement (e.g., Fleiss’ κ, Krippendorff’s α, or pairwise agreement) on Q1 preference or Q2 rationale. With 31 raters on every item, IAA is feasible and load-bearing for trusting plurality labels and the strong/mild contrast; please add overall and stratum-wise agreement, and note how ties/non-decisive rates relate to disagreement.
  2. [§3.4, Table 1, §4.1] Table 1 / §3.4–§4.1: the study-defined strata confound labeling method and source (HateXplain hate→strong, offensive→mild; author-assigned curated items), and the 60-item selection seed is undocumented. The central stratified claim needs a source-disaggregated breakdown (curated vs HateXplain within each stratum, and preferably preference by native HateXplain label on the 30-item subset). Without this, it remains unclear whether the 14/13 vs 1/26 pattern tracks the operational stratum or item-construction differences.
  3. [§3.1, Abstract, §5] §3.1 and §3.5 correctly call the contrast bundled (spans + locality instruction), yet RQ1 and the title foreground “span-guided.” Please keep results language aligned with the bundled implementation everywhere claims are stated (including Abstract and §5), or add a small ablation/discussion clarifying that the human pattern cannot be attributed to span information alone.
minor comments (5)
  1. [§1] §1: line-break artifact yields “abun-dledprompt”; fix to “bundled prompt.”
  2. [Table 3] Table 3: the Q2 “No clear difference” row after a decisive Q1 choice is easy to confuse with the 279 Q1 non-choices; a footnote or renamed category would help.
  3. [§3.7] §3.7: Perspective failure→0.5 and default RoBERTa-large BERTScore settings are documented; state briefly whether any sensitivity to these defaults was checked.
  4. [Figure 3, Appendix C] Figure 3 caption already warns against reading a sampling distribution; consider moving or shortening it if space is tight, since it is illustrative only.
  5. [Appendix E] Appendix E examples are helpful; labeling each with source (curated vs HateXplain) would aid interpretation of the stratum analyses.

Circularity Check

0 steps flagged

No circularity: empirical preference study with measured outcomes, not a derivation that redefines its target.

full rationale

This paper reports a controlled human preference comparison of two prompting strategies (span-guided vs unguided detoxification) under a fixed generator, plus secondary automatic and LLM-judge diagnostics. The central results—item-level pluralities (Table 1), Fisher association with study-defined strata (Table 2), rationale distributions (Table 3), toxicity–similarity geometry (Tables 4–5), and LLM-judge agreement (Table 6)—are measured quantities from blinded annotation and external scorers, not quantities forced by construction from fitted parameters or self-defined identities. Q_λ is explicitly labeled a sensitivity diagnostic, not a human utility model or deployment objective (§3.2, §4.3). The strong/mild strata are heuristic operational labels used only to organize the sample; the paper repeatedly disclaims causal severity interpretation and routing rules (§3.3–3.4, §5.2, §5.4, Conclusion). There is no uniqueness theorem, no load-bearing self-citation chain, no ansatz smuggled in as derivation, and no renaming of a known closed-form result as a first-principles prediction. The work is self-contained against its own human and automatic benchmarks within the stated scope.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 2 invented entities

The load-bearing claim rests on empirical protocol choices rather than a mathematical derivation. Key dependencies are human preference as the primary criterion, heuristic strong/mild strata as analysis slices, oracle-provided harmful spans, a bundled guided prompt (span info + locality), fixed generator/decoding choices, and off-the-shelf toxicity/similarity/LLM judges as secondary diagnostics.

free parameters (3)
  • Q_λ toxicity weight λ = 0.6 (main table)
    Main multi-generator proxy table uses λ=0.6; Appendix C shows guided proxy-win rate swings from 86.7% at λ=0 to 23.3% at λ=1 on the 30-item set, so ranking conclusions depend on this hand-chosen mixture weight.
  • Human-study decoding hyperparameters = temp 0.7, top_p 0.9, max_new_tokens 80
    temperature=0.7, top_p=0.9, max_new_tokens=80, do_sample=True for Qwen2.5-7B-Instruct; one frozen sample per condition makes preference outcomes dependent on this stochastic draw.
  • 60-item corpus construction and stratum assignment = seed undocumented; A/B seed 42
    30 curated + 30 HateXplain items, balanced 30/30 strong/mild; HateXplain mapping hate→strong and offensive→mild; curated items author-assigned; selection seed not documented. These choices define the association being measured.
axioms (5)
  • domain assumption Blinded human pairwise preference is the primary standard for comparing detoxification strategies.
    Stated throughout Sections 1, 3.6, and 5; automatic metrics and LLM judges are explicitly secondary diagnostics.
  • domain assumption Provided item-level harmful spans (gold HateXplain rationales or curated spans) are an acceptable oracle setup for studying localized rewriting behavior.
    Section 3.4 and Ethics/Limitations; removes detector error but makes the guided condition non-deployable as tested.
  • ad hoc to paper Study-defined strong/mild labels are usable operational strata for association testing even though they are not a validated severity taxonomy.
    Sections 3.3–3.4 and 3.9 define c(x)∈{STRONG,MILD} and run Fisher tests on that partition while disclaiming universality.
  • domain assumption Perspective API TOXICITY and BERTScore F1 are adequate diagnostic proxies for mitigation and similarity geometry.
    Section 3.7; paper itself notes they are imperfect and can miss pragmatic reversal or encode bias.
  • ad hoc to paper The bundled guided prompt (span information plus locality instruction) is a legitimate implementation of span-guided detoxification for behavioral comparison.
    Section 1 and 3.1 explicitly say g is the full prompt strategy, not an isolated causal effect of span availability.
invented entities (2)
  • Study-defined strong/mild operational strata no independent evidence
    purpose: Partition the 60-item human set to test association between input grouping and guided vs unguided plurality preference.
    Not an external validated severity scale; constructed in-paper from HateXplain native labels plus author assignment on curated items.
  • Item-level plurality preference outcome (guided / unguided / non-decisive) no independent evidence
    purpose: Aggregate 31 annotator votes into a single strategy label per input for stratified tests.
    Defined by the paper’s counting rule in Section 3.6; standard aggregation device, but the exact plurality/non-decisive thresholding is study-specific.

pith-pipeline@v1.2.0-daily-grok45 · 17491 in / 3758 out tokens · 80302 ms · 2026-07-30T20:50:21.809782+00:00 · methodology

0 comments
read the original abstract

Span-guided rewriting aims to preserve meaning by localizing edits to annotated harmful spans, but the same constraint can leave harmful intent insufficiently mitigated. We present a controlled exploratory comparison of span-guided and unguided detoxification on a mixed-source English evaluation set comprising manually curated inputs and HateXplain test items. We conduct a dense blinded human evaluation under a fixed single-generator setting. Human preferences reveal a trade-off rather than a uniformly superior rewriting strategy. Span-guided outputs are favored when localized editing preserves the original stance and avoids unnecessary modification, whereas unguided outputs are favored when broader rewriting achieves more complete mitigation. This contrast varies substantially across the study-defined strata: the two strategies are competitive in the strong stratum, while unguided rewriting is clearly preferred in the mild stratum. Rationale annotations trace this difference to complementary failure risks: residual harm after localized editing and over-modification after broader rewriting. We treat automatic evaluation as a diagnostic rather than a substitute for human judgment. Toxicity-similarity scalarizations, a multi-generator analysis, and two general-purpose LLM judges reproduce parts of the aggregate tendency but do not yield an analogous stratified contrast. These setting-specific findings do not establish a severity-based routing rule. Instead, they motivate evaluation protocols that assess mitigation sufficiency and meaning preservation separately and report both residual harm and over-modification alongside aggregate scores.

Figures

Figures reproduced from arXiv: 2607.26795 by Kyungwon Park.

Figure 1
Figure 1. Figure 1: From uniform scalar ranking to controlled strategy comparison. A common scalar-ranking setup selects [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Item-level plurality preference by study [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Guided proxy-win rates across one sequence [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

20 extracted references · 2 linked inside Pith

  1. [1]

    IEEE Transactions on Artificial Intelligence , year =

    A Review of Text Style Transfer Using Deep Learning , author =. IEEE Transactions on Artificial Intelligence , year =

  2. [2]

    Computational Linguistics , volume =

    Deep Learning for Text Style Transfer: A Survey , author =. Computational Linguistics , volume =. 2022 , doi =

    2022

  3. [3]

    2022 , publisher =

    Logacheva, Varvara and Dementieva, Daryna and Ustyantsev, Sergey and Moskovskiy, Daniil and Dale, David and Krotova, Irina and Semenov, Nikita and Panchenko, Alexander , booktitle =. 2022 , publisher =

    2022

  4. [4]

    2023 , url =

    Agarwal, Vibhor and Chen, Yu and Sastry, Nishanth , journal =. 2023 , url =

    2023

  5. [5]

    2021 , publisher =

    Krause, Ben and Gotmare, Akhilesh Deepak and McCann, Bryan and Naik, Nitish Shirish and Keskar, Caiming Xiong and Joty, Shafiq and Socher, Richard , booktitle =. 2021 , publisher =

    2021

  6. [6]

    and Choi, Yejin , booktitle =

    Liu, Alisa and Sap, Maarten and Lu, Ximing and Swayamdipta, Swabha and Bhagavatula, Chandra and Smith, Noah A. and Choi, Yejin , booktitle =. 2021 , publisher =

    2021

  7. [7]

    Khondaker, Md Tawkat Islam and Abdul-Mageed, Muhammad and Lakshmanan, Laks V. S. , booktitle =. 2024 , address =

    2024

  8. [8]

    2024 , address =

    Lee, Beomseok and Kim, Hyunwoo and Kim, Keon and Choi, Yong Suk , booktitle =. 2024 , address =

    2024

  9. [9]

    Findings of the Association for Computational Linguistics: ACL 2023 , pages =

    Correction of Errors in Preference Ratings from Automated Metrics for Text Generation , author =. Findings of the Association for Computational Linguistics: ACL 2023 , pages =. 2023 , address =

    2023

  10. [10]

    Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =

    Can Large Language Models Be an Alternative to Human Evaluations? , author =. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages =. 2023 , address =

    2023

  11. [11]

    Wang, Jiaan and Liang, Yunlong and Meng, Fandong and Sun, Zengkui and Shi, Haoxiang and Li, Zhixu and Xu, Jinan and Qu, Jianfeng and Zhou, Jie , booktitle =. Is. 2023 , address =

    2023

  12. [12]

    , booktitle =

    Gehman, Samuel and Gururangan, Suchin and Sap, Maarten and Choi, Yejin and Smith, Noah A. , booktitle =. 2020 , publisher =

    2020

  13. [13]

    2021 , publisher =

    Mathew, Binny and Saha, Prabhat Kumar and Tharad, Hardik and Rajgaria, Shubham and Singhania, Prerna and Maity, Subham and Goyal, Punyajoy and Mukherjee, Animesh , booktitle =. 2021 , publisher =

    2021

  14. [14]

    Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021) , pages =

    Pavlopoulos, John and Sorensen, Jeffrey and Laugier, L. Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021) , pages =. 2021 , publisher =

    2021

  15. [15]

    arXiv preprint arXiv:2403.19836 , year =

    Implicit Hate: Measuring the Unmeasurable? , author =. arXiv preprint arXiv:2403.19836 , year =

    Pith/arXiv arXiv

  16. [16]

    Proceedings of the First Workshop on Cross-Cultural Considerations in NLP , pages =

    Hate Speech Classifiers Are Culturally Insensitive , author =. Proceedings of the First Workshop on Cross-Cultural Considerations in NLP , pages =. 2023 , address =

    2023

  17. [17]

    and Tay, Yi and Sorensen, Jeffrey and Gupta, Jai and Metzler, Donald and Vasserman, Lucy , journal =

    Lees, Alyssa and Tran, Vinh Q. and Tay, Yi and Sorensen, Jeffrey and Gupta, Jai and Metzler, Donald and Vasserman, Lucy , journal =. A New Generation of Perspective. 2022 , url =

    2022

  18. [18]

    and Artzi, Yoav , booktitle =

    Zhang, Tianyi and Kishore, Varsha and Wu, Felix and Weinberger, Kilian Q. and Artzi, Yoav , booktitle =. 2020 , url =

    2020

  19. [20]

    2024 , howpublished =

    Hello. 2024 , howpublished =

    2024

  20. [21]

    arXiv preprint arXiv:2412.15115 , year =

    Qwen2.5 Technical Report , author =. arXiv preprint arXiv:2412.15115 , year =

    Pith/arXiv arXiv