Pith. sign in

REVIEW 2 major objections 6 minor 33 references

Choosing Where and How to Moderate: End-to-End Trade-offs in Filter Placement and Response Rewriting

T0 review · 2 major / 6 minor · reviewed 2026-08-01 · deepseek-v4-flash

Pith's one-line read The paper argues that content-moderation placement and action should be chosen against each deployment's own safety ceiling and latency budget, not by a universal rule, and it supports this with end-to-end metrics that measure final custome

desk verdict A careful, honest study of moderation placement and rewriting: the internal human-labelled results carry the paper, while the public LLM-graded corroboration is the main weak link. read the letter →

arxiv 2607.26200 v1 pith:6BIVINTI submitted 2026-07-28 cs.CL

classification cs.CL
keywords contentmoderationfilterplacementresponserewritingend-to-endevaluationUsefulnessHarmfulExposureguardrailtrade-offsLLMsafety
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that content-moderation classifiers should be judged by final outcomes—whether the user sees a safe, relevant answer—not by classifier accuracy. Using two metrics, Usefulness and Harmful Exposure, it compares placing the filter on the user input, on the model response, or on both. On its internal benchmark and a public chat benchmark, checking only the response was the most useful filter-only placement, while checking both input and response exposed harmful content least often. Replacing response-side blocking with one rewrite attempt recovered almost all blocked turns and produced the same observed number of harmful exposures as blocking alone, although the authors stress this equality is not an equivalence result. The paper concludes that moderation placement and action should be chosen against each deployment's safety ceiling and latency budget rather than by a universal rule.

What carries the argument

The central mechanism is the decision pipeline over final show-or-block outcomes, scored by Usefulness (shown safe relevant responses divided by total turns) and Harmful Exposure (shown harmful responses divided by total turns), with end-to-end Block Rate, end-to-end False Positive Rate, and latency as diagnostics. Rewriting is a one-shot recovery action: a flagged response is rewritten, re-screened by the same filter, and shown only if it passes; otherwise it is hard-blocked. A lightweight probe-based router selects rewrite-or-refuse and domain before regeneration, cutting conditional rewrite time from 13.8 seconds to 0.47 seconds on the internal stack while preserving comparable outcomes.

What would settle it

Have human annotators label all rewrites that passed the public re-screening (202 shown outputs). If any is harmful or irrelevant under the same rubric, the reported equality of 35 harmful exposures between response-only blocking and response + rewrite would no longer hold. A second check: re-run the placement comparison with a different aligned generator; if response-only no longer gives the highest filter-only Usefulness, the ordering is not transferable.

Watch

Extended reading notes

Core claim

The discovery is that filter placement and follow-up action jointly determine outcomes that component accuracy cannot express. Measured end-to-end, response-only filtering composed with an aligned generator blocks far fewer useful turns than input-side filtering, because the generator already handles many risky inputs safely; input+response filtering gives the lowest harmful exposure at the cost of roughly twice the blocking; and response-side rewriting converts most would-be blocks into shown safe answers with the same observed harmful-exposure count as blocking, for the tested operating points. The paper frames this as a constrained comparison, not a universal recipe.

Load-bearing premise

The load-bearing premise is that the LLM grader's toxicity rubric, validated only on prompts, correctly labels generated responses and optimized rewrites in the public evaluation; if those labels misjudge harmful rewrites as safe, the public Harmful Exposure and Usefulness numbers for the rewrite configuration would change.

Editorial extensions

If this is right

  • Response-side filtering is the highest-usefulness filter-only choice for aligned chat generators; input-side filtering over-blocks safe prompts in these settings.
  • Input + response filtering is the only placement that meets the tightest harmful-exposure ceilings, so deployments with strict safety budgets should accept its usefulness cost.
  • Selective rewriting can recover most blocked traffic: the selected rewrite configuration cut end-to-end Block Rate from 9.60% to 0.24% internally while showing the same 30 harmful turns as response-only blocking.
  • Probe-based routing makes rewriting latency-feasible (roughly 29x faster than LLM routing in the internal conditional timing), though user-visible latency must be re-measured on the target stack.
  • Rewrites should aim for calibrated specificity: generalizing trigger words preserves usefulness, but sensitive domains may lose safety-relevant support information.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the placement ordering is explained by the generator's own alignment, the same comparison with a weaker or unaligned generator could shift the frontier toward input-side checks.
  • The equal observed exposure count for rewrite versus blocking invites a formal equivalence test with a prespecified margin; the current sample cannot show that rewriting is as safe as blocking.
  • The public LLM-grader labels are the one transfer that was not human-validated on rewritten outputs; human-reviewing the released rewrites would settle whether the reported public Harmful Exposure and Usefulness figures hold.
  • The framework extends naturally to multi-turn or agentic systems, but response-side checks must then be supplemented by input- or action-side checks where side effects execute before a flag can block.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper proposes an end-to-end evaluation framework for content-moderation deployment, replacing component-level classifier accuracy with two customer-outcome metrics: Usefulness (shown, non-harmful, relevant responses / N) and Harmful Exposure (shown harmful responses / N). Four configurations are compared on a human-labelled internal benchmark (N=1,250) and a public ToxicChat setting (N=5,654): Input only, Response only, Input+response hard blocking, and Response+rewrite. The main findings are that Response only achieves the highest filter-only Usefulness in both settings, Input+response achieves the lowest Harmful Exposure, and replacing Response-only blocking with Response+rewrite recovers most blocked traffic with the same observed Harmful Exposure count for the selected configuration (explicitly not an equivalence result). Probe routing reduces conditional rewrite-stage latency substantially relative to LLM routing. A focused qualitative audit shows both successful safe redirections and cases where sensitive-domain rewrites omit support information.

Significance. If the results hold, the paper makes a useful contribution by shifting moderation evaluation from isolated classifier metrics to deployment-relevant final outcomes. Its strengths include a fully human-labelled internal comparison for the headline placement and rewrite findings, paired bootstrap confidence intervals and exact McNemar tests for differences, explicit non-equivalence caveats, a test-split-only sensitivity analysis for the public filter, and release of code and public artifacts. The framework is practical and the main internal finding—that Response only dominates Input-only and Input+response on Usefulness at the chosen operating points—is credible. The public corroboration is more fragile because its response and rewrite grades come from LLM judges whose rubric was validated only on prompts, not on long generated responses or optimized rewrites. The internal evidence is not affected by this concern, so the central claim is defensible, but the 'both settings' formulation of the abstract is stronger than the public evidence currently supports.

major comments (2)
  1. [Section 4, 'Automatic graders'; Table 3; Table 13; Appendix C] The public ToxicChat results, including the Harmful Exposure counts and the zero-discordance equality between Response only and Response+rewrite, are computed with the toxicity_v10 LLM grader applied to GPT-5 responses (mean 2,901 characters) and to optimized rewrites. The paper states this rubric was validated only on ToxicChat prompts (mean 199 characters) and that the transfer to responses is an approximation. If the grader under-detects harm in long responses or in deliberately generalized rewrites, the public placement ordering and the 'same observed Harmful Exposure count' claim could shift. The internal human-labelled comparison is not affected, which protects the central finding, but the abstract's 'in both settings' and the public rewrite-safety equality currently rest on unvalidated transfer. Please either add a human-validated sample of public responses/rewrites and report gra
  2. [Section 4, 'Public corroboration' and 'Automatic graders'; Table 3; Appendix C, 'Prompt-label stabilization'] The public E2E FP Rate—defined as safe-prompt turns that end in a hard block—uses the LLM-graded toxicity label for the prompt, not ToxicChat's human prompt labels, with a stabilized majority only for the doubly blocked subset. Since the LLM grader has F1 0.856 against human labels on prompts, roughly 14% of prompt-toxicity decisions differ from the human reference. These errors propagate directly into the FP counts used to support the claim that Response only has the lowest E2E FP Rate (1.43% vs. 1.95% for Input only in Table 3). Because E2E FP Rate is a diagnostic rather than a main outcome, this is not fatal, but the paper should either use the available human prompt labels for this metric or provide a sensitivity analysis showing the placement conclusion is unchanged when human labels are used.
minor comments (6)
  1. [Abstract] The abstract says 'yields the same observed Harmful Exposure count as Response only blocking for the selected configuration.' Since the internal count (30 turns) and public count (35 turns) are computed from different label sources, it would be clearer to state which setting is being referenced or to mention both explicitly.
  2. [Table 3] The footnote symbol on 'Response + rewrite†' is not defined in the caption. Please add a footnote defining the selected public configuration.
  3. [Section 4, 'Public corroboration'] The text refers to 'the authors' released fine-tuned T5' but does not give the model identifier or a HuggingFace revision in the body. Appendix O gives some details, but the model card should be cited or linked here for reproducibility.
  4. [Section 2] The phrasing 'Wildflare GuardRail ... already contributes' and 'Constitutional Classifiers ... already contributes' is repetitive. Consider rewording to avoid implying a comparison ranking.
  5. [Figure 2] The configuration labels are placed close together, especially for Input only and Input+response. Direct labeling or a small legend would improve readability.
  6. [Appendix G, Table 8] The column heading 'Route / classify' is not fully spaced and the note about LLM-intent template rows retaining an unused LLM-domain call is important; consider moving it to the table caption for visibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central placement and rewrite comparisons are evaluated with independent harm/relevance labels, and the equal-exposure counts are an observed filter-miss floor rather than a pipeline identity.

full rationale

The derivation chain is not circular. Usefulness and Harmful Exposure are defined from harm and relevance labels rather than from the deployed filter's decisions: 'The filter label drives moderation actions, while the harm and relevance labels provide the evaluation labels against which those actions are scored' (Section 3). The internal headline comparison is fully human-labelled, including all 117 shown rewrites ('All outcomes in the internal comparison in Figure 2a are human-labelled, including harm and relevance for the 117 shown outputs from the selected rewrite configuration'), so the placement ordering and rewrite-recovery gain are not restatements of filter outputs. The public half is explicitly a corroboration using LLM graders, and the paper flags the transfer limitation ('we also apply this prompt-validated rubric to generated responses as an approximation, a transfer that was not human-validated on responses or optimized rewrites'); this is a validity caveat, not a definitional reduction. The observed equality between Response only and Response+rewrite Harmful Exposure is not built into the pipeline: it is explained by the fact that harmful responses were never flagged ('Thirty of 46 human-labelled harmful responses are never flagged at the Medium threshold... leaving those 30 original responses as its observed 2.40% exposure floor'), so rewriting had no harmful cases to alter. The GEPA-optimized rewrite prompts do target T5 pass and relevance in their reward, but the paper reports group-disjoint held-out evaluation ('constructing group-disjoint optimization and held-out splits') and a fully human-labelled internal selected arm, so the recovery result is not merely the optimization objective restated. No load-bearing self-citation or imported uniqueness theorem appears; the acknowledged grader-sensitivity limitation affects the scope of the public corroboration, not the internal human-labelled headline.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities; it uses existing filters, generators, probes, and prompt-optimization methods. The central claims rest on empirical measurements, so the main ledger entries are the hand-chosen thresholds, trained routing probes, and filter-specific prompt packs that define the selected configurations, plus the labeling assumptions that ground the outcome metrics.

free parameters (5)
  • Harm threshold (Medium-or-above) = Any category labelled Medium or High is treated as harmful
    Hand-selected cutoff applied to both filter decisions and human harm labels in the internal benchmark; sensitivity is checked across Low/Medium/High in Appendix N, but the headline results use this single threshold.
  • Relevance threshold = Relevance scores 2 and 3 count as useful; score 1 does not
    Defines the Usefulness numerator. This is a hand-chosen scoring boundary on the paper's three-point relevance scale and directly affects all Usefulness values.
  • Routing probe weights (intent head + 8 domain heads) = Trained attention-pooling probes on frozen Qwen3-4B layer-18 residuals; public weights ~54 KB
    The selected Response + rewrite configuration uses these probes for routing. They are trained on routing labels and are part of the deployed pipeline that produces the latency and rewrite-outcome claims.
  • Per-domain calibration parameters = Learned per-domain temperature and bias before domain argmax
    Applied in the public router before choosing a domain head; these are fitted numbers that affect routing behavior and, indirectly, rewrite prompts and outcomes.
  • GEPA-optimized rewrite and refusal prompt packs = Optimized prompt text for each target filter (Azure and T5)
    The rewrite/refusal prompts are optimized against a composite reward that includes the target filter pass rate. The selected rewrite configuration is only as good as these tuned prompts, and the prompts are filter-specific.
assumptions (5)
  • domain assumption LLM graders transfer to generated responses and optimized rewrites without human validation
    Public ToxicChat outcome metrics use toxicity_v10 and relevance_v01 grades; the paper states this transfer 'was not human-validated on responses or optimized rewrites' (Section 4, Automatic graders).
  • domain assumption Human labels on the internal benchmark are accurate enough to serve as ground truth
    Only 571 of 1,250 internal records were double-labelled; the remaining records were labelled by one annotator after applying refined guidance. The paper reports adjudication for disagreements but does not report inter-annotator agreement statistics (Section 4).
  • domain assumption The evaluated aligned generators already decline or safely handle most harmful inputs
    The authors use this to explain why Response only has the highest filter-only Usefulness (Section 5). The finding is conditioned on modern aligned generators and may not transfer to weaker or less aligned models.
  • domain assumption Single-turn English chat framing captures the relevant deployment space
    The paper limits its framework and experiments to single-turn English chat and explicitly notes that multi-turn, agentic, multilingual, or other-taxonomy systems may change the ordering (Limitations).
  • standard math Standard binomial and paired-test assumptions hold for the outcome rates
    Wilson score intervals, percentile bootstrap resampling at the row level, and exact McNemar tests assume rows are exchangeable draws; the prompts are hand-authored or drawn from ToxicChat, not necessarily i.i.d. from a production distribution, so inference is descriptive of the benchmark rather than of production prevalence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Choosing Where and How to Moderate: End-to-End Trade-offs in Filter Placement and Response Rewriting." pith.science (2026). https://pith.science/paper/6BIVINTI

@misc{pith2026260726200,
  author       = {Pith},
  title        = {Pith review of: Choosing Where and How to Moderate: End-to-End Trade-offs in Filter Placement and Response Rewriting},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6BIVINTI}},
  note         = {Machine review of arXiv:2607.26200}
}
read the original abstract

Content-moderation classifiers are usually evaluated in isolation, but deployment requires choosing where to intervene and what follows a flag. We evaluate these choices using two end-to-end customer-outcome metrics rather than component accuracy: Usefulness, the fraction of turns with a shown, non-harmful, relevant response, and Harmful Exposure, the fraction with a shown harmful response. Latency and error rates are diagnostics. We compare Input only, Response only, and Input + response hard blocking on a human-labelled product benchmark and public ToxicChat evaluation. At the evaluated operating points, Response only achieves the highest filter-only Usefulness in both settings, while Input + response achieves lower Harmful Exposure. Replacing Response only blocking with Response + rewrite recovers most blocked traffic and yields the same observed Harmful Exposure count as Response only blocking for the selected configuration; this equality is not an equivalence result. Probe routing substantially reduces conditional route-and-generation time relative to LLM routing at comparable measured outcomes. A focused output review shows how rewrites balance filter passage with usefulness by generalizing triggering language while retaining benign intent and safe redirection; some sensitive-domain outputs nevertheless omit potentially safety-relevant support information. These results support comparing moderation configurations under deployment-specific safety and latency constraints rather than applying a universal placement rule. Code and public artifacts are available at https://github.com/microsoft/mod-frontier

Figures

Figures reproduced from arXiv: 2607.26200 by the authors.

Figure 1
Figure 1. The four moderation configurations. Green paths show responses; red paths block or, in (d), trigger one [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Usefulness–Harmful Exposure trade-offs; higher and further left is preferred. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Complete decision paths for the four configurations. Each path terminates in a response shown to the [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Rewrite designs after response-side filtering. In the internal Azure setting, the filter produces a block [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 5 linked inside Pith

  1. [1]

    2023 , pages =

    Lin, Zi and Wang, Zihan and Tong, Yongqi and Wang, Yangkun and Guo, Yuxin and Wang, Yujia and Shang, Jingbo , booktitle =. 2023 , pages =

  2. [2]

    Inan, Hakan and Upasani, Kartikeya and Chi, Jianfeng and Rungta, Rashi and Iyer, Krithika and Mao, Yuning and Tontchev, Michael and Hu, Qing and Fuller, Brian and Testuggine, Davide and Khabsa, Madian , journal =

  3. [3]

    Journal of Machine Learning Research , volume =

    Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer , author =. Journal of Machine Learning Research , volume =

  4. [4]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume =

    A Holistic Approach to Undesired Content Detection in the Real World , author =. Proceedings of the AAAI Conference on Artificial Intelligence , volume =

  5. [5]

    2024 , note =

    Harm Categories in. 2024 , note =

  6. [6]

    2024 , note =

    Quickstart: Analyze Text Content with. 2024 , note =

  7. [7]

    , booktitle =

    Gehman, Samuel and Gururangan, Suchin and Sap, Maarten and Choi, Yejin and Smith, Noah A. , booktitle =. 2020 , pages =

  8. [8]

    and Hatfield-Dodds, Zac and Mann, Ben and Amodei, Dario and Joseph, Nicholas and McCandlish, Sam and Brown, Tom and Kaplan, Jared , journal =

    Bai, Yuntao and Kadavath, Saurav and Kundu, Sandipan and Askell, Amanda and Kernion, Jackson and Jones, Andy and Chen, Anna and Goldie, Anna and Mirhoseini, Azalia and McKinnon, Cameron and Chen, Carol and Olsson, Catherine and Olah, Christopher and Hernandez, Danny and Drain, Dawn and Ganguli, Deep and Li, Dustin and Tran-Johnson, Eli and Perez, Ethan an...

Show all 33 references
  1. [9]

    and Moazam, Hanna and Miller, Heather and Zaharia, Matei and Potts, Christopher , booktitle =

    Khattab, Omar and Singhvi, Arnav and Maheshwari, Paridhi and Zhang, Zhiyuan and Santhanam, Keshav and Vardhamanan, Sri and Haq, Saiful and Sharma, Ashutosh and Joshi, Thomas T. and Moazam, Hanna and Miller, Heather and Zaharia, Matei and Potts, Christopher , booktitle =

  2. [10]

    and Tan, Shangyin and Soylu, Dilara and Ziems, Noah and Khare, Rishi and Opsahl-Ong, Krista and Singhvi, Arnav and Shandilya, Herumb and Ryan, Michael J

    Agrawal, Lakshya A. and Tan, Shangyin and Soylu, Dilara and Ziems, Noah and Khare, Rishi and Opsahl-Ong, Krista and Singhvi, Arnav and Shandilya, Herumb and Ryan, Michael J. and Jiang, Meng and Potts, Christopher and Sen, Koushik and Dimakis, Alexandros G. and Stoica, Ion and ...

  3. [11]

    arXiv preprint arXiv:2505.09388 , year =

  4. [12]

    2025 , howpublished =

  5. [13]

    Han, Seungju and Rao, Kavel and Ettinger, Allyson and Jiang, Liwei and Lin, Bill Yuchen and Lambert, Nathan and Choi, Yejin and Dziri, Nouha , booktitle =

  6. [14]

    Ghosh, Shaona and Varshney, Prasoon and Galinkin, Erick and Parisien, Christopher , journal =

  7. [15]

    Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) , year =

    R. Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) , year =

  8. [16]

    Hartvigsen, Thomas and Gabriel, Saadia and Palangi, Hamid and Sap, Maarten and Ray, Dipankar and Kamar, Ece , booktitle =

  9. [17]

    Advances in Neural Information Processing Systems (NeurIPS) , volume =

    Training Language Models to Follow Instructions with Human Feedback , author =. Advances in Neural Information Processing Systems (NeurIPS) , volume =

  10. [18]

    Dai, Josef and Pan, Xuehai and Sun, Ruiyang and Ji, Jiaming and Xu, Xinbo and Liu, Mickel and Wang, Yizhou and Yang, Yaodong , journal =. Safe

  11. [19]

    Cui, Justin and Chiang, Wei-Lin and Stoica, Ion and Hsieh, Cho-Jui , booktitle =

  12. [20]

    arXiv preprint arXiv:2508.09224 , year =

    From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Training , author =. arXiv preprint arXiv:2508.09224 , year =

  13. [21]

    arXiv preprint arXiv:2501.18837 , year =

    Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming , author =. arXiv preprint arXiv:2501.18837 , year =

  14. [22]

    Zeng, Wenjun and Liu, Yuchi and Mullins, Ryan and Peran, Ludovic and Fernandez, Joe and Harkous, Hamza and Narasimhan, Karthik and Proud, Drew and Kumar, Piyush and Radharapu, Bhaktipriya and Sturman, Olivia and Wahltinez, Oscar , journal =

  15. [23]

    Rebedea, Traian and Dinu, Razvan and Sreedhar, Makesh Narsimhan and Parisien, Christopher and Cohen, Jonathan , booktitle =

  16. [24]

    Bridging the Safety Gap: A Guardrail Pipeline for Trustworthy

    Han, Shanshan and Avestimehr, Salman and He, Chaoyang , journal =. Bridging the Safety Gap: A Guardrail Pipeline for Trustworthy

  17. [25]

    Proceedings of the International Conference on Learning Representations (ICLR) , note =

    Reasoned Safety Alignment: Ensuring Jailbreak Defense via Answer-Then-Check , author =. Proceedings of the International Conference on Learning Representations (ICLR) , note =

  18. [26]

    Ren, Kaixuan and Nakov, Preslav and Naseem, Usman , journal =

  19. [27]

    Zhang, Zhihao and Huang, Liting and Wu, Guanghao and Nakov, Preslav and Ji, Heng and Naseem, Usman , journal =

  20. [28]

    Evaluating Prompt Injection Defenses for Educational

    Anonymous , journal =. Evaluating Prompt Injection Defenses for Educational

  21. [29]

    arXiv preprint arXiv:2605.30021 , year =

    Mitigating Diversity Collapse in Preference-Optimized Language Models , author =. arXiv preprint arXiv:2605.30021 , year =

  22. [30]

    arXiv preprint arXiv:1610.01644 , year =

    Understanding Intermediate Layers Using Linear Classifier Probes , author =. arXiv preprint arXiv:1610.01644 , year =

  23. [31]

    Proceedings of the International Conference on Learning Representations (

    A Structured Self-Attentive Sentence Embedding , author =. Proceedings of the International Conference on Learning Representations (

  24. [32]

    BeaverTails: Towards Improved Safety Alignment of

    Ji, Jiaming and Liu, Mickel and Dai, Josef and Pan, Xuehai and Zhang, Chi and Bian, Ce and Chen, Boyuan and Sun, Ruiyang and Wang, Yizhou and Yang, Yaodong , booktitle =. BeaverTails: Towards Improved Safety Alignment of

  25. [33]

    2025 , howpublished =

    Claude Opus 4.1 Model Card , author =. 2025 , howpublished =

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.