{"id":"56a3c0f0-830d-4b2f-a76c-4a6e5a0b2b4e","arxiv_id":"2507.23319","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"GPT-4o-mini systematically reduces the sensitivity of offensive sentences during paraphrasing, and expert annotators confirm a strong shift from taboo and derogatory language to milder classes.","lead":"Researchers asked GPT-4o-mini to paraphrase thousands of offensive sentences and found that it consistently softened them, shifting taboo language into milder categories. This matters because it quantifies how aligned AI systems quietly sanitize speech even when no moderation is requested, a behavior relevant to content moderation and model transparency.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The load-bearing concern is the undocumented annotation layer: no inter-annotator agreement, no adjudication rule, and 231 missing sentences in Section 3.2, so the confusion matrix and Delta values grounding the central claim are unverified ground truth.","rationale":"I read the paper as a serious empirical study of a real and plausible phenomenon. The asymmetric expert-annotated confusion matrix and the strongly significant Bowker test are the paper's best evidence, and the effect size makes it likely that GPT-4o-mini does reduce sensitive language during paraphrasing. However, the manuscript's own Section 3.2 leaves the measurement layer underdocumented: no inter-annotator agreement, no stated aggregation rule, and 231 unexplained missing sentences. Since all quantitative claims flow through those expert labels, this is the weakest load-bearing assumption. I do not see it as grounds for rejection; the missing reporting is fixable and the effect is large enough to survive plausible noise. But the paper should remain conditional until the annotation protocol is fully documented and the key matrix is re-checked on an agreement-restricted subset. I agree with the reader's weakest assumption and would not move the verdict. A secondary concern, the absence of a control condition to attribute the shift specifically to alignment, is real but less fundamental, because the core empirical observation about GPT-4o-mini's behavior does not depend on that causal attribution.","tokens_in":11501,"tokens_out":9481,"duration_ms":104173,"concrete_test":"Reanalyze the 2269 annotated sentences by computing per-sentence Cohen's kappa (or Krippendorff's alpha) on the two independent annotations; report how disagreements were adjudicated; audit the 231 missing sampled sentences. Then recompute the Figure 2 confusion matrix and Bowker's statistic (chi-squared = 138.255, df = 6) twice: once restricted to pairs where both annotators agree, and once with disagreements assigned to a separate uncertain row and column. If the direction and rough magnitude of the shift survive both reanalyses, such as majority Taboo-to-Informal and Derogatory-to-less-sensitive with p < 0.0001, the concern is resolved; if the shift weakens or reverses, the central claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 says each sentence received two independent annotations, but the paper never reports inter-annotator agreement, never states how disagreements were resolved into a final label, and leaves 231 of the 2500 sampled sentences unaccounted for. Every downstream result, including Figure 2's confusion matrix, Table 1's Delta values, and Table 2's classifier evaluation, treats these labels as ground truth. If the two annotators frequently disagree and the aggregation rule correlates with whether a sentence is original or paraphrased, for example by defaulting toward the less sensitive category for the more neutral-sounding paraphrase, the pronounced Taboo-to-Informal (51.6%) and Derogatory-to-Formal/Polite-or-Informal (58.9%) cells could be inflated by annotation bias rather than reflecting GPT-4o-mini's behavior. This is load-bearing because the central claim is exactly that the model systematically shifts sensitivity; an unvalidated measurement layer cannot support that claim. The size of the Bowker statistic makes this a concern about confidence rather than automatic falsity, but it is a concrete and fixable gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether GPT-4o-mini implicitly moderates sensitive content when asked to perform a neutral paraphrase task. The authors collect 23,347 sentences containing sensitive expressions, paraphrase each three times, and then have 10 linguist experts annotate a subset of 1,250 original/paraphrase pairs into four sensitivity categories (Formal/Polite, Informal, Derogatory, Taboo). They report a confusion matrix showing that paraphrases tend to move from Taboo and Derogatory toward less sensitive categories, supported by Bowker's test of symmetry (χ²=138.255, df=6, p<0.0001), and compute a per-class average sensitivity shift Δ showing a mean drop of 1.7 levels for Taboo sentences. The paper also evaluates six open-source LLMs and six traditional classifiers on the same annotation task, finding that traditional classifiers such as MLP and Born Classifier match expert labels better than the zero-shot LLMs, with overall F1 around 0.5. The central conclusion is that GPT-4o-mini systematically sanitizes language even without explicit detoxification instructions.","tokens_in":11661,"tokens_out":7246,"duration_ms":81867,"significance":"If the central claim holds, the paper provides a concrete, quantitative demonstration that a proprietary aligned LLM performs implicit content moderation during a task that does not ask for it. This is a valuable contribution to the study of alignment side-effects and complements prior work that focuses on explicit detoxification or refusal behavior. The paper's strengths include a simple and reproducible experimental pipeline, the use of a neutral paraphrase prompt, a statistically significant test for asymmetric shifts, and the inclusion of diverse automated classifiers as a comparison. The dataset itself, even if limited in size, is a useful resource for studying sensitivity perception in LLMs. The main weakness is that the reliability of the expert-annotation ground truth is not documented, and several procedural details are missing, which currently prevents a full endorsement of the quantitative conclusions.","major_comments":[{"comment":"The annotation layer is not sufficiently documented to support the load-bearing ground truth. The paper states that each sentence received two independent annotations, but it does not report inter-annotator agreement (e.g., Cohen's kappa or Krippendorff's alpha), does not describe how disagreements were resolved into a final label, and does not explain why only 2269 of the 2500 selected sentences have annotations. If annotations are noisy or if the aggregation rule correlates with whether a sentence is original or paraphrased, the confusion matrix in Figure 2, the Bowker statistic, and all Δ values in Table 1 inherit that bias. The 231 missing sentences also require an explanation, because their systematic exclusion (for instance, due to the 'Difficult to say' option) could shift the sensitivity distribution and distort the measured transition probabilities.","section":"§3.2"},{"comment":"The handling of the 232 refusal responses from GPT-4o-mini is unspecified. The paper says these refusals 'were rare' but does not state whether a refused sentence was paraphrased again, whether the refusal was treated as a non-response, or whether such sentences were excluded from the annotation sample. Since the annotation section says 'we randomly selected one of its three paraphrases,' this presupposes that all original sentences have three valid paraphrases. If refusals are concentrated among the most offensive sentences, the annotated sample could be biased away from the top of the sensitivity scale, which would directly affect the estimated Δ for Taboo and Derogatory classes. The authors should report the exact handling of refusals and, if any sentences were excluded, state how this affects the representativeness of the sample.","section":"§3.1"},{"comment":"It is unclear on which data split the Δ values for the traditional classifiers are computed. Section 3.2 describes an 80/20 train/test split for the traditional classifiers, but Table 1 reports Δ for each sensitivity class without specifying whether the predictions come from the 20% test set or the full dataset. If the full dataset is used, the classifiers have already seen 80% of the labels during training, making the Δ values and the ΔMSD comparisons optimistically biased. If only the test set is used, the per-class sample sizes are small and would explain the missing Taboo entries for Nearest Neighbors and Random Forest; this should be clarified, and the test-set class distribution should be reported.","section":"§3.3 and Table 1"},{"comment":"The blinding of the annotators to the original/paraphrased status is not convincingly described. Each annotator's batch contains both the original sentence and its paraphrased version, and the paper only says that 'no explicit indication was provided regarding the sentence's pair.' Because the two sentences often share many lexical items, an annotator can easily infer which is the paraphrase, especially when one version is visibly more polite or neutral. This could subtly influence their sensitivity labels and inflate the measured direction of the shift. The authors should describe how the two versions of a pair were presented to the annotator (e.g., separated across batches, randomized order, or masked formatting) or otherwise justify that the pairing was not inferable.","section":"§3.2"},{"comment":"The confusion matrix in Figure 2 is reported only as row percentages, without the underlying counts or the total number of paired sentences. The Bowker test statistic is reported, but the strength of the evidence cannot be fully assessed without knowing the number of observations per cell (for instance, whether the 87.1% diagonal for Formal/Polite is based on 50 or 500 sentences). Reporting the raw counts or at least the row sums would allow readers to evaluate the precision of the estimated transition probabilities and the robustness of the Bowker test result.","section":"§4.1"}],"minor_comments":[{"comment":"The category definition for Informal contains a typo: 'very conversetional' should be 'very conversational.'","section":"§3.2"},{"comment":"The Δ metric assumes an equal-interval scale across the four sensitivity categories (1=Formal/Polite to 4=Taboo). This is a strong assumption, as the perceptual distance between Derogatory and Taboo may differ from that between Informal and Derogatory; the paper should acknowledge this limitation explicitly.","section":"§3.3"},{"comment":"The paper reports 23,347 sentences collected but does not state how the 1,250 original sentences were sampled 'uniformly with respect to expressions.' Since there are 599 expressions, uniform sampling would give roughly 2 sentences per expression; the authors should clarify whether this was a random sample or a stratified selection, and how this interacts with the at-most-45-sentences-per-expression constraint.","section":"§3.1"},{"comment":"The zero-shot LLM classification evaluation does not describe how the model outputs were parsed (e.g., whether outputs were normalized, whether synonyms like 'formal' or 'taboo' were accepted) or how failed outputs were handled. This information is needed to interpret the reported F1 scores.","section":"§4.2"},{"comment":"The rows for Nearest Neighbors and Random Forest show a dash for the Taboo Δ value, which is unexplained in the text. If these classifiers never predict the Taboo class in the test set, that is an important limitation that should be noted in the discussion of the classifiers' performance.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The central research question is timely and the empirical design is generally sound, but the stress-test concern about the annotation layer is exactly right: the paper currently provides no evidence of annotation reliability, and the missing 231 sentences need a clear explanation. The authors can likely address this in a revision by reporting inter-annotator agreement, describing the adjudication process, and clarifying the treatment of refusals and the data split used for the classifier Δ values. If those points are fixed, I would be willing to consider the paper for acceptance. The comparison with traditional classifiers is somewhat secondary and could be streamlined if the authors wish to focus the paper on the implicit moderation result."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this paper gives a useful measurement of a real phenomenon—GPT-4o-mini systematically softens sensitive language during paraphrasing—but the measurement layer has a transparency gap. I would not bet against the effect, but I would not quote the headline magnitudes without seeing the annotation details.\n\nWhat is new: the paper quantifies implicit sanitization in a paraphrase task using expert labels, and compares six open-source LLMs against six traditional classifiers. The main result is internally consistent: 51.6% of Taboo originals become Informal, 58.9% of Derogatory originals become Formal/Polite or Informal, and Bowker's test rejects symmetry with p < 0.0001. That is a genuine empirical result, not a demo. The classifier comparison is also useful: traditional methods, especially MLP and Born Classifier, track expert Delta better than the open LLMs, while overall accuracy is poor enough to caution against using generic LLMs for moderation automation.\n\nSoft spots: Section 3.2 samples 2500 sentences but reports annotations for only 2269, with no explanation for the missing 231. More importantly, there is no inter-annotator agreement and no stated adjudication rule. With ten annotators and two independent labels per sentence, that is a standard and fixable transparency gap. If disagreements were resolved toward the less sensitive class for paraphrases, the effect could be inflated. I see no evidence that happened, but I cannot rule it out. There is also no control condition, so part of the shift could be ordinary paraphrase behavior rather than alignment-specific sanitization. The authors say the result 'suggests' implicit moderation, which is honest, but the design does not isolate the mechanism. No code or data release makes the measurement hard to verify. These are not fatal—the effect is large enough that even noisy labels would likely show it—but they cap the precision of the claim.\n\nThe citation pattern is fine. Born Classifier is a self-citation, but it is evaluated as a baseline against external human labels, not used to prove the main effect.\n\nRecommendation: worth refereeing. A serious editor should send this out, with the request that the authors document the annotation process, add a control baseline, and release code/data. This is a paper for content moderation researchers, alignment evaluators, and anyone building benchmark tasks for sensitivity classification.","headline":"A solid empirical measurement of a real effect, but the annotation layer is underdocumented and the magnitude should be read with caution until those details are released.","tokens_in":12223,"tokens_out":3188,"would_cite":true,"duration_ms":38001,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GPT-4o-mini systematically flattens sensitive language when asked only to paraphrase it.","keywords":["implicit content moderation","LLM paraphrasing","sanitization","sensitive language","sensitivity annotation","GPT-4o-mini","zero-shot classification","text detoxification"],"falsifier":"Recompute the confusion matrix and Bowker's test using only sentences where the two judges independently gave the same label; if the asymmetry disappears or the average drop for taboo sentences vanishes, the result is an artifact of how labels were combined rather than a real behavior of the model.","tokens_in":11240,"feed_emoji":"🤐","tokens_out":10167,"duration_ms":102922,"temperature":0.7,"pith_summary":"GPT-4o-mini, when asked only to paraphrase a sentence, tends to return a less sensitive version of it, even though no instruction to soften or detoxify was given. The paper claims this implicit moderation is systematic: expert annotators classified original and paraphrased sentences into four sensitivity levels, and the shifts consistently point downward, from Taboo and Derogatory toward Informal and Formal/Polite. The strongest evidence is the confusion matrix in which 51.6% of originally Taboo sentences become Informal after paraphrase, and Bowker's symmetry test rejects the null hypothesis ($\\chi^2 = 138.255$, df = 6, p < 0.0001). The paper also reports that zero-shot open-source LLMs are worse than simple trained classifiers at reproducing expert sensitivity judgments, so automated moderation monitoring is not yet safe to hand to generic LLMs. If correct, the finding matters because aligned models may sanitize language by default in tasks where users expect neutral rewording, silently changing the content being processed.","feed_headline":"Paraphrase prompts make GPT-4o-mini soften taboo language","feed_subtitle":"Expert annotations show 52% of taboo sentences return as informal after a neutral rewrite request.","key_machinery":"The machinery is a four-class ordinal sensitivity schema (Formal/Polite, Informal, Derogatory, Taboo) applied identically to original sentences and to GPT-4o-mini's paraphrases. Shifts are summarized in a 4×4 confusion matrix whose asymmetry is tested by Bowker's test for symmetry, and quantified by $\\Delta$, the average signed change in sensitivity level (1 to 4) from original to paraphrase for each starting class. The schema does the work of making an otherwise fuzzy judgment — how offensive is this sentence? — measurable, so implicit moderation becomes a detectable, statistically testable pattern rather than an impression.","core_discovery":"The paper's central claim is that GPT-4o-mini performs implicit content moderation during paraphrase generation: it systematically lowers the sensitivity level of the text it rewrites. Using a four-level schema (Formal/Polite, Informal, Derogatory, Taboo) applied by ten expert annotators to 2,269 original and paraphrased sentences, the authors find that sensitivity is preserved for less sensitive categories (87.1% of Formal/Polite stay Formal/Polite) but collapses for the most sensitive ones: only 15.6% of Taboo sentences remain Taboo, 51.6% become Informal, and 58.9% of Derogatory sentences become Formal/Polite or Informal. The average shift metric $\\Delta$ is $-1.7$ for original Taboo sentences and $-0.87$ for Derogatory, meaning roughly one to two sensitivity levels of reduction, and Bowker's test ($\\chi^2 = 138.255$, df = 6, p < 0.0001) rejects symmetry of the shift matrix. A secondary claim is that zero-shot open-source LLMs do not reliably replicate expert sensitivity labels: traditional classifiers such as MLP and the Born classifier reach higher F1 scores, and no automated method exceeds roughly 0.53 overall F1.","pith_inferences":["If the same protocol were run on all three paraphrases instead of one per original sentence, the $\\Delta$ estimates could be tested for stability across paraphrase draws; large variance would mean the one-sample annotation design understates the range of possible sanitization.","Treating the four sensitivity levels as evenly spaced is a modeling choice; a weighting that makes the Taboo-to-Derogatory distance larger than other gaps would change the magnitude of $\\Delta$ but not the direction of the shift.","A natural next experiment is to prompt GPT-4o-mini with an explicit instruction to preserve tone and register; if the negative $\\Delta$ shrinks, that would confirm the sanitization is an alignment default rather than an unavoidable property of paraphrasing."],"forward_implications":["A neutral 'paraphrase this sentence' request to GPT-4o-mini is not content-preserving in sensitivity: taboo and derogatory material is routinely downgraded, so downstream uses of such paraphrases inherit a sanitization bias.","The direction of the shift is consistently toward less sensitive classes for every classifier that detects a shift, reinforcing the conclusion that aligned LLMs moderate by default rather than by instruction.","Because even the best automated classifier reaches only 0.53 overall F1, zero-shot or lightly trained models are not yet reliable substitutes for expert judgment in moderation monitoring.","Traditional classifiers trained on expert labels approximate expert sensitivity shifts more closely than zero-shot open-source LLMs, offering a cheaper local alternative for monitoring paraphrase behavior at scale."],"supporting_citations":[{"why":"establishes RLHF as the alignment technique that plausibly installs the implicit moderation behavior.","marker":"Ouyang et al., 2022"},{"why":"documents the model's design goal of avoiding disallowed content, the behavior whose side effect is measured.","marker":"OpenAI, 2024a"},{"why":"supplies the symmetry test used to reject the null hypothesis of no sensitivity shift.","marker":"Bowker, 1948"},{"why":"provides the web corpus from which the sentences were drawn.","marker":"Jakubíček et al., 2013"},{"why":"provides the toxicity benchmark used to contextualize reductions in disallowed output.","marker":"Gehman et al., 2020"},{"why":"supports the zero-shot prompting protocol for open-source LLM classification.","marker":"Alizadeh et al., 2025"}],"fun_headline_variants":["GPT-4o-mini silently downgrades taboo language when asked to paraphrase","Paraphrasing with GPT-4o-mini? Taboo words get softer","GPT-4o-mini quietly strips taboo from paraphrases","Taboo sentences lose their edge when GPT-4o-mini paraphrases","Implicit moderation: GPT-4o-mini cleans up taboo language in paraphrase"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the human judges' labels are correct; the paper does not say how often the two judges disagreed, how disagreements were settled into a final label, or why 231 of the sampled sentences ended up without annotations. If those labels are unreliable, every measured shift inherits that unreliability.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4o-mini silently downgrades taboo language when asked to paraphrase","Paraphrasing with GPT-4o-mini? Taboo words get softer","GPT-4o-mini quietly strips taboo from paraphrases","Taboo sentences lose their edge when GPT-4o-mini paraphrases","Implicit moderation: GPT-4o-mini cleans up taboo language in paraphrase"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000352,"raw_usage":{"total_tokens":1910,"prompt_tokens":928,"completion_tokens":982,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":881}},"tokens_in":544,"tokens_out":982,"duration_ms":9636,"temperature":1.0,"reasoning_tokens":881,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T10:50:33.667048+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the confusion matrix and Bowker's test using only sentences where the two judges independently gave the same label; if the asymmetry disappears or the average drop for taboo sentences vanishes, the result is an artifact of how labels were combined rather than a real behavior of the model.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the symmetry test used to reject the null hypothesis of no sensitivity shift."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supports the zero-shot prompting protocol for open-source LLM classification."}],"review_version":1}