{"id":"20d0b02f-c0e6-4be4-b306-de864ba76d12","arxiv_id":"2505.12090","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"LLM paraphrasing obfuscates authorship unevenly across users, and prompting with each author's top SHAP-identified style feature improves average evasion but does not consistently beat zero-shot paraphrasing.","lead":"The paper tests whether GPT-4 and LLaMA-3.1 can hide a writer's identity by paraphrasing, and finds that success varies sharply by user. It then builds personalized prompts from the stylistic features that best identify each author, which improves average obfuscation and slightly reduces the spread.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Obfuscation success is measured only as a drop in verifier F1; without semantic-preservation or fluency checks, the central claim may reflect text degradation rather than effective style obfuscation.","rationale":"The paper's central claim is that LLM-based author obfuscation works, with user-level variability and an effective personalized extension. Every quantitative result in the paper is expressed as a drop in author-verification F1. The paper itself flags in Section 4.3 that it has not measured semantic preservation, so the reported 'obfuscation success' may be an artifact of the LLM producing text that is merely harder to classify because it is degraded, out-of-distribution, or topically shifted. This is not a peripheral caveat: it determines whether the phenomenon under study is author obfuscation or text corruption. The personalized prompting method intensifies this risk, because it instructs the model to manipulate one hand-picked surface feature, such as punctuation frequency or POS-tag counts, which can easily yield unnatural text while still satisfying a simple feature check. The reader's weakest-assumption analysis identifies the same issue, and the recommended conditional acceptance is appropriate: the concern is serious but addressable with additional quality metrics. I also note secondary concerns—lack of significance testing on the small per-user improvements, the mild circularity of fitting SHAP values to the same verifier used in evaluation, and pooling across datasets in the dip test—but the semantic-preservation confound is the single most load-bearing issue because it could invalidate the interpretation of all reported F1 drops.","tokens_in":13261,"tokens_out":7317,"duration_ms":71873,"concrete_test":"On a sample of obfuscated outputs from both LLMs and both prompt conditions (zero-shot and personalized), compute semantic similarity to the source text (e.g., BERTScore or cosine similarity of SBERT embeddings) and fluency (e.g., perplexity under a held-out language model, or human ratings). Then compute the correlation between these quality scores and the AV F1 drops reported in Tables 4 and 5. If the users with the largest F1 drops also have the lowest similarity or fluency, the evasion results are confounded by text degradation; if quality is preserved for the successful cases, the concern does not land.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that LLM paraphrasing successfully obfuscates authorship—rests entirely on reduced author-verification F1. The paper never measures semantic similarity, fluency, or content preservation of the obfuscated texts. Section 4.3 explicitly concedes: 'we are not evaluating these two LLMs in terms of their outputs' semantic preservation, we can only conclude that LLaMA-3.1 paraphrasing is more effective at evading AV detection, which could come at the expense of text quality and semantic similarity to the original text.' A model that lowers F1 by producing generic, ungrammatical, or topic-shifted text is not a useful obfuscation method, even if it 'evades' the verifier. This confound is especially acute for the personalized prompts, which ask the LLM to increase or decrease a single surface feature (e.g., 'more double quotation marks'), potentially producing unnatural output. If the F1 drops in Tables 4 and 5 are driven by quality collapse rather than by plausible style transfer, the headline claims about personalized obfuscation and bimodality mitigation are not supported as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how effectively LLM-based paraphrasing obfuscates authorship at the level of individual users. The authors train author-verification models on IMDb, Yelp, and Blog data using writeprint logistic regression, XGBoost, and BERT features; evaluate zero-shot GPT-4 and LLaMA-3.1 paraphrasing; observe that verification F1 drops unevenly across users; apply Hartigan's dip test to claim bimodality; and then propose a personalized prompting method in which per-author SHAP values identify a single stylistic feature to be increased or decreased during paraphrasing. They report that personalized prompts lower verifier F1 further than zero-shot prompting for several dataset/model combinations and that the dip-test p-values rise, suggesting reduced bimodality.","tokens_in":13379,"tokens_out":3108,"duration_ms":33560,"significance":"The user-level perspective is a useful contribution to author obfuscation, which is often reported only as an aggregate statistic. The cross-dataset evaluation (Yelp, IMDb, Blog) with two LLMs and two verifier families is a reasonable design, and the use of SHAP values to make the personalization interpretable is appealing. If the findings are sound, the paper provides practical evidence that per-author style-feature prompting can improve evasion and reduce outcome variability. However, the central claim currently rests entirely on verifier F1 reductions, a metric that cannot distinguish successful style transfer from degraded text. The paper explicitly acknowledges this gap in Section 4.3. The personalization loop also uses SHAP values computed on the same logistic-regression verifier that is a principal evaluation target, which makes the improvement partly by construction. The paper does not release code or data, which limits reproducibility of the exact numbers.","major_comments":[{"comment":"The central evidence for obfuscation success is a drop in author-verification F1, without any measurement of semantic preservation, fluency, or readability of the paraphrased outputs. The authors themselves state in Section 4.3 that they are 'not evaluating these two LLMs in terms of their outputs' semantic preservation' and that LLaMA's stronger evasion 'could come at the expense of text quality.' This is not a minor caveat: if the F1 drops are caused by generic, ungrammatical, or topic-shifted text, then the headline claims about 'effective obfuscation' and 'personalized obfuscation' are unsupported. The personalized prompts explicitly instruct the model to increase or decrease a surface feature such as punctuation frequency, which makes quality collapse a concrete risk. I would ask the authors to report semantic-similarity metrics (e.g., BERTScore, sentence embeddings, or NLI-based entailment) and, ideally, a small human evaluation of fluency and meaning preservation on the obfuscated outputs, and to re-examine the conclusions in light of those results.","section":"Section 4.3, Tables 1, 4, 5"},{"comment":"The claimed bimodal distribution is not supported for GPT-4 by the paper's own statistical test: the Hartigan dip-test p-values for GPT-4 are 0.270 (logistic regression) and 0.572 (BERT), both far above the 0.05 threshold, while only LLaMA-3.1 shows significant multimodality (p = 0.000 and p = 0.050). The sentence in Section 4.3 that 'for both LLMs, there are cases where...' is a visual impression, not a tested claim. The abstract's unqualified statement that 'We observe a bimodal distribution of efficacy' therefore overstates the evidence. The authors should either restrict the bimodality claim to LLaMA-3.1 or provide additional statistical support (e.g., a larger set of users or a different multimodality test) before treating GPT-4 as exhibiting the same pattern.","section":"Sections 4.4 and 5, Table 2"},{"comment":"The personalized prompts are designed from SHAP values computed on the same logistic-regression author verifier that is one of the two main evaluation targets in Tables 4 and 5. The logistic-regression F1 improvement is therefore partly by construction: the prompt is explicitly instructed to move the feature that most directly influences that model's decision. The fact that BERT also shows some improvement in some datasets mitigates this concern, but BERT outcomes are inconsistent (e.g., Blog results worsen for personalized prompts in Table 5), so the evidence for genuine, generalizable obfuscation is not established. I recommend evaluating personalized prompts against a held-out verifier that was not used for feature selection, or at minimum reporting the results separately as 'same-verifier' and 'cross-verifier' evidence, with the circularity explicitly acknowledged in the interpretation.","section":"Section 5.1, Tables 4 and 5"},{"comment":"The claims that personalized obfuscation 'outperforms' zero-shot are based on average F1 differences across only 10 or 5 users per dataset, with no variance estimates, confidence intervals, or paired significance tests. For example, the Yelp logistic-regression average for GPT-4 moves from 0.42 to 0.40, and the LLaMA Blog numbers move from 0.56 to 0.52; these differences could easily be within noise for n = 10. Adding per-user differences with a signed-rank test or bootstrap intervals would substantially strengthen the evidential value of Tables 4 and 5 and would also clarify whether the claimed 'partial mitigation' of bimodality in Table 6 is statistically meaningful.","section":"Tables 4 and 5, Section 5.2"}],"minor_comments":[{"comment":"The phrase 'treat AO as a an adversarial attack' contains a typo: 'a an' should be 'an'.","section":"Section 2"},{"comment":"The SHAP values are attributed to 'Hart, 1989' in the text and references, but this should cite Shapley (1953) and Lundberg and Lee (2017) for the SHAP method itself; Hart 1989 is the Shapley value encyclopedia entry and is not the standard source for SHAP.","section":"Section 5.1 and References"},{"comment":"The example prompt instructs the model to produce 'more **double quotation marks**', but the surrounding text says that for a negative SHAP feature the prompt would ask to decrease the feature; the example and the explanatory text should be aligned so the reader can see how the sign of the SHAP value maps to the direction of the instruction.","section":"Section 5.1, prompt example"},{"comment":"The text refers to 'red cells' indicating drops of less than 20%, but the table as printed does not show color; the threshold should be stated explicitly in the caption or a note so the reader can reproduce the classification.","section":"Table 1"},{"comment":"The entries 'unsuccessful increase' and 'successful decrease' are ambiguous. For instance, a feature marked 'Period mark frequency↓' with 'unsuccessful increase' means the LLM increased the feature when asked to decrease it; this should be stated in the caption or made into a clearer success/failure column.","section":"Table 3"},{"comment":"The word 'generalizabilty' is misspelled; it should be 'generalizability'.","section":"Section 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a CL/NLP venue, and the user-wise analysis is timely, but the two main refutations to address are the lack of semantic-preservation evaluation and the same-verifier circularity in the SHAP-based personalization. Both are fixable with additional experiments, so I do not recommend rejection, but the current version overstates its conclusions in the abstract and conclusion. I would also encourage the authors to provide code and data to make the reported F1 numbers reproducible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read: the paper offers a clear user-level analysis of LLM paraphrasing for author obfuscation and a simple personalization mechanism (SHAP-selected stylistic feature prompting). The main thing to know is that the headline claims are weaker than the abstract suggests because the only success metric is a drop in verifier F1. The authors concede in Section 4.3 that they do not measure semantic preservation, so we cannot distinguish real style change from generic or degraded output.\n\nWhat works: the per-user breakdown in Table 1 is genuinely useful – it shows obfuscation is far from uniform (Hitchcoc, User_4 etc. retain high F1), and LLaMA-3.1 consistently evades the verifiers more than GPT-4. That is a practical caution for anyone relying on LLM-based anonymization. The SHAP-based prompt design is concrete and easy to reproduce; Table 3 shows both models can usually shift the targeted feature. The paper is clearly written and the evaluation covers three datasets plus three verifier types.\n\nThe soft spots are proportional. The missing semantic evaluation is the biggest one. That caveat in 4.3 applies to every result, not just the LLaMA-vs-GPT comparison. For personalized prompts, the risk is especially acute: ordering a model to use more double quotes or fewer nouns could push it toward unnatural text. The authors should add at least one semantic-similarity or fluency metric, or explicitly reframe claims as \"evading these specific verifiers.\"\n\nThe bimodality claim is also overstated: Hartigan's dip test is only significant for LLaMA-3.1. GPT-4 p-values are 0.27 and 0.57, which is merely failure to reject unimodality. With 25 users, the test is underpowered, so \"variability across users\" is more accurate than \"bimodal.\"\n\nThe circularity is real but mild: SHAP features come from the logistic regression verifier used in the main evaluation, so gains there are partly white-box fitting. BERT results, which are independent of the feature choice, are mixed and sometimes negative (Blog). No significance tests or confidence intervals appear anywhere.\n\nVerdict: within-subfield useful, not a field-changer. It deserves a serious referee, but the authors should be asked for semantic-quality metrics, a tempering of the bimodality and personalization claims, and proper significance testing. I'd cite this paper for its user-level variance finding.","headline":"Useful user-level analysis of LLM author obfuscation, but the missing semantic evaluation undercuts the headline claims.","tokens_in":14002,"tokens_out":5128,"would_cite":true,"duration_ms":49905,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-paraphrased author obfuscation is bimodal across users, and feature-targeted prompts narrow the divide.","keywords":["author obfuscation","authorship verification","large language models","personalized prompting","SHAP values","writing style","bimodality","paraphrasing"],"falsifier":"Measure semantic similarity and fluency of the obfuscated outputs on the same datasets, for example with a reference-based semantic metric and perplexity or human readability ratings, and compare zero-shot and personalized rewrites per user. If the personalized prompts' additional F1 drop coincides with disproportionately large semantic or fluency loss, the conclusion that targeted stylistic changes, rather than text degradation, drive the evasion would not hold.","tokens_in":12966,"feed_emoji":"🕵️","tokens_out":6986,"duration_ms":65736,"temperature":0.7,"pith_summary":"The paper asks whether large language models can hide a writer's identity by paraphrasing, and whether one generic prompt works for every author. Testing GPT-4 and LLaMA-3.1 against per-user author-verification models on Yelp, IMDb, and blog data, it finds that the average drop in detection hides a two-cluster pattern: some authors are easily hidden while others remain identifiable. The paper proposes a personalized remedy: use SHAP values to find the single most distinctive stylistic marker for each author, then prompt the LLM to deliberately increase or decrease that marker during paraphrasing. Across both verifier types and most datasets, the personalized prompts lower detection scores further than zero-shot paraphrasing, and the statistical evidence of bimodality weakens. The result matters for anyone using paraphrase tools for privacy, suggesting generic obfuscation is unreliable but author-specific targeting can improve it.","feed_headline":"Paraphrase obfuscation fails some authors; targeted prompts close the gap","feed_subtitle":"Per-user tests show paraphrase evasion splits into easy and hard authors; feature-targeted prompts narrow the gap.","key_machinery":"The load-bearing mechanism is a two-step personalization loop built on SHAP values and writeprint features, where writeprint features are measurable stylistic markers such as punctuation frequencies, part-of-speech frequencies, capitalization, and word counts. First, an interpretable logistic-regression author verifier is trained per user on these features, and SHAP values over the validation set rank which single feature most strongly pushes predictions toward that author. Second, the paper builds a user-specific prompt that tells the LLM to paraphrase while deliberately increasing or decreasing exactly that feature, for example adding more double quotation marks or reducing dash usage. The loop closes by testing the rewritten text against the same verifier and against a BERT-based verifier, with the drop in F1 treated as obfuscation success and Hartigan's dip test checking whether per-user performance is still bimodal.","core_discovery":"On the paper's own terms, the central claim is that LLM-based author obfuscation is bimodal at the user level: zero-shot paraphrase prompting drops average author-verification F1 from roughly 0.90 to between 0.38 and 0.63, yet some authors' texts remain almost as attributable as before. The paper further claims that personalization partially fixes this. For each author it ranks writeprint features by SHAP value, picks the top feature, and instructs the model to change that feature's value in the paraphrased output. GPT-4's personalized obfuscation lowers verifier F1 on every dataset compared with its zero-shot prompting, LLaMA-3.1 improves in the less formal Yelp and Blog datasets, and Hartigan's dip test, significant for LLaMA's zero-shot results, no longer indicates bimodality after personalization. The authors frame this as evidence that obfuscation should target the features that identify an author rather than applying a uniform rewrite to everyone.","pith_inferences":["A direct extension the authors did not run is to score the obfuscated outputs for meaning preservation and fluency; doing so could overturn the relative ranking of the two models and even reinterpret the personalization gain as partly a quality loss.","The single-feature design is minimal by the paper's own admission; combining two or three top SHAP features, or choosing features per text rather than per author, is a natural next test that could push detection down further.","A defensive implication follows: if obfuscation means exaggerating or suppressing one measurable feature, an adaptive verifier could learn to flag exactly those one-sided feature anomalies, turning the personalized prompt into a detectable signature.","The same SHAP-guided prompting recipe transfers in principle to hiding other classifier-exposed attributes, such as demographic traits, as long as an interpretable model can rank which textual cues expose them."],"forward_implications":["A writer who uses a generic paraphrase prompt should not assume protection: per-user results in the paper include cases where the verifier's F1 barely drops.","Personalized prompts built against an interpretable writeprint verifier also reduce detection by a BERT verifier on most datasets, so the targeted features are not simply an artifact of one classifier family.","The paper's finding that LLaMA-3.1 evades more than GPT-4, while the paper does not measure semantic preservation, means model choice is a quality-versus-evasion trade-off that remains unresolved.","Style-based personalization shows its weakest, least consistent gains on the Blog dataset, whose posts average about 40 words, implying short texts leave too little stylistic surface to modify."],"supporting_citations":[{"why":"Establishes the zero-shot LLM paraphrasing baseline and provides the IMDb and Yelp user subsets the paper reuses.","marker":"Utpala et al., 2023"},{"why":"StyleRemix is the closest personalized-obfuscation baseline and supplies the standardized Blog dataset and the style-element idea the paper extends.","marker":"Fisher et al., 2024a"},{"why":"Defines the writeprint stylometric feature set used to train the logistic regression and XGBoost author verifiers.","marker":"Abbasi and Chen, 2008"},{"why":"Provides the dip test the paper uses to detect bimodality in obfuscation efficacy across users.","marker":"Hartigan and Hartigan, 1985"},{"why":"Supplies the Shapley-value-based SHAP method used to identify each author's most predictive feature.","marker":"Hart, 1989"},{"why":"Supplies BERT-large, the embedding-based author verifier used as the strong second classifier.","marker":"Devlin et al., 2019"},{"why":"Identifies GPT-4 (gpt-4-turbo), one of the two LLMs whose obfuscation the paper evaluates.","marker":"Achiam et al., 2023"},{"why":"Identifies LLaMA-3.1-8B-Instruct, the smaller model the paper finds more effective at evading verification.","marker":"Dubey et al., 2024"}],"fun_headline_variants":["Bimodal author obfuscation fixed by personalized prompts","Author obfuscation works unevenly; personalization evens it out","LLM obfuscation is bimodal; targeted prompts fix it","Feature-targeted prompts narrow author obfuscation gap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a lower author-verification F1 measures successful style obfuscation, but the paper never measures whether the rewritten text preserves meaning or reads fluently, so the extra evasiveness could come from degraded output rather than genuine stylistic disguise.","fun_headline_variants_meta":{"raw":{"variants":["Bimodal author obfuscation fixed by personalized prompts","Author obfuscation works unevenly; personalization evens it out","LLM obfuscation is bimodal; targeted prompts fix it","Feature-targeted prompts narrow author obfuscation gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00073,"raw_usage":{"total_tokens":3214,"prompt_tokens":839,"completion_tokens":2375,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":455,"completion_tokens_details":{"reasoning_tokens":2300}},"tokens_in":455,"tokens_out":2375,"duration_ms":14940,"temperature":1.0,"reasoning_tokens":2300,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:40:32.718525+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure semantic similarity and fluency of the obfuscated outputs on the same datasets, for example with a reference-based semantic metric and perplexity or human readability ratings, and compare zero-shot and personalized rewrites per user. If the personalized prompts' additional F1 drop coincides with disproportionately large semantic or fluency loss, the conclusion that targeted stylistic changes, rather than text degradation, drive the evasion would not hold.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the writeprint stylometric feature set used to train the logistic regression and XGBoost author verifiers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the dip test the paper uses to detect bimodality in obfuscation efficacy across users."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Shapley-value-based SHAP method used to identify each author's most predictive feature."}],"review_version":1}