REVIEW 4 major objections 6 minor 35 references
Personalized Author Obfuscation with Large Language Models
T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read LLM-paraphrased author obfuscation is bimodal across users, and feature-targeted prompts narrow the divide.
desk verdict Useful user-level analysis of LLM author obfuscation, but the missing semantic evaluation undercuts the headline claims. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a two-step personalization loop built on SHAP values and writeprint features, where writeprint features are measurable stylistic markers such as punctuation frequencies, part-of-speech frequencies, capitalization, and word counts. First, an interpretable logistic-regression author verifier is trained per user on these features, and SHAP values over the validation set rank which single feature most strongly pushes predictions toward that author. Second, the paper builds a user-specific prompt that tells the LLM to paraphrase while deliberately increasing or decreasing exactly that feature, for example adding more double quotation marks or reducing dash usage. The loop closes by testing the rewritten text against the same verifier and against a BERT-based verifier, with the drop in F1 treated as obfuscation success and Hartigan's dip test checking whether per-user performance is still bimodal.
What would settle it
Measure semantic similarity and fluency of the obfuscated outputs on the same datasets, for example with a reference-based semantic metric and perplexity or human readability ratings, and compare zero-shot and personalized rewrites per user. If the personalized prompts' additional F1 drop coincides with disproportionately large semantic or fluency loss, the conclusion that targeted stylistic changes, rather than text degradation, drive the evasion would not hold.
Extended reading notes
Core claim
On the paper's own terms, the central claim is that LLM-based author obfuscation is bimodal at the user level: zero-shot paraphrase prompting drops average author-verification F1 from roughly 0.90 to between 0.38 and 0.63, yet some authors' texts remain almost as attributable as before. The paper further claims that personalization partially fixes this. For each author it ranks writeprint features by SHAP value, picks the top feature, and instructs the model to change that feature's value in the paraphrased output. GPT-4's personalized obfuscation lowers verifier F1 on every dataset compared with its zero-shot prompting, LLaMA-3.1 improves in the less formal Yelp and Blog datasets, and Hartigan's dip test, significant for LLaMA's zero-shot results, no longer indicates bimodality after personalization. The authors frame this as evidence that obfuscation should target the features that identify an author rather than applying a uniform rewrite to everyone.
Load-bearing premise
The load-bearing premise is that a lower author-verification F1 measures successful style obfuscation, but the paper never measures whether the rewritten text preserves meaning or reads fluently, so the extra evasiveness could come from degraded output rather than genuine stylistic disguise.
Editorial extensions
If this is right
- A writer who uses a generic paraphrase prompt should not assume protection: per-user results in the paper include cases where the verifier's F1 barely drops.
- Personalized prompts built against an interpretable writeprint verifier also reduce detection by a BERT verifier on most datasets, so the targeted features are not simply an artifact of one classifier family.
- The paper's finding that LLaMA-3.1 evades more than GPT-4, while the paper does not measure semantic preservation, means model choice is a quality-versus-evasion trade-off that remains unresolved.
- Style-based personalization shows its weakest, least consistent gains on the Blog dataset, whose posts average about 40 words, implying short texts leave too little stylistic surface to modify.
Reading between the lines
- A direct extension the authors did not run is to score the obfuscated outputs for meaning preservation and fluency; doing so could overturn the relative ranking of the two models and even reinterpret the personalization gain as partly a quality loss.
- The single-feature design is minimal by the paper's own admission; combining two or three top SHAP features, or choosing features per text rather than per author, is a natural next test that could push detection down further.
- A defensive implication follows: if obfuscation means exaggerating or suppressing one measurable feature, an adaptive verifier could learn to flag exactly those one-sided feature anomalies, turning the personalized prompt into a detectable signature.
- The same SHAP-guided prompting recipe transfers in principle to hiding other classifier-exposed attributes, such as demographic traits, as long as an interpretable model can rank which textual cues expose them.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies how effectively LLM-based paraphrasing obfuscates authorship at the level of individual users. The authors train author-verification models on IMDb, Yelp, and Blog data using writeprint logistic regression, XGBoost, and BERT features; evaluate zero-shot GPT-4 and LLaMA-3.1 paraphrasing; observe that verification F1 drops unevenly across users; apply Hartigan's dip test to claim bimodality; and then propose a personalized prompting method in which per-author SHAP values identify a single stylistic feature to be increased or decreased during paraphrasing. They report that personalized prompts lower verifier F1 further than zero-shot prompting for several dataset/model combinations and that the dip-test p-values rise, suggesting reduced bimodality.
Significance. The user-level perspective is a useful contribution to author obfuscation, which is often reported only as an aggregate statistic. The cross-dataset evaluation (Yelp, IMDb, Blog) with two LLMs and two verifier families is a reasonable design, and the use of SHAP values to make the personalization interpretable is appealing. If the findings are sound, the paper provides practical evidence that per-author style-feature prompting can improve evasion and reduce outcome variability. However, the central claim currently rests entirely on verifier F1 reductions, a metric that cannot distinguish successful style transfer from degraded text. The paper explicitly acknowledges this gap in Section 4.3. The personalization loop also uses SHAP values computed on the same logistic-regression verifier that is a principal evaluation target, which makes the improvement partly by construction. The paper does not release code or data, which limits reproducibility of the exact numbers.
major comments (4)
- [Section 4.3, Tables 1, 4, 5] The central evidence for obfuscation success is a drop in author-verification F1, without any measurement of semantic preservation, fluency, or readability of the paraphrased outputs. The authors themselves state in Section 4.3 that they are 'not evaluating these two LLMs in terms of their outputs' semantic preservation' and that LLaMA's stronger evasion 'could come at the expense of text quality.' This is not a minor caveat: if the F1 drops are caused by generic, ungrammatical, or topic-shifted text, then the headline claims about 'effective obfuscation' and 'personalized obfuscation' are unsupported. The personalized prompts explicitly instruct the model to increase or decrease a surface feature such as punctuation frequency, which makes quality collapse a concrete risk. I would ask the authors to report semantic-similarity metrics (e.g., BERTScore, sentence embeddings, or NLI-based entailment) and, ideally, a small human evaluation of fluency and meaning preservation on the obfuscated outputs, and to re-examine the conclusions in light of those results.
- [Sections 4.4 and 5, Table 2] The claimed bimodal distribution is not supported for GPT-4 by the paper's own statistical test: the Hartigan dip-test p-values for GPT-4 are 0.270 (logistic regression) and 0.572 (BERT), both far above the 0.05 threshold, while only LLaMA-3.1 shows significant multimodality (p = 0.000 and p = 0.050). The sentence in Section 4.3 that 'for both LLMs, there are cases where...' is a visual impression, not a tested claim. The abstract's unqualified statement that 'We observe a bimodal distribution of efficacy' therefore overstates the evidence. The authors should either restrict the bimodality claim to LLaMA-3.1 or provide additional statistical support (e.g., a larger set of users or a different multimodality test) before treating GPT-4 as exhibiting the same pattern.
- [Section 5.1, Tables 4 and 5] The personalized prompts are designed from SHAP values computed on the same logistic-regression author verifier that is one of the two main evaluation targets in Tables 4 and 5. The logistic-regression F1 improvement is therefore partly by construction: the prompt is explicitly instructed to move the feature that most directly influences that model's decision. The fact that BERT also shows some improvement in some datasets mitigates this concern, but BERT outcomes are inconsistent (e.g., Blog results worsen for personalized prompts in Table 5), so the evidence for genuine, generalizable obfuscation is not established. I recommend evaluating personalized prompts against a held-out verifier that was not used for feature selection, or at minimum reporting the results separately as 'same-verifier' and 'cross-verifier' evidence, with the circularity explicitly acknowledged in the interpretation.
- [Tables 4 and 5, Section 5.2] The claims that personalized obfuscation 'outperforms' zero-shot are based on average F1 differences across only 10 or 5 users per dataset, with no variance estimates, confidence intervals, or paired significance tests. For example, the Yelp logistic-regression average for GPT-4 moves from 0.42 to 0.40, and the LLaMA Blog numbers move from 0.56 to 0.52; these differences could easily be within noise for n = 10. Adding per-user differences with a signed-rank test or bootstrap intervals would substantially strengthen the evidential value of Tables 4 and 5 and would also clarify whether the claimed 'partial mitigation' of bimodality in Table 6 is statistically meaningful.
minor comments (6)
- [Section 2] The phrase 'treat AO as a an adversarial attack' contains a typo: 'a an' should be 'an'.
- [Section 5.1 and References] The SHAP values are attributed to 'Hart, 1989' in the text and references, but this should cite Shapley (1953) and Lundberg and Lee (2017) for the SHAP method itself; Hart 1989 is the Shapley value encyclopedia entry and is not the standard source for SHAP.
- [Section 5.1, prompt example] The example prompt instructs the model to produce 'more **double quotation marks**', but the surrounding text says that for a negative SHAP feature the prompt would ask to decrease the feature; the example and the explanatory text should be aligned so the reader can see how the sign of the SHAP value maps to the direction of the instruction.
- [Table 1] The text refers to 'red cells' indicating drops of less than 20%, but the table as printed does not show color; the threshold should be stated explicitly in the caption or a note so the reader can reproduce the classification.
- [Table 3] The entries 'unsuccessful increase' and 'successful decrease' are ambiguous. For instance, a feature marked 'Period mark frequency↓' with 'unsuccessful increase' means the LLM increased the feature when asked to decrease it; this should be stated in the caption or made into a clearer success/failure column.
- [Section 3] The word 'generalizabilty' is misspelled; it should be 'generalizability'.
Circularity Check
Personalized prompts are fitted to the logistic-regression verifier that also measures success, making the LR gains partly by construction.
-
fitted input called prediction
[Section 5.1 (SHAP feature selection) and Section 5.2 (evaluation on the same logistic regression verifier, Tables 4 and 6)]
"For each author, we found the top features with highest average SHAP values over the validation dataset. This information sheds light on the features that contributed the most to identifying the author in the validation data set. After we learn the top feature with the highest average SHAP value, we use it to generate a personalized prompt for each author."
The top SHAP feature is, by definition, the feature with the largest contribution to the logistic-regression verifier's predictions. The personalized prompt instructs the LLM to change exactly that feature in the direction that reduces the verifier's confidence (e.g., 'Ensure the paraphrased version has more double quotation marks'). Success is then measured as the F1 drop of that same logistic-regression verifier (Table 4). Thus the improvement over zero-shot on the LR evaluation is partly forced: the prompt is constructed from the evaluator's own weights, making the attack white-box rather than an independent prediction of obfuscation quality.
full rationale
The paper contains no load-bearing self-citations: the reference list shows no overlap with the authors' prior work, and the central comparison is against zero-shot prompting rather than a cited 'uniqueness theorem'. The main circular element is the personalized obfuscation loop: SHAP values are computed from the same logistic-regression verifier used to evaluate success, and the prompt explicitly targets that verifier's most influential feature, so the observed LR F1 drop is in part a direct consequence of the fitted feature direction. The BERT verifier results provide independent evidence that the method transfers beyond the fitted model, which prevents a score of 6 or higher. The absence of semantic-preservation and fluency checks is a serious correctness and validity concern, but it is not circularity in the derivation chain; the paper itself concedes in Section 4.3 that it can only conclude the LLM 'is more effective at evading AV detection, which could come at the expense of text quality and semantic similarity.' Overall, one partial circular step, with independent content in the BERT evaluation.
Assumptions & free parameters
free parameters (1)
- Per-author top SHAP feature and direction =
Varies per user; examples include SPACE pos-tag frequency (decrease) for Yelp User_24 and comma frequency (increase…
assumptions (3)
- domain assumption Writeprint features plus logistic regression form an adequate proxy for an authorship adversary.
- domain assumption A drop in author-verification F1 indicates successful obfuscation without measuring semantic preservation or fluency.
- domain assumption LLMs can reliably alter a single requested surface feature while paraphrasing.
Cite this review
Pith. "Pith review of Personalized Author Obfuscation with Large Language Models." pith.science (2026). https://pith.science/paper/5S4SQ67R
@misc{pith2026250512090,
author = {Pith},
title = {Pith review of: Personalized Author Obfuscation with Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/5S4SQ67R}},
note = {Machine review of arXiv:2505.12090}
}
read the original abstract
In this paper, we investigate the efficacy of large language models (LLMs) in obfuscating authorship by paraphrasing and altering writing styles. Rather than adopting a holistic approach that evaluates performance across the entire dataset, we focus on user-wise performance to analyze how obfuscation effectiveness varies across individual authors. While LLMs are generally effective, we observe a bimodal distribution of efficacy, with performance varying significantly across users. To address this, we propose a personalized prompting method that outperforms standard prompting techniques and partially mitigates the bimodality issue.
Figures
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Ahmed Abbasi and Hsinchun Chen. 2008. Writeprints: A stylometric approach to identity-level identification and similarity detection in cyberspace. ACM Transactions on Information Systems (TOIS), 26(2):1--29
work page 2008
-
[4]
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774
arXiv 2023
-
[5]
Mishari Almishari, Ekin Oguz, and Gene Tsudik. 2014. Fighting authorship linkability with crowdsourcing. In Proceedings of the second ACM conference on Online social networks, pages 69--82
work page 2014
-
[6]
Malik Altakrori, Thomas Scialom, Benjamin CM Fung, and Jackie Chi Kit Cheung. 2022. A multifaceted framework to evaluate evasion, content preservation, and misattribution in authorship obfuscation techniques. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2391--2406
work page 2022
-
[7]
Janek Bevendorff, Martin Potthast, Matthias Hagen, and Benno Stein. 2019. Heuristic authorship obfuscation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1098--1108
work page 2019
-
[8]
Haohan Bo, Steven HH Ding, Benjamin Fung, and Farkhund Iqbal. 2019. Er-ae: Differentially private text generation for authorship anonymization. arXiv preprint arXiv:1907.08736
work page Pith review arXiv 2019
Show all 35 references
-
[9]
Daniel Castro-Castro, Reynier Ortega Bueno, and Rafael Munoz. 2017. Author masking by sentence transformation. CLEF (Working Notes), 40
2017
-
[10]
Tianqi Chen and Carlos Guestrin. 2016. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pages 785--794
2016
-
[11]
Evan Crothers, Nathalie Japkowicz, Herna Viktor, and Paula Branco. 2022. Adversarial robustness of neural-statistical features in detection of generative transformers. In 2022 International Joint Conference on Neural Networks (IJCNN), pages 1--8. IEEE
2022
-
[12]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human...
2019
-
[13]
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783
2024 arXiv
-
[14]
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2017. Hotflip: White-box adversarial examples for text classification. arXiv preprint arXiv:1712.06751
2017 arXiv
-
[15]
Oluwaseyi Feyisetan, Borja Balle, Thomas Drake, and Tom Diethe. 2020. Privacy-and utility-preserving textual analysis via calibrated multivariate perturbations. In Proceedings of the 13th international conference on web search and data mining, pages 178--186
2020
-
[16]
Jillian Fisher, Skyler Hallinan, Ximing Lu, Mitchell Gordon, Zaid Harchaoui, and Yejin Choi. 2024 a . Styleremix: Interpretable authorship obfuscation via distillation and perturbation of style elements. arXiv preprint arXiv:2408.15666
2024 arXiv
-
[17]
Jillian Fisher, Ximing Lu, Jaehun Jung, Liwei Jiang, Zaid Harchaoui, and Yejin Choi. 2024 b . Jamdec: Unsupervised authorship obfuscation using constrained decoding over small language models. arXiv preprint arXiv:2402.08761
2024 arXiv
-
[18]
Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. 2018. Black-box generation of adversarial text sequences to evade deep learning classifiers. In 2018 IEEE Security and Privacy Workshops (SPW), pages 50--56. IEEE
2018
-
[19]
Muhammad Haroon, Fareed Zaffar, Padmini Srinivasan, and Zubair Shafiq. 2021. Avengers ensemble! improving transferability of authorship obfuscation. arXiv preprint arXiv:2109.07028
2021 arXiv
-
[20]
Sergiu Hart. 1989. https://doi.org/10.1007/978-1-349-20181-5_25 Shapley Value , pages 210--216. Palgrave Macmillan UK, London
1989 doi
-
[21]
John A Hartigan and Pamela M Hartigan. 1985. The dip test of unimodality. The annals of Statistics, pages 70--84
1985
-
[22]
Georgi Karadzhov, Tsvetomila Mihaylova, Yasen Kiprov, Georgi Georgiev, Ivan Koychev, and Preslav Nakov. 2017. The case for being average: A mediocrity approach to style masking and author obfuscation: (best of the labs track at clef-2017). In Experimental IR Meets Multilingual...
2017
-
[23]
Yashwant Keswani, Harsh Trivedi, Parth Mehta, and Prasenjit Majumder. 2016. Author masking through translation. CLEF (Working Notes), 1609:890--894
2016
-
[24]
Asad Mahmood, Faizan Ahmad, Zubair Shafiq, Padmini Srinivasan, and Fareed Zaffar. 2019. A girl has no name: Automated authorship obfuscation using mutant-x. Proceedings on Privacy Enhancing Technologies
2019
-
[25]
Justus Mattern, Benjamin Weggenmann, and Florian Kerschbaum. 2022. The limits of word level differential privacy. arXiv preprint arXiv:2205.02130
2022 arXiv
-
[26]
Casey Meehan, Khalil Mrini, and Kamalika Chaudhuri. 2022. Sentence-level privacy for document embeddings. arXiv preprint arXiv:2205.04605
2022 arXiv
-
[27]
Fatemehsadat Mireshghallah and Taylor Berg-Kirkpatrick. 2021. Style pooling: Automatic text style obfuscation for improved classification fairness. arXiv preprint arXiv:2109.04624
2021 arXiv
-
[28]
Martin Potthast, Matthias Hagen, and Benno Stein. 2016. Author obfuscation: Attacking the state of the art in authorship verification. CLEF (Working Notes), pages 716--749
2016
-
[29]
Jonathan Schler, Moshe Koppel, Shlomo Argamon, and James W Pennebaker. 2006. Effects of age and gender on blogging. In AAAI spring symposium: Computational approaches to analyzing weblogs, volume 6, pages 199--205
2006
-
[30]
Yanir Seroussi, Ingrid Zukerman, and Fabian Bohnert. 2014. Authorship attribution with topic models. Computational Linguistics, 40(2):269--310
2014
-
[31]
Rakshith Shetty, Bernt Schiele, and Mario Fritz. 2018. \ A4NT \ : Author attribute anonymity by adversarial training of neural machine translation. In 27th USENIX Security Symposium (USENIX Security 18), pages 1633--1650
2018
-
[32]
Robin Staab, Mark Vero, Mislav Balunovi \'c , and Martin Vechev. 2023. Beyond memorization: Violating privacy via inference with large language models. arXiv preprint arXiv:2310.07298
2023 arXiv
-
[33]
Nafis Irtiza Tripto, Saranya Venkatraman, Dominik Macko, Robert Moro, Ivan Srba, Adaku Uchendu, Thai Le, and Dongwon Lee. 2023. A ship of theseus: curious cases of paraphrasing in llm-generated texts. arXiv preprint arXiv:2311.08374
2023 arXiv
-
[34]
Saiteja Utpala, Sara Hooker, and Pin Yu Chen. 2023. Locally differentially private document generation using zero shot prompting. arXiv preprint arXiv:2310.16111
2023 arXiv
-
[35]
Eric Xing, Saranya Venkatraman, Thai Le, and Dongwon Lee. 2024. Alison: Fast and effective stylometric authorship obfuscation. arXiv preprint arXiv:2402.00835
2024 arXiv
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.