REVIEW 4 major objections 5 minor 43 references
Evaluating Apple Intelligence's Writing Tools for Privacy Against Large Language Model-Based Inference Attacks: Insights from Early Datasets
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper presents the first empirical evaluation of Apple Intelligence's writing tools as privacy-preserving mechanisms, showing that the Friendly and Professional tone rewrites substantially degrade LLM-based emotion inference.
desk verdict A genuinely first empirical look at Apple Intelligence writing tools as an emotion-privacy defense, but the headline comparison leaks training data and the privacy claim outruns the evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the set of four on-device text-formatting modes offered by Apple Intelligence—Rewrite, Friendly, Professional, and Concise—applied to emotion-labeled sentences to see whether the rewritten text still betrays the original feeling. The argument runs on two new manually constructed evaluation datasets, one derived from the six-emotion Dair-AI Twitter corpus and one from the seven-emotion DailyDialog corpus, each with 40 instances per emotion per condition, plus a panel of seven attacker models spanning encoder-only, encoder-decoder, and decoder-only LLMs. The performance gap between original and rewritten texts is the evidence that the tools fail the attacker's inference models.
What would settle it
Fine-tune the same seven models on Apple-rewritten texts paired with their original emotion labels. If accuracy on held-out rewritten texts returns to near the original level, say above 80% for anger and surprise, the claim that Friendly and Professional rewriting provides emotional privacy would be refuted.
Extended reading notes
Core claim
The paper's central claim is that Apple Intelligence's tone-modification tools can serve as privacy-preserving mechanisms against LLM-based emotion inference attacks. Specifically, it claims that the Friendly and Professional modes significantly degrade the performance of adversarial emotion classifiers: accuracy that is near 100% on original texts often falls below 50%, and in some cases below 10%, after rewriting, across fine-tuned encoder models (BERT, RoBERTa, DistilBERT, DeBERTa), a sequence-to-sequence model (Flan T5), and prompt-engineered GPT-4o and DeepSeek R1. The Rewrite mode gives moderate degradation and Concise gives little, so the privacy benefit is concentrated in the emotion-neutralizing tone changes rather than in rewriting in general.
Load-bearing premise
The load-bearing premise is that a drop in a static emotion classifier's accuracy means the user's emotional state is actually concealed; an attacker who knows the rewriting tool and retrains, or who reads other cues in the text, might still infer the emotion.
Editorial extensions
If this is right
- If Friendly and Professional rewrites reliably hide emotions from current LLM classifiers, then users concerned about emotional privacy could default to those tone modes in Mail, Messages, and Notes.
- On-device tone rewriting offers a privacy mechanism that never requires the raw text to leave the device, complementing Apple's existing on-device processing approach.
- The near-zero effect of Concise suggests that simple shortening is not a privacy tool; only semantic and emotional rephrasing is, which guides which modes to expose in privacy settings.
- The results support building adaptive rewriting systems that selectively apply emotion-neutralizing rewrites only when the text carries sensitive emotional content.
- Similar tone-rewriting features in other AI assistants could be repositioned as privacy controls rather than purely stylistic features.
Reading between the lines
- The experiments test static classifiers on out-of-distribution rewrites; a determined adversary who knows the tool and fine-tunes on Apple-rewritten texts could plausibly recover much of the lost accuracy, so the privacy guarantee is likely weaker against adaptive attackers than the headline numbers suggest.
- Accuracy drop is a proxy, not proof, of emotional concealment; topic, named entities, and other cues may still leak emotion, so a human-judgment or content-based leakage test would strengthen the claim.
- A natural next experiment is to measure whether combining tone rewriting with other privacy techniques, such as differential privacy or paraphrase randomization across multiple tool outputs, defeats an adversary who trains on rewritten data.
- The datasets are small, with 40 instances per emotion per condition, and manually generated; scaling them or testing on long-form texts could reveal whether the effect persists beyond short sentences.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper evaluates whether Apple Intelligence's writing tools (Rewrite, Friendly, Professional, Concise) can serve as privacy-enhancing mechanisms against LLM-based emotion inference attacks. The authors manually construct two early evaluation datasets by applying these tools to 40 instances per emotion drawn from the Dair-AI Emotion and DailyDialog datasets. They then measure the accuracy and F1 of several fine-tuned encoder models (BERT, RoBERTa, DeBERTa, DistilBERT, Flan T5) and prompt-based models (GPT-4o, DeepSeek) on original versus rewritten texts. The central observational claim is that the Friendly and Professional tools substantially reduce classifier accuracy, while Rewrite and Concise have weaker effects. The paper interprets these accuracy drops as evidence that Apple Intelligence's writing tools can reduce emotional privacy leakage.
Significance. If the results are valid, this would be one of the first empirical studies of a deployed, system-wide text-rewriting feature as a privacy-enhancing technology, and the early datasets could be a useful resource for subsequent research. The paper covers a reasonable spread of model architectures, including encoder-only, sequence-to-sequence, and decoder-only LLMs, and it makes a concrete, falsifiable observational claim about specific tools. However, the significance is conditional: the current evaluation has methodological gaps that directly affect the size and interpretation of the reported effects, and the paper's privacy conclusion extends beyond what a static classifier accuracy drop can establish.
major comments (4)
- [Section 3.4 vs. Sections 3.3 and A.2] The evaluation instances are selected randomly from the train, test, and validation partitions of the same datasets used to train the attack models. Any instances drawn from the training partition are therefore seen verbatim during training, which can inflate the reported 'Original Text' accuracy through memorization. The rewritten texts are out-of-distribution, so the large accuracy drops for Friendly and Professional may partly reflect distribution shift rather than a genuine privacy-preserving transformation. The paper should evaluate on a held-out set of original texts that were never used in training, or use cross-validation, and report original-text accuracy on unseen instances separately.
- [Sections 4 and 5] The central conclusion equates a drop in emotion-classifier accuracy with an increase in emotional privacy. The threat model considers only static classifiers evaluated on out-of-distribution rewrites; it does not consider an attacker who knows the rewriting tool and retrains, or one who exploits other textual cues such as topic, length, or discourse markers. The observed accuracy drops are real distribution shifts but do not by themselves establish that emotional content is concealed. The paper should either reframe the claims as 'reduced accuracy under a fixed classifier threat model' or add an adaptive-attacker experiment.
- [Tables 8-11] Several reported percentages are inconsistent with the stated evaluation size of 40 instances per emotion category. For example, Table 8 reports BERT Anger accuracy of 95.12% and 35.32%, which cannot arise from 40 instances (the corresponding counts would be 38.048 and 14.128). Many other values, such as 86.45% and 9.34%, similarly imply fractional instance counts. Additionally, some rows are identical across all four rewriting tools (e.g., Table 8 GPT-4o Anger: 13.50 for Rewrite, Professional, Friendly, and Concise; DeepSeek Anger: 28.50 for all tools), which undermines the claim that tool-specific effects are being measured. These anomalies need to be resolved before the quantitative accuracy drops can be interpreted.
- [Section 3.4 and Section 6] With only 40 instances per emotion category and no confidence intervals or significance tests, the headline drops are statistically fragile. The paper acknowledges the small dataset size in Section 6, but this limitation is load-bearing because the main claim is a comparison of near-ceiling original accuracy (often 100%) to very low rewritten accuracy. For several GPT-4o and DeepSeek rows the original accuracy is already 45-80%, so the marginal 'privacy gain' is much smaller. The authors should report per-instance results, exact counts, and uncertainty estimates, and should avoid claiming that differences of a few instances are meaningful.
minor comments (5)
- [Section 3.4] The paper should specify how many of the 40 instances per emotion are drawn from train, test, and validation respectively, and should state whether duplicate texts across emotion categories were removed. For example, the same original text 'I talk to dogs as I feel they cannot understand words...' appears in both the Love and Joy tables (Tables 14 and 15), which may indicate duplication in the evaluation set.
- [Table 18] The Friendly output for 'Fine! I want a divorce!' is reported as 'NULL'. This should be explained; a null or failed rewrite should not be treated as a valid transformed text without discussion.
- [Table 12] The Professional rewrite of 'I feel MMF, and I can't be bothered to fight it' is given as 'I am experiencing a strong desire to masturbate...'. This output seems implausible and may be a transcription error or a data-quality issue; it should be verified and either corrected or excluded.
- [Section 4 and Figure captions] The text references 'Fig. 1(d) provides a graphical representation of attacker's inference models' performance on disgust emotions', but Figure 1 is for the Dair-AI dataset, which has no disgust category. Disgust appears only in the DailyDialog-based Figure 3(b). The figure/caption cross-references need to be corrected.
- [Throughout] The manuscript contains numerous typos and grammatical errors that impede readability, e.g., 'repsectively', 'quitre identical', 'the them inferences model's F1 scores get severe degradation'. A thorough copyedit is needed.
Circularity Check
No significant circularity; the accuracy-drop finding is a direct measurement, though an evaluation-leakage concern limits external validity.
full rationale
The paper's central result is an empirical measurement rather than a derived prediction. The fine-tuned inference models are external classifiers (BERT, RoBERTa, DeBERTa, DistilBERT, Flan T5), and the Apple-rewritten evaluation texts are newly generated inputs, not outputs of a model whose parameters encode the conclusion. No parameter is fitted to the evaluation set, no 'prediction' is computed from a fitted quantity, and no load-bearing self-citation or imported uniqueness theorem appears in the derivation. The main validity threat is that Section 3.4 samples evaluation instances 'from train, test and validation set randomly' from the same source corpora used to fine-tune the attack models, so original-text accuracy may be inflated by memorization when instances come from the training partition; this would make the Friendly/Professional accuracy drops look larger than they would be against genuinely unseen original texts. This is a confound or external-validity limitation, not a circular derivation, and the paper's own Section 6 flags the related limitation of small, manually constructed evaluation datasets (40 instances per emotion category) and the absence of comparison to other rewriting systems.
Assumptions & free parameters
assumptions (3)
- domain assumption Manual outputs of Apple Intelligence writing tools are representative of the deployed system's behavior on user text.
- domain assumption A decrease in emotion classification accuracy on rewritten text implies an increase in emotional privacy.
- domain assumption The emotion labels in Dair-AI and DailyDialog are valid ground truth for the author's emotional state.
Cite this review
Pith. "Pith review of Evaluating Apple Intelligence's Writing Tools for Privacy Against Large Language Model-Based Inference Attacks: Insights from Early Datasets." pith.science (2026). https://pith.science/paper/O4AJQPEO
@misc{pith2026250603870,
author = {Pith},
title = {Pith review of: Evaluating Apple Intelligence's Writing Tools for Privacy Against Large Language Model-Based Inference Attacks: Insights from Early Datasets},
year = {2026},
howpublished = {\url{https://pith.science/paper/O4AJQPEO}},
note = {Machine review of arXiv:2506.03870}
}
read the original abstract
The misuse of Large Language Models (LLMs) to infer emotions from text for malicious purposes, known as emotion inference attacks, poses a significant threat to user privacy. In this paper, we investigate the potential of Apple Intelligence's writing tools, integrated across iPhone, iPad, and MacBook, to mitigate these risks through text modifications such as rewriting and tone adjustment. By developing early novel datasets specifically for this purpose, we empirically assess how different text modifications influence LLM-based detection. This capability suggests strong potential for Apple Intelligence's writing tools as privacy-preserving mechanisms. Our findings lay the groundwork for future adaptive rewriting systems capable of dynamically neutralizing sensitive emotional content to enhance user privacy. To the best of our knowledge, this research provides the first empirical analysis of Apple Intelligence's text-modification tools within a privacy-preservation context with the broader goal of developing on-device, user-centric privacy-preserving mechanisms to protect against LLMs-based advanced inference attacks on deployed systems.
Figures
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258
arXiv 2021
-
[4]
Karen L Boyd and Nazanin Andalibi. 2023. Automated emotion recognition in the workplace: How proposed technologies reveal potential futures of work. Proceedings of the ACM on human-computer interaction, 7(CSCW1):1--37
work page 2023
-
[5]
Martin Juan Jos \'e Bucher and Marco Martini. 2024. Fine-tuned'small'llms (still) significantly outperform zero-shot generative ai models in text classification. arXiv preprint arXiv:2406.08660
arXiv 2024
-
[6]
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2024. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology, 15(3):1--45
2024
-
[7]
Yanran Chen and Steffen Eger. 2025. Do emotions really affect argument convincingness? a dynamic approach with llm-based manipulation checks. arXiv preprint arXiv:2503.00024
work page Pith review arXiv 2025
-
[8]
Badhan Chandra Das, M Hadi Amini, and Yanzhao Wu. 2025. Security and privacy challenges of large language models: A survey. ACM Computing Surveys, 57(6):1--39
work page 2025
Show all 43 references
-
[9]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human...
2019
-
[10]
Rui Fan, Shu Li, Tingting He, and Yu Liu. 2025. https://aclanthology.org/2025.coling-main.210/ Aspect-based sentiment analysis with syntax-opinion-sentiment reasoning chain . In Proceedings of the 31st International Conference on Computational Linguistics, pages 3123--3137, Ab...
2025
-
[11]
Tom Gunter, Zirui Wang, Chong Wang, Ruoming Pang, Andy Narayanan, Aonan Zhang, Bowen Zhang, Chen Chen, Chung-Cheng Chiu, David Qiu, et al. 2024. Apple intelligence foundation language models. arXiv preprint arXiv:2407.21075
2024
-
[12]
Jochen Hartmann, Mark Heitmann, Christian Siebert, and Christina Schamp. 2023. More than a feeling: Accuracy and application of sentiment analysis. International Journal of Research in Marketing, 40(1):75--87
2023
-
[13]
Bui Thanh Hung and Nguyen Hoang Minh Thu. 2024. Novelty fused image and text models based on deep neural network and transformer for multimodal sentiment analysis. Multimedia Tools and Applications, 83(25):66263--66281
2024
-
[14]
Lindrit Kqiku, Marvin K \"u hn, and Delphine Reinhardt. 2022. From sentiment to sensitivity: The role of emotions on privacy exposure in twitter. In Proceedings of the 2022 Workshop on Open Challenges in Online Social Networks, pages 10--15
2022
-
[15]
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017. https://aclanthology.org/I17-1099/ D aily D ialog: A manually labelled multi-turn dialogue dataset . In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume ...
2017
-
[16]
Aiwei Liu, Qiang Sheng, and Xuming Hu. 2024 a . Preventing and detecting misinformation generated by large language models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 3001--3004
2024
-
[17]
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692
2019 arXiv
-
[18]
Zhiwei Liu, Kailai Yang, Qianqian Xie, Tianlin Zhang, and Sophia Ananiadou. 2024 b . Emollms: A series of emotional large language models and annotation tools for comprehensive affective analysis. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data...
2024
-
[19]
Massimiliano Luca, Gabriel Lopez, Antonio Longa, and Joe Kaul. 2024. How are you really doing? dig into the wheel of emotions with large language models. In 2024 Artificial Intelligence for Business (AIxB), pages 72--75. IEEE
2024
-
[20]
Weicheng Ma, Chunyuan Deng, Aram Moossavi, Lili Wang, Soroush Vosoughi, and Diyi Yang. 2024. https://doi.org/10.18653/v1/2024.findings-acl.162 Simulated misinformation susceptibility ( SMISTS ): Enhancing misinformation research with large language model simulations . In Findi...
2024 doi
-
[21]
Rui Mao, Qian Liu, Kai He, Wei Li, and Erik Cambria. 2022. The biases of pre-trained language models: An empirical study on prompt-based sentiment analysis and emotion detection. IEEE transactions on affective computing, 14(3):1743--1753
2022
-
[22]
Stephen Meisenbacher, Nihildev Nandakumar, Alexandra Klymenko, and Florian Matthes. 2024. https://aclanthology.org/2024.lrec-main.16/ A comparative analysis of word-level metric differential privacy: Benchmarking the privacy-utility trade-off . In Proceedings of the 2024 Joint...
2024
-
[23]
Md Saef Ullah Miah, Md Mohsin Kabir, Talha Bin Sarwar, Mejdl Safran, Sultan Alfarhood, and MF Mridha. 2024. A multimodal approach to cross-lingual sentiment analysis with ensemble of transformer and llm. Scientific Reports, 14(1):9603
2024
-
[24]
Eldesouki, and Yasser Hifny
Ahmad Mortadi, Waleed Nazih, Mohamed I. Eldesouki, and Yasser Hifny. 2025. Intelligent de-identification of medical discharge summaries using hybrid nlp techniques. ACM Transactions on Asian and Low-Resource Language Information Processing
2025
-
[25]
Maximilian Mozes, Xuanli He, Bennett Kleinberg, and Lewis D Griffin. 2023. Use of llms for illicit purposes: Threats, prevention measures, and vulnerabilities. arXiv preprint arXiv:2308.12833
2023 arXiv
-
[26]
David Pissarra, Isabel Curioso, Jo \ a o Alveira, Duarte Pereira, Bruno Ribeiro, Tom \'a s Souper, Vasco Gomes, Andr \'e Carreiro, and Vitor Rolla. 2024. https://aclanthology.org/2024.privatenlp-1.8/ Unlocking the potential of large language models for clinical text anonymizat...
2024
-
[27]
Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020. Pre-trained models for natural language processing: A survey. Science China technological sciences, 63(10):1872--1897
2020
-
[28]
Alec Radford and Karthik Narasimhan. 2018. https://api.semanticscholar.org/CorpusID:49313245 Improving language understanding by generative pre-training
2018
-
[29]
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. https://api.semanticscholar.org/CorpusID:160025533 Language models are unsupervised multitask learners
2019
-
[30]
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1--67
2020
-
[31]
Margarita Rodr \' guez-Ib \'a nez, Antonio Cas \'a nez-Ventura, F \'e lix Castej \'o n-Mateos, and Pedro-Manuel Cuenca-Jim \'e nez. 2023. A review on sentiment analysis from social media platforms. Expert Systems with Applications, 223:119862
2023
-
[32]
Kat Roemmich, Florian Schaub, and Nazanin Andalibi. 2023. Emotion ai at work: Implications for workplace surveillance, emotional labor, and emotional privacy. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1--20
2023
-
[33]
Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. 2018. https://doi.org/10.18653/v1/D18-1404 CARER : Contextualized affect representations for emotion recognition . In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Proc...
2018 doi
-
[34]
Lei Shu, Liangchen Luo, Jayakumar Hoskere, Yun Zhu, Yinxiao Liu, Simon Tong, Jindong Chen, and Lei Meng. 2024. Rewritelm: An instruction-tuned large language model for text rewriting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18970--18980
2024
-
[35]
Yanqi Song, Ruiheng Liu, Shu Chen, Qianhao Ren, Yu Zhang, and Yongqi Yu. 2024. Securesql: Evaluating data leakage of large language models as natural language interfaces to databases. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 5975--5990
2024
-
[36]
Samuel Sousa and Roman Kern. 2022. https://doi.org/10.1007/s10462-022-10204-6 How to keep text private? a systematic review of deep learning methods for privacy-preserving natural language processing . Artif. Intell. Rev., 56(2):1427–1492
2022 doi
-
[37]
Xiaofei Sun, Xiaoya Li, Shengyu Zhang, Shuhe Wang, Fei Wu, Jiwei Li, Tianwei Zhang, and Guoyin Wang. 2023. Sentiment analysis through llm negotiations. arXiv preprint arXiv:2311.01876
2023 arXiv
-
[38]
Yan Wang, Wei Song, Wei Tao, Antonio Liotta, Dawei Yang, Xinlei Li, Shuyong Gao, Yixuan Sun, Weifeng Ge, Wei Zhang, et al. 2022. A systematic review on affective computing: Emotion models, databases, and recent advances. Information Fusion, 83:19--52
2022
-
[39]
Aijia Yuan, Edlin Garcia Colato, Bernice Pescosolido, Hyunju Song, and Sagar Samtani. 2025. Improving workplace well-being in modern organizations: A review of large language model-based mental health chatbots. ACM Transactions on Management Information Systems, 16(1):1--26
2025
-
[40]
Marcos Zampieri, Damith Premasiri, and Tharindu Ranasinghe. 2024. https://aclanthology.org/2024.trac-1.2/ A federated learning approach to privacy preserving offensive language identification . In Proceedings of the Fourth Workshop on Threat, Aggression & Cyberbullying @ LREC-...
2024
-
[41]
Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing. 2023. Sentiment analysis in the era of large language models: A reality check. arXiv preprint arXiv:2305.15005
2023 arXiv
-
[42]
Zixing Zhang, Liyizhe Peng, Tao Pang, Jing Han, Huan Zhao, and Bj \"o rn W Schuller. 2024. Refashioning emotion recognition modelling: The advent of generalised large models. IEEE Transactions on Computational Social Systems
2024
-
[43]
Yang Zhao, Masayasu Muraoka, Issei Yoshida, Bishwaranjan Bhattacharjee, and Hiroshi Kanayama. 2025. https://aclanthology.org/2025.coling-main.107/ A simple-yet-efficient instruction augmentation method for zero-shot sentiment classification . In Proceedings of the 31st Interna...
2025
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.