REVIEW 4 major objections 5 minor 33 references
Recipient Profiling: Predicting Characteristics from Messages
T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The gender of a message's intended recipient can be predicted from text alone.
desk verdict New task formulation worth a look; the SWDA evidence may be an artifact, so the feasibility claim rests on MDC and TIC. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machine that carries the argument is the recipient-profiling classifier $f: \mathcal{U}_{A\to R} \to S$, a map from messages addressed by authors to recipients into a sensitive-attribute space such as gender. Because individual messages are often too short to be informative, the paper concatenates utterances from the same conversation to a minimum length before classification. The empirical machinery consists of fine-tuning three pretrained text encoders—BERT, MPNet, and DeBERTa—with a single linear classification layer, training and testing on recipient-disjoint splits so that no recipient appears in more than one split, and scoring with balanced accuracy. The transfer experiments then ask whether a classifier trained on one corpus still classifies recipients in another, testing whether the learned signal is domain-general.
What would settle it
Mask all direct identifiers in the test messages—recipient names, gendered honorifics and titles, and third-person references to the recipient—then retrain and evaluate the same models. If balanced accuracy drops to chance on all three datasets, the claimed recipient-specific signal beyond direct identifiers is not supported; if it stays above chance, the accommodation signal is real.
Extended reading notes
Core claim
The paper claims that messages carry a recoverable signal about the intended recipient, not just the author, and that this signal is strong enough to classify the recipient's gender above chance from text alone. Fine-tuned BERT, MPNet, and DeBERTa models reach balanced accuracies of about 0.77 on Switchboard, 0.60 on the Movie Dialog Corpus, and 0.68 on the Tennis Interviews Corpus, and models trained on one corpus exceed chance (0.51–0.58) on the other two. The authors take this as evidence that recipient profiling is feasible and that it constitutes a new privacy risk inherent in textual messages, one that runs parallel to the better-studied risk of author profiling.
Load-bearing premise
The above-chance accuracy is evidence about recipients only if it comes from how people write to different recipients, not from direct identifiers such as recipient names, 'sir'/'ma'am', or other labels already present in the message; the paper states that it did not check this distinction.
Editorial extensions
If this is right
- Recipient gender classification is feasible on at least three distinct conversation genres, so recipient profiling can be treated as a measurable task rather than a hypothetical one.
- Because models transfer across datasets at above-chance accuracy, at least part of the signal generalizes beyond a single corpus's vocabulary or topic distribution.
- Message collections that include recipient metadata now count as privacy-relevant in the same way author-labeled text does, and recipient-side anonymization becomes a concrete design target.
- Recipient-profiling models offer a quantitative probe for communication accommodation theory, turning a documented stylistic phenomenon into a testable predictor.
Reading between the lines
- The paper leaves untested whether the signal comes from direct identifiers (names, 'sir'/'ma'am') or from indirect stylistic accommodation; masking those direct identifiers and retraining would show which channel drives the result.
- The higher accuracy for female recipients is left unexplained; comparing messages whose topics are held fixed across recipient genders would indicate whether the asymmetry is about the recipient or about topic conventions.
- Transfer accuracies sit only slightly above chance, so the practical privacy risk may be domain-dependent; a realistic attacker evaluation would need messages from unseen communication channels, not just held-out recipients from the same corpus.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a new NLP task, Recipient Profiling: predicting a sensitive attribute (here, gender) of the intended recipient of a message, as opposed to the author. The authors formalize the task, then provide proof-of-concept experiments on three conversation datasets (SWDA, MDC, TIC) using three pretrained encoders (BERT, MPNet, DeBERTa) fine-tuned with a linear classification head. They report above-chance balanced accuracy on all same-domain settings, with the largest effects on SWDA (0.75–0.77) and TIC (0.67–0.68), and weaker but above-chance cross-dataset transfer (0.51–0.58). The paper also reports a female-vs-male accuracy gap and a between-model agreement analysis. The authors frame the result as evidence that messages leak information about the recipient, raising new privacy concerns.
Significance. If the results hold, the paper introduces a genuinely new task that is a natural counterpart to author profiling and has plausible privacy implications. The authors provide a clean task formalization, use recipient-based held-out splits, report balanced accuracy with standard deviations over three seeds, and release the preprocessed datasets on HuggingFace, which supports reproducibility. The central claim, however, is stronger than the evidence: the above-chance accuracy could be driven by trivial cues (recipient names, titles) or by a preprocessing artifact in SWDA, and the paper explicitly acknowledges that direct vs. indirect identifiers were not separated. The significance therefore hinges on resolving these confounds; as it stands, the paper is a useful proof-of-concept of the task's feasibility, but the privacy-leak interpretation is not yet established.
major comments (4)
- [Section 4 (SWDA preprocessing)] The preprocessing description states that 'we concatenated segments from the same conversations until we reached a limit of 1,000 characters' without specifying that the concatenation is restricted to a single speaker's turns. In the two-party Switchboard conversations, consecutive dialogue acts normally alternate between the two speakers; if each 1,000-character chunk contains both the author's and the recipient's utterances, then the classifier can infer the recipient's gender from the recipient's own words, turning the SWDA experiment into author profiling rather than recipient profiling. This would also explain why SWDA accuracy (0.75–0.77) is far above MDC (0.59–0.60) and TIC (0.67–0.68). Please clarify whether the concatenation grouped by speaker, and ideally add an experiment that removes the recipient's own turns from the input (or reports results on author-only segments) to confirm that the signal resides in the author's messages.
- [Sections 5.1, 6, and 7] The conclusion that recipient profiling 'constitutes a new privacy risk' assumes that the predictions are based on indirect linguistic accommodation rather than direct identifiers such as recipient names, pronouns, or titles like 'sir'/'ma'am'. Section 7 explicitly acknowledges that the paper 'did not investigate ... whether their predictions were due to direct or indirect identifiers.' This is a load-bearing gap because direct identifiers would make the task trivial and would not support the stated accommodation-based interpretation. I recommend adding a simple ablation (e.g., masking names and gendered address terms, or comparing against a model trained only on such features) or explicitly rephrasing the privacy claims to reflect that the source of the signal is unknown.
- [Section 5.2] The cross-dataset transfer claim rests on balanced accuracies in the narrow interval [0.51, 0.58] being 'better than chance,' but no significance tests are reported and the text does not list the standard deviations for the transfer matrix (they appear only in the figure). With three seeds, a 0.51 mean may not be reliably above 0.5. The authors should report confidence intervals or a per-cell permutation/binomial test against chance to support the transferability claim.
- [Section 5 (overall)] Throughout the results, the evidence for 'above-chance' performance is limited to means and standard deviations over three seeds. No statistical significance tests are provided for any comparison to chance or between models. Given that the central claims are 'better-than-chance' outcomes, adding a minimal significance test (e.g., a bootstrap or permutation test on the balanced accuracy) would materially strengthen the conclusions.
minor comments (5)
- [Section 7] The sentence 'There is two main aspects to our work' should be 'There are two main aspects'.
- [Section 4] The split description says '80% of recipients in the train set and 20% distributed in the test and validation sets,' but Table 1 gives 352/18/70 recipients out of 440, which is 80.0% train, 4.1% validation, and 15.9% test; please clarify the exact proportions.
- [Abstract and Section 1] The data link appears as '[data:HF-datasets ]' with an incomplete bracket; ensure the URL and its formatting are correct in the final version.
- [Equation (1)] The inline expression for kappa is ambiguous: 'κi,j = Pi,j − Ri,j / 1 − Ri,j' should be properly typeset as (Pi,j − Ri,j)/(1 − Ri,j).
- [Figure 3] The caption contains a typo: 'The models were train and test with in the same domain' should be 'trained and tested in the same domain.'
Circularity Check
No significant circularity: the recipient profiling result is an empirical feasibility finding evaluated on held-out recipients and cross-domain transfer, with no equation-level reduction to its inputs.
full rationale
The paper's central claim is an empirical feasibility result, not an analytic derivation. The evaluation uses a recipient-based group split (Section 4: 'We split the datasets into train, validation, and test sets with a recipient-based group split, so that no recipient appears in multiple splits, to prevent overfitting'), held-out test recipients, standard supervised fine-tuning, and cross-dataset transfer experiments. No fitted parameter is renamed as a prediction, and no equation in the paper reduces the reported balanced accuracy to its own input by construction. The only self-citation, Sileo (2023) as the DeBERTa training source, is a model choice rather than a load-bearing premise, and the same conclusion is obtained with BERT and MPNet as well. The Section 7 limitation that the authors 'did not investigate ... whether their predictions were due to direct or indirect identifiers' is an interpretability and validity caveat, not a circularity, because it does not make the measured result equivalent to the training signal. The possible SWDA preprocessing concern raised by the reviewer, in which concatenating consecutive segments from a two-party conversation could mix author and recipient turns, is a data-construction artifact that could threaten the empirical claim, but it is not a derivation that reduces to its own input, so it does not raise the circularity score under the stated rules. The derivation chain is therefore self-contained with respect to circularity.
Assumptions & free parameters
free parameters (3)
- SWDA concatenation length threshold =
1000 characters
- Majority-class subsampling =
random subsample to class size
- Fine-tuning hyperparameters =
learning rate 2e-5, 3 epochs, Adam
assumptions (3)
- domain assumption Recipients are labeled with an accurate binary gender in all three datasets.
- domain assumption The concatenated or raw message text preserves the signal about the recipient.
- ad hoc to paper A linear classification head on a pretrained text encoder is an adequate model family for the task.
Cite this review
Pith. "Pith review of Recipient Profiling: Predicting Characteristics from Messages." pith.science (2026). https://pith.science/paper/6W36WA6V
@misc{pith2026241212954,
author = {Pith},
title = {Pith review of: Recipient Profiling: Predicting Characteristics from Messages},
year = {2026},
howpublished = {\url{https://pith.science/paper/6W36WA6V}},
note = {Machine review of arXiv:2412.12954}
}
read the original abstract
It has been shown in the field of Author Profiling that texts may inadvertently reveal sensitive information about their authors, such as gender or age. This raises important privacy concerns that have been extensively addressed in the literature, in particular with the development of methods to hide such information. We argue that, when these texts are in fact messages exchanged between individuals, this is not the end of the story. Indeed, in this case, a second party, the intended recipient, is also involved and should be considered. In this work, we investigate the potential privacy leaks affecting them, that is we propose and address the problem of Recipient Profiling. We provide empirical evidence that such a task is feasible on several publicly accessible datasets (https://huggingface.co/datasets/sileod/recipient_profiling). Furthermore, we show that the learned models can be transferred to other datasets, albeit with a loss in accuracy.
Figures
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Bartholomew Ayeni. 2021. https://doi.org/10.7575/aiac.ijalel.v.10n.1p.80 Language choices and its effect in a culturally diversified nigeria business places: Adopting giles’ communication accommodation theory . International Journal of Applied Linguistics and English Literature
-
[4]
Brau, James Cicon, and Stephen R
James C. Brau, James Cicon, and Stephen R. Owen. 2022. https://doi.org/10.3390/ijfs10020025 A textual analysis of logograms in chinese ipo roadshows: How agreement between investors and management relates to pricing and performance . International Journal of Financial Studies, 10(2):25
-
[5]
K.H. Brodersen, C.S. Ong, K.E. Stephan, and J.M. Buhmann. 2010. The balanced accuracy and its posterior distribution. In Proceedings of the 20th International Conference on Pattern Recognition, pages 3121--3124
work page 2010
-
[6]
Sandra Cornbleet and Ronald Carter. 2002. The Language of Speech and Writing. Routledge. Creative Commons license (2001). Available at http://ndl.ethernet.edu.et/bitstream/123456789/8674/1/46.pdf.pdf
arXiv 2001
-
[7]
Cristian Danescu-Niculescu-Mizil and Lillian Lee. 2011. Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, ACL 2011
2011
-
[8]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 Bert: Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and...
Show all 33 references
-
[9]
P Kingma Diederik. 2014. Adam: A method for stochastic optimization. (No Title)
2014
-
[10]
Sara El Manar and Ismail Kassou. 2014. Authorship analysis studies: A survey. International Journal of Computer Applications, 86(12):22--29
2014
-
[11]
Ma \"e l Fabien, Esau Villatoro-Tello, Petr Motlicek, and Shantipriya Parida. 2020. https://aclanthology.org/2020.icon-main.16 B ert AA : BERT fine-tuning for authorship attribution . In Proceedings of the 17th International Conference on Natural Language Processing (ICON), pa...
2020
-
[12]
Simona Fer. 2018. https://doi.org/10.2139/ssrn.3128115 Verbal communication as a two-way process in connecting people . SSRN Electronic Journal. Available at SSRN
2018 doi
-
[13]
Jillian Fisher, Ximing Lu, Jaehun Jung, Liwei Jiang, Zaid Harchaoui, and Yejin Choi. 2024. https://doi.org/10.18653/v1/2024.naacl-long.87 JAMDEC : Unsupervised authorship obfuscation using constrained decoding over small language models . In Proceedings of the 2024 Conference ...
2024 doi
-
[14]
Frey and Derek R
Timothy K. Frey and Derek R. Lane. 2021. https://doi.org/10.1080/03634523.2021.1903521 Cat in the classroom: A multilevel analysis of students’ experiences with instructor nonaccommodation . Communication Education, 70(3):223--246
2021
-
[15]
Liye Fu, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2016. Tie-breaker: Using language models to quantify gender bias in sports journalism. In Proceedings of the IJCAI workshop on NLP meets Journalism
2016
-
[16]
Edwards, and Joseph B
Howard Giles, America L. Edwards, and Joseph B. Walther. 2023. https://doi.org/10.1016/j.langsci.2023.101571 Communication accommodation theory: Past accomplishments, current trends, and future prospects . Language Sciences, 99:101571. Accessed: 2024-08-14
2023
-
[17]
Howard Giles and Tania Ogay. 2007. Communication accommodation theory. In B. B. Whaley and W. Samter, editors, Explaining Communication: Contemporary Theories and Exemplars, pages 293--310. Lawrence Erlbaum Associates Publishers
2007
-
[18]
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. https://arxiv.org/abs/2006.03654 Deberta: Decoding-enhanced bert with disentangled attention . Preprint, arXiv:2006.03654
2021 arXiv
-
[19]
Corina Koolen and Andreas van Cranenburgh. 2017. https://doi.org/10.18653/v1/W17-1602 These are not the stereotypes you are looking for: Bias and fairness in authorial gender attribution . In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing , page...
2017 doi
-
[20]
Miroslav Kubat and Stan Matwin. 1997. Addressing the curse of imbalanced training sets: One-sided selection. In Proceedings of the Fourteenth International Conference on Machine Learning (ICML '97), pages 179--186, San Francisco, CA, USA. Morgan Kaufmann
1997
-
[21]
Pierre Lison, Ildik \'o Pil \'a n, David S \'a nchez, Montserrat Batet, and Lilja vrelid. 2021. Anonymisation models for text data: State of the art, challenges and future directions. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics an...
2021
-
[22]
Gabriel Loiseau, Damien Sileo, Damien Riquet, Maxime Meyer, and Marc Tommasi. 2024. Tarot: Task-oriented authorship obfuscation using policy optimization methods. arXiv preprint arXiv:2407.21630
2024 arXiv
-
[23]
Nuno Mamede, Jorge Baptista, and Francisco Dias. 2016. Automated anonymization of text documents. In 2016 IEEE congress on evolutionary computation (CEC), pages 1287--1294. IEEE
2016
-
[24]
Mary L. McHugh. 2012. Interrater reliability: The kappa statistic. Biochemia Medica (Zagreb), 22(3):276--282
2012
-
[25]
Babatunde Onikoyi, Nonso Nnamoko, and Ioannis Korkontzelos. 2023. https://doi.org/10.1016/j.nlp.2023.100018 Gender prediction with descriptive textual data using a machine learning approach . Natural Language Processing Journal, 4:100018. Accessed: 2024-08-12
2023
-
[26]
Sarra Ouni, Faten Fkih, and Mohamed Nazih Omri. 2023. https://doi.org/10.1007/s11042-023-14711-8 A survey of machine learning-based author profiling from texts analysis in social networks . Multimedia Tools and Applications, 82:36653--36686
2023 doi
-
[27]
Carla Perez-Almendros, Luis Espinosa-Anke, and Steven Schockaert. 2022. https://doi.org/10.18653/v1/2022.semeval-1.38 S em E val-2022 task 4: Patronizing and condescending language detection . In Proceedings of the 16th International Workshop on Semantic Evaluation (SemEval-20...
2022 doi
-
[28]
Francisco Rangel and Paolo Rosso. 2013. https://www.semanticscholar.org/paper/Use-of-Language-and-Author-Profiling- In 7th International Conference on Corpus Linguistics. Accessed: 2024-07-09
2013
-
[29]
Francisco Rangel, Paolo Rosso, Moshe Koppel, Efstathios Stamatatos, and Giacomo Inches. 2013. Overview of the author profiling task at pan 2013. In CLEF conference on multilingual and multimodal information access evaluation, pages 352--365. CELCT
2013
-
[30]
Damien Sileo. 2023. https://arxiv.org/abs/2301.05948 tasksource: Structured dataset preprocessing annotations for frictionless extreme multi-task learning and evaluation . arXiv preprint arXiv:2301.05948
2023 arXiv
-
[31]
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. https://arxiv.org/abs/2004.09297 Mpnet: Masked and permuted pre-training for language understanding . Preprint, arXiv:2004.09297
2020 arXiv
-
[32]
Stolcke, N
A. Stolcke, N. Coccaro, R. Bates, P. Taylor, C. Van Ess-Dykema, K. Ries, E. Shriberg, D. Jurafsky, R. W. Martin, and M.. Meteer. 2000. https://doi.org/10.1162/089120100561737 Dialogue act modeling for automatic tagging and recognition of conversational speech . Computational L...
-
[33]
Brian M. Watson. 2020. https://doi.org/10.1007/978-3-030-41668-3_9 Communication accommodation theory as an intervention tool to improve interprofessional practice in healthcare . In Louise Mullany, editor, Professional Communication. Communicating in Professions and Organizat...
2020 doi
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.