Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

LLM-as-a-Judge for Privacy Evaluation? Exploring the Alignment of Human and LLM Perceptions of Privacy in Textual Data

T0 review · 3 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper argues that LLMs can judge how private a text is, because their ratings align with the average human rating per text more closely than human raters align with each other.

desk verdict First sizable empirical test of LLM-as-a-Judge for privacy, with good artifacts; the central alignment claim needs within-dataset per-text agreement before it fully lands. read the letter →

arxiv 2508.12158 v1 pith:TDVA2IKR submitted 2025-08-16 cs.CL

Stephen Meisenbacher , Alexandra Klymenko , Florian Matthes This is my paper
classification cs.CL
keywords privacyevaluationLLM-as-a-Judgesensitivitytextdatahumanperceptioninter-rateragreementKrippendorff'salphauserstudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether LLMs can replace human raters as judges of how private or sensitive a piece of text is. To answer it, the authors built a set of 250 user-written texts from 10 domains, had 13 LLMs rate each text on a five-point privacy-sensitivity scale, and ran a survey in which 677 people each rated 20 of the texts. They report that human raters agree with each other only weakly, with an inter-rater agreement of 0.39 on Krippendorff's alpha, while the LLMs' ratings align much more closely with the average human rating for each text. The conclusion is that LLM-as-a-Judge is a feasible, low-cost way to approximate a global human judgment of text privacy, although it cannot capture individual or demographic variation. This matters because privacy-preserving NLP currently lacks scalable, human-grounded evaluation methods.

What carries the argument

The load-bearing mechanism is a two-level comparison built on a five-point Likert scale of privacy sensitivity, anchored by descriptions from '1: harmless' to '5: extremely private'. Agreement is measured twice: once between each LLM and the rounded average human rating per text, and once pairwise between an LLM and individual human raters. The first measure produces the paper's positive result; the second reveals its limits. Krippendorff's alpha is used throughout, chosen because it handles the between-subjects survey design in which each participant rated only 20 of the 250 texts. An improved prompt with an explicit scale and requested rationale gives higher inter-LLM agreement (0.58 vs 0.

What would settle it

A replication in which each text is rated by at least 200 humans and by LLMs, measuring agreement with unrounded averages and with medians, would settle the claim: if LLM-to-average agreement is no better than average human-to-human agreement once sampling noise is removed, or if rounding to integers creates the apparent alignment, then the claim that LLMs capture a global privacy perspective fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that, under its operationalization of privacy as content sensitivity on a 1–5 scale, LLMs can accurately model a global human privacy perspective. Across 250 texts, the 13 LLMs' averaged ratings track the per-text average human rating (about 55 raters per text) substantially better than individual humans track each other; for every dataset, at least one LLM matches the human average within ±0.1. This holds even though LLMs as a group skew toward higher, more private scores than humans, and even though pairwise LLM–human agreement is only comparable to pairwise human–human agreement. The authors interpret the high agreement with the average as evidence that LLMs c

Load-bearing premise

The result depends on treating the rounded per-text average of about 55 human ratings as a meaningful 'global human privacy perspective,' even though human raters disagree with one another substantially (Krippendorff's alpha = 0.39, with per-text standard deviations around 1.3 points).

Editorial extensions

If this is right

  • If LLM-as-a-Judge works for privacy, privacy-preserving NLP evaluation could shift from proxy metrics such as attack success or semantic distance to human-grounded sensitivity ratings at far lower cost: under $20 for 16,250 LLM responses versus £2,031 for the 13,540 human ratings.
  • Model choice matters: proprietary and large open models such as Llama-3.3-70B and gemma-3-12b align with humans, while 1B–4B open models do not, so reliable local privacy judging still requires capable hardware or API access.
  • The improved prompt with an explicit scale and requested rationale should be the default for privacy judging, since it raises inter-LLM agreement from 0.54 to 0.58 and is the configuration used in human comparisons.
  • Domain is a first-order factor: mental-health, medical, and confession-style texts are rated most private, while reviews and tweets are rated least, so privacy evaluation should be domain-stratified.
  • LLM judges are best used to estimate aggregate privacy sensitivity rather than to stand in for any individual user's privacy opinion, because pairwise LLM–human agreement is no better than pairwise human–human agreement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not run: check whether LLM ratings predict the full distribution of human votes, such as the fraction of 4–5 ratings, and not just the mean; if they only hit the average, they may be reproducing central tendency rather than a genuinely shared privacy concept.
  • Because the survey sample is dominated by Europe (42%) and Africa (35%), the 'global human perspective' is a weighted average; re-weighting by country or testing LLM agreement within demographic subgroups would show how global the alignment really is.
  • The adversarial-confidence sampling may inflate the spread of text sensitivity in ways that make mean-agreement easier; an unselected random corpus would test whether the LLM-to-average advantage persists at natural base rates.
  • Human reasoning themes (embarrassment, 'private or public', risk of harm) are more varied than LLM reasoning (sensitivity plus identifiability), suggesting an LLM privacy judge with explicit rationale could double as an explainable privacy labeler, though the paper does not claim this.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper investigates whether LLM-as-a-Judge can be used to evaluate the privacy sensitivity of textual data. It compares privacy ratings from 13 LLMs (two prompt variants, five runs each) with ratings from 677 human participants on 250 texts drawn from 10 datasets. The paper reports low inter-human agreement (Krippendorff's alpha = 0.39), higher inter-LLM agreement with the improved prompt (0.58), and claims that LLMs align strongly with the average human rating per text while showing lower pairwise agreement with individual humans. It additionally analyzes human and LLM reasoning patterns and discusses cost and feasibility of LLM-based privacy evaluation.

Significance. If the central claim holds, this is a valuable contribution to privacy-preserving NLP: it would be the first systematic evaluation of LLM-as-a-Judge for privacy sensitivity, backed by a large human study (677 participants, 10 datasets, 13 LLMs) and public release of prompts, survey application, and data. The paper also offers a useful qualitative comparison of human and LLM reasoning about privacy. The empirical design is strong: adversarial text sampling, attention checks, five-run LLM averaging, and careful survey administration. However, the main alignment claim currently rests on pooled, dataset-level comparisons, and the manuscript does not report the per-text, within-dataset agreement that a privacy judge would need. The conclusions are therefore not yet fully supported, though the missing analysis is directly checkable from the released artifacts.

major comments (3)
  1. [§4.3, Fig 4, Table 3] The headline claim that LLMs 'accurately model a global human privacy perspective' is based on LLM agreement with the average human rating per text, computed over all 250 texts pooled across 10 datasets. Because dataset-level human means range from 1.4 to 3.3 (Table 3) and the sample was deliberately selected to span the full vulnerability range (§3.1.2), high pooled agreement can be driven primarily by dataset identity (e.g., mental-health posts vs. Yelp reviews) rather than by accurate per-text ordering. The manuscript reports no within-dataset per-text agreement between LLM ratings and human means. Since a privacy judge must rate individual texts, the central claim requires within-dataset per-text alignment. This is a missing-analysis concern, not a claim that the result is false; it is directly checkable from the released data and prompts. Please report per-dataset agreement (e.g., p
  2. [§4.2, §4.3, §5.1] The comparison treats the rounded average of ~55 human ratings per text as a meaningful 'global human privacy perspective,' despite inter-human Krippendorff's alpha of only 0.39. Table 3 reports dataset-level means and their standard deviations, but the paper does not characterize the per-rater dispersion or the stability of the per-text means. More importantly, no human-to-mean baseline is computed: the paper does not report how well individual human raters agree with the mean of the remaining raters (e.g., split-half reliability or leave-one-out agreement). If LLM-to-mean agreement is not higher than this human-to-mean baseline, the statement in §5.1 that LLMs are 'a more objective evaluator than humans' is not supported. This baseline is computable from the released survey responses.
  3. [§4.3] The magnitude of the key result—LLM agreement with the average human rating—is never reported numerically in the text; Figure 4 is described only qualitatively ('very strong agreement'). Given that the paper's central quantitative comparison is between this value and the inter-human alpha of 0.39, the agreement scores for each LLM (and pooled) should be stated explicitly, together with confidence intervals or a statistical test of the difference. Without these numbers, the reader cannot assess whether the claimed advantage over inter-human agreement is meaningful or within sampling error.
minor comments (4)
  1. [§4.1] In the model-size comparison, 'large models (Llama-3.3-70B and gemma-3-1b)' appears to be a typo; the second model should likely be gemma-3-12b-it. The small-models group also lists gemma-3-1b twice.
  2. [§3.3.1 and §3.2.2] The survey instructions told participants that 'privacy is subjective and there are no wrong answers,' while the LLM prompt did not include this caveat. The paper claims the humans and LLMs received the same instrument; this instruction difference should be acknowledged and its potential effect on ratings discussed.
  3. [§4.1 and §4.3] The procedure of averaging five LLM runs and then rounding to the nearest integer is described briefly; please specify how ties and non-parseable outputs are handled, and whether the same rounding is applied to the human ratings before agreement calculation.
  4. [Limitations] The Limitations section acknowledges dataset representativeness, data contamination, and the role of the privacy definition, but does not mention the dataset-pooling issue identified in this report. Adding a sentence on this limitation, or presenting the within-dataset analysis, would strengthen the paper.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the LLM–human agreement result is an independent empirical measurement, not a fitted or self-referential derivation.

full rationale

The paper's central result is an empirical comparison: 13 LLMs and 677 humans rated the same 250 texts on the same 5-point privacy-sensitivity scale, and the authors compare LLM ratings with the per-text average human rating. Nothing in the LLM ratings is fitted to the human ratings; the LLM outputs are independent inferences from the prompts, and the human average is an external aggregate computed from survey responses. The 'global human privacy perspective' is explicitly operationalized as the average human rating (Sec 5.1: "if the goal is to represent the 'global human privacy opinion', i.e., the average human rating, then LLMs achieve very high agreement"), so the abstract's wording is a label on the measured quantity rather than a second quantity derived from the first. The shared 5-point scale and instructions are a deliberate comparability choice, not a circular reduction. Self-citations ([42], [43], [44], [45]) are used only as dataset sources or background context and are not load-bearing for the agreement analysis. The Limitations section acknowledges that the operationalized definition limits generalizability. The skeptical concern that pooled LLM-to-average agreement may be inflated by dataset-level differences is a missing-analysis/validity issue (no within-dataset per-text agreement is reported in Sec 4.3 or Table 3), not a circularity: the reported numbers would still be what they are, just less informative. Therefore no circular step is exhibited.

Assumptions & free parameters 0 free parameters · 7 assumptions · 0 invented entities

The central empirical result depends on the operational definition of privacy (a hand-built 5-point scale), the assumption that averaging noisy human ratings yields a meaningful target, and the domain assumptions about sampling, agreement metrics, and contamination. No numeric free parameters are fit to the alignment result itself.

assumptions (7)
  • domain assumption A single privacy-sensitivity score on a hand-built 5-point Likert scale can meaningfully represent the privacy of an entire text.
    Sec 3.2.1 defines the scale and Sec 3.3.1 gives it to both LLMs and humans; all agreement measures inherit this design choice.
  • domain assumption The average of human ratings per text is a meaningful global human privacy perspective despite low agreement.
    Sec 4.3 and Sec 5.1 interpret LLM alignment with this average as modeling global opinion, though inter-human alpha is 0.39 and per-text standard deviations are high.
  • standard math Krippendorff's alpha applied to rounded average ratings with missing data is a valid measure of agreement.
    Sec 4.1 uses the krippendorff library; rounding average ratings to integers and pooling among raters on shared texts is assumed acceptable without validation.
  • standard math Averaging multiple ratings per text and rounding to the nearest integer preserves agreement for Krippendorff's alpha.
    Sec 4.1 states that average ratings are rounded to integers for agreement calculations; this transformation is not validated against the raw ratings.
  • domain assumption Adversarial classifier confidence is a valid proxy for privacy vulnerability for representative text selection.
    Sec 3.1.2 fine-tunes deberta-v3-base and uses prediction confidence to select 25 texts per dataset; if confidence does not track human privacy perceptions, the sample is biased toward model-specific signals.
  • domain assumption The 10 selected public datasets and the Prolific participant pool are sufficiently representative for general claims.
    Sec 3.1 and Sec 3.4; the authors themselves acknowledge in Sec 6 that the datasets and geographic distribution of participants limit generalizability.
  • domain assumption No data contamination: the tested LLMs have not memorized the public test texts in ways that affect privacy ratings.
    Acknowledged as unaddressed in Sec 6 Limitations; if contamination occurs, the LLM ratings may reflect training-set context rather than a general privacy judgment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM-as-a-Judge for Privacy Evaluation? Exploring the Alignment of Human and LLM Perceptions of Privacy in Textual Data." pith.science (2026). https://pith.science/paper/TDVA2IKR

@misc{pith2026250812158,
  author       = {Pith},
  title        = {Pith review of: LLM-as-a-Judge for Privacy Evaluation? Exploring the Alignment of Human and LLM Perceptions of Privacy in Textual Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TDVA2IKR}},
  note         = {Machine review of arXiv:2508.12158}
}
abstract

Despite advances in the field of privacy-preserving Natural Language Processing (NLP), a significant challenge remains the accurate evaluation of privacy. As a potential solution, using LLMs as a privacy evaluator presents a promising approach $\unicode{x2013}$ a strategy inspired by its success in other subfields of NLP. In particular, the so-called $\textit{LLM-as-a-Judge}$ paradigm has achieved impressive results on a variety of natural language evaluation tasks, demonstrating high agreement rates with human annotators. Recognizing that privacy is both subjective and difficult to define, we investigate whether LLM-as-a-Judge can also be leveraged to evaluate the privacy sensitivity of textual data. Furthermore, we measure how closely LLM evaluations align with human perceptions of privacy in text. Resulting from a study involving 10 datasets, 13 LLMs, and 677 human survey participants, we confirm that privacy is indeed a difficult concept to measure empirically, exhibited by generally low inter-human agreement rates. Nevertheless, we find that LLMs can accurately model a global human privacy perspective, and through an analysis of human and LLM reasoning patterns, we discuss the merits and limitations of LLM-as-a-Judge for privacy evaluation in textual data. Our findings pave the way for exploring the feasibility of LLMs as privacy evaluators, addressing a core challenge in solving pressing privacy issues with innovative technical solutions.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Can LLMs Make (Personalized) Access Control Decisions?

    cs.CR 2025-11 unverdicted novelty 5.0 of 10

    LLMs reflect users' privacy preferences in access control decisions with up to 86% agreement and can promote safer behavior, but personalization trades off higher individual match for potentially less secure results w...

Reference graph

Works this paper leans on

82 extracted references · 10 canonical work pages · cited by 1 Pith paper

  1. [1]

    Aras Alkis and Tekin Kose. 2022. Privacy concerns in consumer E-commerce activities and response to social media advertising: Empirical evidence from Europe. Computers in Human Behavior 137 (2022), 107412. https://doi.org/10. 1016/j.chb.2022.107412

    arXiv 2022

  2. [2]

    Lori Baker-Eveleth, Robert Stone, and Daniel Eveleth. 2022. Understanding social media users’ privacy-protection behaviors. Information & Computer Security 30, 3 (2022), 324–345

    2022

  3. [3]

    Ruwan Bandara, Mario Fernando, and Shahriar Akter. 2020. Privacy concerns in E-commerce: A taxonomy and a future research agenda. Electronic Markets 30, 3 (2020), 629–647

    2020

  4. [4]

    Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, et al . 2025. LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks. arXiv:cs.CL/2406.18403 https://arxiv.org/abs/2406.18403

    arXiv 2025

  5. [5]

    Sravani Boinepelli, Tathagata Raha, Harika Abburi, Pulkit Parikh, Niyati Chhaya, and Vasudeva Varma. 2022. Leveraging Mental Health Forums for User-level Depression Detection on Social Media. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cie...

    2022

  6. [6]

    Iva Bojic, Jessica Chen, Si Yuan Chang, Qi Chwen Ong, Shafiq Joty, and Josip Car

  7. [7]

    Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr. 2022. What Does it Mean for a Language Model to Preserve Privacy?. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency (FAccT ’22). Association for Computing Machinery, New York, NY, USA, 2280–2292. https://doi.org/10.1145/3531146.3534642

    arXiv 2022

  8. [8]

    Santiago Castro. 2017. Fast Krippendorff: Fast computation of Krippendorff’s alpha agreement measure. https://github.com/pln-fing-udelar/fast-krippendorff

    2017

Show all 82 references
  1. [9]

    Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang

  2. [10]

    Tianhui Chen and Jing Chen and. 2025. Exploring College Students’ Per- ceptions of Privacy and Security in Online Learning: A Comprehensive Questionnaire-Based Study. International Journal of Human–Computer Interac- tion 41, 10 (2025), 6481–6494. https://doi.org/10.1080/104473...

    2025

  3. [11]

    Yang Cheng and Hua Jiang. 2020. How do AI-driven chatbots impact user experience? Examining gratifications, perceived privacy risk, satisfaction, loyalty, and continued use. Journal of Broadcasting & Electronic Media 64, 4 (2020), 592–614. https://doi.org/10.1080/08838151.2020.1834296

    2020

  4. [12]

    Aparna Elangovan, Ling Liu, Lei Xu, Sravan Babu Bodapati, and Dan Roth. 2024. ConSiDERS-The-Human Evaluation Framework: Rethinking Human Evaluation for Generative Large Language Models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (...

    2024 doi

  5. [13]

    David Elliott and Eldon Soifer. 2022. AI technologies, privacy, and security. Frontiers in Artificial Intelligence 5 (2022), 826737

    2022

  6. [14]

    Enryu. 2023. Dataset of 100m tweets. https://doi.org/10.5281/zenodo.15086029

    2023 doi

  7. [15]

    Marco Furini, Silvia Mirri, Manuela Montangero, and Catia Prandi. 2020. Pri- vacy perception when using smartphone applications. Mobile Networks and Applications 25 (2020), 1055–1061. https://doi.org/10.1007/s11036-020-01529-z

    2020 doi

  8. [16]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Ab- hishek Kadian, Ahmad Al-Dahle, et al . 2024. The Llama 3 Herd of Models. arXiv:cs.AI/2407.21783 https://arxiv.org/abs/2407.21783

    2024 arXiv

  9. [17]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, et al. 2025. A Survey on LLM-as-a-Judge. arXiv:cs.CL/2411.15594 https://arxiv. org/abs/2411.15594

    2025 arXiv

  10. [18]

    Quang-An Ha, Jengchung Victor Chen, Ha Uy Uy, and Erik Paolo Capistrano and

  11. [19]

    Ivan Habernal. 2021. When differential privacy meets NLP: The devil is in the detail. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (Eds.). Association for Co...

    2021 doi

  12. [20]

    Ivan Habernal, Fatemehsadat Mireshghallah, Patricia Thaine, Sepideh Ghanavati, and Oluwaseyi Feyisetan. 2023. Privacy-Preserving Natural Language Processing. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics: Tutoria...

    2023 doi

  13. [21]

    It’s the Company, the Gov- ernment, You and I

    Julie Haney, Yasemin Acar, and Susanne Furman. 2021. "It’s the Company, the Gov- ernment, You and I": User Perceptions of Responsibility for Smart Home Privacy and Security. In 30th USENIX Security Symposium (USENIX Security 21) . USENIX Association, 411–428. https://www.useni...

    2021

  14. [22]

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. DeBERTa: Decoding-enhanced BERT with Disentangled Attention. In International Confer- ence on Learning Representations. https://openreview.net/forum?id=XPZIaotutsD

    2021

  15. [23]

    Dirk Hovy, Anders Johannsen, and Anders Søgaard. 2015. User Review Sites as a Resource for Large-Scale Sociolinguistic Studies. In Proceedings of the 24th International Conference on World Wide Web (WWW ’15) . International World Wide Web Conferences Steering Committee, Republ...

    2015

  16. [24]

    Lijie Hu, Ivan Habernal, Lei Shen, and Di Wang. 2024. Differentially Private Natural Language Models: Recent Advances and Future Directions. In Findings of the Association for Computational Linguistics: EACL 2024 , Yvette Graham and Matthew Purver (Eds.). Association for Compu...

    2024

  17. [25]

    There will be less privacy, of course

    Patrick Gage Kelley, Celestina Cornejo, Lisa Hayes, Ellie Shuo Jin, Aaron Sed- ley, Kurt Thomas, et al. 2023. "There will be less privacy, of course": How and why people in 10 countries expect AI will affect privacy in the future. In Nine- teenth Symposium on Usable Privacy an...

    2023

  18. [26]

    Oleksandra Klymenko, Stephen Meisenbacher, and Florian Matthes. 2022. Differ- ential Privacy in Natural Language Processing: The Story So Far. In Proceedings of the Fourth Workshop on Privacy in Natural Language Processing , Oluwaseyi Feyisetan, Sepideh Ghanavati, Patricia Tha...

    2022 doi

  19. [27]

    Josephine Lau, Benjamin Zimmerman, and Florian Schaub. 2018. Alexa, Are You Listening? Privacy Perceptions, Concerns and Privacy-seeking Behaviors with Smart Speakers. Proc. ACM Hum.-Comput. Interact. 2, CSCW, Article 102 (Nov. 2018), 31 pages. https://doi.org/10.1145/3274371

    2018 doi

  20. [28]

    Anna Leschanowsky, Silas Rech, Birgit Popp, and Tom Bäckström. 2024. Evaluat- ing privacy, security, and trust perceptions in conversational AI:: A systematic review. (2024)

    2024

  21. [29]

    Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, et al. 2024. LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods. arXiv:cs.CL/2412.05579 https://arxiv.org/abs/2412.05579

    2024 arXiv

  22. [30]

    Haoran Li, Dadi Guo, Donghao Li, Wei Fan, Qi Hu, Xin Liu, et al. 2024. PrivLM- Bench: A Multi-level Privacy Evaluation Benchmark for Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , Lun-Wei Ku...

    2024 doi

  23. [31]

    Jonathan Li, Rohan Bhambhoria, and Xiaodan Zhu. 2022. Parameter-Efficient Legal Domain Adaptation. InProceedings of the Natural Legal Language Processing Workshop 2022, Nikolaos Aletras, Ilias Chalkidis, Leslie Barrett, Cătălina Goant,ă, and Daniel Preot,iuc-Pietro (Eds.). Ass...

    2022

  24. [32]

    Tianshi Li, Sauvik Das, Hao-Ping (Hank) Lee, Dakuo Wang, Bingsheng Yao, and Zhiping Zhang. 2024. Human-Centered Privacy Research in the Age of Large Language Models. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (CHI EA ’24) . Association fo...

    2024

  25. [33]

    Pierre Lison, Ildikó Pilán, David Sanchez, Montserrat Batet, and Lilja Øvrelid

  26. [34]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG Evaluation using Gpt-4 with Better Human Alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2511–2522

    2023

  27. [35]

    Tesfay, Vanessa Bracamonte, and Toru Nakamura

    Sascha Löbner, Welderufael B. Tesfay, Vanessa Bracamonte, and Toru Nakamura

  28. [36]

    Ying Ma, Cherie Sew, Zhanna Sarsenbayeva, Jarrod Knibbe, and Jorge Goncalves

  29. [37]

    Anonymisation Models for Text Data: State of the art, Challenges and Future Directions. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)...

    2021 doi

  30. [38]

    Abdul Majeed and Sungchang Lee. 2021. Anonymization Techniques for Privacy Preserving Data Publishing: A Comprehensive Survey. IEEE Access 9 (2021), 8512–8545. https://doi.org/10.1109/ACCESS.2020.3045700

    2021

  31. [39]

    Giacomo Marzi, Marco Balzano, and Davide Marchiori. 2024. K-Alpha calculator– krippendorff’s alpha calculator: a user-friendly tool for computing krippendorff’s alpha inter-rater reliability coefficient. MethodsX 12 (2024), 102545. https: //doi.org/10.1016/j.mex.2023.102545

    2024

  32. [40]

    In 2023 20th Annual International Conference on Privacy, Security and Trust (PST)

    Systematizing the State of Knowledge in Detecting Privacy Sensitive Information in Unstructured Texts using Machine Learning. In 2023 20th Annual International Conference on Privacy, Security and Trust (PST) . 1–7. https://doi. HAIPS ’25, October 13–17, 2025, Taipei, Taiwan Me...

    2023

  33. [41]

    Matthieu Meeus, Shubham Jain, Marek Rei, and Yves-Alexandre de Montjoye

  34. [42]

    Understanding Users’ Perspectives on Location Privacy Management on iPhones. Proc. ACM Hum.-Comput. Interact. 8, MHCI, Article 282 (Sept. 2024), 25 pages. https://doi.org/10.1145/3676529

    2024 doi

  35. [43]

    Darshini Mahendran, Changqing Luo, and Bridget T. Mcinnes. 2021. Review: Privacy-Preservation in the Context of Natural Language Processing.IEEE Access 9 (2021), 147600–147612. https://doi.org/10.1109/ACCESS.2021.3124163

    2021

  36. [44]

    Stephen Meisenbacher and Florian Matthes. 2024. Thinking Outside of the Differential Privacy Box: A Case Study in Text Privatization with Language Model Prompting. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , Yaser Al-Onaizan, Moh...

    2024 doi

  37. [45]

    Stephen Meisenbacher, Nihildev Nandakumar, Alexandra Klymenko, and Florian Matthes. 2024. A Comparative Analysis of Word-Level Metric Differential Privacy: Benchmarking the Privacy-Utility Trade-off. In Proceedings of the 2024 Joint International Conference on Computational Li...

    2024

  38. [46]

    Madeth May, Georgios Fessakis, Angelique Dimitracopoulou, and Sébastien George. 2012. A Study on User’s Perception in E-learning Security and Pri- vacy Issues. In 2012 IEEE 12th International Conference on Advanced Learning Technologies. 88–89. https://doi.org/10.1109/ICALT.2012.145

    2012 doi

  39. [47]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, et al. 2024. GPT-4 Technical Report. arXiv:cs.CL/2303.08774 https: //arxiv.org/abs/2303.08774

    2024 arXiv

  40. [48]

    In33rd USENIX Security Symposium (USENIX Security 24)

    Did the Neurons Read your Book? Document-level Membership Inference for Large Language Models. In33rd USENIX Security Symposium (USENIX Security 24). USENIX Association, Philadelphia, PA, 2369–2385. https://www.usenix.org/ conference/usenixsecurity24/presentation/meeus

  41. [49]

    Stephen Meisenbacher, Maulik Chevli, and Florian Matthes. 2025. On the Impact of Noise in Differentially Private Text Rewriting. InFindings of the Association for Computational Linguistics: NAACL 2025, Luis Chiruzzo, Alan Ritter, and Lu Wang (Eds.). Association for Computation...

    2025 doi

  42. [50]

    Stephen Meisenbacher, Alexandra Klymenko, Alexander Karpp, and Florian Matthes. 2025. Investigating User Perspectives on Differentially Private Text Pri- vatization. In Proceedings of the Sixth Workshop on Privacy in Natural Language Processing, Ivan Habernal, Sepideh Ghanavat...

    2025 doi

  43. [51]

    Xudong Pan, Mi Zhang, Shouling Ji, and Min Yang. 2020. Privacy Risks of General- Purpose Language Models. In 2020 IEEE Symposium on Security and Privacy (SP) . 1314–1331. https://doi.org/10.1109/SP40000.2020.00095

    2020

  44. [52]

    Nandita Pattnaik, Shujun Li, and Jason R. C. Nurse. 2023. A Survey of User Perspectives on Security and Privacy in a Home Networking Environment. ACM Comput. Surv. 55, 9, Article 180 (Jan. 2023), 38 pages. https://doi.org/10.1145/ 3558095

    2023

  45. [53]

    Preksha Nema, Pauline Anthonysamy, Nina Taft, and Sai Teja Peddinti. 2022. Analyzing user perspectives on mobile app privacy at scale. In Proceedings of the 44th International Conference on Software Engineering (ICSE ’22) . Association for Computing Machinery, New York, NY, US...

    2022

  46. [54]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP-IJ...

    2019 doi

  47. [55]

    Goucher, Adam Perelman, Aditya Ramesh, et al

    OpenAI, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, et al . 2024. GPT-4o System Card. arXiv:cs.CL/2410.21276 https: //arxiv.org/abs/2410.21276

    2024 arXiv

  48. [56]

    Ozlem Ozmen Garibay, Brent Winslow, Salvatore Andolina, Margherita Antona, Anja Bodenschatz, Constantinos Coursaris, et al . 2023. Six human-centered artificial intelligence grand challenges. International Journal of Human–Computer Interaction 39, 3 (2023), 391–437. https://do...

    2023

  49. [57]

    Qian Pan, Zahra Ashktorab, Michael Desmond, Martín Santillán Cooper, James Johnson, Rahul Nair, et al. 2024. Human-Centered Design Recommendations for LLM-as-a-judge. In Proceedings of the 1st Human-Centered Large Language Modeling Workshop, Nikita Soni, Lucie Flek, Ashish Sha...

    2024 doi

  50. [58]

    Eicher-Miller, Toby Jia-Jun Li, Meng Jiang, and Ronald A

    Annalisa Szymanski, Noah Ziems, Heather A. Eicher-Miller, Toby Jia-Jun Li, Meng Jiang, and Ronald A. Metoyer. 2025. Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks. In Proceedings of the 30th International Conference on Intellige...

    2025

  51. [59]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, et al. 2025. Gemini: A Family of Highly Capable Multimodal Models. arXiv:cs.CL/2312.11805 https://arxiv.org/abs/2312.11805

    2025 arXiv

  52. [60]

    Ildikó Pilán, Pierre Lison, Lilja Øvrelid, Anthi Papadopoulou, David Sánchez, and Montserrat Batet. 2022. The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymization. Computational Linguistics 48, 4 (Dec. 2022), 1053–1101. https:/...

    2022 doi

  53. [61]

    Welderufael B Tesfay, Jetzabel M Serna, and Sebastian Pape. 2016. Challenges in Detecting Privacy Revealing Information in Unstructured Text.. In PrivOn@ ISWC. https://ceur-ws.org/Vol-1750/paper-05.pdf

    2016

  54. [62]

    Sakib Shahriar, Rozita Dara, and Rajen Akalu. 2025. A comprehensive review of current trends, challenges, and opportunities in text data privacy. Computers & Security 151 (2025), 104358. https://doi.org/10.1016/j.cose.2025.104358

    2025

  55. [63]

    Samuel Sousa and Roman Kern. 2023. How to keep text private? A sys- tematic review of deep learning methods for privacy-preserving natural lan- guage processing. Artificial Intelligence Review 56, 2 (2023), 1427–1492. https: //doi.org/10.1007/s10462-022-10204-6

    2023 doi

  56. [64]

    Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krimmel, et al. 2024. jina-embeddings-v3: Multilingual Embeddings With Task LoRA. arXiv:cs.CL/2409.10173 https://arxiv.org/abs/2409.10173

    2024 arXiv

  57. [65]

    Xuequn Wang, Xiaolin Lin, and Bin Shao. 2024. Security and Privacy Protection in Developing Ethical AI: A Mixed-Methods Study from a Marketing Employee Perspective. Journal of Business Ethics (2024), 1–20. https://doi.org/10.1007/ s10551-024-05894-7

    2024

  58. [66]

    Jurgen Willems, Moritz J Schmid, Dieter Vanderelst, Dominik Vogel, and Falk Ebinger. 2023. AI-driven public services and the privacy paradox: do citizens really care about their privacy? Public Management Review 25, 11 (2023), 2116–

    2023

  59. [67]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupati- raju, Shreya Pathak, et al. 2024. Gemma: Open Models Based on Gemini Research and Technology. arXiv:cs.CL/2403.08295 https://arxiv.org/abs/2403.08295

    2024 arXiv

  60. [68]

    Biwei Yan, Kun Li, Minghui Xu, Yueyan Dong, Yue Zhang, Zhaochun Ren, et al

  61. [69]

    Mina Tsay-Vogel, James Shanahan, and Nancy Signorielli. 2018. Social media cultivating perceptions of privacy: A 5-year analysis of privacy at- titudes and self-disclosure behaviors among Facebook users. New Media & Society 20, 1 (2018), 141–161. https://doi.org/10.1177/146144...

    2018 doi

  62. [70]

    Saiteja Utpala, Sara Hooker, and Pin-Yu Chen. 2023. Locally Differentially Private Document Generation Using Zero Shot Prompting. In Findings of the Association for Computational Linguistics: EMNLP 2023 , Houda Bouamor, Juan Pino, and Kalika Bali (Eds.). Association for Comput...

    2023

  63. [71]

    Dorothy Zhao, Mikako Inaba, and Andrés Monroy-Hernández. 2022. Under- standing Teenage Perceptions and Configurations of Privacy on Instagram. Proc. ACM Hum.-Comput. Interact. 6, CSCW2, Article 550 (Nov. 2022), 28 pages. https://doi.org/10.1145/3555608

    2022 doi

  64. [72]

    M Vimalkumar, Sujeet Kumar Sharma, Jang Bahadur Singh, and Yogesh K Dwivedi. 2021. ‘Okay google, what about my privacy?’: User’s privacy per- ceptions and acceptance of voice based digital assistants. Computers in Human Behavior 120 (2021), 106763. https://doi.org/10.1016/j.ch...

    2021

  65. [76]

    Steven M Williamson and Victor Prybutok. 2024. Balancing privacy and progress: a review of privacy challenges, systemic oversight, and patient perceptions in AI-driven healthcare. Applied Sciences 14, 2 (2024), 675. https://doi.org/10.3390/ app14020675

    2024

  66. [78]

    arXiv:cs.CR/2403.05156 https://arxiv.org/abs/2403.05156

    On Protecting the Data Privacy of Large Language Models (LLMs): A Survey. arXiv:cs.CR/2403.05156 https://arxiv.org/abs/2403.05156

    arXiv

  67. [79]

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019)

    2019 arXiv

  68. [80]

    Zhiping Zhang, Bingcan Guo, and Tianshi Li. 2025. Privacy Leakage Overshad- owed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agent. arXiv:cs.HC/2411.01344 https://arxiv.org/abs/2411.01344

    2025

  69. [82]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, et al . 2023. Judging LLM-as-a-judge with MT-bench and Chatbot Arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems (NIPS ’23) . Curran Associa...

    2023

  70. [2021]

    International Journal of Human–Computer Interaction 37, 6 (2021), 512–527

    Exploring the Privacy Concerns in Using Intelligent Virtual Assistants under Perspectives of Information Sensitivity and Anthropomorphism. International Journal of Human–Computer Interaction 37, 6 (2021), 512–527. https://doi.org/10. 1080/10447318.2020.1834728 arXiv:https://do...

    2021

  71. [2023]

    arXiv preprint arXiv:2310.01917 (2023)

    Hierarchical evaluation framework: Best practices for human evaluation. arXiv preprint arXiv:2310.01917 (2023)

    2023 arXiv

  72. [2024]

    In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.)

    Humans or LLMs as the Judge? A Study on Judgement Bias. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing , Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 8301–83...

    2024

  73. [2134]

    https://doi.org/10.1080/14719037.2022.2063934

    2022

  74. [8457]

    https://doi.org/10.18653/v1/2023.findings-emnlp.566

    2023 doi

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.