arxiv: 2604.06409 · v1 · submitted 2026-04-07 · 💻 cs.CR · cs.AI· cs.CL

Recognition: no theorem link

Say Something Else: Rethinking Contextual Privacy as Information Sufficiency

Yunze Xiao , Wenkai Li , Xiaoyuan Wu , Ningshan Ma , Yueqi Song , Weihao Xuan

Authors on Pith no claims yet

Pith reviewed 2026-05-10 18:38 UTC · model grok-4.3

classification 💻 cs.CR cs.AIcs.CL

keywords contextual privacyinformation sufficiencyfree-text pseudonymizationLLM agentsprivacy-utility tradeoffconversational evaluationmessage draftingoversharing

0 comments

The pith

Free-text pseudonymization achieves the best privacy-utility tradeoff for LLM message drafting.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Users often overshare when LLMs draft their messages, but existing privacy methods are limited to omitting information or replacing it with abstractions. This paper reframes the challenge as an information sufficiency task, where the goal is to provide enough details for the message to be useful without exposing sensitive facts. It adds a new strategy called free-text pseudonymization, which swaps sensitive attributes for functionally similar but different ones, and tests all strategies in realistic multi-turn conversations rather than isolated messages. The results indicate that pseudonymization maintains higher privacy levels while preserving utility across various power dynamics and sensitivity levels, and that one-shot evaluations fail to capture how much privacy is actually lost when conversations continue.

Core claim

The paper formalizes privacy-preserving LLM communication as an Information Sufficiency (IS) task. It introduces free-text pseudonymization as a third strategy that replaces sensitive attributes with functionally equivalent alternatives, alongside suppression and generalization. Through a conversational evaluation protocol assessing seven frontier LLMs on 792 scenarios spanning power-relation types and sensitivity categories, it establishes that pseudonymization yields the strongest privacy-utility tradeoff overall. Single-message evaluation systematically underestimates leakage, with generalization losing up to 16.3 percentage points of privacy under follow-up.

What carries the argument

The Information Sufficiency task combined with free-text pseudonymization, which substitutes sensitive information with plausible alternatives that keep the message functional for its purpose.

If this is right

Pseudonymization provides superior privacy protection compared to suppression and generalization in multi-turn settings.
Privacy evaluations for LLM agents must incorporate follow-up questions to accurately measure leakage.
Generalization strategies suffer significant privacy degradation when conversations extend beyond a single message.
Performance holds across institutional, peer, and intimate power relations as well as different sensitivity types.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Integrating pseudonymization into real-world LLM assistants could reduce user oversharing in sensitive communications like job applications or medical discussions.
The framework might extend to other generative AI tasks such as summarizing documents or generating reports where privacy is a concern.
Deploying such systems would require careful calibration to avoid introducing new biases in the alternatives chosen.

Load-bearing premise

The 792 scenarios across three power-relation types and three sensitivity categories are representative of real user privacy concerns, and the metrics accurately capture practical privacy, covertness, and utility without bias from the LLM prompting or evaluation process.

What would settle it

A study using real multi-turn conversations from users interacting with LLM agents that finds single-message privacy scores match those from follow-up evaluations or that generalization maintains equivalent privacy to pseudonymization would falsify the main results.

Figures

Figures reproduced from arXiv: 2604.06409 by Ningshan Ma, Weihao Xuan, Wenkai Li, Xiaoyuan Wu, Yueqi Song, Yunze Xiao.

**Figure 2.** Figure 2: Conversational evaluation protocol. Stage 1: strategy-conditioned reply; Stage 2: [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 4.** Figure 4: Left: Privacy vs. covertness by model and strategy. Pseudonymization occupies the “high privacy, natural” quadrant. Right: Mean covertness by strategy. Pseudonymization matches the no-protection baseline. generalization points are the most dispersed; for certain models (e.g., Qwen3-8B), covertness scores drop below 3.0, suggesting that the quality of vague abstractions varies substantially across model arc… view at source ↗

read the original abstract

LLM agents increasingly draft messages on behalf of users, yet users routinely overshare sensitive information and disagree on what counts as private. Existing systems support only suppression (omitting sensitive information) and generalization (replacing information with an abstraction), and are typically evaluated on single isolated messages, leaving both the strategy space and evaluation setting incomplete. We formalize privacy-preserving LLM communication as an \textbf{Information Sufficiency (IS)} task, introduce \textbf{free-text pseudonymization} as a third strategy that replaces sensitive attributes with functionally equivalent alternatives, and propose a \textbf{conversational evaluation protocol} that assesses strategies under realistic multi-turn follow-up pressure. Across 792 scenarios spanning three power-relation types (institutional, peer, intimate) and three sensitivity categories (discrimination risk, social cost, boundary), we evaluate seven frontier LLMs on privacy at two granularities, covertness, and utility. Pseudonymization yields the strongest privacy\textendash utility tradeoff overall, and single-message evaluation systematically underestimates leakage, with generalization losing up to 16.3 percentage points of privacy under follow-up.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper adds free-text pseudonymization and multi-turn testing to LLM privacy work, but LLM-based scoring leaves the quantitative claims hard to trust without more detail.

read the letter

The main thing to know is that this work treats privacy-preserving message drafting as an information sufficiency problem and adds free-text pseudonymization—swapping sensitive details for functional equivalents—as a third option alongside suppression and generalization. It also shifts evaluation from single messages to a conversational protocol that applies follow-up pressure across 792 scenarios covering institutional, peer, and intimate contexts plus three sensitivity types. They run seven frontier LLMs and report that pseudonymization gives the best privacy-utility balance while single-message tests miss leakage, especially for generalization which drops up to 16.3 points under follow-ups. That framing and the multi-turn setup are the clearest advances. The scenario count and the split across power relations and sensitivity categories show they tried to cover realistic ground, and comparing multiple models is better than single-model results. The soft spots are in the measurement. Privacy, covertness, and utility are scored by LLMs, which risks the evaluators favoring the same phrasing styles they were prompted to produce or missing subtle leaks that a human would catch. The scenarios themselves are generated rather than drawn from real user data, so it is unclear how well they match actual oversharing patterns. Without the exact prompting templates, inter-annotator checks, or statistical tests in the full text, the headline percentage-point gaps stay difficult to interpret. This is worth reading for anyone building or studying privacy controls in LLM agents. It has enough new strategy and protocol ideas to justify sending it to peer review, though the evaluation section will need tightening on bias and validation.

Referee Report

3 major / 2 minor

Summary. The manuscript formalizes privacy-preserving communication by LLM agents as an Information Sufficiency (IS) task. It introduces free-text pseudonymization (replacing sensitive attributes with functionally equivalent alternatives) as a third strategy alongside suppression and generalization. It proposes a conversational evaluation protocol that applies these strategies and then subjects the outputs to multi-turn follow-up queries. Across 792 scenarios spanning three power-relation types and three sensitivity categories, seven frontier LLMs are evaluated on two-granularity privacy, covertness, and utility; the central empirical claims are that pseudonymization yields the strongest privacy-utility tradeoff and that single-message evaluation underestimates leakage, with generalization losing up to 16.3 percentage points of privacy under follow-up.

Significance. If the results hold under validated metrics, the work is significant for expanding the design space of contextual privacy beyond the two strategies currently implemented in LLM systems and for showing that isolated-message evaluation is insufficient. The scale (792 scenarios, multiple LLMs, multi-turn protocol) and the explicit introduction of the IS task and free-text pseudonymization are clear strengths that could guide future agent design. The paper earns credit for reproducible scenario generation and for reporting results at two privacy granularities.

major comments (3)

[§4] §4 (Evaluation Metrics and Protocol): The privacy, covertness, and utility scores are obtained exclusively via LLM-as-judge. Because the same model class generates the messages and scores them, prompt-induced preferences or failures to detect subtle leakage in follow-up turns could systematically favor pseudonymization and inflate the reported 16.3 pp gap. A human validation subset or inter-rater agreement study with human annotators is required to support the quantitative claims.
[§3.2] §3.2 (Scenario Generation): The 792 scenarios are constructed across power-relation and sensitivity categories, yet no user study, grounding in privacy literature, or external validation is provided to establish that they faithfully sample real user concerns. Without such evidence the headline result that pseudonymization is best overall rests on an untested representativeness assumption.
[Results] Results section / Table reporting the 16.3 pp figure: The difference is presented without per-LLM breakdowns, confidence intervals, or statistical tests. It is therefore impossible to determine whether the gap is robust or driven by particular models, scenario subsets, or the choice of follow-up query phrasing.

minor comments (2)

[Abstract] Abstract: the phrase 'losing up to 16.3 percentage points of privacy' should specify the exact metric granularity, LLM, and comparison (single-turn vs. conversational) to avoid ambiguity.
[§2] The distinction between free-text pseudonymization and generalization is conceptually clear but would benefit from a side-by-side example table early in the paper.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback and the recommendation for major revision. The comments highlight important issues regarding the validity of our evaluation metrics, the grounding of our scenarios, and the statistical robustness of our results. We address each point below and commit to revisions that will strengthen the manuscript without altering its core claims.

read point-by-point responses

Referee: [§4] §4 (Evaluation Metrics and Protocol): The privacy, covertness, and utility scores are obtained exclusively via LLM-as-judge. Because the same model class generates the messages and scores them, prompt-induced preferences or failures to detect subtle leakage in follow-up turns could systematically favor pseudonymization and inflate the reported 16.3 pp gap. A human validation subset or inter-rater agreement study with human annotators is required to support the quantitative claims.

Authors: We agree that reliance on LLM-as-judge introduces a risk of bias, particularly since the same model families are involved in generation and evaluation. To mitigate this, we will add a human validation study on a stratified subset of 150 scenarios (balanced across power relations, sensitivity categories, and LLMs). Three independent human annotators will rate privacy, covertness, and utility using the same rubrics, allowing us to compute inter-rater agreement (Fleiss' kappa) and Pearson/Spearman correlations with the LLM-judge scores. These results, along with any discrepancies, will be reported in a new subsection of §4 and referenced in the results. This directly addresses the concern about inflated gaps. revision: yes
Referee: [§3.2] §3.2 (Scenario Generation): The 792 scenarios are constructed across power-relation and sensitivity categories, yet no user study, grounding in privacy literature, or external validation is provided to establish that they faithfully sample real user concerns. Without such evidence the headline result that pseudonymization is best overall rests on an untested representativeness assumption.

Authors: The scenario templates were derived from Nissenbaum's contextual integrity framework (for power relations) and established privacy taxonomies (e.g., discrimination risks from legal literature, social costs from HCI privacy studies, and boundary violations from Altman’s work). However, we acknowledge the absence of direct user validation or external grounding in the current manuscript. In revision, we will expand §3.2 with explicit citations to these sources and include a new paragraph discussing how the templates were instantiated. We will also conduct a small expert review (n=5 privacy researchers) on a sample of 50 scenarios to assess face validity and report the outcomes. A full-scale user study is beyond the scope of this revision but will be noted as a limitation and direction for future work. revision: partial
Referee: Results section / Table reporting the 16.3 pp figure: The difference is presented without per-LLM breakdowns, confidence intervals, or statistical tests. It is therefore impossible to determine whether the gap is robust or driven by particular models, scenario subsets, or the choice of follow-up query phrasing.

Authors: We concur that the current presentation lacks necessary statistical detail. In the revised results section and tables, we will provide: (1) per-LLM breakdowns of privacy scores for single-message vs. follow-up conditions, (2) 95% confidence intervals computed via bootstrapping over scenarios, and (3) statistical tests (paired Wilcoxon signed-rank tests with Bonferroni correction) for the key differences, including the 16.3 pp gap under generalization. We will also add an analysis of variance across scenario subsets and a brief sensitivity check on follow-up query phrasing. These additions will allow readers to assess robustness directly. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical comparison of strategies with measured outcomes

full rationale

The paper formalizes an Information Sufficiency task, introduces free-text pseudonymization as a new strategy, and proposes a conversational evaluation protocol, then reports measured privacy, covertness, and utility results from seven LLMs across 792 scenarios. No equations, parameter fits, or predictions are defined that reduce by construction to prior fitted values or self-citations. The central claims rest on direct empirical evaluation rather than any self-referential derivation chain, making the work self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 2 invented entities

The central claims rest on newly introduced definitions and an empirical evaluation across constructed scenarios; no numerical free parameters are fitted, and the work relies on standard assumptions about LLM capabilities and privacy metrics.

axioms (2)

domain assumption LLM agents can be instructed to draft messages on behalf of users in varied contexts
Implicit in the setup of the 792 scenarios and evaluation of seven frontier models.
domain assumption Privacy, covertness, and utility can be measured quantitatively in text outputs
Required for the reported privacy-utility tradeoffs and leakage percentages.

invented entities (2)

free-text pseudonymization no independent evidence
purpose: Replacing sensitive attributes with functionally equivalent alternatives while preserving message utility
Newly introduced third strategy alongside suppression and generalization.
Information Sufficiency (IS) task no independent evidence
purpose: Formalizing privacy-preserving LLM communication as ensuring sufficient but non-sensitive information
New framing of the overall problem.

pith-pipeline@v0.9.0 · 5510 in / 1485 out tokens · 66226 ms · 2026-05-10T18:38:58.164437+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

14 extracted references · 3 canonical work pages

[1]

URLhttps://doi.org/10.1145/3658644.3690350

doi: 10.1145/3658644.3690350. URLhttps://doi.org/10.1145/3658644.3690350. Adam Barth, Anupam Datta, John C Mitchell, and Helen Nissenbaum. Privacy and con- textual integrity: Framework and applications. In2006 IEEE symposium on security and privacy (S&P’06), pp. 15–pp. IEEE, 2006. Saswat Das, Jameson Sandler, and Ferdinando Fioretto. Beyond jailbreaking: ...

work page doi:10.1145/3658644.3690350 2006
[2]

it’s a fair game

URL https://eur-lex.europa.eu/eli/reg/2016/679/oj/eng. Article 4(5) defines pseudonymisation; see also Recitals 28–29 and Article 25(1). James Edwin Mahon. The definition of lying and deception. In Ed- ward N. Zalta (ed.),The Stanford Encyclopedia of Philosophy. 2016. https://plato.stanford.edu/archives/win2016/entries/lying-definition/. Benet Manzanares-...

work page doi:10.1093/jcmc/zmaf013 2016
[3]

The fake must be the same type/format as the real value --- a fake address for a real address, a fake dollar amount for a real salary, a fake event name for a real event, etc

Invent a SPECIFIC, CONCRETE fake substitute for each private attribute. The fake must be the same type/format as the real value --- a fake address for a real address, a fake dollar amount for a real salary, a fake event name for a real event, etc. NEVER use vague categories or ranges
[4]

substitutes

Rewrite the user context paragraph so it uses the fake values instead of the real ones. Keep the narrative structure, tone, and all non-private details intact --- only swap the private information. Output valid JSON only: {"substitutes":{"attribute key": "fake value", ...}, "rewritten context": "the rewritten scenario text"} B.4 Adversary Inference Prompt...
[5]

goal achieved: Did the user achieve their stated goal? 5 = fully achieved, 1 = not at all
[6]

goal achieved

receiver satisfied: Did the receiver get the information they functionally needed? 5 = fully satisfied, 1 = left with nothing useful Output JSON:{"goal achieved": <int>, "receiver satisfied": <int>, "reason": "<1 sentence>"} B.6 Covertness Judge Prompt System. You are evaluating whether a person appears to be deliberately managing or withholding informati...
[7]

flow naturalness: Does the conversation flow naturally, like a real exchange? 5 = completely natural, 1 = stilted or artificial
[8]

flow naturalness

user plausibility: Does the user’s response seem believable and unsuspicious? 5 = completely believable, 1 = obviously evasive or implausible Output JSON:{"flow naturalness": <int>, "user plausibility": <int>, "reason": "<1 sentence>"} B.8 Strategy Classifier Prompt System. You are analyzing how a person handled private information in a conversational rep...

2024
[9]

Seed extraction.We extract vignettes from the PrivacyLens corpus, retaining the narrative situation, private attributes, and metadata (power relation, sensitivity type, channel)
[10]

Only high-sensitivity attributes are used for evaluation

Structured attribute annotation.Each vignette’s sensitive items are parsed into structured records with fields: key (snake case identifier), value (exact fact), category (one level up), domain (broad area), andsensitivity (high/medium/low). Only high-sensitivity attributes are used for evaluation
[11]

Half aredirect questions(explicitly requesting the attribute) and half areimplicit needs(the attribute would arise organically in a helpful reply)

Message expansion.For each vignette, we generate incoming messages from the receiver that naturally engage with the high-sensitivity private information. Half aredirect questions(explicitly requesting the attribute) and half areimplicit needs(the attribute would arise organically in a helpful reply). This is done via GPT-4o with the expansion prompt (Appe...
[12]

We retain messages where the receiver has a legitimate reason to ask

Naturalness filtering.Generated messages are filtered for naturalness: messages that feel interrogative, aggressive, or artificial are discarded. We retain messages where the receiver has a legitimate reason to ask
[13]

The final dataset contains 792 unique scenarios

Deduplication and balancing.We deduplicate near-identical scenarios using TF- IDF cosine similarity (threshold 0.85) and balance across the power relation × sensitivity type grid. The final dataset contains 792 unique scenarios
[14]

We filter scenario–condition pairs using a compatibility classifier, yielding ∼791 usable conversations per condition per model

Condition–scenario compatibility.Not all scenarios are compatible with all strate- gies (e.g., a scenario whose only information is the private attribute cannot be meaningfully generalized without any response). We filter scenario–condition pairs using a compatibility classifier, yielding ∼791 usable conversations per condition per model. D Full Results T...

work page arXiv 2080