Beyond Direct Identifiers: Probabilistic Privacy Risk Estimation for Privacy-Conscious LLM Query Delegation

Julia Hirschberg; Li Siyan; Zhou Yu

REVIEW 3 major objections 5 minor 15 references

Beyond Direct Identifiers: Probabilistic Privacy Risk Estimation for Privacy-Conscious LLM Query Delegation

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adding k-anonymity to prompt optimization improves privacy-utility balance in LLM query delegation.

desk verdict This is an honest pilot study with a useful new dataset, but the experiments never isolate the k-anonymity term, so the paper's central claim is unsupported as written. read the letter →

arxiv 2608.09140 v1 pith:4ECM42KH submitted 2026-08-10 cs.CR cs.CL

Li Siyan , Zhou Yu , Julia Hirschberg This is my paper · ORCID

classification cs.CRcs.CL

keywords privacy-consciousdelegationk-anonymityself-disclosurepromptoptimizationLLMprivacyquasi-identifiersprobabilisticriskestimationPUPA-SD

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that privacy in LLM delegation cannot be secured by removing explicit identifiers alone, because quasi-identifying traits can still single users out. It augments the Privacy-Conscious Delegation (PCD) pipeline with an LLM-driven probabilistic estimate of k-anonymity, treating a larger anonymity set as a second objective alongside response quality. To train and test this objective, the authors build PUPA-SD, a small dataset of 166 naturalistic user queries containing self-disclosures, and optimize PAPILLON-style prompts on it. Their preliminary results show that the optimized prompts improve response quality on held-out conversations for most local models, with Llama-3.2-3B achieving the strongest privacy-utility balance.

What carries the argument

The load-bearing mechanism is an efficient, LLM-based estimator of k-anonymity adapted from BRANCH. Given a rewritten query, it extracts estimable disclosures, orders them into cumulative conditioning groups, and estimates each conditional probability from the LLM's pretraining knowledge of population statistics, with a fallback population of 400 million English speakers when information is insufficient. Multiplying the population by the estimated percentages yields the expected number of people matching the disclosed traits. This scalar is inserted into the PAPILLON optimization objective as the k-anonymity reward, so prompt optimization can trade off utility against the size of the anonymity set.

What would settle it

Take a set of disclosures with known population counts (for example, from census microdata or voter records), compute the true k-anonymity, and compare it with the estimator's output on the same text. If the estimated values are uncorrelated or systematically wrong, or if optimizing on the estimated k leaves true re-identification risk unchanged, the central argument for using the metric fails.

Watch

Extended reading notes

Core claim

The central claim is that k-anonymity, estimated probabilistically from text by an LLM, is a useful auxiliary metric for privacy-conscious delegation. On the task of rewriting a user query before sending it to a remote model, the paper shows that adding a BRANCH-based k-anonymity term to the optimization objective, measured as log2(k)/log2(400M), transfers to unseen conversations: after optimizing on PUPA-SD and evaluating on PUPA-TNB, most models improve quality, and Llama-3.2-3B reduces PII leakage from 29.1 to 11.0 while raising quality from 54.2 to 67.4. The authors also find that smaller models struggle to jointly optimize quality and privacy, and that k-anonymity scores stay largely stable, suggesting rewritten queries often fall back to the full population estimate.

Load-bearing premise

The whole approach depends on the LLM's k-anonymity estimates approximating real re-identification risk, but the paper states plainly that the estimators are not calibrated against or grounded in real population statistics.

Editorial extensions

If this is right

Optimizing PAPILLON prompts on PUPA-SD improves response quality on held-out PUPA-TNB conversations for most local models tested.
Llama-3.2-3B-Instruct achieves the best privacy-utility balance, with quality rising from 54.2 to 67.4 and PII leakage falling from 29.1 to 11.0.
Smaller models such as Qwen-2.5-0.5B and Qwen-2.5-1.5B struggle to optimize quality and privacy together, with leakage increasing after optimization.
Because k-anonymity scores remain largely stable across conditions, the objective's variation may come mainly from the quality term; future work should test optimizing on both k-anonymity and PII leakage.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the estimator is uncalibrated, optimizing against it may lower a meaningless number; a natural extension is to ground estimates in real census or registry data via retrieval.
The near-stable k-anonymity values suggest the metric may be easy to satisfy by removing all estimable disclosures, which could push rewrites toward vagueness and hurt utility on tasks needing specific detail.
The method could be tested as a defense against actual linkability: measure whether optimized rewrites reduce the success of an external re-identification attack, not just the estimated k-anonymity.
The PUPA-SD dataset is tiny (166 instances); scaling extraction could turn the objective into a training signal for smaller models that currently fail to balance privacy and utility.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: the new thing here is PUPA-SD, a small hand-inspected set of self-disclosing queries, plus an integration of BRANCH-style k-anonymity estimation into PAPILLON's prompt-optimization objective. That integration is a natural next step, and the paper is refreshingly honest about what it does and does not show. But the headline claim—that adding a k-anonymity objective improves delegation quality and privacy-utility balance—is not supported by the reported experiments.

What is genuinely useful: the dataset (166 queries from WildChat and LMSYS-Chat-1M, with self-disclosures extracted and manually checked), the full prompts in the appendix, and a simplified BRANCH estimator that is cheap enough to use inside optimization. The sanity check showing negative correlation between PII count and estimated k-anonymity is correctly labeled as a monotonicity check rather than validation. The authors do not oversell the estimator's calibration; they flag it as a limitation.

The soft spots are load-bearing. The experiments compare before vs. after optimization using a composite metric that includes k-anonymity, but there is no ablation that optimizes with the quality metric alone. Since k-anonymity scores remain largely flat across all models and conditions, and since PII leakage is explicitly excluded from the optimization metric, the observed quality gains are entirely consistent with SIMBA simply optimizing response quality. The authors essentially acknowledge this in Section 5.2 when they say the objective is driven largely by the quality metric. The reported leakage improvement for Llama-3.2-3B is not optimized-for and could be incidental. The estimator is uncalibrated, the fallback population and log normalization introduce free parameters, and PUPA-SD is only 166 instances, so the quantitative results are pilot-level. None of these problems are hidden, but they are not cosmetic; they directly undermine the central claim.

Who gets value: researchers working on privacy-preserving delegation or LLM-based privacy risk estimation may want PUPA-SD and the simplified estimator as building blocks. The paper is a reasonable pilot, not a demonstration.

My recommendation: I would not desk-reject this—it is a serious attempt with useful artifacts—but I would send it to peer review only with the expectation of major revision. The authors should add an ablation without the k-anonymity term, report variance, and either calibrate the estimator or reframe the paper as proposing a metric rather than showing that the metric helps optimization.

Referee Report

3 major / 5 minor

Summary. The paper proposes PCD-k, a variant of Privacy-Conscious Delegation that augments the PAPILLON prompt-optimization objective with an LLM-estimated k-anonymity term. The authors introduce PUPA-SD, a small dataset of 166 user queries with self-disclosure, and implement a BRANCH-based k-anonymity estimator. They report before/after optimization results on the held-out PUPA-TNB set across six local models, claiming that quality improves on unseen conversations and that Llama-3.2-3B achieves the best privacy-utility balance. The central contribution is the proposal to use probabilistic k-anonymity as an auxiliary metric and optimization target for PCD.

Significance. If the central claim were established, this would be a useful step beyond PII-only redaction for privacy-conscious delegation, and the paper's emphasis on quasi-identifiers and self-disclosure is well motivated. The paper also contributes a new dataset and a cheaper BRANCH-style estimator, and it is commendably explicit about several limitations. However, the experiments as designed do not isolate the effect of the k-anonymity objective, and the k-anonymity estimator is not validated against any ground truth. Since the same uncalibrated estimator is used both as the optimization target and as the privacy outcome measure, the reported privacy-utility balance does not support the paper's claim. The contribution is therefore not established at the level required for publication.

major comments (3)

[Section 5.2, Table 2] The experiments compare only 'before optimization' with 'after optimization' under the composite objective MetricPAPILLON + k_Anon(q'). There is no ablation that optimizes with MetricPAPILLON alone, and no control condition with a constant or random k-anonymity term. Since k-anonymity scores 'remain largely stable across all models and conditions' and the paper itself states that 'the objective is driven largely by the quality metric,' the observed quality gains (e.g., Llama-3.2-3B +13.2, Qwen-2.5-1.5B +17.3) are entirely consistent with SIMBA optimizing response quality alone. The abstract's claim that adding a k-anonymity objective improves delegation quality is therefore not identified by the reported experiments.
[Section 4.2 and Limitations] The load-bearing measurement instrument is the BRANCH-based k-anonymity estimator, which relies on GPT-5-mini's pretraining knowledge, a fallback population of 400M, and a simplified cumulative-conditioning structure. The Limitations section explicitly states that 'our estimators are not calibrated against or grounded in real population statistics.' Because the same kind of estimator is used to compute the optimization objective and to report the privacy outcome, the reported k-anonymity values cannot be interpreted as evidence about real re-identification risk. The sanity check in Section 5.1 (Spearman rho = -0.4045) only confirms monotonic behavior by construction, as the paper acknowledges. Without external validation against ground-truth population statistics or a concrete re-identification attack, the privacy-utility balance claim is unsupported.
[Section 4.3, optimization metric] The optimization metric is MetricPAPILLON + k_Anon(q'), with PII leakage deliberately excluded. The k-anonymity term is normalized as log2(k)/log2(400M), and the paper does not report sensitivity to this normalization or to the weights within MetricPAPILLON. Since k_Anon contributes little variation, the effective objective is quality-driven; any privacy improvements in Table 2 (e.g., Llama-3.2-3B leakage 29.1 to 11.0) are not attributable to the proposed mechanism, and Qwen models show increased leakage after optimization (e.g., Qwen-2.5-1.5B 54.40 to 62.95). The claim of 'best privacy-utility balance' for Llama-3.2-3B is therefore not tied to the proposed k-anonymity objective.

minor comments (5)

[Section 3.1] 'An GPT-5-based LLM judge' should be 'A GPT-5-based LLM judge.'
[Section 2.2] The acronym 'BRANCH' is never expanded; please define it or state that it is the system name from Zheng et al. (2025).
[Section 5.2] The composite metric is described as 'normalized,' but the normalization procedure is not specified; please state how MetricPAPILLON and k_Anon(q') are combined and normalized.
[Figure 2] The axes of Figure 2 are not labeled; the distribution of PII counts would be clearer with explicit x- and y-axis titles.
[References] Several references use 'and 1 others' or truncated author lists (e.g., Carlini et al. 2021, Zheng et al. 2023); please provide complete author lists or use the journal's abbreviation style consistently.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: acknowledged estimator limitations and a missing control are validity concerns, not circular reductions.

full rationale

This paper's derivation chain is not circular. The central empirical claim—that SIMBA optimization of PAPILLON on PUPA-SD improves held-out response quality and yields a favorable quality/leakage trade-off for Llama-3.2-3B—is evaluated on held-out PUPA-TNB with response-quality and PII-leakage measurements that are separate from the k-anonymity estimator used in the objective. The k-anonymity estimator is admittedly uncalibrated, as the Limitation states: 'our estimators are not calibrated against or grounded in real population statistics.' The Section 5.1 sanity check is explicitly labeled 'expected by construction' and 'not formal validation.' These are validity limitations, not circular reductions: the paper does not claim to predict external re-identification risk from the estimator. The absence of a MetricPAPILLON-only control in Section 5.2 means the causal contribution of the k-anonymity term is not identified, but that is an experimental confounding issue, not a case where a result is equivalent to its input by definition. Self-citations to PAPILLON/PUPA (Siyan et al., 2025a) supply the baseline and judges, but the paper's novel contribution—adding a BRANCH-based k-anonymity objective—is tested against held-out, externally sourced conversations rather than against the estimator's own outputs. No load-bearing step reduces to its own inputs.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim depends on an unvalidated LLM-based estimator, several hand-chosen scaling constants, and a small hand-filtered dataset. No new physical or theoretical entities are introduced.

free parameters (5)

Fallback population size (English speakers) = 400M
Used as the prior population and denominator in k-anonymity normalization when the estimator cannot determine a better base; chosen by hand, not from data.
k-anonymity normalization logarithm = log2(k)/log2(400M)
Hand-chosen rescaling to map k-anonymity to [0,1] while preserving distinctions at low k. Changes the objective's effective gradient.
Cosine similarity cache threshold = 0.95
Queries with cosine similarity above 0.95 reuse cached LLM estimates; affects the estimator's outputs by merging near-duplicate disclosures.
MetricPAPILLON weights = not specified
The weighted sum of response quality and prompt well-formedness is not defined numerically in the paper; the relative weight between utility and k-anonymity is therefore not reproducible.
Number of SIMBA demonstrations = 1
A single in-context demonstration is used during optimization; this hyperparameter affects prompt updates.

assumptions (4)

domain assumption BRANCH's LLM-elicited probability estimates accurately reflect real population statistics
Section 4.2 and Limitations: the estimator relies entirely on LLM-encoded census knowledge, which the authors state may be outdated or imprecise and is not calibrated.
domain assumption The cumulative conditioning groups capture the statistical dependencies among disclosures
Section 4.2: the paper simplifies BRANCH's Bayesian network to a sequence of cumulative groups; the authors acknowledge the independence assumption can under-estimate k-anonymity.
domain assumption The PII leakage and quality judges from Siyan et al. (2025a) are valid proxies
Section 4.1: 'we employ the same validated LLM judges from Siyan et al. (2025a)'; no new validation is reported in this paper.
domain assumption Optimization on PUPA-SD transfers to held-out PUPA-TNB
Section 5.2: the experimental design assumes that prompt updates learned on the 166-instance PUPA-SD improve held-out privacy-utility outcomes; without a control objective or error bars this is not established.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Beyond Direct Identifiers: Probabilistic Privacy Risk Estimation for Privacy-Conscious LLM Query Delegation." pith.science (2026). https://pith.science/paper/4ECM42KH

@misc{pith2026260809140,
  author       = {Pith},
  title        = {Pith review of: Beyond Direct Identifiers: Probabilistic Privacy Risk Estimation for Privacy-Conscious LLM Query Delegation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4ECM42KH}},
  note         = {Machine review of arXiv:2608.09140}
}

read the original abstract

Recent work on protecting privacy during user-LLM interactions often focuses on direct, explicit identifiers: the personally-identifiable information (PII) captured by standard detectors. One such approach is Privacy-Conscious Delegation (PCD), where a local LLM acts as an intermediary. However, privacy risk does not stem solely from explicit identifiers but also PII-free self-disclosures, leaving users identifiable through combinations of quasi-identifying traits. We investigate a probabilistic variant of PCD, where we augment its objectives with an LLM-driven probabilistic estimation of k-anonymity. To facilitate this, we first create the PUPA-SD dataset, which contains naturalistic user queries with self-disclosure. Our preliminary results indicate that optimizing PAPILLON on PUPA-SD improves quality on unseen conversations across a variety of local models and produces the best privacy-utility balance for Llama-3.2-3B, while smaller models struggle to jointly optimize quality and privacy. We propose k-anonymity as a useful auxiliary metric for tackling PCD.

Figures

Figures reproduced from arXiv: 2608.09140 by the authors.

**Figure 2.** The distribution of the number of PII units in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** The Pareto line plot (before [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 13 canonical work pages

[1]

These seed Group 1

Identify INDEPENDENT disclosures -- those with no statistical dependencies \ (typically location, age, gender). These seed Group 1

work page
[2]

age given location+gender)

Identify FIRST-WAVE disclosures -- those that depend ONLY on the independent disclosures \ from step 1 (e.g. age given location+gender). Add them to Group 1 as well. --> Group 1 = independent disclosures + first-wave disclosures

work page
[3]

Add them to form the next group

For each subsequent group: find all remaining disclosures whose EVERY dependency is \ already present in the previous group. Add them to form the next group. --> Group k = ALL items from Group k-1 PLUS the newly resolved disclosures

work page
[4]

"" disclosures:str= dspy.InputField(description=

Repeat until all disclosures are placed. **Critical format rule**: every group is CUMULATIVE -- it repeats every disclosure from \ all prior groups and then appends the new ones. Do NOT create a new group for items that \ can be resolved within the current group's context. **Example**: Disclosures: New York City (location), woman (gender), 25 year old (ag...

work page
[5]

This can help search engines understand the structure of your website and improve your website’s SEO

Use internal linking: Internal linking is the practice of linking to other pages on your website from within your content. This can help search engines understand the structure of your website and improve your website’s SEO

work page
[6]

Include your local keywords and information about the con- tent of the image in your alt tags

Use alt tags: Alt tags are short descriptions of the images on your website. Include your local keywords and information about the con- tent of the image in your alt tags

work page
[7]

This will help your website show up in local search results and attract people looking for piano stores in the area

Use local keywords: Incorporate local key- words related to Nashville and Middle Ten- nessee into your content. This will help your website show up in local search results and attract people looking for piano stores in the area

work page
[8]

"best piano store in Nashville

Use long-tail keywords: In addition to lo- cal keywords, use long-tail keywords that are more specific to your business. For example, if you’re a piano store in Nashville, use key- words like ""best piano store in Nashville"" or ""piano lessons in Nashville.""

work page

Show all 15 references

[9]

Make sure your content includes valuable information about your pi- ano store and the products and services you offer

Use quality content: High-quality content is essential for SEO. Make sure your content includes valuable information about your pi- ano store and the products and services you offer. This will help establish your website as an authority in the piano industry and attract more visitors
[10]

Include your local keywords and information about the content of the page in your meta descriptions

Use meta descriptions: Meta descriptions are brief snippets of text that appear under each of your website’s pages in search results. Include your local keywords and information about the content of the page in your meta descriptions
[13]

doing business as

Use social media: Use social media to pro- mote your website and reach a wider audience. Share your content on your social media plat- forms and encourage your followers to share your content with their own followers. I hope these tips help you create engaging and localized co...
[14]

KPop Dance Sensation
[15]

Consider what sets you apart from other K-Pop dance content creators and use that as inspiration for your brand name

K-Pop Starstruck It’s important to choose a name that reflects your brand and resonates with your target au- dience. Consider what sets you apart from other K-Pop dance content creators and use that as inspiration for your brand name
[2022]

InThe Eleventh International Confer- ence on Learning Representations

Quantifying memorization across neural lan- guage models. InThe Eleventh International Confer- ence on Learning Representations. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-V oss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erl...

2021 arXiv
[2025]

a nurse who livesin

Probabilistic reasoning with llms for privacy risk estimation. InThe Thirty-ninth Annual Confer- ence on Neural Information Processing Systems. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P Xing, an...

2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

These seed Group 1

Identify INDEPENDENT disclosures -- those with no statistical dependencies \ (typically location, age, gender). These seed Group 1

work page

[2] [2]

age given location+gender)

Identify FIRST-WAVE disclosures -- those that depend ONLY on the independent disclosures \ from step 1 (e.g. age given location+gender). Add them to Group 1 as well. --> Group 1 = independent disclosures + first-wave disclosures

work page

[3] [3]

Add them to form the next group

For each subsequent group: find all remaining disclosures whose EVERY dependency is \ already present in the previous group. Add them to form the next group. --> Group k = ALL items from Group k-1 PLUS the newly resolved disclosures

work page

[4] [4]

"" disclosures:str= dspy.InputField(description=

Repeat until all disclosures are placed. **Critical format rule**: every group is CUMULATIVE -- it repeats every disclosure from \ all prior groups and then appends the new ones. Do NOT create a new group for items that \ can be resolved within the current group's context. **Example**: Disclosures: New York City (location), woman (gender), 25 year old (ag...

work page

[5] [5]

This can help search engines understand the structure of your website and improve your website’s SEO

Use internal linking: Internal linking is the practice of linking to other pages on your website from within your content. This can help search engines understand the structure of your website and improve your website’s SEO

work page

[6] [6]

Include your local keywords and information about the con- tent of the image in your alt tags

Use alt tags: Alt tags are short descriptions of the images on your website. Include your local keywords and information about the con- tent of the image in your alt tags

work page

[7] [7]

This will help your website show up in local search results and attract people looking for piano stores in the area

Use local keywords: Incorporate local key- words related to Nashville and Middle Ten- nessee into your content. This will help your website show up in local search results and attract people looking for piano stores in the area

work page

[8] [8]

"best piano store in Nashville

Use long-tail keywords: In addition to lo- cal keywords, use long-tail keywords that are more specific to your business. For example, if you’re a piano store in Nashville, use key- words like ""best piano store in Nashville"" or ""piano lessons in Nashville.""

work page