Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

PRvL: Quantifying the Capabilities and Risks of Large Language Models for PII Redaction

T0 review · 5 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Instruction-tuned open-source LLMs, led by DeepSeek-Q1, can redact personally identifiable information from free text at over 99 percent accuracy while leaking almost nothing, with training strategy mattering more than model scale.

desk verdict Useful broad benchmark for LLM-based PII redaction, but the headline numbers rest on an unspecified edit-distance alignment and contradict its own tables; treat rankings as provisional pending a major revision. read the letter →

arxiv 2508.05545 v1 pith:JF3MLIHF submitted 2025-08-07 cs.CR

Leon Garza , Anantaa Kotal , Aritran Piplai , Lavanya Elluri , Prajit Das , Aman Chadha This is my paper
classification cs.CR
keywords PIIredactionlargelanguagemodelsinstructiontuningprivacyleakagenamedentityrecognitionretrieval-augmentedgenerationcross-domaingeneralizationSPriV
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to establish that open-source large language models can be adapted into high-accuracy PII redaction systems, and that the way they are adapted matters more than raw scale. It benchmarks dense, small, mixture-of-experts, long-context, and state-space models under fine-tuning, instruction tuning, and retrieval-augmented generation, measuring span-correct and label-exact redaction, output fidelity, and privacy leakage. The headline result is that instruction-tuned DeepSeek-Q1 reaches 0.994 accuracy and 0.981 recall under span-correct evaluation, and instruction tuning consistently beats fine-tuning and RAG across metrics. The paper also releases PRvL, an open-source suite of tuned models and evaluation tools, so organizations can redact PII in self-hosted environments without sending data to third parties.

What carries the argument

The evaluation machinery is a structural edit-distance alignment that converts differences between a model's generated output and the ground-truth redacted text into insertions, deletions, and substitutions, which are then counted as false negatives, false positives, and mislabels. On top of this alignment the paper defines two evaluation settings: span-correct (whether the right span was masked, regardless of label) and label-exact (whether both span and label match). This distinction, plus the SPriV leakage score, carries the argument by separating 'the PII is hidden' from 'the tag is right'. The training machinery is LoRA-based fine-tuning and instruction tuning.

What would settle it

Re-annotate a random sample of model outputs by human reviewers, judging each masked span without using the automatic edit-distance alignment, and compare human precision and recall to the paper's numbers. Also run the same models on real-world documents such as clinical notes or court records containing PII not generated from templates; if either the human-alignment numbers diverge by more than a few points or the accuracy on real documents falls well below the reported span-correct accuracy, the central claim is weakened.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that instruction tuning is the decisive adaptation strategy for LLM-based PII redaction, with instruction-tuned models outperforming fine-tuned and RAG counterparts on span detection, label precision, structural fidelity, and privacy leakage. The best configuration, instruction-tuned DeepSeek-Q1, achieves the highest span-correct accuracy (0.994) and recall (0.981), and also the lowest SPriV leakage (0.002), while a compact model remains competitive with much larger ones. The paper further shows that retrieval augmentation improves output fluency but is less reliable for strict redaction, and that cross-domain and cross-lingual generalization holds

Load-bearing premise

The evaluation pipeline assumes the edit-distance alignment between model output and ground truth correctly translates insertions, deletions, and substitutions into false-negative, false-positive, and mislabel counts, and that the AI4Privacy test sets are genuinely out-of-distribution for the models being evaluated.

Editorial extensions

If this is right

  • Organizations can deploy self-hosted open-source models, such as instruction-tuned DeepSeek-Q1, to reach near-0.99 redaction accuracy without sending data to third-party APIs.
  • Instruction tuning should be preferred over fine-tuning or retrieval augmentation when the goal is strict, low-leakage redaction; RAG is better reserved for fluency or domain grounding.
  • Compact models in the 3B-8B parameter range can match or beat much larger proprietary models, so high redaction accuracy does not require the largest available model.
  • The high mislabel counts in label-exact evaluation (thousands per model) indicate that typed redaction for compliance needs additional work even when span redaction is nearly perfect.
  • Cross-domain transfer to Spanish and Italian suggests that instruction-tuned models can be extended to new languages with little or no task-specific tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's edit-distance metric counts over-masking as a pure false positive, so a conservative redactor that masks extra tokens will appear less precise even if it never leaks PII; deployment choices should weight recall and leakage separately.
  • Because the evaluation datasets are synthetic, a natural next test is on real-world corpora such as clinical notes or legal filings, where accuracy on organic PII distributions may be lower than the reported 0.99 figures.
  • If instruction tuning is the active ingredient, the same recipe should transfer to other text-transformation tasks such as de-identification of clinical notes, anonymization of chat logs, or redaction of court transcripts, where instruction pairs can be generated from existing annotations.
  • The finding that RAG helps fluency but hurts redaction reliability suggests retrieval should be aimed at entity-type definitions or redaction policies rather than full example sentences.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes PRvL, an open-source suite and evaluation framework for using LLMs as PII redactors. It compares dense, small, MoE, long-range, and state-space model architectures under fine-tuning, instruction tuning, and RAG, with additional zero-shot baselines. The headline empirical claims are that instruction-tuned models, especially DeepSeek-Q1, achieve the highest span-correct and label-exact accuracy, that instruction tuning is critical for effective redaction, and that RAG improves fluency but is less reliable for strict redaction. The paper also reports cross-domain and cross-lingual results on Spanish, Italian, and an external English dataset, and releases code, checkpoints, and evaluation tools for reproducibility.

Significance. If the central claims hold, the paper provides valuable practical guidance for deploying open-source, self-hosted LLM-based PII redaction systems, with a broad comparison across architectures and training paradigms. The release of the toolkit and checkpoints is a concrete contribution, and the inclusion of both span-correct and label-exact evaluations, as well as a leakage metric, is useful for compliance-oriented applications. However, the numerical foundation of the claims is currently not verifiable: the edit-distance alignment underlying all span-level metrics is underspecified, several tables are internally inconsistent, and one claim about instruction tuning is contradicted by the paper's own tables. These issues must be resolved before the ranking can be accepted.

major comments (5)
  1. [Section V.A, Eqs. (2)-(6)] The definition of TP/FP/FN/TN is not operational. The text says the minimum edit distance is computed, but it never specifies the alignment algorithm, the costs of insertion/deletion/substitution, tie-breaking, or how token-level operations are aggregated into span-level counts. Accuracy is defined as (TP+TN)/(TP+FP+FN+TN), but TN is never defined at span level. Since the headline comparison separates DeepSeek-Q1 instruction-tuned from fine-tuned by only 0.001 accuracy (0.994 vs 0.993) in Table II, a different alignment could plausibly change the ranking. The evaluation code or an exact algorithmic description must be provided before the metrics are uniquely determined.
  2. [Section VI.B vs Tables III and IV] The text contradicts its own tables. Section VI.B states that fine-tuned Llama3.1-8B and DeepSeek-Q1 show mislabel counts of '3120 and 2764 in Table III', but Table III lists 2974 and 3033. It also states that Llama3.2-3B has SPriV '0.75 in Table IV', but Table IV lists 0.010 for instruction-tuned and 0.205 for RAG. These are not rounding differences; they are different numbers. The reader cannot tell which set of measurements is reported in the paper.
  3. [Table V vs Tables II and III] The paradigm comparison in Table V is internally inconsistent with the main results. For Llama3.2-3B (IT), Table V reports Span Acc 0.992, Label Acc 0.992, and Mislabels 2968, whereas Tables II and III report 0.983, 0.982, and 2673. The value 2968 is exactly the mislabel count of Llama3.1-8B (IT) in Table III. This suggests that rows were copied or mislabeled, and it makes the 'instruction tuning consistently improves' conclusion unreliable.
  4. [Section VI.A and Table III] The claim that 'instruction-tuned variants reduce this number' of mislabels is not supported by Table III. For DeepSeek-Q1, mislabels are 3033 (fine-tuned) and 3047 (instruction-tuned), so instruction tuning increases the count. For Llama3.2-3B, the counts are 2005 (fine-tuned) and 2673 (instruction-tuned), also an increase. Only Llama3.1-8B shows a small decrease (2974 to 2968). The summary statement is factually contradicted by the presented data.
  5. [Section III.E and Section VI.H] The RAG evaluation may be partially in-sample. Section III.E states that the retrieval corpus includes 'previously annotated redaction pairs or curated templates', but the paper does not state whether the retrieval index for the main AI4Privacy evaluation is disjoint from the test set. Section VI.H explicitly says the external English dataset was 'not used during training or RAG construction', which implies the same care was needed for the main evaluation. If test examples or their annotated outputs are present in the retrieval index, the RAG condition is a retrieval-memorization benchmark rather than a test of context-aware redaction. The paper must state the overlap status and, if necessary, remove overlapping examples.
minor comments (5)
  1. [Table IV] Typo in Llama3.2-3B (Fine-Tuned) row: ROUGE-1 is listed as '0..602' with a double period.
  2. [Section VI.D] The text says 'Instruction-tuned models generally show reduced F1', but Section VI.A and Figures 3-5 consistently report instruction-tuned models with higher or comparable F1. Please clarify which comparison is intended.
  3. [Section III.D.2] The Instruction-Tuning Example is labeled with a 'Fine-Tuning Example' header, which is confusing. The example also appears to be duplicated from the fine-tuning section.
  4. [Section VI.B.2] The example of underredaction says Llama3.2-3B exhibits SPriV '0.75 in Table IV'; Table IV shows no 0.75 value. This appears to reference a different evaluation or a typo.
  5. [References] Several reference entries are incomplete or imprecise: [13] lacks an arXiv identifier, [15] and [16] describe DeepSeek-R1 and DeepSeek-Q1 with generic 'retrieval-augmented' descriptions that do not match the models used.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical benchmark whose claims are evaluated with externally defined metrics and standard held-out testing; no load-bearing self-citation or definitional reduction is present.

full rationale

The paper's central claims are empirical comparisons of LLM redaction performance across architectures and training strategies. There is no derivation chain that reduces a predicted quantity to a fitted input by construction. The evaluation uses held-out test splits (Section IV.B), external datasets for cross-domain testing (Section V.D), and established or clearly operationalized metrics: ROUGE/BLEU for sequence fidelity, SPriV from cited prior work for leakage, and a custom edit-distance alignment for span-level correctness. The custom metric defines TP/FP/FN via edit operations (Section V.A), which is an evaluation protocol rather than a fitted parameter; it is under-specified and may affect the reproducibility of exact numbers, but it is not circular because it is not defined in terms of the models' outputs or trained parameters. No uniqueness theorem, ansatz, or load-bearing argument is imported from the authors' own prior work; the self-citations present are ordinary references to prior empirical results. The RAG retrieval corpus is described as containing 'previously annotated redaction pairs' (Section III.E.2), and the paper does not state whether the index excludes the test split; that is a possible contamination risk, not a demonstrated circular reduction. Internal inconsistencies (e.g., Table V's Llama3.2-3B IT numbers differing from Tables II/III) are correctness concerns, not circularity. Overall, the paper's conclusions rest on external benchmarks and held-out evaluations rather than on assumptions that already contain the conclusions.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the correctness of the AI4Privacy benchmark labels, the validity of the custom edit-distance metric, and the assumption that translated synthetic templates constitute genuine cross-domain tests. The LoRA and RAG hyperparameters are untracked free parameters that could change the rankings. No new physical or conceptual entities are introduced.

free parameters (3)
  • LoRA rank/alpha/learning rate/warmup/epochs = not reported
    Section IV.A.2 says hyperparameters were selected via grid search per model class; none are listed, and every reported score depends on them.
  • RAG retrieval top-k and similarity threshold = not reported
    Section III.E.1 specifies top-k exemplars from a pre-encoded index and optional filters but never states k, the index, or the thresholds used in Tables II-IV.
  • Number of training and instruction-tuning examples = not reported
    Section III.D describes fine-tuning on parallel corpora and instruction tuning on a curated set of examples, but the corpus size, mix, and example count are unspecified.
assumptions (4)
  • domain assumption AI4Privacy-300K/500K annotations are accurate and representative
    Section IV.B states evaluation uses these synthetic datasets with rich PII annotations; the entire benchmark inherits their label schema and error distribution.
  • domain assumption Edit-distance alignment between model output and reference correctly identifies false positives/negatives
    Section V.A defines TP/FP/FN through insertions, deletions, substitutions in a minimum edit sequence, but the alignment algorithm is not specified, so results assume it is correct.
  • domain assumption Spanish and Italian test sets are valid out-of-domain data
    Section V.D says examples are professionally translated or synthetically generated from English templates with spans re-aligned; this makes cross-lingual transfer partially a template-following exercise.
  • domain assumption SPriV from PrivacyMind [11] correctly measures privacy leakage
    Section V.C adopts SPriV as the leakage metric without revalidation in the redaction setting.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PRvL: Quantifying the Capabilities and Risks of Large Language Models for PII Redaction." pith.science (2026). https://pith.science/paper/JF3MLIHF

@misc{pith2026250805545,
  author       = {Pith},
  title        = {Pith review of: PRvL: Quantifying the Capabilities and Risks of Large Language Models for PII Redaction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JF3MLIHF}},
  note         = {Machine review of arXiv:2508.05545}
}
read the original abstract

Redacting Personally Identifiable Information (PII) from unstructured text is critical for ensuring data privacy in regulated domains. While earlier approaches have relied on rule-based systems and domain-specific Named Entity Recognition (NER) models, these methods fail to generalize across formats and contexts. Recent advances in Large Language Models (LLMs) offer a promising alternative, yet the effect of architectural and training choices on redaction performance remains underexplored. LLMs have demonstrated strong performance in tasks that require contextual language understanding, including the redaction of PII in free-form text. Prior work suggests that with appropriate adaptation, LLMs can become effective contextual privacy learners. However, the consequences of architectural and training choices for PII Redaction remain underexplored. In this work, we present a comprehensive analysis of LLMs as privacy-preserving PII Redaction systems. We evaluate a range of LLM architectures and training strategies for their effectiveness in PII Redaction. Our analysis measures redaction performance, semantic preservation, and PII leakage, and compares these outcomes against latency and computational cost. The results provide practical guidance for configuring LLM-based redactors that are accurate, efficient, and privacy-aware. To support reproducibility and real-world deployment, we release PRvL, an open-source suite of fine-tuned models, and evaluation tools for general-purpose PII Redaction. PRvL is built entirely on open-source LLMs and supports multiple inference settings for flexibility and compliance. It is designed to be easily customized for different domains and fully operable within secure, self-managed environments. This enables data owners to perform redactions without relying on third-party services or exposing sensitive content beyond their own infrastructure.

Figures

Figures reproduced from arXiv: 2508.05545 by the authors.

Figure 1
Figure 1. End-to-end decision workflow for training or deploying a PII-aware redaction language model. The diagram outlines multiple adaptation paths: Fine-Tuning, Instruction Tuning, and Retrieval￾Augmented Generation (RAG) and model selection spanning both proprietary (P) and open-source (OS) architectures. open problem—especially for downstream tasks like redaction, where private information may surface implicitly through … view at source ↗
Figure 2
Figure 2. Overview of the RAG-based redaction pipeline: retrieved examples inform context-aware masking. 1) Retrieval-Augmented Generation (RAG): RAG enhances redaction performance by explicitly conditioning the model on retrieved context relevant to the input. This enables the model to resolve ambiguous cases, handle rare PII types, and follow domain-specific redaction conventions. The RAG pipeline involves three stages (ill… view at source ↗
Figure 3
Figure 3. This graph visualizes the trade-off between GPU usage and model performance. All experiments were run on two 48GB NVIDIA RTX 6000 GPUs. The plot is divided into quadrants: the top-left represents the optimal trade-off—high performance with low GPU usage. The top-right indicates high performance at high computational cost, while the bottom-left reflects both low resource usage and low performance [PITH_FULL_IMAGE:fi… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: This plot illustrates the trade-off between inference latency (ms) and F1 score for generating 150 tokens. Models are categorized by training strategy (FT: fine-tuned, IT: instruction-tuned). The top￾left quadrant indicates the optimal balance—high F1 with low latency.…
Figure 5
Figure 5. Figure 5: This plot illustrates the trade-off between model size (in bil￾lions of parameters) and F1 score. The x-axis uses a log scale. Models in the top-left quadrant achieve high F1 with compact architectures, including DeepSeek-Q1(IT), LLaMA 3.2–3B(FT), and DeepSeek￾Q1(FT). …

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PromptPET: Privacy-Utility Optimized Prompt Obfuscation

    cs.CR 2026-07 conditional novelty 7.0 of 10

    PromptPET selectively applies four obfuscation actions (including novel noising) via an OPRO-style rule optimizer to match single-action privacy-utility frontiers and outperform prior prompt-minimization methods on Wi...

Reference graph

Works this paper leans on

55 extracted references · 37 canonical work pages · cited by 1 Pith paper

  1. [1]

    Summary of the hipaa privacy rule — hhs.gov,

    “Summary of the hipaa privacy rule — hhs.gov,” 3 2025, [Online; accessed 2025-06-20]. [Online]. Available: https://www.hhs.gov/hipaa/ for-professionals/privacy/laws-regulations/index.html

    work page 2025

  2. [2]

    Automated de-identification of free-text medical records,

    I. Neamatullah, M. M. Douglass, L.-W. H. Lehman, A. Reisner, M. Vil- larroel, W. J. Long, P. Szolovits, G. B. Moody, R. G. Mark, and G. D. Clifford, “Automated de-identification of free-text medical records,” BMC medical informatics and decision making , vol. 8, pp. 1–17, 2008

    work page 2008

  3. [3]

    Legal aid data breach leaks millions of sensitive records, moj’s poor cybersecurity practices slammed - cpo magazine,

    A. Hope, “Legal aid data breach leaks millions of sensitive records, moj’s poor cybersecurity practices slammed - cpo magazine,” 5 2025, [Online; accessed 2025-06-20]. [Online]. Available: https: //tinyurl.com/53d697hz

    work page 2025

  4. [4]

    Extracting training data from large language models,

    N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-V oss, K. Lee, A. Roberts, T. Brown, D. Song, U. Erlingsson et al., “Extracting training data from large language models,” in 30th USENIX security symposium (USENIX Security 21) , 2021, pp. 2633–2650

    2021

  5. [5]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Pro- ceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers) , 2019, pp. 4171–4186

    2019

  6. [6]

    Introduction to the conll-2003 shared task: Language-independent named entity recognition,

    E. F. Sang and F. De Meulder, “Introduction to the conll-2003 shared task: Language-independent named entity recognition,” arXiv preprint cs/0306050, 2003

    arXiv 2003

  7. [7]

    Ontonotes: A unified relational semantic representa- tion,

    S. S. Pradhan, E. Hovy, M. Marcus, M. Palmer, L. Ramshaw, and R. Weischedel, “Ontonotes: A unified relational semantic representa- tion,” in International Conference on Semantic Computing (ICSC 2007), 2007, pp. 517–526

    work page 2007

  8. [8]

    Natural Language Processing – Amazon Comprehend,

    Amazon Web Services, “Natural Language Processing – Amazon Comprehend,” AWS, 2025, [Online]. Available: URL [Accessed: Date]. [Online]. Available: https://aws.amazon.com/comprehend/

    work page 2025

Show all 55 references
  1. [9]

    Presidio - data protection and de-identification sdk

    Microsoft, “Presidio - data protection and de-identification sdk.” [Online]. Available: https://github.com/microsoft/presidio

  2. [10]

    Cloud data loss prevention — google cloud,

    “Cloud data loss prevention — google cloud,” [Online; accessed 2025- 06-21]. [Online]. Available: https://cloud.google.com/security/products/ dlp

    2025

  3. [11]

    Privacymind: large language models can be contextual privacy protection learners,

    Y . Xiao, Y . Jin, Y . Bai, Y . Wu, X. Yang, X. Luo, W. Yu, X. Zhao, Y . Liu, Q. Gu et al. , “Privacymind: large language models can be contextual privacy protection learners,” arXiv preprint arXiv:2310.02469 , 2023

    2023 arXiv

  4. [12]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar et al. , “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  5. [13]

    Gpt-4 technical report,

    OpenAI, “Gpt-4 technical report,” https://openai.com/research/gpt-4, 2023

    2023

  6. [14]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel et al., “Exploring the limits of transfer learning with a unified text-to-text transformer,” JMLR, 2020

    2020

  7. [15]

    Mixtral of experts,

    M. AI, “Mixtral of experts,” https://mistral.ai/news/mixtral-of-experts/, 2023

    2023

  8. [16]

    Deepseek-r: Retrieval-augmented language models,

    DeepSeek, “Deepseek-r: Retrieval-augmented language models,” https: //github.com/deepseek-ai, 2024

    2024

  9. [17]

    Deepseek-q: Mixture of experts for multitask language under- standing,

    ——, “Deepseek-q: Mixture of experts for multitask language under- standing,” https://github.com/deepseek-ai, 2024

    2024

  10. [18]

    Falconmamba: Combining falcon and mamba for efficient long-context modeling,

    T. I. Institute, “Falconmamba: Combining falcon and mamba for efficient long-context modeling,” https://huggingface.co/tiiuae/ falcon-mamba, 2024

    2024

  11. [19]

    Rule-based information extraction is dead! long live rule-based information extraction systems!

    L. Chiticariu, Y . Li, and F. Reiss, “Rule-based information extraction is dead! long live rule-based information extraction systems!” EMNLP 2013 - 2013 Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference , vol. October, pp. 827–832, 01 2013

    2013

  12. [20]

    Automated pii extraction from social media for raising privacy awareness: A deep transfer learning approach,

    Y . Liu, F. Y . Lin, M. Ebrahimi, W. Li, and H. Chen, “Automated pii extraction from social media for raising privacy awareness: A deep transfer learning approach,” in 2021 IEEE International Conference on Intelligence and Security Informatics (ISI) . IEEE, 2021, pp. 1–6

    2021

  13. [21]

    Automatic de-identification of textual documents in the electronic health record: a review of recent research,

    S. M. Meystre, F. J. Friedlin, B. R. South, S. Shen, and M. H. Samore, “Automatic de-identification of textual documents in the electronic health record: a review of recent research,” BMC medical research methodology, vol. 10, pp. 1–16, 2010

    2010

  14. [22]

    The mitre identification scrubber toolkit: design, training, and assessment,

    J. Aberdeen, S. Bayer, R. Yeniterzi, B. Wellner, C. Clark, D. Hanauer, B. Malin, and L. Hirschman, “The mitre identification scrubber toolkit: design, training, and assessment,” International journal of medical informatics, vol. 79, no. 12, pp. 849–859, 2010

    2010

  15. [23]

    Protected health information filter (philter): accurately and securely de- identifying free-text clinical notes,

    B. Norgeot, K. Muenzen, T. A. Peterson, X. Fan, B. S. Glicksberg, G. Schenk, E. Rutenberg, B. Oskotsky, M. Sirota, J. Yazdany et al. , “Protected health information filter (philter): accurately and securely de- identifying free-text clinical notes,” NPJ digital medicine , vol....

    2020

  16. [24]

    Computer-assisted de-identification of free text in the mimic ii database,

    M. Douglass, G. D. Clifford, A. Reisner, G. B. Moody, and R. G. Mark, “Computer-assisted de-identification of free text in the mimic ii database,” in Computers in Cardiology, 2004. IEEE, 2004, pp. 341–344

    2004

  17. [25]

    Scanning electronic documents for personally identifiable information,

    T. Aura, T. A. Kuhn, and M. Roe, “Scanning electronic documents for personally identifiable information,” in Proceedings of the 5th ACM workshop on Privacy in electronic society , 2006, pp. 41–50

    2006

  18. [26]

    Unmasking the reality of pii masking models: Performance gaps and the call for accountability,

    D. Singh and S. Narayanan, “Unmasking the reality of pii masking models: Performance gaps and the call for accountability,”arXiv preprint arXiv:2504.12308, 2025

    2025 arXiv

  19. [27]

    A review of automatic end-to-end de-identification: Is high accuracy the only metric?

    V . Yogarajan, B. Pfahringer, and M. Mayo, “A review of automatic end-to-end de-identification: Is high accuracy the only metric?” Applied Artificial Intelligence, vol. 34, no. 3, pp. 251–269, 2020

    2020

  20. [28]

    De-identification of patient notes with recurrent neural networks,

    F. Dernoncourt, J. Y . Lee, O. Uzuner, and P. Szolovits, “De-identification of patient notes with recurrent neural networks,”Journal of the American Medical Informatics Association , vol. 24, no. 3, pp. 596–606, 2017

    2017

  21. [29]

    De-identification of electronic health record using neural network,

    T. Ahmed, M. M. Al Aziz, and N. Mohammed, “De-identification of electronic health record using neural network,” Scientific Reports , vol. 10, no. 1, pp. 1–11, 2020

    2020

  22. [30]

    Deidentification of free- text medical records using pre-trained bidirectional transformers,

    A. E. Johnson, L. Bulgarelli, and T. J. Pollard, “Deidentification of free- text medical records using pre-trained bidirectional transformers,” in Proceedings of the ACM Conference on Health, Inference, and Learning. ACM, 2020, pp. 214–221

    2020

  23. [31]

    Building a best-in-class automated de-identification tool for electronic health records through ensemble learning,

    K. Murugadoss, A. Rajasekharan, B. Malin, V . Agarwal, S. Bade, J. R. Anderson, J. L. Ross, W. A. J. Faubion, J. D. Halamka, V . Soundararajan et al. , “Building a best-in-class automated de-identification tool for electronic health records through ensemble learning,” Patterns...

    2021

  24. [32]

    Privacy-preserving prompt tuning for large language model services,

    Y . Li, Z. Tan, and Y . Liu, “Privacy-preserving prompt tuning for large language model services,” arXiv preprint arXiv:2304.11635 , 2023

    2023 arXiv

  25. [33]

    Enhancing small medical learners with privacy-preserving contextual prompting,

    X. Zhang, S. Li, X. Yang, C. Tian, Y . Qin, and L. R. Petzold, “Enhancing small medical learners with privacy-preserving contextual prompting,” arXiv preprint arXiv:2308.04621 , 2023

    2023 arXiv

  26. [34]

    Propile: Probing privacy leakage in large language models,

    S. Kim, S. Yun, H. Lee, M. Gubri, S. Yoon, and S. J. Oh, “Propile: Probing privacy leakage in large language models,” in Advances in Neural Information Processing Systems , 2023

    2023

  27. [35]

    Analyzing leakage of personally identifiable information in language models,

    N. Lukas, A. Salem, R. Sim, S. Tople, L. Wutschitz, and S. Zanella- B´eguelin, “Analyzing leakage of personally identifiable information in language models,” arXiv preprint arXiv:2305.17303 , 2023

    2023 arXiv

  28. [36]

    Privacy as contextual integrity,

    H. Nissenbaum, “Privacy as contextual integrity,” Washington Law Review, vol. 79, pp. 119–158, 2004

    2004

  29. [37]

    Can llms keep a secret? testing privacy implications of language models via contextual integrity theory,

    N. Mireshghallah, H. Kim, X. Zhou, Y . Tsvetkov, M. Sap, R. Shokri, and Y . Choi, “Can llms keep a secret? testing privacy implications of language models via contextual integrity theory,” in International Conference on Learning Representations (ICLR) , 2024

    2024

  30. [38]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, P. Wallis, Z. Allen-Zhu et al. , “Lora: Low-rank adaptation of large language models,” in International Conference on Learning Representations (ICLR), 2022

    2022

  31. [39]

    The power of scale for parameter-efficient prompt tuning,

    B. Lester, R. Al-Rfou, and N. Constant, “The power of scale for parameter-efficient prompt tuning,” in Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing , 2021, pp. 3045–3059

    2021

  32. [40]

    Prefix-tuning: Optimizing continuous prompts for generation,

    X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” arXiv preprint arXiv:2101.00190 , 2021

    2021 arXiv

  33. [41]

    Finetuned language models are zero-shot learners,

    J. Wei, M. Bosma, V . Y . Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V . Le, “Finetuned language models are zero-shot learners,” arXiv preprint arXiv:2109.01652 , 2021

    2021 arXiv

  34. [42]

    Stanford alpaca: An instruction-following llama model,

    R. Taori, I. Gulrajani, T. Zhang, Y . Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto, “Stanford alpaca: An instruction-following llama model,” 2023

    2023

  35. [43]

    Retrieval- augmented generation for knowledge-intensive nlp tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W.-t. Yih, T. Rockt ¨aschel et al. , “Retrieval- augmented generation for knowledge-intensive nlp tasks,” Advances in neural information processing systems , vol. 33, pp. 9459–9474, 2020

    2020

  36. [44]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al. , “Training language models to follow instructions with human feedback,” Advances in neural information processing systems , vol. 35, pp. 27 730–27 744, 2022

    2022

  37. [45]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,” Journal of Machine Learning Research, vol. 21, no. 140, pp. 1–67, 2020

    2020

  38. [46]

    Mistral 7b,

    B. Jiang et al., “Mistral 7b,” arXiv preprint arXiv:2310.06825 , 2023

    2023 arXiv

  39. [47]

    Mixtral of experts: Sparse mixture of experts models are extremely effective,

    E. Beeching, G. Izacard, H. Touvron et al., “Mixtral of experts: Sparse mixture of experts models are extremely effective,” arXiv preprint arXiv:2312.15842, 2023

    2023 arXiv

  40. [48]

    Mamba: Linear- time sequence modeling with selective state spaces,

    A. Gu, T. Dao, Y . He, A. R. Fu, C. Re et al. , “Mamba: Linear- time sequence modeling with selective state spaces,” arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  41. [49]

    Improving language models by retrieving from trillions of tokens,

    S. Borgeaud, A. Mensch, J. Hoffmann et al. , “Improving language models by retrieving from trillions of tokens,” in Proceedings of the 39th International Conference on Machine Learning , 2022

    2022

  42. [50]

    Quantifying memorization across neural language models,

    N. Carlini, D. Ippolito, M. Jagielski et al., “Quantifying memorization across neural language models,” arXiv preprint arXiv:2202.07646, 2022

    2022 arXiv

  43. [51]

    Large language models can be strong differentially private learners,

    X. Li, F. Tramer, P. Liang, and T. Hashimoto, “Large language models can be strong differentially private learners,” in International Conference on Learning Representations , 2022

    2022

  44. [52]

    Privacylens: Evaluating privacy norm awareness of language models in action,

    Y . Shao, T. Li, W. Shi, Y . Liu, and D. Yang, “Privacylens: Evaluating privacy norm awareness of language models in action,” arXiv preprint arXiv:2409.00138, 2024

    2024 arXiv

  45. [53]

    Can large language models really recognize your name?

    D. Pham, P. Kairouz, N. Mireshghallah, E. Bagdasarian, C. M. Pham, and A. Houmansadr, “Can large language models really recognize your name?” arXiv preprint arXiv:2505.14549 , 2025

    2025 arXiv

  46. [54]

    ai4privacy/pii-masking-300k · datasets at hugging face,

    “ai4privacy/pii-masking-300k · datasets at hugging face,” 6 2024, [Online; accessed 2025-06-21]. [Online]. Available: https://huggingface. co/datasets/ai4privacy/pii-masking-300k

    2024

  47. [55]

    ai4privacy/open-pii-masking-500k-ai4privacy · datasets at hugging face,

    “ai4privacy/open-pii-masking-500k-ai4privacy · datasets at hugging face,” 3 2025, [Online; accessed 2025-06-21]. [Online]. Available: https: //huggingface.co/datasets/ai4privacy/open-pii-masking-500k-ai4privacy VIII. A PPENDIX TABLE VIII: Training resource requirements: GPU ho...

    2025

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.