REVIEW 3 major objections 5 minor 42 references
LegalGuardian: A Privacy-Preserving Framework for Secure Integration of Large Language Models in Legal Practice
T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read LegalGuardian claims a masking/unmasking pipeline lets lawyers use external LLMs without leaking client PII, while keeping outputs nearly unchanged.
desk verdict A plausible sanitization pipeline whose own recall numbers contradict the confidentiality guarantee; the circular evaluation and tiny synthetic dataset make the quantitative claims unsupported. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is an entity-dictionary masking layer: each detected PII span is replaced with a consistent typed placeholder such as [PERSON 1] or [ADDRESS 1], and the same dictionary is used after the LLM call to substitute the original values back into the output. Two detectors feed this layer: GLiNER, a BERT-based NER model adapted for custom entity types, and Qwen2.5-14B, a local LLM prompted one-shot to output JSON entity spans. The integrity check is semantic similarity between outputs from masked and unmasked prompts, measured with cosine, Jaro-Winkler, and Levenshtein metrics.
What would settle it
Run LegalGuardian on a corpus of real, de-identified immigration-lawyer prompts with ground-truth PII annotations, then measure entity-level F1 and cosine similarity between masked and unmasked outputs; if F1 falls materially below the reported 93% and 97% or cosine similarity drops well below 0.97, the framework's effectiveness claim is not general.
Extended reading notes
Core claim
The paper claims that a lightweight masking/unmasking pipeline can preserve both client confidentiality and output quality when lawyers interact with external LLMs. In tests on 50 synthetic immigration-law prompts containing 460 entities, GLiNER achieves 93% F1 and Qwen2.5-14B achieves 97% F1 for PII detection; GLiNER shows perfect precision while Qwen shows higher recall. Cosine similarity between outputs generated from masked and unmasked prompts is about 0.98, which the paper reads as evidence that masking does not materially distort meaning.
Load-bearing premise
The evaluation assumes the 50 synthetic prompts, built from Faker-generated data and fixed templates, are representative of real immigration-lawyer workflows; if genuine prompts contain different entity distributions, phrasings, or domain jargon, the measured detection and fidelity numbers may not transfer.
Editorial extensions
If this is right
- Legal professionals could send masked prompts to external LLM services without manually reviewing each prompt for confidential information.
- A hybrid configuration, using GLiNER as the precision-focused primary detector and Qwen2.5-14B as a recall-boosting supplement, would combine the reported strengths of both models.
- The masking/unmasking pattern could extend beyond immigration law to other legal practice areas and to cloud-based LLM services, as the paper states in its future-work section.
- The reported cosine similarity of about 0.98 suggests that downstream tasks such as summarization, translation, and drafting would retain most of their semantic content after masking.
- If the framework is adopted, lawyers could comply with professional confidentiality duties while still gaining the productivity benefits of generative AI.
Reading between the lines
- Editorial inference: the synthetic prompts built from Faker data and fixed templates likely underestimate the variety of real lawyer prompts, so the reported F1 and similarity numbers should be re-tested on genuine, de-identified case materials.
- Editorial inference: placeholder-based masking preserves cross-reference structure but may still leak information through statistical patterns, so a stronger privacy claim would require adversarial re-identification testing beyond the paper's current metrics.
- Editorial inference: the same masking/unmasking architecture could transfer to other regulated domains such as healthcare or finance, where similar confidentiality duties and PII classes exist, but the paper does not test those settings.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces LegalGuardian, a framework that masks personally identifiable information (PII) in legal prompts before they are sent to an external LLM, then unmaps the placeholders in the LLM response using a maintained entity dictionary. The framework is evaluated on 50 synthetic immigration-law prompts generated with Faker data and Qwen2.5-14B, comparing a GLiNER-based NER pipeline with a Qwen2.5-14B one-shot prompted pipeline. The reported results are F1 scores of 93% for GLiNER and 97% for Qwen2.5-14B for PII detection, and high semantic similarity between outputs from masked and unmasked prompts. The authors conclude that legal professionals can use external LLMs without compromising client confidentiality or output quality.
Significance. The problem addressed is important: lawyers need practical, lightweight safeguards when using third-party LLM services, and the proposed masking/unmasking pipeline is a reasonable design direction. The manuscript is clearly written and the pipeline is simple enough to be reproducible. However, the significance is conditional on the evaluation supporting the confidentiality claim. The central claim that LegalGuardian protects client confidentiality 'before any external interaction' is undermined by the paper's own entity-level recall numbers, and the evaluation is largely circular because the same model family generates the test data, performs the detection, and produces the outputs used for similarity measurement. If revised with an independent gold-standard evaluation and a more careful formulation of the privacy guarantee, the framework could be a useful contribution to privacy-preserving LLM use in legal practice.
major comments (3)
- [Results, Table 1 and Figure 3] The central confidentiality claim is contradicted by the entity-level recall reported in Table 1 and Figure 3. For example, GLiNER recall for person is 0.72 and for location is 0.65, and Qwen2.5-14B recall for person is 0.77; these values mean that on the paper's own test set, a substantial fraction of client names and locations are not masked and are therefore transmitted to the external LLM. The abstract and conclusion state that LegalGuardian safeguards data 'before any external interaction' and that lawyers can use LLMs 'without compromising client confidentiality,' but the reported false-negative rates imply the opposite for a nontrivial fraction of entities. The paper should explicitly define an acceptable residual-risk threshold, report the per-prompt false-negative rate, and discuss the operational consequences of a missed entity. Without this, the headline F1 scores (93% and 97%) are misleading for a privacy-protection system, where a false negative is a direct privacy failure rather than a mere performance loss.
- [Methodology, Synthetic Data and Content Generation; Results, Overall Accuracy] The evaluation is circular: Qwen2.5-14B is used to generate the 50 synthetic prompts from Faker data and templates, the same Qwen2.5-14B model is then used as the PII detector, and the same model also generates both the baseline outputs and the masked-prompt outputs used for semantic similarity. The reported detection performance is therefore partly a measure of Qwen's ability to find entities in text that Qwen itself produced, which does not transfer to real lawyer-written prompts. The manuscript needs an independent gold-standard evaluation, ideally on real or human-annotated legal prompts, with detection and evaluation performed by models different from the data generator, and with confidence intervals or significance tests. Without such evidence, the claimed F1 scores cannot support the framework's deployment-oriented conclusions.
- [Results, Semantic Similarity] The semantic similarity evaluation is not a valid proxy for utility with a real external LLM. The paper states in the Secure Prompting Layer that a local instance of Qwen2.5-14B was used 'treating it as an external LLM to mimic real-world conditions,' but this does not measure the behavior of commercial or cloud-based LLMs, which are the actual target systems. Moreover, both the baseline and the masked-prompt outputs come from the same model, so the high cosine similarity (0.98 for GLiNER, 0.97 for Qwen) may partly reflect the model's tendency to preserve template structure rather than genuine semantic fidelity. The authors should evaluate with at least one actual external LLM, report per-task-type similarity statistics rather than only means, and discuss cases where the external model fails to preserve placeholders, since such failures would break the unmasking step.
minor comments (5)
- [Appendix, Listings 1 and 2] The appendix contains two listings both labeled 'Listing 1' (one for fake PII generation and one for fake prompt generation); renumber the listings and reference them explicitly in the main text.
- [Methodology, Evaluation Layer] The text reads 'stored and organized using thePandas library' and 'calculated usingSpaCy'; add spaces and proper formatting for the library names.
- [Throughout] The model name is written inconsistently as 'Qwen-2.5 14B' and 'Qwen2.5-14B'; use a single consistent notation.
- [Introduction to LegalGuardian Framework] The claim that LegalGuardian is 'the first privacy protection framework explicitly designed to safeguard attorney-client confidentiality' is too strong given the related work on PP-TS and Hide-and-Seek discussed later in the paper; soften this claim to avoid overstatement.
- [Results, Table 2] Table 2 reports Levenshtein Distance as a similarity metric, but a higher Levenshtein distance indicates less similarity; clarify the direction of the metric in the table caption or in the text.
Circularity Check
No significant circularity: the evaluation is an empirical measurement with manually curated ground truth, not a derivation equivalent to its inputs.
full rationale
The paper's central claims are that LegalGuardian masks PII before external LLM interaction and that masking preserves output utility. Neither claim is derived by definition from the inputs. PII detection performance is measured against manually curated ground-truth annotations of 460 entities in 50 prompts (Results & Findings, Overall Accuracy), with true positives, false positives, and false negatives identified by human review; no parameter is fitted to the evaluation set and then renamed as a prediction. The semantic similarity analysis compares baseline outputs from unmasked prompts with unmasked outputs from the masked pipeline using cosine, Jaro-Winkler, and Levenshtein metrics, which is an empirical measurement rather than an identity. The main methodological weakness—that Qwen2.5-14B generated the synthetic prompts and also served as the PII detector and as the 'external' LLM for similarity evaluation—is a limitation on external validity and a potential same-model bias, but it does not make the reported F1 or similarity scores equivalent to the paper's inputs by construction. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via citation. The confidentiality claim is undermined by recall values below 1.0 in Table 1, but that is a correctness or safety concern, not circularity. Therefore the appropriate circularity score is 0.
Assumptions & free parameters
assumptions (4)
- domain assumption The 50 synthetic prompts generated by Qwen2.5-14B are representative of real immigration law practice.
- domain assumption The entity types selected (person, address, date, etc.) cover the PII that must be protected in legal prompts.
- ad hoc to paper Using the same LLM (Qwen2.5-14B) to generate test data, detect entities, and evaluate outputs does not bias the results.
- domain assumption Semantic similarity metrics are a valid proxy for downstream utility in legal tasks.
Cite this review
Pith. "Pith review of LegalGuardian: A Privacy-Preserving Framework for Secure Integration of Large Language Models in Legal Practice." pith.science (2026). https://pith.science/paper/4AZSR7K7
@misc{pith2026250110915,
author = {Pith},
title = {Pith review of: LegalGuardian: A Privacy-Preserving Framework for Secure Integration of Large Language Models in Legal Practice},
year = {2026},
howpublished = {\url{https://pith.science/paper/4AZSR7K7}},
note = {Machine review of arXiv:2501.10915}
}
read the original abstract
Large Language Models (LLMs) hold promise for advancing legal practice by automating complex tasks and improving access to justice. However, their adoption is limited by concerns over client confidentiality, especially when lawyers include sensitive Personally Identifiable Information (PII) in prompts, risking unauthorized data exposure. To mitigate this, we introduce LegalGuardian, a lightweight, privacy-preserving framework tailored for lawyers using LLM-based tools. LegalGuardian employs Named Entity Recognition (NER) techniques and local LLMs to mask and unmask confidential PII within prompts, safeguarding sensitive data before any external interaction. We detail its development and assess its effectiveness using a synthetic prompt library in immigration law scenarios. Comparing traditional NER models with one-shot prompted local LLM, we find that LegalGuardian achieves a F1-score of 93% with GLiNER and 97% with Qwen2.5-14B in PII detection. Semantic similarity analysis confirms that the framework maintains high fidelity in outputs, ensuring robust utility of LLM-based tools. Our findings indicate that legal professionals can harness advanced AI technologies without compromising client confidentiality or the quality of legal documents.
Figures
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
American Bar Association . 1983. Model Rules of Professional Conduct. American Bar Association website
work page 1983
-
[4]
American Bar Association . 2024 a . Formal Opinion 512 . American Bar Association website
work page 2024
-
[5]
American Bar Association . 2024 b . S tate A I T ask F orce I nformation. https://www.americanbar.org/groups/centers_commissions/center-for-innovation/state-ai-task-force-information/
work page 2024
-
[6]
Cardellino, C.; Teruel, M.; Alemany, L. A.; and Villata, S. 2017. A low-cost, high-coverage legal named entity recognizer, classifier and linker. Proceedings of the 16th edition of the International Conference on Articial Intelligence and Law
work page 2017
-
[7]
Chalamala, S. R.; Kummari, N. K.; Singh, A. K.; Saibewar, A.; et al. 2022. Federated learning to comply with data protection regulations. CSI Transactions on ICT, 10(1): 47--60
work page 2022
-
[8]
Chalkidis, I.; Androutsopoulos, I.; and Aletras, N. 2019. Neural Legal Judgment Prediction in E nglish. In Korhonen, A.; Traum, D.; and M \`a rquez, L., eds., Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 4317--4323. Florence, Italy: Association for Computational Linguistics
work page 2019
Show all 42 references
-
[9]
Chalkidis, I.; Fergadiotis, M.; Malakasiotis, P.; Aletras, N.; et al. 2020. LEGAL - BERT : The Muppets straight out of Law School. In Cohn, T.; He, Y.; and Liu, Y., eds., Findings of the Association for Computational Linguistics: EMNLP 2020, 2898--2904. Online: Association for...
2020
-
[10]
Chen, Y.; Li, T.; Liu, H.; and Yu, Y. 2023. Hide and Seek (HaS): A Lightweight Framework for Prompt Privacy Protection. arXiv:2309.03057
2023 arXiv
-
[11]
V.; and Kim, M
Chien, C. V.; and Kim, M. 2024. Generative AI and Legal Aid: Results from a Field Study and 100 Use Cases to Bridge the Access to Justice Gap. Loyola of Los Angeles Law Review, forthcoming
2024
-
[12]
Coavoux, M.; Narayan, S.; and Cohen, S. B. 2018. Privacy-preserving Neural Representations of Text. arXiv:1808.09408
2018 arXiv
-
[13]
Dentons. 2023. D entons to launch client secure version of C hat G P T --- dentons.com. https://www.dentons.com/en/about-dentons/news-events-and-awards/news/2023/august/dentons-to-launch-client-secure-version-of-chatgpt. [Accessed 30-11-2024]
2023
-
[14]
Dwork, C. 2006. Differential Privacy . In Bugliesi, M.; Preneel, B.; Sassone, V.; and Wegener, I., eds., Automata, Languages and Programming , 1--12. Berlin, Heidelberg: Springer Berlin Heidelberg. ISBN 978-3-540-35908-1
2006
-
[15]
Edemacu, K.; and Wu, X. 2024. Privacy Preserving Prompt Engineering : A Survey
2024
-
[16]
Faraglia, D.; and et. al. 2016. Faker
2016
-
[17]
Fischel, D. R. 1998. Lawyers and Confidentiality. University of Chicago Law Review, 65: 1
1998
-
[18]
Gentry, C.; Halevi, S.; and Smart, N. P. 2012. Homomorphic Evaluation of the AES Circuit. Cryptology ePrint Archive, Paper 2012/099
2012
-
[19]
E.; Ré, C.; et al
Guha, N.; Nyarko, J.; Ho, D. E.; Ré, C.; et al. 2023. LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models. arXiv:2308.11462
2023 arXiv
-
[20]
Honnibal, M.; Montani, I.; Van Landeghem, S.; and Boyd, A. 2020. spaCy: Industrial-strength Natural Language Processing in Python
2020
-
[21]
Iezzi, M. 2020. Practical Privacy-Preserving Data Science With Homomorphic Encryption: An Overview. 2020 IEEE International Conference on Big Data (Big Data), 3979--3988
2020
-
[22]
B.; Avent, B.; Bellet, A.; et al
Kairouz, P.; McMahan, H. B.; Avent, B.; Bellet, A.; et al. 2021. Advances and Open Problems in Federated Learning. arXiv:1912.04977
2021 arXiv
-
[23]
Kan, Z.; Qiao, L.; Yu, H.; Peng, L.; et al. 2023. Protecting User Privacy in Remote Conversational Systems: A Privacy-Preserving framework based on text sanitization. arXiv:2306.08223
2023 arXiv
-
[24]
Kifer, D.; and Machanavajjhala, A. 2011. No free lunch in data privacy. In ACM SIGMOD Conference
2011
-
[25]
Klinefelter, A. 2011. When to Research is to Reveal : The Growing Threat to Attorney and Client Confidentiality from Online Tracking . Virginia Journal of Law and Technology , 16(01)
2011
-
[26]
Li, D.; Shao, R.; Wang, H.; Guo, H.; et al. 2023. MPCFormer: fast, performant and private Transformer inference with MPC. arXiv:2211.01452
2023 arXiv
-
[27]
Li, X.; Tram \`e r, F.; Liang, P.; and Hashimoto, T. B. 2021. Large Language Models Can Be Strong Differentially Private Learners. ArXiv, abs/2110.05679
2021 arXiv
-
[28]
Lin, G.; Hua, W.; and Zhang, Y. 2024. EmojiCrypt: Prompt Encryption for Secure Communication with Large Language Models. arXiv:2402.05868
2024 arXiv
-
[29]
Lindell, Y. 2020. Secure Multiparty Computation (MPC). IACR Cryptol. ePrint Arch., 2020: 300
2020
-
[30]
Mamakas, D.; Tsotsi, P.; Androutsopoulos, I.; and Chalkidis, I. 2022. Processing Long Legal Documents with Pre-trained Transformers: Modding LegalBERT and Longformer. arXiv:2211.00974
2022 arXiv
-
[31]
B.; Moore, E.; Ramage, D.; and y Arcas, B
McMahan, H. B.; Moore, E.; Ramage, D.; and y Arcas, B. A. 2016. Federated Learning of Deep Networks using Model Averaging. CoRR, abs/1602.05629
2016 arXiv
-
[32]
Medvedeva, M.; and Mcbride, P. 2023. Legal Judgment Prediction: If You Are Going to Do It, Do It Right. In Preo t iuc-Pietro, D.; Goanta, C.; Chalkidis, I.; Barrett, L.; Spanakis, G.; and Aletras, N., eds., Proceedings of the Natural Legal Language Processing Workshop 2023, 73...
2023
-
[33]
OpenAI. 2023. M arch 20 C hat G P T outage: H ere’s what happened. https://openai.com/index/march-20-chatgpt-outage/
2023
-
[34]
OpenAI; et al. 2024. GPT-4 Technical Report. arXiv:2303.08774
2024 arXiv
-
[35]
Qwen Team . 2024. Qwen2.5: A Party of Foundation Models
2024
-
[36]
What Can ChatGPT Do ?
Taecharungroj, V. 2023. “ What Can ChatGPT Do ?” Analyzing Early Reactions to the Innovative AI Chatbot on Twitter . Big Data and Cognitive Computing, 7(1): 35
2023
-
[37]
Trautmann, D. 2023. Large Language Model Prompt Chaining for Long Legal Document Classification. arXiv:2308.04138
2023 arXiv
-
[38]
Yao, A. C. 1982. Protocols for secure computations. In 23rd Annual Symposium on Foundations of Computer Science (sfcs 1982), 160--164
1982
-
[39]
A.; Mudgerikar, A.; Singla, A.; Papapanagiotou, I.; et al
Yavuz, A. A.; Mudgerikar, A.; Singla, A.; Papapanagiotou, I.; et al. 2017. Real-Time Digital Signatures for Time-Critical Networks. IEEE Transactions on Information Forensics and Security, 12: 2627--2639
2017
-
[40]
Yu, F.; Quartey, L.; and Schilder, F. 2023. Exploring the Effectiveness of Prompt Engineering for Legal Reasoning Tasks. In Rogers, A.; Boyd-Graber, J.; and Okazaki, N., eds., Findings of the Association for Computational Linguistics: ACL 2023, 13582--13596. Toronto, Canada: A...
2023
-
[41]
Zaratiana, U.; Tomeh, N.; Holat, P.; and Charnois, T. 2023. GLiNER: Generalist Model for Named Entity Recognition using Bidirectional Transformer. arXiv:2311.08526
2023 arXiv
-
[42]
Zhou, X.; Lu, J.; Gui, T.; Ma, R.; et al. 2022. T ext F usion: Privacy-Preserving Pre-trained Model Inference via Token Fusion. In Goldberg, Y.; Kozareva, Z.; and Zhang, Y., eds., Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 8360--837...
2022
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.