{"id":"5b4fb169-7568-4d16-a1c0-e255405b62dd","arxiv_id":"2412.12040","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Language models leak personally identifiable information when summarizing medical and legal documents, and fine-tuning on pseudonymized data substantially reduces leakage in open-weight models.","lead":"This study tests whether language models accidentally reveal personal information such as names, ages, and locations when summarizing medical and legal documents. Across six models and expert-written private summaries, it finds that privacy is unreliable under standard prompting, but fine-tuning on pseudonymized data helps smaller open-weight models catch up.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated GPT-4o PII detector, shown by the paper itself to miss gender pronouns (the most leaked category), underpins all PTR/LDR/TPR metrics; human-LM comparisons may use different annotation instruments.","rationale":"The reader identified the unvalidated GPT-4o detector as the weakest assumption, and the paper's own qualitative results provide direct evidence that the detector misses the most commonly leaked PII category. I agree this is the single most load-bearing concern. The synthetic-data concern is real but less damaging: even if the injected PII is synthetic, the task still tests whether models copy identifiable information from the input, which is a meaningful proxy. Missing significance tests and error bars are also secondary: they affect the strength of specific comparisons but not the core direction. The detector issue, by contrast, undermines every quantitative number in Figure 4 and Table 4, and the human-vs-LM asymmetry in Table 4 could invalidate the abstract's central contrast if the detector's per-category errors are large. The proposed test directly addresses this by using human annotations as the gold standard for the same documents and models. While the qualitative direction of the findings is likely to survive, the paper's quantitative claims and rankings cannot be accepted without this validation, so the original CONDITIONAL verdict remains appropriate.","tokens_in":24222,"tokens_out":7886,"duration_ms":66752,"concrete_test":"Use the 74 gold-standard Discharge Me! documents and generate summaries with GPT-4o, DeepSeek-Chat, and Llama-3.3-70B (the models in Table 4). Have the same two medical-expert annotators plus adjudicator, following the §4.5 protocol, mark PII spans in these summaries. Compute per-category PTR/LDR/TPR from the human labels and compare with the GPT-4o-detector-based scores, focusing on gender/pronoun recall. If human-based scores differ materially (e.g., GPT-4o gender TPR rises from 8% to above 20%) or any model ranking reverses, the quantitative conclusions and the claim of closeness to human performance require revision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative evidence rests on PTR, LDR, and TPR scores computed by a GPT-4o-based automatic PII detector described in §4.4 and prompted in Figure 5. This detector is never validated against human annotations, yet the paper's own qualitative analysis (Table 3, rows 5–6) demonstrates that both GPT-4o and Presidio fail to identify gender pronouns as PII, even though the detector prompt explicitly instructs tagging pronouns as GENDER. §6 and Table 4 identify gender as the most commonly leaked category. Consequently, the reported leakage rates for all models are likely to undercount the dominant PII type, and model rankings may be distorted because models differ in which PII types they leak. A second, compounding issue is that the human-vs-LM comparison in Table 4 appears to use expert annotations for the doctors' summaries but the unvalidated GPT-4o detector for the LM summaries on the same 74 documents: the human evaluation in §4.5 covers only DeepSeek-Chat and IFT+Llama-3.3-70B, not GPT-4o or Llama-3.3-70B, so TPR for those models cannot come from the human annotation protocol. The paper does not explain how these model TPRs were computed. This asymmetry makes the headline contrast between human and LM privacy protection, and the 'frontier LMs close to human performance' conclusion in §6, not reliably measurable from the reported numbers. The central claim may still hold qualitatively, but its quantitative support is insecure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper asks whether language models leak personally identifiable information when summarizing medical and legal documents. The authors construct pseudonymized versions of the Discharge Me! and AsyLex datasets by injecting synthetic profiles with GPT-4o, then evaluate two closed-source and four open-weight models under six prompting methods and instruction fine-tuning. They measure leakage with a GPT-4o-based PII detector using PTR, LDR, and TPR, and supplement this with a human evaluation of two selected models. The central claims are that LMs frequently leak PII, that expert doctors produce substantially more private summaries, and that instruction fine-tuning on the pseudonymized data brings open-weight models close to frontier closed-source models.","tokens_in":24493,"tokens_out":5845,"duration_ms":43150,"significance":"If these claims hold, the paper makes a useful contribution: it releases new pseudonymized datasets, expert-written private summaries, and annotations; it provides the first systematic human-expert comparison for privacy in summarization; and it proposes a practical fine-tuning recipe for open-weight models. The human evaluation supports the qualitative direction of the findings. However, the headline quantitative comparisons rest on an unvalidated automatic PII detector that the paper itself shows misses the most frequently leaked PII type, and several text/table inconsistencies undermine the strength of the stated conclusions. The contribution is significant enough to warrant revision rather than rejection, provided the measurement and reporting issues are addressed.","major_comments":[{"comment":"The PTR, LDR, and TPR metrics all rely on the GPT-4o-based detector described in §4.4 and prompted in Figure 5, but this detector is never validated against human annotations. The paper's own qualitative analysis (Table 3, rows 5 and 6) shows that both GPT-4o and Presidio fail to identify gender pronouns as PII even though the detection prompt explicitly targets GENDER, and §6 with Table 4 reports gender as the most leaked category. Consequently, the reported leakage rates undercount the dominant PII type, and model rankings can be distorted because models differ in which PII types they leak. Please validate the automatic detector (e.g., precision/recall against human annotations on a random sample) or recompute the headline metrics from human annotations for all models and conditions.","section":"§4.4, Figure 5, Table 3"},{"comment":"The gold-standard comparison of doctors' summaries with GPT-4o and Llama-3.3-70b appears to mix annotation instruments. The human evaluation described in §4.5 covers only DeepSeek-Chat and IFT+Llama-3.3-70B, so the TPR values for GPT-4o and Llama-3.3-70b in Table 4 cannot come from that protocol; they can only come from the unvalidated GPT-4o detector applied to the same 74 documents. This asymmetry undermines the headline contrast between human and LM privacy protection and the conclusion in §6 that frontier LMs are close to human performance. Please state explicitly how each TPR in Table 4 was computed and, ideally, measure humans and LMs with the same annotation instrument.","section":"§6, Table 4 versus §4.5"},{"comment":"The text claims that 'PTR decreases across all models in the medical domain' after IFT, but the appendix shows the opposite for several model/prompt combinations (e.g., IFT-Llama-3.1 8B 0-Shot Priv Sum PTR 25.74 vs Llama-3.1 8B 17.60; IFT-Qwen2.5 14B 18.52 vs Qwen2.5 14B 6.65 in Table 11). Similarly, 'Deepseek-Chat exhibiting a 2% increase in PTR when using in-context samples' is not supported by the reported values (1.85 to 1.89 in Table 11; 1.79 to 1.91 in Table 12). Please either correct these statements or clarify which baselines and conditions are being compared, since the abstract's claim that IFT makes smaller models comparable to frontier models depends on these comparisons.","section":"§5.2 and Tables 11–12"},{"comment":"All headline comparisons are point estimates without confidence intervals, significance tests, or effect-size measures. Given that leakage rates are low and the test sets are relatively small (e.g., 74 documents for the gold standard, 100 for human evaluation), the claim that GPT-4o is close to human performance (8% vs 4% gender TPR) is not distinguishable from noise. Please add uncertainty quantification (e.g., bootstrap confidence intervals) or appropriate statistical tests for the main privacy comparisons.","section":"§5.1–§5.3, §6"}],"minor_comments":[{"comment":"The phrase 'expert human expert performance' contains a duplicated word; it should read 'expert human performance' or 'human expert performance.'","section":"Abstract"},{"comment":"The instruction 'The data has been redacted' is duplicated in the pseudonymization prompt template; please remove the redundant sentence.","section":"Figure 6"},{"comment":"The BLEU threshold of 20% is introduced after manual checking of 200 documents, but no justification or sensitivity analysis is provided; please clarify how this threshold was chosen and whether the main results are robust to it.","section":"§3.2"},{"comment":"The model name 'Llama-3.3-70b' uses inconsistent capitalization compared to 'Llama-3.3-70B' elsewhere in the paper; please standardize.","section":"Table 4"},{"comment":"The reference for P˘ais, et al. appears to have a formatting artifact in the author field; please fix the bibliographic entry.","section":"References"},{"comment":"The paper states that 'we perform our initial analysis using Presidio' but the main evaluation later switches to GPT-4o; please clarify whether the dataset stratification PII counts come from Presidio and whether the GPT-4o detector uses the same PII taxonomy.","section":"§3.3"}],"recommendation":"major_revision","confidential_remarks":"I support a major revision rather than rejection because the qualitative finding is supported by human evaluation and the dataset/resource contribution is valuable. However, the quantitative claims in the abstract and §6 cannot be trusted until the detector is validated or replaced, and the internal inconsistencies in §5.2 are fixed. Please also consider whether the paper should temper the 'comparable to frontier models' claim to the specific settings where it actually holds."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper deserves a serious look. It ships new pseudonymized medical and legal summarization data, expert-written private summaries, and a broad sweep of six models under prompting and IFT. The central qualitative finding—unconstrained LMs leak PII and expert humans are much more private—is plausible and partly confirmed by the human evaluation (95 of 100 summaries judged free of source-derived PII). The dataset release alone is a real contribution.\n\nThe soft spot is the measurement instrument. All PTR/LDR/TPR numbers depend on a GPT-4o-based PII detector that is never validated against human annotations, and the paper itself shows (Table 3, rows 5-6) that both GPT-4o and Presidio miss gender pronouns—the most commonly leaked category (Table 4). That means the headline leakage rates likely undercount the dominant PII type, and model rankings could be distorted if models differ in which types they leak. Table 4 also appears to mix annotation instruments: doctors' TPR comes from expert annotation, while GPT-4o and Llama-3.3-70B TPRs cannot come from the §4.5 protocol, which only covered DeepSeek-Chat and IFT+Llama-3.3-70B. The paper never explains how those model TPRs were computed. That asymmetry makes the \"frontier LMs close to human performance\" conclusion not reliably measurable from the reported numbers. There are also no confidence intervals or significance tests anywhere, so differences like the 2% PTR increases are often within noise.\n\nThese are addressable, not fatal. The core claim probably survives a re-run with a validated detector. But as published, the quantitative precision is overstated, and §5.2's \"improvements across all models\" claim is contradicted by Appendix K (e.g., IFT+Llama-3.1-8B gets worse LDR under 0-Shot Private on the medical task).\n\nFor a referee: send it out. The questions matter, the data is reusable, and the flaws are fixable. I'd ask for external validation of the detector on a human-annotated sample, error bars or at least a per-model breakdown of leakage by PII type, and a clear statement of which annotation instrument produced each TPR in Table 4.","headline":"Solid empirical study with a reusable dataset; the headline leakage numbers rest on an unvalidated detector that misses the most common PII type, so the quantitative claims need another pass.","tokens_in":25047,"tokens_out":1993,"would_cite":false,"duration_ms":16750,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Language models routinely leak personal identifiers when summarizing medical and legal documents, while expert human summaries are much more protective.","keywords":["privacy-preserving summarization","personally identifiable information","abstractive summarization","instruction fine-tuning","pseudonymization","medical NLP","legal NLP","large language models"],"falsifier":"Re-annotate a random sample of machine-generated and doctor-written summaries with trained human annotators using the same five PII categories, then recompute PTR, LDR, and TPR from the human spans. If humans find models are no leakier than the doctors, the central gap claim fails; if humans find even more leakage, especially of gender pronouns, the claim is strengthened.","tokens_in":23982,"feed_emoji":"🔒","tokens_out":7768,"duration_ms":60423,"temperature":0.7,"pith_summary":"The paper asks whether language models can be trusted to summarize medical records and legal case files without copying out personal details. By taking real anonymized records, reinserting realistic synthetic personal information, and asking models to summarize them, the authors find that models regularly leak names, dates, locations, ages, and especially gender pronouns into their summaries. Two medical doctors who wrote private summaries of the same documents leaked almost nothing, apart from occasional pronouns. The authors also show that instruction fine-tuning on their pseudonymized data makes smaller open-weight models about as private as large closed-source models, while improving summary quality. If correct, the result means current summarization systems are not safe for direct deployment in privacy-sensitive settings without dedicated anonymization steps.","feed_headline":"LMs leak personal data in medical and legal summaries","feed_subtitle":"Expert doctors leak almost nothing; instruction fine-tuning closes much of the gap for open models.","key_machinery":"The evaluation machinery is a three-part pipeline. First, source documents are pseudonymized: a synthetic-profile generator creates realistic profiles with name, age, gender, race, birth date, birth location, and residence, and GPT-4o reinserts those values into redacted clinical and legal texts, with a BLEU threshold plus manual checks used as quality control. Second, leakage is quantified by three metrics: PTR, the share of private tokens from the source that appear in the summary; LDR, the share of summaries containing at least one leaked private token; and TPR, the share of source PII spans that reappear in the summary, all computed with a GPT-4o-based PII detector prompted in the style of prior de-identification work. Third, the same pseudonymized data is used to instruction fine-tune open-weight models, with separate medical and legal models, so that privacy-preserving behavior is learned rather than prompted. The gold-standard comparison comes from two medical doctors who wrote private summaries of 74 documents and annotated leaked spans, providing the expert baseline.","core_discovery":"The paper claims that language models, when prompted to summarize pseudonymized medical and legal source documents, frequently copy personally identifiable information from the source into their summaries, whereas expert-written privacy-preserving summaries leak far less. The authors construct this comparison by reinserting synthetic PII into two publicly released anonymized datasets, recruiting medical doctors to write gold-standard private summaries and to annotate leaked spans, and evaluating six models under six prompting strategies plus instruction fine-tuning. Their headline result is that leakage is common and uneven: gender pronouns are the most frequently leaked category, chain-of-thought prompting increases leakage relative to dedicated anonymization steps, and instruction fine-tuning on the new pseudonymized data lets open-weight models such as Llama-3.3-70B reach privacy protection comparable to larger closed-source models while improving summary quality. A human evaluation of 100 document-summary pairs found 95 summaries free of source-linked PII, with slight leakage attributed to the fine-tuned open model and none to the closed-source model, and participants preferred the open model's summaries.","pith_inferences":["Because the paper's own qualitative analysis shows the automatic detector and an off-the-shelf tagger miss gender pronouns, the reported leakage figures likely undercount the most common leakage category; a benchmark built on human-annotated spans would give a sterner test and may widen the human-model gap.","The order effect the paper observes — summarizing before anonymizing beats anonymizing before summarizing — suggests a practical pipeline design: let the model summarize the original text, then run a dedicated redaction pass, rather than relying on the model to redact first.","The pseudonymization recipe of synthetic profiles inserted by GPT-4o with a BLEU sanity threshold is a reusable way to create privacy-preserving summarization training data in other sensitive domains, but its realism has only been spot-checked, so transfer should be validated by domain experts."],"forward_implications":["Unconstrained zero-shot summarization leaks PII in nearly all documents, with leaked-document ratios close to 100 percent in the medical task.","Explicit privacy instructions reduce leakage but do not eliminate it, and adding in-context examples can increase leakage for some closed-source models.","Anonymizing after summarizing is more effective at suppressing PII than anonymizing before summarizing.","Chain-of-thought prompting improves summary quality but is less privacy-safe than the two-stage anonymization approaches.","Instruction fine-tuning on the pseudonymized data gives open-weight models privacy metrics comparable to or better than frontier closed-source models while raising ROUGE and BERTScore."],"supporting_citations":[{"why":"Supplies the Discharge Me! electronic health record summarization task with anonymized document-summary pairs.","marker":"(Xu, 2024)"},{"why":"Supplies the AsyLex refugee-case summarization task with judgment-summary pairs.","marker":"(Barale et al., 2023)"},{"why":"Provides the MIMIC-IV-Note deidentified clinical notes from which Discharge Me! is derived.","marker":"(Johnson et al., 2023)"},{"why":"Provides the GPT-4o PII-detection prompt template and the redaction-augmentation approach the authors adapt for leakage detection.","marker":"(Kim et al., 2024)"},{"why":"GPT-4o is the model used for pseudonymization, automatic PII detection, and as a closed-source baseline.","marker":"(OpenAI et al., 2024)"},{"why":"Supplies Cohen's kappa as the inter-annotator agreement measure for the human evaluation.","marker":"(Artstein and Poesio, 2008)"},{"why":"Provides the LoRA method used for instruction fine-tuning of open-weight models.","marker":"(Hu et al., 2022)"},{"why":"Prior work on LMs revealing sensitive information that this paper extends to PII leakage in medical and legal summarization.","marker":"(Mireshghallah et al., 2024)"},{"why":"Prior work on instruction-tuning to reduce PII leakage in question answering, which the paper contrasts and extends to summarization.","marker":"(Xiao et al., 2024)"}],"fun_headline_variants":["LMs leak personal data even from pseudonymized sources","Chain-of-thought boosts privacy leaks in LM summaries","Fine-tuning closes privacy gap for open-weight LMs","Human experts outshine LMs on privacy in summaries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison depends on the automatic GPT-4o PII detector catching every leaked identifier in the summaries, and the paper's own examples show that detector misses gender pronouns, the category it finds most often leaked, so the measured leakage rates may understate the true rates.","fun_headline_variants_meta":{"raw":{"variants":["LMs leak personal data even from pseudonymized sources","Chain-of-thought boosts privacy leaks in LM summaries","Fine-tuning closes privacy gap for open-weight LMs","Human experts outshine LMs on privacy in summaries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000264,"raw_usage":{"total_tokens":1591,"prompt_tokens":918,"completion_tokens":673,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":609}},"tokens_in":534,"tokens_out":673,"duration_ms":6570,"temperature":1.0,"reasoning_tokens":609,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:20:40.642643+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of machine-generated and doctor-written summaries with trained human annotators using the same five PII categories, then recompute PTR, LDR, and TPR from the human spans. If humans find models are no leakier than the doctors, the central gap claim fails; if humans find even more leakage, especially of gender pronouns, the claim is strengthened.","supporting_citations":[],"review_version":1}