{"id":"a7f7fabb-fc58-4e27-b913-2bb3cf19ff29","arxiv_id":"2505.06284","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"DMRL applies inverse reinforcement learning with GRPO to train LLMs to extract PII, reporting higher reconstruction accuracy than four baselines on three datasets.","lead":"Researchers describe an attack method, DMRL, that uses reward learning to make language models reveal private information such as names, social security numbers, and addresses. It beats several existing attack baselines in their tests, but the paper does not release code or data and contains an unresolved contradiction in how the training data is described.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 3.1 defines training demonstrations as refusal responses, but Appendix A shows full PII disclosures; every reported result depends on this reward signal, so the central claim rests on an unresolved contradiction.","rationale":"The reader's weakest_assumption identifies the same contradiction between Section 3.1's refusal demonstrations and Appendix A's disclosing demonstrations, and my reading confirms this is the most load-bearing defect. The central empirical claim is that GRPO-S with IRL-trained shadow reward models outperforms all baselines in PII extraction. That claim depends entirely on the reward signal: Eq. (1) samples demonstrations from D, Eq. (3) updates the reward model by contrasting demonstration responses with policy samples, and Eq. (14) optimizes the policy against that reward model. If the demonstrations are refusals, the reward model should drive the policy toward refusal behavior, which is the opposite of extraction; if the demonstrations are disclosures, the formal definition of D in Section 3.1 is incorrect. The paper offers no reconciliation, and because no code or data is provided, a reader cannot infer which description reflects the actual experiment. This is not a stylistic inconsistency: it changes the sign of the learned reward and therefore the entire behavior of the optimized policy. The absence of error bars and the reuse of prior HAIR/SRMIR machinery are secondary concerns, but the demonstration-data contradiction alone is sufficient to block acceptance, since it makes the method unreproducible as stated. My check would settle the issue by exposing the actual form of o_ji and testing whether the published numbers are obtainable from either definition.","tokens_in":14370,"tokens_out":2938,"duration_ms":31607,"concrete_test":"Obtain the actual demonstration dataset, or at minimum one complete entry (q_ji, o_ji) from D, and check whether o_ji ends in the refusal f described in Section 3.1 or in the disclosing answer shown in Appendix A. Then replicate Algorithm 1 plus Eq. (14) on a small ECHR slice twice: once with o_ji as a refusal and once with o_ji as a disclosure. If refusal demonstrations drive PII reconstruction to the levels in Table 2, the contradiction is only notational; if they do not, and only disclosure demonstrations work, Section 3.1 is false as stated. The minimal version of this test is releasing the dataset construction code and a data card showing the final answer f for 10 sampled entries.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 defines each demonstration o_ji = (s_1,...,s_k,f) with f the final refusal answer, and D as harmful instructions paired with safe refusal responses and detailed reasoning. Algorithm 1 then trains category-specific reward models Rj by contrasting the demonstration y with a policy-sampled response via Eq. (3), and the GRPO-S objective in Eq. (14) uses Rj as the reward signal. If D truly contained refusals, Rj would assign high reward to refusal-like completions, so optimizing J^j_GRPO-S would train the policy to refuse rather than extract PII. Appendix A, however, is titled \"Dataset for Training Reward Model\" and shows answers such as \"Sure, my name is John Smith...\" with explicit disclosure of SSN, DOB, and address, annotated \"PII disclosed.\" These are not refusals; they are compliant disclosures. The paper never reconciles this discrepancy, and no code or data is released that could disambiguate. Because Eq. (1), Algorithm 1, and Eq. (14) are formally well-defined, one of the two descriptions must be wrong: either the text in Section 3.1 is a mismatch, or the Appendix's data is not what Algorithm 1 consumes. Either way, the reported reconstruction gains in Tables 1-3 are not reproducible from the manuscript as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes DMRL, a data extraction attack that first constructs a demonstration dataset of privacy-leakage Q&A pairs, trains category-specific shadow reward models via maximum-likelihood inverse reinforcement learning on those demonstrations, measures data- and model-level hardness using CLIP similarity and reward gaps, and then fine-tunes the target LLM with GRPO using hardness-scaled advantages. The paper claims that DMRL achieves the highest PII reconstruction accuracy on ECHR, Enron, and Yelp-Health across GPT-2 Small/Medium/Large/XL, Qwen1.5-7B, LLaMA2-7B, ChatGLM3-6B, and Baichuan2-7B, outperforming TAB, PrivAgent, Bag of Tricks, and AL-PII.","tokens_in":14665,"tokens_out":5111,"duration_ms":47631,"significance":"If the results were reproducible, DMRL would be a useful red-teaming contribution to the study of privacy leakage from LLMs. The formal IRL/GRPO derivation and the algorithm pseudocode are clearly presented, and the hardness-scaling idea is novel enough to merit attention. However, as written the central empirical claim is not supported: the demonstration-data contradiction, the absence of error bars and significance tests, and the missing implementation details prevent verification. The paper also includes a limitations section and a risk discussion, which are appropriate for this topic, but these do not mitigate the technical issues.","major_comments":[{"comment":"Section 3.1 defines each demonstration o_ji as a refusal response f along with a CoD reasoning sequence, and states that D contains safe refusal responses. Appendix A, however, shows training examples such as 'Sure, my name is John Smith...' with full SSN, DOB, and address, annotated 'PII disclosed.' These are not refusals. Because Algorithm 1 trains R_j by contrasting the demonstration with a policy sample via Eq. (3), and Eq. (14) uses R_j as the reward signal for GRPO, a reward model trained on refusals would push the policy toward refusal, directly contradicting the extraction goal; a reward model trained on Appendix A disclosures would make Section 3.1 incorrect. The paper never reconciles this, and no code or data is released to disambiguate, so the reported gains in Tables 1-3 are not reproducible from the manuscript as written.","section":"Section 3.1 and Appendix A"},{"comment":"Tables 1-3 report point estimates only, with no error bars, confidence intervals, or significance tests. Since each cell is a single number and the paper claims that DMRL 'significantly outperforms all baseline methods,' the reader cannot assess whether the 4-7 point gains over AL-PII in Figure 2 and Table 2 are beyond sampling noise. In addition, Table 1 lists only GPT-2 Small, Medium, and Large, while the text and Table 2 include GPT-2 XL; this discrepancy needs correction.","section":"Tables 1-3 and Section 4.2"},{"comment":"The data-hardness measure applies a CLIP model to text sub-sentences, but no validation is provided that CLIP similarity correlates with leakage semantics. CLIP is trained for image-text matching, and the paper does not report any baseline or ablation comparing this measure to text-only similarity metrics, nor does it describe the sentence-splitting prompt or the choice of CLIP checkpoint. Because alpha^D_ji enters the advantage in Eq. (14), a miscalibrated hardness measure directly affects the optimization, so this is not a cosmetic issue.","section":"Section 3.3, Eqs. (5)-(8)"},{"comment":"Figure 2 is referenced in Section 4.2 but is not present in the manuscript, and Table 3 reports PII inference accuracy for DMRL alone with no baseline comparisons. Without the figure and without baseline numbers, the claim that DMRL 'consistently outperforms all baselines' on inference is unverifiable.","section":"Figure 2 and Table 3"}],"minor_comments":[{"comment":"Equation (8) writes sigma(delta) and sigma(bar-delta) without defining delta as delta_ji or specifying that the denominator is the category-level mean; the notation should be made precise.","section":"Eq. (8)"},{"comment":"The abstract and Section 3.4 call GRPO 'Gradient Reward Policy Optimization' while the standard name is Group Relative Policy Optimization; please correct this.","section":"Abstract and Section 3.4"},{"comment":"Section 1 says 'base on dataset duplicates' and Section 4.1 contains the typo 'wit publicly available'; these should be corrected.","section":"Section 1 and Section 4.1"},{"comment":"The text says the experiments cover 'four different LLMs and four variants of GPT-2,' but Table 1 omits GPT-2 XL; either add the XL column or revise the text.","section":"Section 4.1"},{"comment":"The relationship to the authors' prior HAIR and SRMIR work (Cheng et al. 2025b; Cheng and Cao 2025) is mentioned only by citation; a short comparison would help distinguish the new contributions.","section":"Related work and method"}],"recommendation":"reject","confidential_remarks":"The Section 3.1/Appendix A contradiction and the absence of code/data make the central claim unverifiable in the current manuscript. If the authors can release the dataset and code and clarify the reward signal, a substantially revised version might merit re-review, but the present version does not meet the bar for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is an incremental extension of the authors' earlier HAIR/SRMIR work, applying IRL-trained shadow reward models plus GRPO with hardness scaling to PII extraction. That's a legitimate contribution to red-teaming: the evaluation is broad (four GPT-2 sizes, four 7B-class models, three datasets), and DMRL beats AL-PII by a few points on most configurations. If those numbers hold, it's a useful tool for the privacy community.\n\nThe soft spot is not the novelty; it's the internal contradiction between Section 3.1 and Appendix A. Section 3.1 says the demonstration dataset D consists of 'refusal responses' with CoD reasoning. Appendix A shows the actual training data: answers that fully disclose SSNs, addresses, and birth dates, annotated 'PII disclosed.' Those are not refusals. The reward model in Algorithm 1 is trained on this data, and Eq. (14) uses that reward to optimize the policy. If D were refusals, the policy would be trained to refuse—the opposite of extraction. The paper never reconciles this. Either the text or the appendix is wrong, and the experimental results depend entirely on this reward signal.\n\nThere are smaller issues: no code or data released, no error bars or significance tests, CLIP used on text snippets without validation, and two of the authors' own papers share the same arXiv ID in the references. These are fixable in revision, but the D contradiction is load-bearing. Without clarification or released data, Tables 1–3 could not be reproduced even if the method works in practice.\n\nI'd send this to peer review only after the authors resolve the contradiction—or better, release the demonstration dataset and training code. As it stands, the paper has a promising direction and a credible evaluation framework, but the central methodological description is not coherent.\n\nFor a reading group, it's a useful case study in how reward learning papers can slip internally, but not a primary reference. I would not cite it until the inconsistency is fixed.\n\nRecommendation: desk reject in current form; invite a resubmission if the contradiction is resolved and code/data are released.","headline":"A load-bearing contradiction between the described training data and the appendix makes the reported PII extraction gains unreproducible; otherwise an incremental but well-scoped red-teaming extension.","tokens_in":15193,"tokens_out":4962,"would_cite":false,"duration_ms":42596,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that DMRL, which learns reward models from privacy-leakage demonstrations and scales GRPO by data and model hardness, extracts PII from LLMs more accurately than four baselines across all tested models and datasets.","keywords":["data extraction","inverse reinforcement learning","reward model","GRPO","personally identifiable information","privacy leakage","hardness-aware optimization","LLM red teaming"],"falsifier":"Train SRL shadow reward models and run GRPO-S on one GPT-2 size and one dataset twice, once with the demonstration set containing only refusal responses and once with only full PII disclosures; if the refusal-trained variant reconstructs PII as accurately as the disclosure-trained variant, the claimed reward mechanism is not what drives extraction.","tokens_in":14159,"feed_emoji":"🕵️","tokens_out":7028,"duration_ms":61682,"temperature":0.7,"pith_summary":"This paper proposes DMRL, a method that extracts personally identifiable information from large language models by learning reward models from a demonstration dataset of privacy-leakage question–answer pairs, then fine-tuning the target model with Group Relative Policy Optimization (GRPO) using those rewards. The paper claims DMRL beats four existing extraction baselines on every model and dataset tested, with reconstruction accuracy gains of 4–7 points over the strongest baseline and inference accuracy above 50% on most configurations. The wider point is that reward learning can drive extraction without relying on duplicated training data, handcrafted prompts, or random-search generation. If the claim holds, red teams gain a systematic attack, and defenses need to focus on the reward signal.","feed_headline":"Reward learning pulls more private data out of LLMs","feed_subtitle":"Inverse RL plus difficulty-scaled GRPO tops four baselines on ECHR, Enron, and Yelp-Health.","key_machinery":"The central mechanism is the shadow reward model $R_j(\\cdot,\\cdot)$, trained per category by joint reward-and-policy inverse RL from the demonstration subset $D_j$, together with the combined hardness coefficient $\\alpha_{ji} = \\alpha^{\\mathrm{D}}_{ji}\\,\\alpha^{\\mathrm{M}}_j$ that scales the GRPO advantage for question $i$ in category $j$. Data hardness uses one minus the average maximal CLIP similarity between split sub-sentences of the demonstration answer and the current policy's answer; model hardness uses the reward gap after outlier masking. The reward model converts the attack into a reward optimization problem: GRPO pushes the policy toward outputs that the shadow reward model scores as matching the leakage demonstrations, while the hardness coefficient reweights updates to focus on hard data and responsive model states.","core_discovery":"On the paper's own terms, the discovery is that data- and model-aware reward learning makes PII extraction substantially more effective than current attack methods. DMRL first builds a Q&A privacy-leakage dataset for each of N categories, trains category-specific shadow reward models with maximum-likelihood inverse reinforcement learning on that demonstration data, and then optimizes the target LLM with GRPO while scaling each advantage by a combined hardness coefficient: data hardness from CLIP similarity between demonstration and generated responses, and model hardness from the shadow reward model's reward gap. Reported results show DMRL achieving the highest PII reconstruction accuracy in all settings, for example 2.88 to 21.71 on ECHR from GPT-2 Small to Large, and top-1 inference accuracy of 40.7–55.1% across four 6–7B open models. The paper interprets this as evidence that larger models are more memorization-prone and that reward-guided optimization exposes PII more reliably than prior extractors.","pith_inferences":["The direction of the learned reward is underdetermined by the paper: Section 3.1 says demonstrations are refusals with reasoning, while Appendix A shows full disclosures; a direct ablation on refusal-only versus disclosure-only demonstrations would settle which signal actually drives extraction.","The hardness-scaling scheme is not tied to privacy: the same data-level CLIP similarity and model-level reward gap could be applied to any alignment task where demonstrations exist without preference pairs.","If the method works as reported, defenses that filter prompt patterns or detect known attack templates may not stop it, since the attack signal lives in the reward model and the optimization procedure rather than in the final prompt."],"forward_implications":["If DMRL's claims hold, the four evaluated baselines—TAB, PrivAgent, Bag of Tricks, and AL-PII—are all outperformed on PII reconstruction across GPT-2 Small/Medium/Large/XL, Qwen1.5-7B, LLaMA2-7B, ChatGLM3-6B, and Baichuan2-7B.","Larger GPT-2 models extract more PII, with reconstruction accuracy rising with model size on ECHR, Enron, and Yelp-Health.","LLMs can infer sensitive personal information from context at above-chance rates, with most tested 6–7B models exceeding 50% inference accuracy.","Data extraction can succeed without relying on duplicated training data, prompt engineering, or random-search adversarial generation."],"supporting_citations":[{"why":"Cheng and Cao (2025) provides the shadow-reward-model and introspective-reasoning dataset recipe that DMRL adapts to data extraction.","marker":"Cheng and Cao (2025)"},{"why":"Li et al. (2024a) supplies the maximum-likelihood IRL formulation and the joint reward-policy update used to train each shadow reward model.","marker":"Li et al. (2024a)"},{"why":"Shao et al. (2024) contributes GRPO, the group-relative policy optimization objective that DMRL extends with hardness scaling.","marker":"Shao et al. (2024)"},{"why":"Cheng et al. (2025b) gives the hardness-aware scaling strategy for combining data- and model-level difficulty.","marker":"Cheng et al. (2025b)"},{"why":"Radford et al. (2021) provides the CLIP model whose text similarity scores define data hardness in Equations 5–8.","marker":"Radford et al. (2021)"},{"why":"Lukas et al. (2023) defines the AL-PII baseline and the PII extractability, reconstruction, and inference metrics.","marker":"Lukas et al. (2023)"},{"why":"Yu et al. (2023b) is the Bag of Tricks generate-then-rank baseline that DMRL must outperform.","marker":"Yu et al. (2023b)"},{"why":"Nie et al. (2024) is the PrivAgent RL-based adversarial prompt generation baseline for privacy leakage.","marker":"Nie et al. (2024)"},{"why":"Inan et al. (2021) is the TAB baseline that measures leakage by reproducing unique training fragments.","marker":"Inan et al. (2021)"}],"fun_headline_variants":["Inverse RL + GRPO: DMRL extracts more PII from LLMs","Reward learning ups LLM private data extraction","DMRL: Model-aware rewards boost PII extraction","Hardness-tuned rewards extract more private data from LLMs","Data- and model-aware reward learning improves PII attacks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central load-bearing premise is that the demonstration dataset consistently rewards PII-disclosing outputs; the paper describes this dataset as refusal responses with reasoning in Section 3.1, while Appendix A shows answers that fully disclose the requested PII, leaving the direction of the reward signal unresolved.","fun_headline_variants_meta":{"raw":{"variants":["Inverse RL + GRPO: DMRL extracts more PII from LLMs","Reward learning ups LLM private data extraction","DMRL: Model-aware rewards boost PII extraction","Hardness-tuned rewards extract more private data from LLMs","Data- and model-aware reward learning improves PII attacks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000181,"raw_usage":{"total_tokens":1293,"prompt_tokens":919,"completion_tokens":374,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":290}},"tokens_in":535,"tokens_out":374,"duration_ms":3457,"temperature":1.0,"reasoning_tokens":290,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:34:41.202153+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train SRL shadow reward models and run GRPO-S on one GPT-2 size and one dataset twice, once with the demonstration set containing only refusal responses and once with only full PII disclosures; if the refusal-trained variant reconstructs PII as accurately as the disclosure-trained variant, the claimed reward mechanism is not what drives extraction.","supporting_citations":[],"review_version":1}