{"id":"685f22c7-74f4-4ec4-86e9-a249b8e66578","arxiv_id":"2608.12911","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A dynamic relational unlearning framework and benchmark show that document MLLMs jointly leak correlated private fields under weak visual evidence, and that the framework lowers the leakage rate while partly preserving extraction utility.","lead":"Document understanding AI models can leak multiple private fields, such as a person's name and document number, when shown blank or meaningless images. This paper introduces a benchmark to measure that relational leakage and a dynamic unlearning method that suppresses it while keeping normal document extraction mostly intact.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluation protocol draws probing, forgetting, and leakage scoring from the same 5,000 training identities (Supp. D.1), so the reported suppression could be targeted pair erasure. A held-out identity split is the missing control.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing gap I found: the dynamic forget set and the leakage evaluation are drawn from the same training identity library, so the reported suppression may be targeted erasure rather than evidence of general relational unlearning. This is not a disagreement with the paper's motivating observation; Table 1 and the qualitative outputs in Fig. 5 do show that document MLLMs reproduce memorized field associations under weak visual evidence, and the static-forget-set ablation in Supp. F.1 suggests the dynamic update helps. The issue is specifically whether the central mitigation claim, that DRUF generalizes to non-probed leakage points as stated in Section 3.2, is supported. The text of Supp. D.1 makes the overlap explicit, and no identity-level split is described anywhere in the main text or supplementary material. Because the central contribution is the unlearning method, not just the benchmark, this missing control is load-bearing. A held-out identity evaluation would settle it cleanly: if leak rate on held-out identities remains low, the concern is resolved; if not, the method reduces to memorized erasure of probed pairs. The paper has real independent support in the form of the benchmark construction, the consistent leakage phenomenon across models, and the training-dynamics curves in Supp. Fig. 7, so I would not move the verdict to REJECT. CONDITIONAL remains appropriate, conditioned on the held-out evaluation and the other reviewer-identified details such as error bars and threshold sensitivity. My read therefore leaves the reader's verdict unchanged.","tokens_in":22285,"tokens_out":5971,"duration_ms":59038,"concrete_test":"Split the 5,000 training identities into an 80% probe/forget pool and a 20% held-out evaluation pool. Build DRUF's dynamic forget set F(t) only from pairs detected on the 80% pool; never include held-out identities in F(t). Then evaluate the unlearned model with the Image-Driven protocol of Section 5.3 on the held-out identities, scoring only against held-out identities in the training identity library. Run three seeds. If held-out Acc@1.0 remains near the base model level (e.g., above 0.05), DRUF suppresses only probed pairs; the general relational unlearning claim fails. If held-out Acc@1.0 drops to approximately 0.001, the generalization claim is supported. Rerun SCRUB under the same split to verify the 4.8-point comparison survives the held-out control.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of the paper is that DRUF is a general relational unlearning method, and Section 3.2 explicitly relies on the assumption that unlearning on exposed pairs generalizes to other data points that have not been probed. The supporting evaluation does not currently demonstrate that generalization. The dynamic forget set F(t) is built from pairs leaked on probing inputs, and the leakage evaluation compares outputs against the same training identity library used for training and probing. Supplementary D.1 states that the probing/evaluation test images are 6,857 face crops derived from the same 5,000 DocXPand-25k samples used to train the target model, and no identity-level split between probe/forget targets and evaluation identities is described. Consequently, the headline result in Table 2 (Image-Driven Acc 0.642 to 0.001, Prompt-Driven to 0.000) can be explained entirely by erasing the exact associations that were placed into F(t), rather than by a capability to suppress relational leakage on unseen identities. A static-forget-set ablation in Supp. F.1 and the test-set-size study in Supp. D.2 do not resolve this, because both still draw their leakage targets from the same identity pool used for scoring. The comparison among unlearning methods is internally fair because all methods receive the same cached targets, but that fairness does not establish the general relational unlearning capability that distinguishes DRUF from sample-level erasure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies relational privacy leakage in document-understanding multimodal LLMs (MLLMs): when an input image lacks valid visual evidence, the model may fall back on memorized field associations and jointly generate multiple correlated sensitive fields (e.g., family name and document number) from training identities. The authors propose the Dynamic Relational Unlearning Framework (DRUF), which probes the current model under blank-image inputs to build a dynamic forget set of leaked field pairs, then applies a relation-level forgetting loss (the product of span-level KL divergences) to suppress the joint generation of those pairs while a teacher–student retain branch preserves KIE utility. They also introduce DocPrivacyBench, a benchmark for measuring such leakage under Image-Driven and Prompt-Driven settings at multiple similarity thresholds, and they report experiments on three document MLLMs and six unlearning baselines. The headlined results show DRUF reducing leakage accuracy to near zero on the tested LLaVA-1.5-hf/DocXPand-25k configuration while keeping KIE LC around 0.808.","tokens_in":22557,"tokens_out":7588,"duration_ms":71068,"significance":"The problem is well motivated and timely: relational leakage under weak visual evidence is a realistic privacy risk for document-processing MLLMs, and the paper offers a concrete benchmark (DocPrivacyBench) and a new unlearning formulation (pair-level KL-coupled forgetting with a dynamic forget set). The release of code and benchmark is a positive artifact. If the central claims survive a stricter evaluation, the paper would provide a useful starting point for privacy-preserving document MLLMs. However, the current evidence is not yet sufficient to support the claimed generality of DRUF: the unlearning evaluation is performed on one model–dataset pair, and the probing/forgetting/evaluation identities overlap, so the reported suppression may reflect targeted erasure of the exact pairs placed in the forget set rather than a general relational unlearning capability.","major_comments":[{"comment":"The claim that lower data quality generally leads to higher privacy leakage is not robust to the leakage threshold. For LLaVA-1.5-hf Image Driven, Acc@0.8 goes from 0.874 (DocXPand-25k) to 0.910 (IDNet) to 1.000 (IDNet with noise), while Acc@1.0 goes from 0.835 to 0.004 to 0.346, and LC goes from 0.972 to 0.809 to 0.882. Under the 0.8 threshold, the cleaner IDNet appears more leaky than DocXPand, whereas under the 1.0 threshold it is drastically less leaky, reversing the paper's data-quality narrative. The paper should justify the threshold choice and report the similarity-score distribution or a threshold-free metric, and it should explain the sharp transition from Acc@0.8=0.910 to Acc@0.9=0.004 for this cell.","section":"Section 5.3, Table 1"},{"comment":"The unlearning evaluation does not separate the identities used to build the dynamic forget set from those used to score leakage. F(t) is constructed from pairs leaked by the current model on probing inputs, and the leakage evaluation matches outputs against the same training identity library; Supp D.1 states that the 6,857 probing/evaluation face crops are derived from the same 5,000 DocXPand-25k samples used to train the target model. With no identity-level split between probe/forget targets and evaluation identities, the near-zero leakage in Table 2 can be explained entirely by erasing the exact associations placed in F(t), and the Section 3.2 claim that suppression 'generalizes to other data points that have not been probed' is unsupported. Please add a held-out-identity control: train on set A, probe and forget on a subset of A, and evaluate leakage on the remaining training identities that were never probed, plus a false-positive control using identities not in training.","section":"Section 3.2, Supp D.1, Table 2"},{"comment":"The unlearning comparison is run on a single model (LLaVA-1.5-hf) and a single dataset (DocXPand-25k), but the abstract states that 'we evaluate three MLLMs and six unlearning methods using this benchmark'. No unlearning results are reported for Xgen-Phi3, Idefics2, or for the IDNet datasets. The six-method comparison therefore supports conclusions only for one model–dataset pair, and the headline 4.8-percentage-point improvement over SCRUB is a single-cell result. Please either add unlearning experiments on at least one more model and one more dataset, or restrict the claims to the actual scope.","section":"Section 5.4, Table 2"},{"comment":"The ablation designed to show the benefit of the dynamic forget set is confounded. The 'Static Forget Set + RDU' row in Table 5 achieves Prompt-Driven/Image-Driven Acc of 0.002/0.006 with KIE LC 0.996, nearly identical to DRUF's 0.000/0.000 with KIE LC 0.996. The 'substantial degradation' (KIE LC 0.055) comes from a separate experiment with 500 static targets and 500 training steps, which is not a controlled comparison with the DRUF configuration (which uses the standard training schedule and dynamic updates at intervals). Thus the paper does not currently demonstrate that the dynamic update mechanism materially improves leakage suppression or utility preservation over a static set. Please report a matched static baseline using the same training schedule, forget-set size, and number of updates.","section":"Supp F.1, Table 5"}],"minor_comments":[{"comment":"The abrupt transition from Acc@0.8=0.910 to Acc@0.9=0.004 for LLaVA-1.5-hf on IDNet Image Driven is surprising; please report the similarity-score distribution and the exact matching rule at each threshold.","section":"Table 1"},{"comment":"The parenthetical numbers, e.g., '(55)', are said in the text to be numbers of leaked pairs, but the caption does not define them; please define the notation and clarify whether these counts are distinct identities.","section":"Table 2 caption"},{"comment":"The caption contains the literal text 'family\\u name' (with a backslash-u escape); this appears to be a rendering error and should be fixed.","section":"Figure 2"},{"comment":"The unlearning evaluation switches to a dual-private-field setting (given name and document number) at threshold 1.0, which differs from the three-field, multi-threshold setting of Table 1; please state this explicitly in the main text and justify the change, since it complicates direct comparison of leakage rates between sections.","section":"Section 5.4"},{"comment":"The '4.8 percentage points' improvement should be identified as the Image-Driven leakage-accuracy improvement over SCRUB for LLaVA-1.5-hf on DocXPand-25k, and the scope of the claim should be stated.","section":"Abstract"},{"comment":"The heading 'Experimental Setup.' is duplicated immediately before the description of the training-process analysis.","section":"Supp C.1"},{"comment":"Reference [7] is cited to support the claim that unlearning on exposed leakage points generalizes to unprobed data points; this is a central assumption, and the citation alone does not demonstrate that the dynamic forget-set approach generalizes across identities. Please either provide direct evidence or cite a study that establishes this generalization mechanism for similar settings.","section":"Section 3.2, reference [7]"},{"comment":"The forgetting objective L_forget = -K_a K_b is non-positive, and its minimization drives the product of span-level KL divergences upward; please state the intended optimization direction explicitly and clarify that the goal is to maximize the product of the two KL shifts.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of the venue and the problem is timely, but the main claims need stronger support before publication. In my view the most serious issue is the identity overlap between probing/forgetting and leakage evaluation; without a held-out identity split, the reported suppression is not convincing evidence of general relational unlearning. The threshold instability in Table 1 also needs to be resolved. These are fixable with additional experiments rather than a change in the paper's core idea."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear [Colleague],\n\nThe one thing to know: this paper identifies a genuinely important risk—document MLLMs that memorize field associations can leak several sensitive fields at once when the input image gives no usable visual evidence—and it ships a benchmark (DocPrivacyBench) plus a concrete unlearning method (DRUF). That part is worth your time. But the headline claim that DRUF generalizes relational unlearning is not supported by the evaluation as designed. The probe set, the dynamic forget set, and the leakage test all draw from the same 5,000 training identities (Supp. D.1 describes the test images as face crops derived from those same samples). No identity-level split is described anywhere. So the near-zero post-unlearning leakage in Table 2 is exactly what you'd expect if the method simply erased the pairs it found during probing; it does not demonstrate suppression on unseen identities. The static-forget-set ablation and the test-set-size study don't repair this, because they also score against the same identity pool.\n\nWhat the paper does well: the relational, multi-field framing is new and practical for KIE. The dynamic forget set and the multiplicative span-level KL objective are concrete and not in prior work. The baseline comparison is internally fair, since every unlearning method gets the same cached targets. The code and benchmark are released, which makes follow-up possible.\n\nThe soft spots, in order of severity. First, the missing held-out identities is load-bearing—without it the central generalization claim is unproven. Second, the leakage metric is fragile: in Table 1, LLaVA on IDNet Image Driven scores Acc@0.8 = 0.910 but Acc@1.0 = 0.004. That is a 227-fold swing from a threshold change, and it means the 'noisier data leaks more' narrative can reverse depending on where you set the bar. The paper needs a clearly primary metric and a sensitivity analysis. Third, the unlearning experiments run on a single model and dataset (LLaVA on DocXPand-25k), so even a clean identity split would leave breadth questions. Minor: no error bars on the main tables, and the span-level KL computation under-specifies how the sensitive spans are located in the generated output.\n\nMy take: the risk is real and the benchmark is a useful community asset, but the paper currently overclaims the unlearning result. I'd send it to review—the topic and the benchmark deserve referee time—but I'd expect major revision before any acceptance: report a held-out identity evaluation, stabilize the leakage metric, and add statistical grounding. If you're working on privacy or document MLLMs, read it for the benchmark and the phenomenon, not yet for the unlearning numbers.","headline":"A real relational-leakage risk and a useful benchmark, but the unlearning claims rest on an evaluation that erases known training pairs rather than demonstrating generalization to held-out identities.","tokens_in":23121,"tokens_out":3571,"would_cite":false,"duration_ms":34915,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Document MLLMs leak correlated identity fields when visual evidence is missing, and the proposed DRUF framework suppresses this leakage.","keywords":["Multimodal Large Language Models","Privacy Leakage","Machine Unlearning","Key Information Extraction","Document Understanding","Relational Privacy Leakage","Dynamic Relational Unlearning","Identity Documents"],"falsifier":"Train a document MLLM on DocXPand-25k, run DRUF using probing pairs detected on one split of the training identities, then evaluate leakage on a separate split of identities that never contributed to the dynamic forget set; if leakage accuracy remains near zero and KIE LC stays above 0.8, the relational-unlearning claim generalizes, and if leakage returns on those unseen identities, the reported suppression is targeted erasure of known pairs.","tokens_in":22054,"feed_emoji":"🪪","tokens_out":10376,"duration_ms":89195,"temperature":0.7,"pith_summary":"Document MLLMs trained for key information extraction on identity documents memorize associations between fields such as family name, given name, and document number. When the input image provides no valid visual evidence, the paper argues, these models fall back on the memorized relations and jointly generate multiple sensitive fields from a single training identity — a risk it calls relational privacy leakage, which prior single-field analyses missed. To counter it, the paper builds DRUF, an unlearning framework whose forgetting objective couples the two leaked spans, and DocPrivacyBench, a benchmark that measures leakage under image-driven and prompt-driven attacks. Across three MLLMs and six unlearning baselines, DRUF reduces Image-Driven leakage accuracy from 0.642 to 0.001, brings Prompt-Driven leakage to 0.000, and keeps normal KIE extraction quality at 0.808, a gain of 4.8 percentage points over the strongest baseline.","feed_headline":"Document AI leaks name and ID when images go blank","feed_subtitle":"New framework DRUF drops correlated-field leakage from 64% to near zero while preserving extraction quality.","key_machinery":"The load-bearing object is the product coupling of span-level KL divergences, $L_{\\mathrm{forget}} = -K_a K_b$, where $K_a = \\mathrm{KL}_{\\mathrm{span}}(s_a^i)$ and $K_b = \\mathrm{KL}_{\\mathrm{span}}(s_b^i)$ are the average teacher–student KL shifts over the token spans of two sensitive fields jointly exposed by the model. Because $\\partial L_{\\mathrm{forget}}/\\partial K_a = -K_b$ and $\\partial L_{\\mathrm{forget}}/\\partial K_b = -K_a$, each span's update is weighted by the other span's shift, which couples the two fields into one relation-level forgetting target and suppresses their joint recoverability under blank or visually uninformative inputs without erasing whole documents. Around this core, DRUF adds a retain branch that aligns the student with a frozen teacher on normal KIE samples, and a dynamic forget set $F(t)$ rebuilt periodically from probing rounds so that the forgetting targets follow the model's current leakage behavior rather than a static list.","core_discovery":"The paper's central discovery is that relational privacy leakage is a distinct failure mode of document MLLMs: under weak-evidence inputs such as blank or noise-degraded document images, these models reproduce correlated sensitive field pairs — for instance, given name and document number — that co-occurred in the identities they were trained on, instead of grounding their output in the image. The proposed framework treats each exposed field pair as a coupled forgetting unit: Relational Decoupling Unlearning computes span-level KL shifts $K_a$ and $K_b$ between a frozen teacher and a trainable student over the two sensitive spans and applies the objective $L_{\\mathrm{forget}} = -K_a K_b$, whose gradients weight each span by the other's shift and thereby break the stable co-generation of the pair. The forgetting targets are not preset: a dynamic set $F(t)$ collects the pairs the current model actually leaks on probing inputs and is refreshed across unlearning rounds, so the suppression tracks the model's evolving behavior. On DocPrivacyBench, the framework lowers Image-Driven leakage accuracy from 0.642 to 0.001 and Prompt-Driven leakage to 0.000 in the dual-field setting, keeps the normal KIE LC at 0.808, and improves leakage suppression by 4.8 percentage points over the strongest baseline.","pith_inferences":["The benchmark and probing procedure draw on the same 5,000-sample training identity library; a held-out identity split would show whether DRUF suppresses leakage on identities it never probed or merely erases the exact pairs it was told to forget.","The pair-coupling objective is a general recipe for structured outputs: receipts, medical forms, and financial documents could use the same $-K_aK_b$ coupling to stop correlated fields from being co-recoverable when inputs are degraded.","Leakage matching at token-level similarity with two-of-three private fields means a model that shuffles values between identities or lightly paraphrases would lower the measured score without being genuinely safe; testing output membership against the full training library would close that gap.","Because Image-Driven attacks leak more than Prompt-Driven attacks in almost every configuration, deployment-time defenses should treat abnormal or blank images as the primary attack surface rather than focusing on prompt hardening."],"forward_implications":["On noisier, lower-quality datasets the same model leaks far more: the strongest-leaking tested model goes from an Image-Driven Acc@1.0 of 0.004 on clean IDNet to 0.346 after noise is added, and 0.835 on DocXPand-25k.","Aggressive sample-level unlearning (GA, GA+KL) drives leakage to near zero but collapses normal KIE utility to LC 0.119–0.259, while static preference or fine-tuning baselines leave leakage rates of 0.175–0.676, so pair-level relational forgetting is the granularity that preserves utility.","A static forget set built once is not enough: with RDU but no dynamic update, residual leakage remains at 0.002 (Prompt-Driven) and 0.006 (Image-Driven), whereas refreshing the forget set reaches 0.000.","The probing test set should be moderate: 100 samples suppresses leakage to 0.001 with KIE LC 0.808, while 500 samples collapses KIE LC to 0.306 without reducing leakage further.","Training-similar face inputs leak over more unique identities (55) than unrelated photos (32), so the risk persists under both irrelevant and distribution-similar visual triggers."],"supporting_citations":[{"why":"Supplies the prior result that document-based VQA models reproduce training answers after visual evidence is removed, which motivates the weak-evidence leakage setting.","marker":"[20]"},{"why":"SCRUB is the strongest baseline and the teacher-student unlearning architecture that DRUF's retain/forget branches build on.","marker":"[13]"},{"why":"Gradient ascent baseline that shows aggressive knowledge unlearning destroys KIE utility, motivating pair-level relational forgetting.","marker":"[11]"},{"why":"DocXPand-25k provides the identity-card and passport samples used for training, probing, and the noisy-data leakage comparison.","marker":"[14]"},{"why":"IDNet provides the driver's-license samples and the clean-versus-noisy comparison used to show data quality affects leakage.","marker":"[27]"},{"why":"Supports the generalization assumption that unlearning updates on probed leakage points extend to unprobed samples.","marker":"[7]"},{"why":"Establishes that document KIE tasks carry privacy leakage risks and lack tailored privacy-enhancement methods.","marker":"[22]"}],"fun_headline_variants":["Blank scans make doc AI spill correlated ID fields","Weak visual cues leak name-number pairs in document MLLMs","DRUF unlearns leaked field pairs, preserving extraction accuracy","DocPrivacyBench exposes AI identity leaks from weak images","No visual? Document MLLMs gamble on memorized relations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes the pairs it discovers during probing and the identities it tests afterward are not the same memorized records; if they overlap, the reported suppression is targeted deletion of known pairs, not a general ability to stop leakage.","fun_headline_variants_meta":{"raw":{"variants":["Blank scans make doc AI spill correlated ID fields","Weak visual cues leak name-number pairs in document MLLMs","DRUF unlearns leaked field pairs, preserving extraction accuracy","DocPrivacyBench exposes AI identity leaks from weak images","No visual? Document MLLMs gamble on memorized relations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001195,"raw_usage":{"total_tokens":4989,"prompt_tokens":1067,"completion_tokens":3922,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":683,"completion_tokens_details":{"reasoning_tokens":3841}},"tokens_in":683,"tokens_out":3922,"duration_ms":27661,"temperature":1.0,"reasoning_tokens":3841,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:45:56.077661+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a document MLLM on DocXPand-25k, run DRUF using probing pairs detected on one split of the training identities, then evaluate leakage on a separate split of identities that never contributed to the dynamic forget set; if leakage accuracy remains near zero and KIE LC stays above 0.8, the relational-unlearning claim generalizes, and if leakage returns on those unseen identities, the reported suppression is targeted erasure of known pairs.","supporting_citations":[{"cited_title":"DocXPand-25k: a large and diverse benchmark dataset for identity documents analysis","cited_arxiv_id":"2407.20662","evidence_quote":"DocXPand-25k provides the identity-card and passport samples used for training, probing, and the noisy-data leakage comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"IDNet provides the driver's-license samples and the clean-versus-noisy comparison used to show data quality affects leakage."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes that document KIE tasks carry privacy leakage risks and lack tailored privacy-enhancement methods."}],"review_version":1}