{"id":"5ba7b223-490a-4373-b003-b194fd2d4d45","arxiv_id":"2508.04208","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"DP-DocLDM fine-tunes a latent diffusion model under differential privacy to generate synthetic document images, but the public pretraining set already contains the private benchmark datasets, invalidating the claimed privacy protection.","lead":"Researchers combined differentially private latent diffusion models with document-specific conditioning to generate synthetic document images for training downstream classifiers, reporting large gains on the small Tobacco3482 dataset. A critical data overlap, RVL-CDIP and Tobacco3482 are subsets of the IIT-CDIP pretraining set, undermines the privacy claim and the validity of the evaluation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Public pretraining data (IIT-CDIP) includes the supposed private datasets (RVL-CDIP, Tobacco3482), so the autoencoder and diffusion pretraining already saw the private images non-privately; DP fine-tuning cannot provide the claimed privacy guarantee.","rationale":"The reader's weakest assumption is exactly the load-bearing flaw: the public pretraining dataset is not disjoint from the private evaluation datasets. This is not a disagreement with consensus or a stylistic complaint; it is an internal correctness issue. The paper's own Section 4.1 claims no privacy risk from the autoencoder because it is trained on the public dataset, and Section 5.1 identifies that public dataset as IIT-CDIP, which is the source collection for RVL-CDIP and Tobacco3482. Therefore the most privacy-sensitive component—an autoencoder trained on all 11M images—has already seen the private images non-privately. Even if the diffusion model's pretraining filter excluded some private classes, the autoencoder alone is sufficient to invalidate the end-to-end DP claim, because the generated latents are decoded through that non-privately trained autoencoder. The downstream improvements, especially on Tobacco3482, are exactly what one would expect from a model that has memorized the private distribution during pretraining. The reader's rejection is warranted; no verdict change is needed. I also note the paper does release code, which is useful for the proposed hash-overlap and retraining checks, but code availability cannot repair the data-overlap problem.","tokens_in":17062,"tokens_out":4145,"duration_ms":45098,"concrete_test":"Run a provenance/hash overlap check: download the IIT-CDIP metadata and compute perceptual or cryptographic hashes (e.g., SHA-1/MD5) for the RVL-CDIP and Tobacco3482 images; count how many appear in the 11M IIT-CDIP set used for autoencoder pretraining. If the overlap is nonzero, the pretraining set contained private images. For causal impact, retrain the autoencoder and diffusion model on IIT-CDIP with all RVL-CDIP and Tobacco3482 images removed, rerun the Table 3 downstream evaluation, and compare with the reported DP-Adam baselines; if accuracy drops toward those baselines, the reported gains are leakage artifacts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central privacy claim depends on a strict public/private split. Section 4.1 asserts that because the autoencoder is trained on the public dataset, 'it does not introduce any privacy risks'; Section 5.1 defines that public dataset as IIT-CDIP Test Collection 1.0, using all 11M images for the autoencoder and a filtered subset for diffusion pretraining. However, RVL-CDIP is a labeled subset of IIT-CDIP, and Tobacco3482 is drawn from the same IIT-CDIP collection. Thus the pretraining phase has already observed the exact private images without any DP protection. The DP guarantee from DPDM/DP-Promise applies only to the fine-tuning step; the full pipeline—public pretraining on private images plus private fine-tuning—does not satisfy (ε,δ)-DP with respect to those records. Any fidelity of generated samples to the private distribution, including the substantial gains on Tobacco3482 in Table 3, may be explained by this non-private exposure rather than by the private fine-tuning. The manuscript's statement that the autoencoder 'does not introduce any privacy risks' is therefore unsupported and, given the known provenance of these benchmarks, false.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DP-DocLDM, a method to generate synthetic document images under differential privacy. It first non-privately pretrains a class- and layout-conditioned latent diffusion model on IIT-CDIP Test Collection 1.0, then privately fine-tunes the model on private document datasets (RVL-CDIP and Tobacco3482) using DPDM or DP-Promise, and finally uses the generated synthetic samples to train downstream classifiers. The authors report FID scores and downstream classification accuracies at privacy levels ε ∈ {1, 5, 10}, and claim that the approach outperforms directly applying DP-Adam, especially on the small-scale Tobacco3482 dataset.","tokens_in":17322,"tokens_out":3044,"duration_ms":35026,"significance":"If the privacy guarantee were valid, the paper would address an important problem: replacing sensitive document images with synthetic counterparts that preserve utility while satisfying differential privacy. The paper also provides a fairly extensive ablation of pretraining setups, private fine-tuning strategies, and autoencoder choices, and it releases source code. However, the central privacy claim is undermined by a data-provenance error: the 'public' pretraining corpus (IIT-CDIP) contains the very datasets used as 'private' benchmarks (RVL-CDIP and Tobacco3482 are subsets of IIT-CDIP). Consequently, the reported gains cannot be attributed to the DP fine-tuning, and the claim that the pretrained autoencoder 'does not introduce any privacy risks' (Section 4.1) is incorrect. The evaluation is therefore not a valid demonstration of differentially private document generation.","major_comments":[{"comment":"The public/private data split is invalid. Section 4.1 states that the autoencoder is trained on the public dataset and therefore introduces no privacy risk, and Section 5.1 defines the public dataset as IIT-CDIP Test Collection 1.0 (all 11M images for the autoencoder). However, RVL-CDIP is a labeled subset of IIT-CDIP, and Tobacco3482 is drawn from the same IIT-CDIP collection. Thus the autoencoder and the diffusion pretraining already observed the exact 'private' images without any DP protection. The DP guarantee (from DPDM/DP-Promise) applies only to the fine-tuning step, so the full pipeline does not provide (ε,δ)-DP with respect to those records. The assertion that pretraining 'does not introduce any privacy risks' is false as stated.","section":"Sections 4.1 and 5.1"},{"comment":"The downstream evaluation is further contaminated by the same overlap. The DiT-B classifier is described as pretrained in a self-supervised manner on IIT-CDIP, which includes the RVL-CDIP and Tobacco3482 training (and test) images. Therefore the downstream classifier has already seen the private data non-privately, making the accuracy numbers in Table 3 an unreliable measure of the utility of privately fine-tuned synthetic data. This is a separate leakage path from the pretraining of the diffusion model.","section":"Section 5.3 and Table 3"},{"comment":"The headline claim that the approach brings 'substantial performance improvements' on small-scale datasets (Tobacco3482) is not supported by the evidence. Since the pretraining corpus contains the private evaluation data, the high downstream accuracies in Table 3 (e.g., 80-82% on Tobacco3482) can be explained by non-private exposure to the target distribution during pretraining, not by the DP fine-tuning. The comparison with DP-Adam [52] is therefore not an apples-to-apples comparison of privacy-preserving methods.","section":"Abstract and Section 6"}],"minor_comments":[{"comment":"Tobacco3482 is cited only via a Kaggle URL in a footnote; the original dataset reference should be provided.","section":"Section 5.1"},{"comment":"Typo: 'Romback et al.' should be 'Rombach et al.'","section":"Section 4.1"},{"comment":"The abstract says 'DP-SGD' while Section 5.5 and Table 3 compare with 'DP-Adam' from [52]; please clarify the terminology consistently.","section":"Abstract and Section 5.5"},{"comment":"The paper says DP-Promise requires manual tuning of σ for each scenario (Section 5.4), but Section 5.2 states that σ is computed by numerical optimization for DPDM only. Please specify how σ was set for DP-Promise in all experiments.","section":"Section 5.4"}],"recommendation":"major_revision","confidential_remarks":"The data-overlap problem is severe enough that one could justify a reject, because the current experimental evidence does not support the central privacy or utility claims. However, the flaw is not in the method itself but in the experimental design; if the authors re-run the entire evaluation with a public pretraining corpus that is truly disjoint from the private benchmarks (or otherwise account for the overlap), the manuscript could become publishable. Given the scope of the required rework, I recommend major revision rather than reject, but the authors must demonstrate a genuine public/private split and redo all tables and qualitative claims accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, here's my read.\n\nThe genuinely new thing is the system: latent diffusion fine-tuned under DP specifically for document images, with OCR-derived layout conditioning and per-label private fine-tuning. That combination hasn't been done before, and the engineering is sensible. The paper is clearly written, the code is public, and the visual samples at low epsilon look plausible. Credit where due.\n\nBut the central empirical claim collapses on inspection. The \"public\" pretraining dataset is IIT-CDIP 1.0, and the autoencoder is trained on all 11M images. RVL-CDIP is a labeled subset of IIT-CDIP, and Tobacco3482 is drawn from the same collection. So the model already saw the exact private images without any DP protection before fine-tuning. The DP guarantee covers only the fine-tuning step, not the pretraining that already encoded those documents into the autoencoder and diffusion weights. The statement in Section 4.1 that the autoencoder \"does not introduce any privacy risks\" is therefore wrong, and the improvements on Tobacco3482 in Table 3 may simply be memorization of the private training distribution.\n\nThat flaw is load-bearing. The paper's purpose is to show that DP fine-tuning of a pretrained model can generate useful synthetic documents. If the pretraining already leaked the private data, the result tells you nothing about whether the DP fine-tuning is the source of utility. It could be mostly leakage. Without a no-fine-tuning ablation (generate from the pretrained model alone) or a truly disjoint public dataset, the empirical story is uninterpretable.\n\nSecondary issues: no error bars or repeated runs anywhere; configuration selection appears to happen on the same test sets used for the final numbers; and the comparison to DP-Adam is not apples-to-apples because the downstream classifiers in this paper use heavy augmentation and pretrained backbones that the DP-Adam baselines in [52] did not have. Those are softer but still matter.\n\nThe paper is not a toy. The conditioning ideas and the per-label strategy could be worth adapting. But as presented, the evidence for the headline claim is invalidated. I would not cite it for the numbers. I would, however, send it to peer review proper: a competent referee should catch the data provenance issue, and the methodological novelty is enough to warrant the attention. It deserves a serious look, just not acceptance in its current form.","headline":"First DP latent diffusion paper for documents, but the public/private split is invalid—the pretraining corpus contains the private testbeds, so the headline privacy/utility claim does not survive contact with the data provenance.","tokens_in":17830,"tokens_out":1506,"would_cite":false,"duration_ms":20995,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A class- and layout-conditioned latent diffusion model, fine-tuned under differential privacy on private document datasets, generates synthetic images that substitute for real data in downstream document classification, and on small dataset","keywords":["differential privacy","latent diffusion models","document image classification","synthetic document generation","DPDM","DP-Promise","layout conditioning"],"falsifier":"Run a nearest-neighbour or membership audit comparing synthetic images against the original IIT-CDIP images used in pretraining. If many generated samples are near-duplicates of specific RVL-CDIP or Tobacco3482 documents that the supposedly private fine-tuning could not have memorized alone, then the effective privacy is much weaker than the fine-tuning budget suggests, and the claim that the synthetic set substitutes for private data under strict DP fails.","tokens_in":16908,"feed_emoji":"🔐","tokens_out":7664,"duration_ms":81637,"temperature":0.7,"pith_summary":"The paper tries to show that the usual pain of differential privacy — degraded accuracy and awkward training — can be sidestepped by replacing private document images with synthetic ones. It pretrains a latent diffusion model on public documents, privately fine-tunes it on private data, and then uses the generated samples to train ordinary classifiers. The claim is that this pipeline yields realistic, class-specific documents across privacy budgets ($\\varepsilon \\in \\{1,5,10\\}$), and that on a small benchmark it substantially outperforms the direct DP-Adam baseline while keeping the same formal privacy guarantee at the fine-tuning step.","feed_headline":"Synthetic documents substitute for private data under DP","feed_subtitle":"Classifiers trained on synthetic images match or beat direct DP training on a small document benchmark.","key_machinery":"The load-bearing object is a latent diffusion model (LDM): a diffusion model that adds and removes noise in the compressed latent space of a pretrained autoencoder, rather than in pixel space. It is conditioned on class embeddings and on a binary layout mask derived from OCR text bounding boxes, and it is privately fine-tuned with DPDM (DP-SGD with noise multiplicity) or DP-Promise, with one model per label for the best-performing configuration. The same conditioning layout is applied at sampling time, and synthetic images are then used to train standard classifiers with standard, non-private procedures.","core_discovery":"DP-DocLDM's central discovery is that per-label private fine-tuning — training one small latent diffusion model per document class with differential privacy — plus OCR-extracted layout conditioning produces synthetic documents good enough to train downstream classifiers. In the comparison study, layout- and class-conditioned pretraining combined with per-label DPDM fine-tuning reached the best downstream accuracy (77–78%) on RVL-CDIP, and in the full evaluation on the small Tobacco3482 dataset, classifiers trained on synthetic images reached 78–82% accuracy across privacy levels, well above the 44–74% range of direct DP-Adam training. The authors present this as evidence that synthetic data","pith_inferences":["The authors leave implicit that the quoted $\\varepsilon$ protects only the private fine-tuning step; because RVL-CDIP and Tobacco3482 are subsets of the IIT-CDIP pretraining corpus, the non-private pretraining already saw the 'private' images, so the end-to-end guarantee is weaker than the fine-tuning budget alone suggests.","A direct test this suggests: a nearest-neighbour or membership audit of generated images against IIT-CDIP. If generated samples reproduce fragments of specific private documents that only pretraining could have seen, the synthetic set is not the clean private substitute the paper claims.","The same generate-then-train pipeline should transfer to other document tasks — key information extraction, layout analysis, form parsing — wherever DP training is expensive, provided the conditioning channels (class, layout, text) align with what the target task needs.","Because all setups used one fixed hyperparameter configuration, the DPDM-versus-DP-Promise ordering may be an artifact of that configuration; repeating the comparison with per-method tuning would show which algorithm actually carries the result."],"forward_implications":["On small-scale private document sets, DP-generated synthetic data can give downstream classifiers an edge over training with DP directly: Tobacco3482 accuracy jumps from roughly 44–74% to 78–82%.","Class-conditional generation from a single global model struggles to separate similar document classes; per-label models appear to be the more reliable private generation strategy.","Layout conditioning, extracted cheaply by OCR, contributes the largest single performance gain among the pretraining options tested.","Stronger privacy ($\\varepsilon=1$) costs little in visual quality on the larger dataset but visibly degrades FID on the small dataset.","Self-supervised document-specific pretraining of the downstream classifier (DiT-B) exploits the synthetic data best."],"supporting_citations":[{"why":"Supplies the DPDM algorithm, DP-SGD with noise multiplicity, used for private fine-tuning and privacy accounting.","marker":"[13]"},{"why":"Supplies the alternative DP-Promise training algorithm and the GDP accountant compared in the study.","marker":"[62]"},{"why":"Defines the latent diffusion model and pretrained autoencoder backbone the whole pipeline builds on.","marker":"[48]"},{"why":"Provides the DP-SGD/DP-Adam algorithm and gradient clipping/noising mechanism behind the direct-training baseline.","marker":"[1]"},{"why":"Introduces the RVL-CDIP dataset used as a private benchmark for fine-tuning and downstream evaluation.","marker":"[25]"},{"why":"Supplies the public IIT-CDIP corpus used for non-private autoencoder and diffusion pretraining.","marker":"[56]"},{"why":"Provides the direct DP-Adam baseline numbers and the 80/20 split for Tobacco3482 used in comparisons.","marker":"[52]"},{"why":"Defines classifier-free guidance, the conditioning method used in class-conditional private fine-tuning and sampling.","marker":"[29]"}],"fun_headline_variants":["DP diffusion-generated docs beat direct DP training","Per-label DP fine-tuning improves synthetic document quality","Synthetic docs from DP latent diffusion match or beat DP-Adam","Layout-conditioned DP diffusion boosts downstream classifier performance","Privacy-preserving synthetic docs outperform direct DP on small datasets"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The public IIT-CDIP corpus used for non-private pretraining is assumed to be separate from the private datasets, but RVL-CDIP and Tobacco3482 are in fact subsets of IIT-CDIP, so the stated privacy guarantee protects only the fine-tuning step, not the pretraining exposure.","fun_headline_variants_meta":{"raw":{"variants":["DP diffusion-generated docs beat direct DP training","Per-label DP fine-tuning improves synthetic document quality","Synthetic docs from DP latent diffusion match or beat DP-Adam","Layout-conditioned DP diffusion boosts downstream classifier performance","Privacy-preserving synthetic docs outperform direct DP on small datasets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000182,"raw_usage":{"total_tokens":1172,"prompt_tokens":791,"completion_tokens":381,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":319}},"tokens_in":535,"tokens_out":381,"duration_ms":4679,"temperature":1.0,"reasoning_tokens":319,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T00:47:39.877342+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a nearest-neighbour or membership audit comparing synthetic images against the original IIT-CDIP images used in pretraining. If many generated samples are near-duplicates of specific RVL-CDIP or Tobacco3482 documents that the supposedly private fine-tuning could not have memorized alone, then the effective privacy is much weaker than the fine-tuning budget suggests, and the claim that the synthetic set substitutes for private data under strict DP fails.","supporting_citations":[{"cited_title":"In: USENIX Se- curity Symposium (2024), https://www.usenix.org/conference/usenixsecurity24/ presentation/wang-haichen","cited_arxiv_id":null,"evidence_quote":"Supplies the alternative DP-Promise training algorithm and the GDP accountant compared in the study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the DPDM algorithm, DP-SGD with noise multiplicity, used for private fine-tuning and privacy accounting."},{"cited_title":"In: International Conference on Document Analysis and Recognition (ICDAR)","cited_arxiv_id":null,"evidence_quote":"Introduces the RVL-CDIP dataset used as a private benchmark for fine-tuning and downstream evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the public IIT-CDIP corpus used for non-private autoencoder and diffusion pretraining."},{"cited_title":"International Journal on Document Analysis and Recognition (IJDAR) 27(3), 475–499 (jun 2024), http://dx.doi.org/10.1007/s10032-024-00469-8","cited_arxiv_id":null,"evidence_quote":"Provides the direct DP-Adam baseline numbers and the 80/20 split for Tobacco3482 used in comparisons."}],"review_version":1}