{"id":"197dbc41-a51d-4f11-bb83-43d312f341bf","arxiv_id":"2508.01749","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Dosser improves differentially private dataset distillation by decoupling sampling from optimization and projecting signals into a learned subspace.","lead":"This paper introduces a framework called Dosser that makes privacy-preserving synthetic datasets more accurate by separating the private-data sampling step from the optimization step and by projecting training signals into a lower-dimensional subspace. It reports substantial accuracy gains on CIFAR-10, MNIST, and FashionMNIST under a fixed differential privacy budget.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lemma 1's claimed sensitivity reduction under projection is invalid: for a fixed orthogonal projection, worst-case L2 sensitivity equals the original, so the reduced noise scale is unjustified and, if implemented, breaks the stated DP guarantee.","rationale":"The reader's weakest assumption is correct and is the most load-bearing point in the paper. The central novelty of SER is a theoretical privacy-utility gain from subspace projection; that gain is derived in Theorem 1 through Lemma 1's sensitivity ratio. Lemma 1's proof substitutes a single adjacent-pair norm for the worst-case sensitivity, which is not valid for a DP guarantee. The same flaw propagates into Theorem 1. This is a correctness risk for the privacy claim: if the code reduces sigma based on projected dimension, the published CIFAR-10 numbers may not satisfy (10,10^-5)-DP. I would not call this a rejection of the empirical method: DOS is a plausible way to avoid repeated privacy queries, and Appendix B's MSE curves suggest SER can reduce estimation error even with an unreduced per-coordinate noise scale through rank reduction, though the effect size and interpretation would change. The concrete test distinguishes those two mechanisms. Secondary issues remain: Table 2 ablates DOS and SER but not PEA, so part of the improvement over NDPDC may come from PEA; and the claim that LAION-5B contains no CIFAR-10-related information is unsupported and possibly false, which would weaken the no-leakage argument for the SD-based auxiliary set. Appendix A also treats epsilon=P^T P mu - mu as a zero-mean random vector in the MSE expansion, although it is deterministic given P and mu; that is a secondary algebraic issue. These are addressable, but the Lemma 1 issue must be fixed before the theory can be trusted. I therefore do not move the reader's conditional verdict.","tokens_in":17639,"tokens_out":11479,"duration_ms":135936,"concrete_test":"Recompute the L2 sensitivity of the projected mean query f_proj(D)=P*mean(D) under the exact adjacency model used in the paper. Construct two adjacent datasets differing only by a vector v lying in the row space of P with norm equal to the clipping bound; then ||P v||=||v||, so the worst-case sensitivity is unchanged. Then recompute Theorem 1's MSE difference with sigma_proj=sigma_orig rather than sigma_orig*sqrt(k/d) and check whether the predicted noise-reduction term remains positive. To settle the practical impact, rerun the CIFAR-10 IPC=50 comparison with the noise scale forced to the original sensitivity calibration; if SER still helps, the method can be salvaged as rank-reduction, but the theoretical claim and any privacy accounting based on a reduced sigma must be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing concern is Lemma 1 in Appendix A. It claims that projecting the sample mean with an orthonormal-row matrix P reduces the L2 sensitivity by a factor sqrt(k/d), permitting a proportionally smaller Gaussian noise scale under the same (epsilon,delta) budget. The proof computes the sensitivity ratio for a single adjacent pair's difference vector, but DP requires the supremum over all adjacent pairs. For any fixed P, sup_{adjacent D,D'} ||P(mu(D)-mu(D'))||/n = sup_{||v||<=B} ||P v||/n = B/n, since the supremum is attained by v lying in the row space of P. This equals the original sensitivity, so sigma_proj cannot be reduced while preserving (epsilon,delta)-DP. The 'Dimensional Reduction Effect' term in Theorem 1 inherits this error via the lemma. If the implementation calibrates noise from the projected dimension, as Sections 3.3-3.4 and Lemma 1 suggest, the claimed (10,10^-5)-DP guarantee may not hold. The empirical MSE improvement of SER may still survive when sigma_proj=sigma_orig because the reconstructed noise has rank k rather than d, but that is a different mechanism and does not justify a smaller noise scale. This must be corrected before the theoretical contribution and the privacy accounting can be trusted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Dosser, a framework for differentially private dataset distillation that combines Decoupled Optimization and Sampling (DOS) with Subspace-based Error Reduction (SER). DOS first samples a fixed set of DP-protected training signals and then performs many optimization steps on the synthetic dataset using those stored signals, avoiding the need to add fresh noise at every optimization step. SER projects the extracted training signals onto a low-dimensional subspace learned from an auxiliary dataset (generated by Stable Diffusion for CIFAR-10, or by a DP diffusion model for MNIST/FashionMNIST), with the goal of reducing the impact of DP noise. The paper reports substantial accuracy gains over prior private distillation methods on MNIST, FashionMNIST, and CIFAR-10 under (10,10^-5)-DP, including a 10.0% improvement on CIFAR-10 at IPC=50 and an 8.3% improvement with one-fifth the distilled set size. The theoretical contribution is an MSE decomposition (Theorem 1) that separates projection residual, dimensional-reduction, and projection-error terms, supported by a lemma about the sensitivity of the projected sample mean (Lemma 1) and a composition theorem for the auxiliary-data budget (Theorem 2).","tokens_in":17970,"tokens_out":8625,"duration_ms":106900,"significance":"If the empirical results are correct, the paper would make a useful practical contribution to private dataset distillation: the decoupling idea is simple and effective, and the subspace-projection mechanism is well motivated. The authors provide code, extensive ablations, and comparisons with several baselines, which strengthens the empirical part. However, the core theoretical justification for SER, specifically the claim that projecting the signal reduces the L2 sensitivity and therefore permits a smaller Gaussian noise scale, is flawed. Because differential privacy is the central promise of the paper, this error is load-bearing: the stated privacy guarantee is not established for the SER component as described. The empirical improvements might still be real under a corrected post-processing interpretation, but the current manuscript overstates its theoretical contribution and needs a substantial revision of the privacy analysis.","major_comments":[{"comment":"The sensitivity reduction claimed in Lemma 1 is invalid. For a fixed matrix P with orthonormal rows (P P^T = I_k), the L2 sensitivity of the projected sample mean f(D) = P * (1/n) * sum clip(phi(x_i)) under add/remove adjacency is sup_{||v|| <= B} ||P v|| / n = B/n, because the supremum is attained by v lying in the row space of P (e.g., v = P^T y with ||y|| = B). This equals the original sensitivity, not sqrt(k/d) * B/n. The proof incorrectly computes the norm of a single difference vector rather than the supremum over all adjacent pairs. Consequently, the ratio sigma_orig / sigma_proj derived in the lemma is unjustified, and if the implementation sets sigma_proj = sqrt(k/d) sigma_orig, the Gaussian mechanism with that reduced noise scale does not satisfy the stated (epsilon, delta)-DP guarantee.","section":"Appendix A, Lemma 1"},{"comment":"The 'Dimensional Reduction Effect' term in Theorem 1 inherits the error of Lemma 1: it relies on the false claim that the DP noise scale can be reduced by a factor related to the subspace dimension. The MSE decomposition therefore overstates the benefit of subspace projection for reducing DP noise. To make the theorem valid, the authors should either (a) add the noise in the original d-dimensional space and treat projection as post-processing, in which case sigma_proj = sigma_orig and the reconstruction noise variance is reduced because the noise covariance after projection and reconstruction is sigma^2 * P^T P, or (b) provide a correct sensitivity analysis for a mechanism that genuinely adds noise in the projected space. As written, the theorem's second term is unsupported and the claimed noise-efficiency gain is not established.","section":"Theorem 1 and Section 3.3"},{"comment":"The description of the sampling stage is ambiguous about the order of projection, clipping, and noise addition. The text says the noisy mean in the projected space is obtained by projecting and then adding Gaussian noise with scale sigma_proj, and Section 3.3 explicitly allows sigma_proj to be smaller than sigma_orig. If the implementation follows this description, the privacy guarantee fails. If the implementation instead adds noise in the original space and then projects (which would be valid as post-processing), the text and the MSE analysis must be revised to reflect that sigma_proj = sigma_orig. The paper needs to state the exact mechanism unambiguously and give a correct sensitivity analysis for that mechanism.","section":"Section 3.4 and Algorithm 1"}],"minor_comments":[{"comment":"Calling (10, 10^-5) a 'strict privacy budget' is misleading, since epsilon=10 is a high privacy parameter (weak privacy guarantee). Consider rephrasing and reporting results for lower epsilon values, e.g., epsilon in {1,2,5}, to support the privacy-preserving claim.","section":"Abstract and Section 4.1"},{"comment":"The display of Theorem 1 is difficult to parse because several symbols (e.g., the exact form of the 'Dimensional Reduction Effect' term) are not clearly defined in the main text. Please re-typeset the equation with explicit notation and define every quantity used in the decomposition.","section":"Theorem 1 equation"},{"comment":"The claim that Stable Diffusion pretrained on LAION-5B 'contains no information related to CIFAR-10' is not empirically substantiated. Since CIFAR-10 is a widely used public dataset, web-scraped corpora may contain images from it; a brief discussion of potential distribution overlap would strengthen the privacy argument.","section":"Appendix D.1"},{"comment":"The caption of Table 3 should explicitly state the total privacy budget (epsilon=10, delta=10^-5) and clarify that the split satisfies epsilon1 + epsilon2 = 10, so the comparison of budget splits is easy to follow.","section":"Appendix B.1, Table 3"},{"comment":"The table reports mean accuracies without confidence intervals or standard deviations; adding these would help assess the statistical significance of the reported improvements, especially for the 8.3% and 10.0% claims.","section":"Section 4.2, Table 1"}],"recommendation":"major_revision","confidential_remarks":"The reader's stress-test concern about Lemma 1 is correct and is the central issue. I agree that the paper cannot be accepted as is, because the privacy accounting is invalid if the SER mechanism is implemented as written. However, the empirical framework may still be salvageable by reframing SER as post-processing noise reduction (add full-scale noise, then project), and the DOS contribution appears sound. I recommend a major revision with a careful re-derivation of the privacy analysis, and I would suggest having the revised version reviewed by a researcher with specific expertise in differential privacy. The manuscript also tends to overstate the 'strictness' of the privacy budget, which should be toned down."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the DOS half is a clean, useful trick for private dataset distillation, but the SER half is theoretically mis-justified and the paper as written has a privacy-accounting error that needs correction.\n\nWhat's new and good: decoupling the sampling stage from the optimization stage in matching-based privacy-preserving DD is simple and effective. The paper shows that sampling 10k DP-protected signals and then optimizing for 200k iterations with the stored signals beats the coupled approach, and the ablations support this. The SER idea (projecting signals onto a low-dimensional subspace learned from auxiliary data before matching) is also sensible as a dimensionality-reduction heuristic, and the empirical gains over NDPDC are real. The authors release code, which helps reproducibility.\n\nThe soft spot is Lemma 1 and the Dimensional Reduction Effect term. The lemma claims that projecting the sample mean with an orthonormal matrix reduces the L2 sensitivity by a factor sqrt(k/d), allowing a proportionally smaller Gaussian noise scale. That is not right. For a fixed orthogonal projection P, the worst-case sensitivity of f(D)=P*mean(D) equals the original sensitivity B/n, because an attacker can pick an adjacent pair whose difference vector lies in the row space of P. You cannot reduce the per-coordinate noise scale while keeping the same (eps,delta) guarantee. The MSE benefit of projection actually comes from reducing the number of noise coordinates (rank k rather than rank d), which is a legitimate but different mechanism. If the implementation calibrated noise from the projected dimension as the text suggests, the claimed (10,10^-5)-DP guarantee would be broken. If the authors instead used the original sensitivity, the empirical results may survive, but then the theoretical contribution is wrong and needs rewriting.\n\nOther, smaller issues: the privacy-budget split for auxiliary generation is chosen empirically and should be framed that way; the headline numbers include PEA, so the ablation must isolate PEA's contribution. These are minor.\n\nBottom line: this is not a desk-reject. The empirical work is substantial and the DOS idea is publishable. But the SER proof has a load-bearing flaw. Send it to review with a request for a corrected sensitivity analysis and a re-run of experiments under the correct noise calibration.","headline":"DOS is a solid empirical increment, but Lemma 1's sensitivity reduction claim is wrong and the DP accounting needs fixing before the paper can be trusted.","tokens_in":18446,"tokens_out":4034,"would_cite":false,"duration_ms":51252,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Dosser, a framework for differentially private dataset distillation, achieves a 10.0% accuracy gain on CIFAR-10 at 50 images per class and an 8.3% gain at one-fifth the distilled set size by decoupling sampling from optimization and…","keywords":["differential privacy","dataset distillation","synthetic data generation","subspace projection","signal-to-noise ratio","matching-based distillation","decoupled optimization","privacy-preserving machine learning"],"falsifier":"Compute the $\\ell^2$ sensitivity of the projected mean function $f_{\\text{proj}}(D)=\\tfrac{1}{n}\\sum_i P\\phi(x_i)$ under the add/remove-one adjacent-dataset model; if it equals the original sensitivity instead of the claimed smaller value, Lemma 1's noise-scale reduction fails. A simple control experiment would replace SER's PCA projection with a random orthonormal projection of the same dimension: if accuracy stays the same, the gain is dimensionality reduction rather than the learned informative directions.","tokens_in":17476,"feed_emoji":"🔒","tokens_out":10184,"duration_ms":111027,"temperature":0.7,"pith_summary":"This paper tries to show that the main waste in privacy-preserving dataset distillation is not the privacy budget itself but how the budget is spent: existing matching-based methods add fresh differentially private noise at every sampling step and match raw signals from randomly initialized networks, so much of the added noise carries little information. The proposed framework, Dosser, separates the sampling of private training signals from the optimization of synthetic images, and projects signals into a low-dimensional informative subspace before noise is added. With the same privacy budget, this yields a cleaner training signal and a much smaller distilled dataset. If the claim is correct, private synthetic data becomes more practical: more accurate compact datasets under a fixed privacy guarantee, or the same accuracy under a tighter one.","feed_headline":"Private distilled data gains 10% accuracy under same privacy budget","feed_subtitle":"Decoupling optimization from sampling shrinks private distilled sets to one-fifth the size with better accuracy.","key_machinery":"The load-bearing mechanism has two parts. DOS (Decoupled Optimization and Sampling) first samples and stores $N_s$ DP-protected aggregated training signals, one per iteration, and then runs $N_o \\gg N_s$ optimization steps that reuse those stored signals, so extra convergence costs no extra privacy. SER (Subspace-based Error Reduction) learns a projection matrix $P$ by PCA on an auxiliary dataset and applies it before clipping and Gaussian noise, so matching happens in a $k$-dimensional subspace; the paper's Theorem 1 decomposes the mean-squared error of mean estimation into projection residual, dimensional-reduction effect, and projection error to show when projection helps. The optimization loss is the squared $\\ell^2$ distance between synthetic and noisy real signals in the projected space, and the stored tuples also carry the augmentation seed and random network initialization used to extract each signal.","core_discovery":"On the paper's own terms, the central discovery is that noise efficiency, not raw signal quantity, is what decides how much accuracy private dataset distillation can keep. For matching-based distillation under $(10,10^{-5})$-DP, Dosser reports a 10.0% accuracy improvement on CIFAR-10 with 50 images per class and an 8.3% increase with just one-fifth the distilled set size of previous state-of-the-art methods, while staying close to the non-private distribution-matching baseline. The gain comes from two changes: DOS decouples the sampling of DP-protected signals from the optimization of synthetic images so that many optimization steps can reuse a fixed set of privatized signals, and SER projects those signals into a subspace learned from auxiliary data, concentrating informative structure before Gaussian noise is added. The auxiliary data can come from a public generator at no privacy cost, or from a DP generative model trained under a split budget, with the total guarantee preserved by composition.","pith_inferences":["A likely reason SER helps in practice, even if its sensitivity-reduction argument is set aside, is that DP Gaussian noise is isotropic while signal variance is not: PCA concentrates high-variance signal into few coordinates, so the retained coordinates have a higher signal-to-noise ratio; the paper's own observation that keeping all 2048 dimensions still helps is consistent with variance concentra","The same 'sample private signals once, optimize forever' pattern could be lifted into other DP pipelines, such as private fine-tuning or private data generation, whenever a fixed set of privatized gradients or features can support many update steps.","A testable prediction from the error decomposition is that SER's benefit should grow as the privacy budget tightens and shrink as the budget loosens, matching the U-shaped curves the paper reports in its appendix."],"forward_implications":["Under the paper's results, a fixed DP budget buys more accurate distilled datasets: on CIFAR-10, 50 images per class gain 10.0% over the previous state of the art, and an 8.3% accuracy increase is obtained with one-fifth the distilled set size.","Because DOS lets optimization run far longer than sampling, extra compute can improve synthetic-image convergence without spending additional privacy budget.","Because SER's auxiliary dataset can come from a public generator, natural-image domains get the subspace benefit at zero additional privacy cost, while specialized domains can pay for auxiliary data out of the same budget through a DP generative model and still keep the overall guarantee by composition.","The framework accepts any matching signal, features or gradients, so the same decoupling and subspace projection can be applied to other matching-based private distillation methods."],"supporting_citations":[{"why":"Supplies the NDPDC feature-matching baseline and the noise-calculation protocol that Dosser builds on.","marker":"[39]"},{"why":"Defines the synchronized sampling-optimization matching approach whose coupling DOS is designed to remove, and is a key baseline.","marker":"[4]"},{"why":"Supplies the differential privacy definitions, the Gaussian mechanism, and the post-processing property used for the privacy guarantees.","marker":"[10]"},{"why":"Provides the public latent diffusion model used to generate the auxiliary dataset for SER on CIFAR-10.","marker":"[22]"},{"why":"Provides the differentially private diffusion model used to generate auxiliary data for SER on MNIST and FashionMNIST under a split budget.","marker":"[8]"},{"why":"Supplies the subsampled Rényi differential privacy accounting used to track cumulative privacy loss.","marker":"[32]"},{"why":"Provides the differentiable augmentation and signal-extraction conventions used when sampling private training signals.","marker":"[37]"}],"fun_headline_variants":["Noise-efficient DP distillation boosts accuracy by 10%","DP distillation gains 10% accuracy with less noise","Decoupled private distillation: 10% better, 5x smaller","Private data distillation: 10% gain via noise efficiency"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that projecting training signals into a lower-dimensional subspace reduces the worst-case influence any single person's data can have on the released signal, so less Gaussian noise is needed for the same privacy guarantee; if projection does not actually shrink that worst-case influence, the theoretical component of the noise-saving argument weakens.","fun_headline_variants_meta":{"raw":{"variants":["Noise-efficient DP distillation boosts accuracy by 10%","DP distillation gains 10% accuracy with less noise","Decoupled private distillation: 10% better, 5x smaller","Private data distillation: 10% gain via noise efficiency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000416,"raw_usage":{"total_tokens":2165,"prompt_tokens":979,"completion_tokens":1186,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":1117}},"tokens_in":595,"tokens_out":1186,"duration_ms":11718,"temperature":1.0,"reasoning_tokens":1117,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T05:24:43.417842+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the $\\ell^2$ sensitivity of the projected mean function $f_{\\text{proj}}(D)=\\tfrac{1}{n}\\sum_i P\\phi(x_i)$ under the add/remove-one adjacent-dataset model; if it equals the original sensitivity instead of the claimed smaller value, Lemma 1's noise-scale reduction fails. A simple control experiment would replace SER's PCA projection with a random orthonormal projection of the same dimension: if accuracy stays the same, the gain is dimensionality reduction rather than the learned informative directions.","supporting_citations":[{"cited_title":"Differentially private dataset condensation","cited_arxiv_id":null,"evidence_quote":"Supplies the NDPDC feature-matching baseline and the noise-calculation protocol that Dosser builds on."},{"cited_title":"Private set generation with discriminative information","cited_arxiv_id":null,"evidence_quote":"Defines the synchronized sampling-optimization matching approach whose coupling DOS is designed to remove, and is a key baseline."},{"cited_title":"The algorithmic founda- tions of differential privacy","cited_arxiv_id":null,"evidence_quote":"Supplies the differential privacy definitions, the Gaussian mechanism, and the post-processing property used for the privacy guarantees."},{"cited_title":"High-resolution image synthesis with latent diffusion models","cited_arxiv_id":null,"evidence_quote":"Provides the public latent diffusion model used to generate the auxiliary dataset for SER on CIFAR-10."},{"cited_title":"Subsampled r ´enyi differential privacy and analytical moments accountant","cited_arxiv_id":null,"evidence_quote":"Supplies the subsampled Rényi differential privacy accounting used to track cumulative privacy loss."},{"cited_title":"Dataset condensation with gradient matching","cited_arxiv_id":null,"evidence_quote":"Provides the differentiable augmentation and signal-extraction conventions used when sampling private training signals."}],"review_version":1}