{"id":"0c72dbe7-2245-4f69-af3f-b3ba33f58334","arxiv_id":"2510.05416","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Using Hessian eigenvalues from public data to design correlated noise for DP-SGD improves accuracy by 1–4% over current DP-MF methods.","lead":"This paper proposes NoiseCurve, a way to choose the correlated noise in privacy-preserving AI training using the model's curvature, estimated from public unlabeled images. In tests on vision datasets, it improves accuracy by 1–4% over existing DP-SGD and DP-BandMF methods at the same privacy budget.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unaccounted private-validation hyperparameter tuning invalidates the equal-privacy-budget claim; reported ε is not the privacy loss of tuning+training, so the central accuracy comparison is not at the stated privacy level.","rationale":"The reader's weakest assumption (Hessian-spectrum drift) is plausible but partly mitigated by the paper's own Table 7, which shows NoiseCurve still improves without pretraining even when the spectrum is not stabilized. The more direct, load-bearing concern is the unaccounted private-validation tuning: the paper's own protocol uses 10% of the private training data to select hyperparameters and charges no privacy cost. This undermines the central 'equal privacy budget' claim more fundamentally than spectrum drift, because it affects every reported number as a DP claim. The Theorem 1 proof algebra checks out, and the empirical gains are consistent across tables, so I do not recommend rejection; a conditional verdict is appropriate until the privacy-accounting test is run. The reader's rationale does list the validation-tuning issue, but their named weakest assumption is spectrum drift, hence partial agreement.","tokens_in":18682,"tokens_out":13355,"duration_ms":106846,"concrete_test":"Run the main CIFAR-10 (and, if feasible, ChestX-ray14) comparisons under a privacy-accounted protocol: use a standard private selection mechanism such as Report-Noisy-Max on the validation split, add its Rényi-DP privacy cost to the training budget, and compare NoiseCurve vs DP-BandMF at the same total ε. If NoiseCurve's accuracy advantage remains ≥1 point across ε, the concern is resolved; if it shrinks or reverses, the equal-privacy claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5 and Appendix D.1 state: 'we split 10% of data samples from training sets as validation sets and tune hyperparameters on validation sets.' The tuning grid includes learning rate, clip norm, band size, and in effect the p+/μp+ tail parameters. This validation set is drawn from the private training data, but the paper charges no privacy cost for the selection. The reported (ε,δ) guarantee therefore applies only to the training run after hyperparameters are fixed; the full tuning+training procedure is not (ε,δ)-DP with the reported ε. Since the central claim is 'improvements at equal privacy budget,' the comparison is not at the stated privacy level. The limitations section discusses public-data dependence and eigenvalue approximation but does not mention this privacy-accounting gap. The fact that DP-SGD and DP-BandMF baselines are tuned the same way does not restore the guarantee; it only makes the comparison equally optimistic in absolute privacy terms. If the validation-selection privacy cost is properly accounted, NoiseCurve's gains could shrink or vanish.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NoiseCurve, a method for improving DP-SGD by correlating the privacy noise across iterations using Hessian curvature information. In the idealized quadratic-loss case, Theorem 1 derives the expected loss gap between correlated-noise gradient descent and noise-free gradient descent in terms of the Hessian eigenvalues and the matrix X = C^T C, yielding Problem P1: minimize Tr((V^T M V)X^{-1}) subject to diag(X)=1, X positive definite, and b-banded. The authors then propose practical solutions for using public unlabeled data to estimate eigenvalues, coping with non-constant Hessians, zeroing negative eigenvalues, and approximating spectra via curve fitting. Experiments on CIFAR-10 and ChestX-ray14 with various models and privacy budgets report consistent accuracy improvements over DP-SGD and DP-BandMF. The paper explicitly acknowledges limitations regarding public-data dependence and eigenvalue approximation.","tokens_in":18962,"tokens_out":5381,"duration_ms":47474,"significance":"If the empirical comparisons are at the claimed privacy level, this is a useful contribution: the quadratic derivation is clean and the proposed objective is a natural, principled replacement for the DP-MF heuristic. The paper also includes a self-contained proof of Theorem 1, a drop-in integration with the DP-BandMF framework, and a range of experiments. However, the central equal-privacy-budget claim is currently undermined by validation-set hyperparameter tuning on private data that is not charged to the privacy budget. The representative-spectrum assumption is also load-bearing and is only empirically supported in a limited vision setting, as the limitations section itself concedes. With corrections to the privacy accounting and a sensitivity analysis of the spectrum assumption, the contribution would be significant.","major_comments":[{"comment":"The privacy accounting for hyperparameter selection is missing. The paper states 'we split 10% of data samples from training sets as validation sets and tune hyperparameters on validation sets' and Appendix D.1 says Optuna searches learning rate, clip norm, and band size (and, in effect, p+ and mu_p+). These validation data are drawn from the private training set. The reported (epsilon,delta) guarantee is computed for the final training run only; the selection of hyperparameters is a data-dependent computation on private data with no privacy charge. Therefore the claim that NoiseCurve improves over DP-SGD and DP-BandMF at the same privacy budget is not currently supported. The fact that all baselines are tuned the same way does not restore the absolute guarantee. Please either (a) include the selection procedure in the privacy analysis and report the total epsilon, (b) tune on public dat","section":"§5, §D.1"},{"comment":"The load-bearing premise that a single Hessian eigenspectrum computed once from public data after SimCLR pretraining is representative along the private training trajectory is supported only by Figure 3 for one small CNN, and the limitations section explicitly states there is no formal analysis and it is unclear whether it holds outside vision. Since Problem P1 fixes M, an unrepresentative spectrum will miscalibrate C. I ask for a sensitivity analysis: vary the spectrum (e.g., using spectra taken at different training epochs, or perturbing the eigenvalues) and report the resulting accuracy. This is a correctness-risk concern, not a circularity objection, but it is central to the claim that NoiseCurve is a generally applicable drop-in replacement.","section":"§4.2, §6"},{"comment":"The curve-fitting validation in Table 3 is not convincing as presented. The 'Fit' row values (122, 5.59, 479, 20, 140, 285 for k=100,...,3000) are non-monotone in k, and no error bars, standard deviations, or number of runs are given. Some entries make it appear that more eigenvalues can hurt the fitted solution. Please clarify what these numbers represent (one run? mean over runs?), report variance, and explain the non-monotonicity. This matters because S4 is the scalability contribution for large models.","section":"§4.4, Table 3"},{"comment":"Replacing negative eigenvalues with zero is justified empirically by the observation that they are small after pretraining, but Problem P1 is derived for a positive semidefinite Hessian. The paper should state explicitly that clamping negative eigenvalues to zero is a heuristic modification of the quadratic-loss objective, and provide evidence on how sensitive P1's solution is to the exact truncation. An ablation with different thresholds or with small positive floors would clarify whether the convexification step introduces a significant gap between Theorem 1 and the algorithm used in practice.","section":"§4.3"}],"minor_comments":[{"comment":"The row entries in Table 3 lack column separators, and the introductory sentence 'reduction-in-objective of different with respect to k' is garbled. Please reformat and rephrase.","section":"Table 3 / Appendix C"},{"comment":"State the domain of alpha (presumably negative) and clarify that C and alpha are fitted constants. Note that at i = p+, the formula is independent of C and alpha, so the endpoint mu_p+ is a free parameter.","section":"Eq. (2)"},{"comment":"The caption 'epoch 0 and epoch 1' is ambiguous; clarify whether these are checkpoints at initialization and after one epoch of training, and whether pretraining is included.","section":"Figure 3"},{"comment":"Table 5, which tunes on the test set, is explicitly acknowledged as bad practice. Please add a sentence noting that these results are not comparable to the main tables and should not be used to support the central equal-privacy claim.","section":"Appendix D.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a sound theoretical core in the quadratic case and a plausible empirical story, but the equal-privacy-budget claim is invalidated as written by unaccounted validation-set tuning. This is fixable in a revision by either changing the experimental protocol or adding the selection privacy cost. The spectrum-stability assumption should also be stress-tested. I would not reject, but the authors need to address the privacy accounting head-on."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the NoiseCurve paper. The new thing is real: instead of minimizing the noise norm like DP-BandMF, they derive an objective that includes the Hessian, so the mixing matrix is chosen to account for curvature. The quadratic result in Theorem 1 is clean, and P1 is a sensible convex relaxation that slots into the existing DP-BandMF machinery. Credit where due: the public-data eigenvalue estimation, the pretraining stability argument, and the tail approximation are all attempts to deal with real deployment issues, and the experiments cover convex, small nonconvex, and large-model regimes with consistent 1-4% gains.\n\nThe soft spots are the ones the paper half admits. Most important: they tune hyperparameters on a 10% split of the private training set and charge no privacy cost for that selection. The reported epsilon therefore applies to the single training run after hyperparameters are fixed, not to the full tuning+training procedure. That makes the 'equal privacy budget' comparison optimistic in a way that could shrink or erase the gains. Tuning the baselines the same way makes the comparison fair between methods, but it does not restore the advertised DP guarantee for the overall workflow. This needs to be fixed or explicitly discussed.\n\nThe tail-fitting approximation is also shakier than the main theorem. Table 3 shows the fitted objective error is non-monotonic in k, which suggests the curve-fit and the choice of p+, mu_p+ are fragile. The paper acknowledges this in the limitations. The representative-spectrum assumption is likewise empirical and may not transfer outside vision. These are minor-to-moderate; they don't kill the idea.\n\nNo code or artifacts released, which limits reproducibility. I'd like to see the implementation before trusting the exact accuracy numbers.\n\nWho is this for? People working on DP-SGD, DP-MF, and privacy-preserving deep learning. It deserves a serious referee—the central idea is novel, the theory is honest for the quadratic case, and the empirical story is plausible. But the privacy accounting for hyperparameter tuning is a load-bearing issue, and the tail approximation needs more scrutiny. Send it to review with those concerns flagged.","headline":"Genuinely new curvature-aware DP-MF objective with a clean quadratic theorem, but the reported privacy budget ignores the cost of tuning on a private validation split, so the equal-privacy comparison is optimistic.","tokens_in":19446,"tokens_out":2024,"would_cite":true,"duration_ms":15514,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Curvature-aware noise correlations improve DP-SGD accuracy by 1–4% at the same privacy budget.","keywords":["differential privacy","DP-SGD","DP-MF","noise correlation","Hessian eigenvalues","model curvature","matrix factorization","privacy-preserving training"],"falsifier":"Measure the actual Hessian spectra along the private training trajectory on a language or tabular task and compare the accuracy of correlations optimized from the public-data spectrum against the previous data-independent objective; if spectra drift markedly or the accuracy gain disappears, the central claim fails.","tokens_in":18567,"feed_emoji":"📈","tokens_out":6897,"duration_ms":53753,"temperature":0.7,"pith_summary":"DP-SGD lags behind ordinary SGD because the noise injected into each gradient update distorts not only the current step but also every later step, since later gradients are evaluated at shifted parameters. The paper shows that, for a quadratic loss, this compounding effect is captured exactly by a trace term involving the Hessian eigenvalues and the noise correlation matrix. It replaces the existing data-independent DP-MF objective with this curvature-aware objective, called NoiseCurve, and solves the same banded matrix factorization problem. Across CIFAR-10 and ChestX-ray14, with several model families and privacy budgets, the resulting noise correlations yield 1–4% higher accuracy than DP-SGD and the state-of-the-art DP-BandMF at the same privacy guarantee. The practical recipe is to estimate the Hessian spectrum once from public unlabeled data after self-supervised pretraining, truncate negative eigenvalues, and fit a power-law tail for large models.","feed_headline":"Curvature-aware noise lifts DP-SGD accuracy by 1-4%","feed_subtitle":"A drop-in objective for DP-MF accounts for how privacy noise bends later gradients, gaining accuracy at no extra privacy cost.","key_machinery":"The load-bearing object is the identity of Theorem 1: E[L(w̃_T)] − E[L(w_T)] = (η²/2) Tr(X⁻¹(Vᵀ M V)). Here X = CᵀC is the Gram matrix of the noise-mixing matrix, M = Diag(µ₀,…,µ_{p−1}) holds the Hessian eigenvalues, and V[i,j] = (1−ηµᵢ)^{T−j−1} propagates earlier noise through the iterates. This identity reduces the design of noise correlations to a convex trace-minimization problem with banded, diagonal-one, positive-definite constraints, and it is why only Hessian eigenvalues, not the full Hessian, are needed. The supporting machinery makes this practical: pretraining on public data yields a stable representative spectrum, zeroing negative eigenvalues restores convexity, and a power-law c","core_discovery":"The paper's central claim is that cross-iteration noise correlation for DP-SGD should minimize a curvature-aware trace objective rather than the data-independent energy objective used by prior DP-MF methods. For a quadratic loss the paper proves E[L(w̃_T)] − E[L(w_T)] = (η²/2) Tr(X⁻¹(Vᵀ M V)), where X is the Gram matrix of the noise-mixing matrix, M is the diagonal matrix of Hessian eigenvalues, and V encodes the powers (1−ηµᵢ)^{T−j−1}. Minimizing this trace over diagonal-one, positive-definite, banded X yields the mixing matrix C. The paper then argues a single representative Hessian—estimated from public unlabeled data after self-supervised pretraining, with negative eigenvalues zeroed and","pith_inferences":["Inference: If the Hessian spectrum is indeed governed mostly by architecture rather than data, the same optimized correlation matrix could be reused across tasks for a fixed architecture, turning the one-time public-data computation into a reusable artifact.","Inference: The quadratic derivation suggests the curvature-weighted trace objective could also improve other correlated-noise mechanisms, not only the banded form tested here.","Inference: The paper's stability evidence is short-horizon; a natural extension is to re-estimate the spectrum periodically during training and check whether adaptive correlation matrices beat the fixed one.","Inference: The power-law tail fit, treated by the authors as an orthogonal contribution, could serve other applications that need Hessian spectral information; its main risk is sensitivity to the chosen endpoint (p₊, µ_p₊)."],"forward_implications":["Using NoiseCurve as the objective inside the DP-MF framework yields 1–4% higher test accuracy than DP-SGD and the previous banded-MF baseline across every privacy budget tested, with no additional privacy loss.","Curvature information from public unlabeled data—even from a different domain, such as a general vision dataset for chest X-rays—remains informative enough to improve noise correlations.","The method works across convex last-layer finetuning, small non-convex CNNs, full VGG training, and LoRA finetuning of a vision transformer, indicating it is not tied to one architecture.","Because the change is only in the objective and the feasible set is unchanged, NoiseCurve can be dropped into existing DP-MF implementations and privacy accounting.","The eigenvalue curve-fitting approximation extends the method to models where full Hessian eigen-decomposition is infeasible, at a small accuracy cost compared to using the true spectrum."],"fun_headline_variants":["Curvature-aware noise correlation boosts DP-SGD accuracy","NoiseCurve uses Hessian to lift DP-SGD accuracy","Model curvature sharpens DP noise correlation for accuracy","Curvature-based noise mixing gives DP-SGD accuracy gain","DP-SGD noise correlation improved via Hessian curvature"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The Hessian eigenvalue spectrum estimated once from public data after pretraining remains representative of the spectra seen during private training; the paper gives empirical support but no formal analysis, and its own limitations section says it is unclear whether this holds beyond vision.","fun_headline_variants_meta":{"raw":{"variants":["Curvature-aware noise correlation boosts DP-SGD accuracy","NoiseCurve uses Hessian to lift DP-SGD accuracy","Model curvature sharpens DP noise correlation for accuracy","Curvature-based noise mixing gives DP-SGD accuracy gain","DP-SGD noise correlation improved via Hessian curvature"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00027,"raw_usage":{"total_tokens":1452,"prompt_tokens":722,"completion_tokens":730,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":466,"completion_tokens_details":{"reasoning_tokens":663}},"tokens_in":466,"tokens_out":730,"duration_ms":6156,"temperature":1.0,"reasoning_tokens":663,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T11:19:46.716359+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the actual Hessian spectra along the private training trajectory on a language or tabular task and compare the accuracy of correlations optimized from the public-data spectrum against the previous data-independent objective; if spectra drift markedly or the accuracy gain disappears, the central claim fails.","supporting_citations":[],"review_version":1}