REVIEW 3 major objections 5 minor 19 references
Diffusion models can respect PDE constraints through a soft denoiser adjustment instead of loss regularizers — and stay unbiased when the physics is imperfect.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 15:53 UTC pith:ZPK57SMF
load-bearing objection A practical architecture-level trick for softly constraining diffusion denoisers, with a real but unresolved distributional-bias question in the Darcy results. the 3 major comments →
Softly Constrained Denoisers for Diffusion Models Applied to Partial Differential Equations
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central discovery is that guidance-style constraint corrections can be folded directly into the denoiser parameterization, with a learnable scale controlling how much the constraint is followed. The proposed softly constrained denoiser is Dθ(xt, t) = Dorig_θ(xt, t) + γθ(xt, t) σ(t)² ∇_Dorig log l_c(Dorig), where l_c = exp(−‖R(·)‖) is a relaxed constraint function derived from a PDE residual. The training loss and sampling procedure remain identical to standard diffusion models, so in the infinite-data limit the optimal denoiser still equals E[x0|xt] and the score estimate stays valid. The authors prove that adding a residual regularizer to the loss, as in prior physics-informed diffusion
What carries the argument
The central object is Eq. (9), the softly constrained denoiser: Dθ(xt, t) = Dorig_θ(xt, t) + γθ(xt, t) σ(t)² ∇ log l_c(Dorig_θ(xt, t)), where l_c is a relaxed constraint function (e.g., exp(−|R(·)|)) and γθ is a small learnable MLP that outputs a nonnegative scaling. This parameterization is derived from a guidance-style approximation to the tilted score for the distribution l_c(x0)p(x0), replacing the expensive vector-Jacobian product of full guidance with a direct gradient of the constraint evaluated at the denoiser output. Because the standard denoising score-matching loss is used unchanged, the architecture carries the inductive bias without forfeiting the Tweedie‑based connection betwee
Load-bearing premise
The load-bearing premise is that the learned scaling γθ can zero out or correctly weight the constraint gradient so that the SCD parameterization remains capable of representing the true conditional expectation E[x0|xt] when the constraint is misspecified — a property that depends on finite network capacity and optimization succeeding.
What would settle it
Train an SCD model on a dataset while deliberately using a constraint residual that is pure noise (e.g., a random scalar field unrelated to the data), then measure the trained γθ outputs and compare the sampled distribution to a vanilla model; if γθ stays large and the distribution visibly shifts, the flexibility guarantee fails in practice.
If this is right
- Constraint knowledge can be incorporated into a diffusion model without changing the loss or sampler, preserving the standard distribution-modeling guarantee in the population limit.
- Regularizer-based physics-informed diffusion models are provably biased: their optimal denoiser is shifted from E[x0|xt] and their ELBO is strictly worse than vanilla, explaining their sensitivity to misspecified constraints.
- SCD delivers lower PDE residuals than vanilla diffusion and guidance baselines while maintaining NLL close to vanilla on Helmholtz and within a few bits per dim on Darcy flow.
- The method is agnostic to the base denoiser architecture and adds modest training/sampling overhead, making it a practical drop-in for diffusion models applied to PDEs and other differentiable constraints.
- Because any differentiable constraint can define l_c, the same soft inductive bias could be applied beyond PDEs, e.g., to hard physical bounds or learned quality scores.
Where Pith is reading between the lines
- If the learned γθ behaves as intended, an SCD model trained with a completely uninformative random constraint should learn to set γθ near zero and recover the vanilla distribution; that would be a direct, testable confirmation of the flexibility claim.
- The unbiasedness guarantee is asymptotic; the Darcy Flow results (NLL −5.8 vs. vanilla −10.5 bits/dim) suggest that finite capacity and optimization can leave a residual bias, so practitioners should monitor NLL alongside residuals.
- The derivation relies on a diagonal covariance approximation for p(x0|xt); using better covariance estimates or Monte Carlo integration, as the limitations section hints, could yield improved SCD variants with stronger constraint compliance.
- SCD's soft bias could be combined with a final hard-constraint projection for applications requiring exact satisfaction, while keeping the training-phase distribution unbiased.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes Softly Constrained Denoisers (SCD), an architecture modification for diffusion models that appends a learned, noise-level-dependent multiple of the constraint gradient to the denoiser output (Eq. 9). The training loss and sampling procedure are left unchanged, which the authors argue preserves the standard diffusion-model guarantee of converging to the data distribution while providing a soft inductive bias toward constraint satisfaction. The paper proves that regularizer-based training biases the optimal denoiser away from the conditional mean (Props. 3.1–3.2), and reports experiments on toy circle data, Darcy flow, and the Helmholtz equation, measuring residual compliance and NLL. The central claims are that SCD improves constraint satisfaction over vanilla diffusion and guidance baselines while maintaining low distributional bias and robustness to misspecified constraints.
Significance. If the claims are substantiated, SCD would be a practically attractive alternative to physics-informed regularization and inference-time guidance for PDE-constrained generation, with low overhead and an architecture-agnostic construction. The formal result that residual regularizers bias the optimal denoiser is clean and a useful contribution to the literature. The toy and Helmholtz experiments show clear benefits for SCD in terms of residual reduction without NLL degradation. However, the Darcy Flow results, which are the main PDE benchmark, show a substantial NLL regression for SCD, directly undercutting the core 'low distributional bias' claim. The paper also ships reproducible details and a transparent limitation statement, which is commendable.
major comments (3)
- [Table 2, §5.2] The Darcy Flow NLL results directly contradict the paper's central claim of low distributional bias. At the correctly specified constraint level (fmax=7.5), SCD achieves a validation NLL of −5.8 bits/dim versus −10.5 for vanilla diffusion, a degradation of 4.7 bits/dim. Under misspecification (fmax=12.5/15), SCD remains at −5.9/−5.8, whereas vanilla stays at −10.5. This is not a small gap, and it is the paper's own metric for distributional fidelity. The Limitations section (Sec. 6) concedes that 'the parameterization may push the model towards particular biases in practice.' This is evidence that the learned γθ does not reliably down-weight the constraint gradient when it conflicts with data, so the 'flexibility to deviate' that distinguishes SCD from regularizers/guidance is not demonstrated on this benchmark.
- [Eq. (9), §3] The claim that condition (ii) — convergence to the data distribution — is 'achieved by keeping the standard diffusion loss function and sampling loop' is only an argument about the global optimum in the infinite-data, infinite-capacity limit. In practice, the parameterization Dθ = Dorig + γθ σ² ∇ log l_c(Dorig) may have local minima or optimization paths that do not reach the vanilla solution even though γθ=0 is representable. The paper provides no analysis of the learned γθ values, no ablation setting γθ to zero, and no diagnostic of the optimization trajectory. Given the Darcy NLL gap, the assertion that SCD 'maintains enough flexibility to deviate' is unsupported without such evidence. Please report statistics of γθ or a counterfactual where the correction is disabled.
- [§3, Eqs. (6)–(9)] The derivation of the SCD parameterization rests on a sequence of approximations: Dirac-delta approximation to p(x0|xt), diagonal covariance Λ_t, and finally replacing Λ_t by a learned scalar γθ. The paper frames this as a 'principled way to embed the constraint residual,' but the final form is heuristic and its relationship to the exact guidance update (Eq. 4) is loose. This is acceptable if the empirical results are strong, but given the Darcy NLL regression, the heuristic nature should be acknowledged more prominently and the failure modes analyzed. For instance, the constraint gradient ∇ log l_c(Dorig) may have large magnitude at certain noise levels, and the scalar γθ may be unable to modulate it appropriately across the whole trajectory.
minor comments (5)
- [Abstract] Typo: 'thesesoftly constrained denoisers' should be 'these softly constrained denoisers'.
- [§5.1, Fig. 4] The figure reports DPS results, but Table 1 does not include DPS. Please clarify whether DPS is a guidance baseline and why it is omitted from the table.
- [Eq. (5)] The residual function R is used in the regularizer definition before being formally introduced in Sec. 3. Consider defining it earlier or adding a forward reference.
- [Table 2/3 captions] The captions say 'Left: Illustration of the Darcy Flow field' but the table layout combines an illustration with the results; please reformat or clarify the caption.
- [Sec. F.5] The runtime table reports only vanilla and SCD; it would be useful to include PIDM and guidance runtimes for context, since the paper claims minimal overhead over the baseline.
Circularity Check
No significant circularity: the SCD construction is an architecture inspired by guidance approximations, with the scaling γθ learned from the standard diffusion objective; key claims are empirical and self-citations are not load-bearing.
full rationale
The paper's central construction, Eq. (9), is presented as an architecture that embeds a constraint gradient into the denoiser output, motivated by a chain of acknowledged approximations to the guidance formula Eq. (4). This is a design choice, not a derivation whose conclusion is identical to its input: γθ is a learnable scaling trained with the standard denoising objective Eq. (3), and the paper does not claim to derive the empirical constraint-compliance improvements from the parameterization alone. The flexibility claim rests on the possibility γθ can down-weight the correction, which is an empirical/architectural question, not a circular definition. The proofs in Propositions 3.1 and 3.2 analyze the regularizer baseline mathematically and are not circular. The self-citations to Rissanen et al. (2025) appear in contextual lists of denoiser-covariance and Tweedie-formula approximations; they are not used as a uniqueness theorem or as the sole justification for a central claim, and the paper explicitly treats the covariance approximation as one of 'many alternative formulations.' The Limitations section's admission that 'the parameterization may push the model towards particular biases in practice' is an honest empirical caveat about possible distributional bias, not evidence that any prediction reduces by construction to a fitted quantity. The experimental evaluation uses residuals for constraint compliance and NLL for distributional fidelity; the residual metric coincides with the constraint used in training, but that is standard evaluation of constraint satisfaction, not a renamed prediction. Overall, the derivation chain is self-contained and the central claims are empirically testable, so no circular step is established.
Axiom & Free-Parameter Ledger
free parameters (4)
- γθ scaling network parameters =
learned (not reported)
- Constraint relaxation design l_c = exp(-||R||) and norm choice =
varies per experiment
- Truncated log-normal noise schedule parameters =
μ=-2, σ=1.7, lower limit a=-4
- Guidance scale for guided vanilla baseline =
0.03
axioms (4)
- domain assumption R(x) is continuously differentiable and l_c = exp(-||R||) supplies useful gradient information at the denoiser output
- ad hoc to paper Approximate p(x0|xt) as a Dirac at Dθ and covariance as diagonal (Λ_t), then replace Λ_t by learned scalar γθ
- domain assumption Standard denoising loss with the SCD parameterization preserves convergence to E[x0|xt] as data and optimization go to infinity
- domain assumption Finite-difference stencils approximate PDE residuals accurately for both training and evaluation
read the original abstract
Diffusion models have become a powerful generative prior for solutions of partial differential equations (PDEs). Existing approaches enforce physical constraints either by adding the PDE residuals as loss regularizers or through inference-time adjustments. These methods bias the model away from the true data distribution, which is especially problematic when the governing PDE is misspecified. To circumvent these issues while making the most out of the PDE constraint, we introduce soft inductive biases into the denoiser architecture derived from the PDEs. We show that these softly constrained denoisers exploit constraint knowledge to improve compliance over standard denoisers, while maintaining enough flexibility to deviate from it in case of misspecification with respect to observed data.
Figures
Reference graph
Works this paper leans on
-
[2]
Karnakov, P., Litvinov, S., and Koumoutsakos, P
doi: 10.1137/24M1636071. Karnakov, P., Litvinov, S., and Koumoutsakos, P. Solving inverse problems in physics by optimizing a discrete loss: Fast and accurate learning without neural networks.PNAS Nexus, 3(1):pgae005, January
-
[4]
Following these results, we use the truncated log-normal distribution for all our experiments
We use a mean of -1.5 and standard deviation of 1.2 for the log-normal loss and a mean of -2, standard deviation of 1.7 and truncation lower limit of -4 for the truncated log-normal loss. Following these results, we use the truncated log-normal distribution for all our experiments. 17 Softly Constrained Denoisers for Diffusion Models Table 4.Residuals obt...
2022
-
[5]
Helmholtz Equation For the Helmholtz Equation we used the UNet implementation by Karras et al
Table 5.Architecture hyperparameters for the Darcy Flow experiments Hyperparameter Value Model channels 24 Number of residual blocks 8 Per-resolution multipliers [1, 2, 3, 4] Attention resolutions [16, 8] F.4. Helmholtz Equation For the Helmholtz Equation we used the UNet implementation by Karras et al. (2023). We use the Heun sampler implementation by Ka...
2023
-
[7]
Table 7.Runtimes of vanilla and our method on training and sampling on an NVIDIA H200 GPU
We note that our method sees the most impact at sampling time, since the overhead duplicates per each sampling iteration because the Heun sampler makes two neural function evaluations per iteration. Table 7.Runtimes of vanilla and our method on training and sampling on an NVIDIA H200 GPU. Sampling is done for eight samples at a time using the Heun sampler...
2022
-
[9]
Bastek et al
and an inference-time adjustment where the final samples are optimized to have a small PDE residual. Bastek et al. (2025) present Physics-Informed Diffusion Models (PIDM), a framework to train DDPM-based diffusion models with a PDE residual as a regularizer term to minimize along the loss function. Several works utilize DPS-like guidance (Chung et al., 20...
2025
-
[10]
Injecting Measurement Structure for Training Inverse Problem SolversMathematically, the closest work is the likelihood-informed Doob’s h-transform by Denker et al
or GNNs (Valencia et al., 2025), as it remains compatible with any base architecture. Injecting Measurement Structure for Training Inverse Problem SolversMathematically, the closest work is the likelihood-informed Doob’s h-transform by Denker et al. (2024), who finetune adapters using observation gradients ∇x0 p(y|x 0), similar to our constraint-informed ...
2025
-
[11]
dual path
for Bayesian inference from noisy observations, whereas wedefine lc(x0) to restrict generation to a constrained subset. Their goal is an alternative to inference-time adjustment, while we seek to 13 Softly Constrained Denoisers for Diffusion Models alleviate distributional biases and constraint misspecification. Furthermore, we train from scratch, whereas...
2022
-
[12]
We take p(xT ) to be approximately N(0, σ2 maxI), i.e
= logp(x T ) + Z T 0 ∇xt ·g(x t, t)dt,(20) where g(t) =∇ xt logp(x t) is the drift of the probability flow ODE, as well as the score function, which is approximated as ∇xt logp(x t)≈ Dθ (xt,t)−xt σ2 t for score matching diffusion models (Song et al., 2021b; Karras et al., 2022), and T=σ 2 max the maximum noise level of the diffusion process. We take p(xT ...
2022
-
[13]
fine-grained
and f is the net flow of fluid through a given point.Netflow means that if there is the same amount fluid entering and exiting at a given point, then the net flow is zero. As a more concrete example, we can use Darcy flow to describe how water will flow through a body of sand. We can expect more water to flow at the areas where we apply more pressure to s...
2017
-
[14]
We then take the approximation: p(x0|xt)≈ N(x0|µ,Σ),(34) with mean parameter µ and covariance parameter Σ, as this allows us to use Tweedie’s formula, connecting the score function with the exact moments ofp(x 0|xt) (Efron, 2011; Rissanen et al., 2025): µ=E[x 0|xt] =x t +σ 2 t ∇xt logp(x t)(35) Σ =Cov[x 0|xt] =σ 2 t σ2 t ∇2 xt logp(x t)| {z } Hessian ...
2011
-
[100]
intermediate
The input for either task is the denoiser output Dθ(xt, t)and the diffusion time t. Between each layer there is an ELU activation function to ensure that the scaling factor remains positive. F.1. Modified Loss Function The loss function by Karras et al. (2022) has the form seen in Eq. (3) with the choice of distributions t∼ LogNormal(µtrain, σ2 train) and...
2022
-
[128]
For the “Dent” variant of misspecification, we use the following parametric curve: C(θ) =(r(θ) cos(θ), r(θ) sin(θ))(39) r(θ) =1−0.25·β wrap(θ− π 2 ) 1.2 ,5 · 1 + 0.6 1−2 wrap(θ− π 2 ) 1.2 2!! (40) wrap(θ) = ((θ+π) mod 2π)−π(41) β(u,5) = ( (1−u 2)5 :|u|<1 0otherwise, (42) where C defines the coordinates of every point in the curve in polar coordinatese. F....
2023
-
[2015]
Song, J., Vahdat, A., Mardani, M., and Kautz, J
PMLR. Song, J., Vahdat, A., Mardani, M., and Kautz, J. Pseudoinverse-Guided Diffusion Models for Inverse Prob- lems. InInternational Conference on Learning Represen- tations (ICLR), 2023a. Song, J., Zhang, Q., Yin, H., Mardani, M., Liu, M.-Y ., Kautz, J., Chen, Y ., and Vahdat, A. Loss-guided diffu- sion models for plug-and-play controllable generation. I...
2024
-
[2020]
C., Azizzadenesheli, K., and Anandkumar, A
Yao, J., Mammadov, A., Berner, J., Kerrigan, G., Ye, J. C., Azizzadenesheli, K., and Anandkumar, A. Guided dif- fusion sampling on function spaces with applications to pdes.arXiv preprint arXiv:2505.17004,
-
[2021]
Kong, X., Brekelmans, R., and Steeg, G. V . Information- theoretic diffusion. InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5,
2023
-
[2022]
Analyzing and Improving the Training Dy- namics of Diffusion Models.2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp
Karras, T., Aittala, M., Lehtinen, J., Hellsten, J., Aila, T., and Laine, S. Analyzing and Improving the Training Dy- namics of Diffusion Models.2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 24174–24184,
2024
-
[2023]
Oommen, V ., Bora, A., Zhang, Z., and Karniadakis, G. E. Integrating neural operators with diffusion models im- proves spectral representation in turbulence modeling. arXiv preprint arXiv:2409.08477,
-
[2024]
Proofs Proposition 3.1.Let D∗ reg(xt, t)be the denoiser that minimizes the regularized objective Lreg
12 Softly Constrained Denoisers for Diffusion Models Appendices A. Proofs Proposition 3.1.Let D∗ reg(xt, t)be the denoiser that minimizes the regularized objective Lreg. The optimal denoiser output is shifted as follows D∗ reg(xt, t) =E[x0|xt]−λ[∇ yR(y)]⊤ R(y) y=D∗reg ,(11) meaning that the optimal denoiser output D∗ reg(xt, t)is shifted from the conditio...
2021
-
[2025]
Bastek, J.-H., Sun, W., and Kochmann, D
preprint: arXiv:2506.08604. Bastek, J.-H., Sun, W., and Kochmann, D. Physics-informed diffusion models. InInternational Conference on Learn- ing Representations (ICLR),
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.