{"id":"d379bba5-a26e-466f-9bf6-55ea497446fa","arxiv_id":"2502.05416","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Conditioning the Gaussian latent or output distribution of a deep generative model on a linear equality constraint, with the conditional mean as the gradient proxy, enforces the constraint exactly and improves generation quality over post-hoc projection baselines.","lead":"This paper proposes a way to make deep generative models respect hard linear equality constraints, such as charge neutrality or mass balance, by conditioning the model's Gaussian latent distribution on the constraint before sampling. The authors compare several gradient estimators and report that using the conditional expectation as a differentiable proxy yields better generation quality than existing projection-based methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The marginal-expectation gradient estimator in Eq. 5 omits all covariance derivatives of the constrained Gaussian, so for nonlinear decoders it is a biased proxy for ∇θL; the synthetic validation in §4.2 uses identity decoders and L1/L2 losses, so the 'probabilistically sound' claim is unsupported.","rationale":"I chose the Eq. 5 estimator rather than the closed-form L2 notation issue or the inference-only nature of the diffusion experiments because Eq. 5 is the stated mechanism for 'end-to-end differentiable learning' and is used in the VAE and regression experiments; the closed-form results are confined to special identity-decoder settings. The concern is not that the paper is empirically wrong, but that the central theoretical claim ('probabilistically sound') depends on a proxy whose bias is unquantified and demonstrably nonzero in general. The reader's weakest_assumption identified the same step; I agree and add the concrete covariance-derivative mechanism. The proposed test is feasible with the paper's own architecture and would distinguish a healthy heuristic from an unsupported central claim. Secondary issues (notation in Prop. 5.1, missing code, diffusion baseline comparisons) are real but less load-bearing.","tokens_in":23085,"tokens_out":15378,"duration_ms":159345,"concrete_test":"Run the MNIST brightness-constrained VAE of §6.2 with two training estimators: (a) the proposed Marginal Expectation; (b) an unbiased pathwise estimator obtained by reparameterizing z=μc+Lcε and backpropagating through μc and Lc (Cholesky of Σc) for the reconstruction term, with the same ELBO and hyperparameters. Also compare both against a high-sample Monte Carlo estimate of the true expected reconstruction loss. If (b) achieves the same or better NELBO/RL while (a) fails to reduce the MC-estimated loss, the Eq. 5 proxy is biased in a way that matters; if their gradients agree closely on the nonlinear decoder, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation (5) replaces ∇θE_{z∼pθ(z|Az=k)}[ℓ(fu(z),y)] with ∂θm(θ)∇zℓ, where m(θ)=E[z|Az=k]=μc. Exact constrained sampling is z=μc+Lcε with LcLc^T=Σc, so the true gradient contains ∂θμc^T E[∇zℓ] plus terms involving ∂θLc (equivalently ∂θΣc). The proposed proxy keeps only the μc path, evaluated at a single sample rather than at the conditional mean; all covariance gradients are discarded. For identity decoder with L2 loss the omitted term is ∂θtr(Σc), which is generally nonzero; for L1 loss even the mean-path term is biased because E[sign(z-y)]≠sign(μc-y). The §4.2 synthetic comparison therefore cannot validate the estimator for nonlinear decoders (sigmoid VAE, U-Net features, MPNN) or for NLL/ELBO losses, where the covariance term is first-order. No theorem in the paper bounds this bias, so the statement that the constrained distribution is 'differentiably learned' in a probabilistically sound way is unsupported. The reported gains in Tables 2, 5, 6 may reflect properties of this particular proxy rather than of the constrained expected loss.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes to enforce hard linear equality constraints in deep generative models by constraining the Gaussian latent or output distribution itself, i.e., sampling from pθ(z | Az = k) rather than drawing from an unconstrained distribution and projecting post hoc. It derives exact sampling from the Gaussian conditional and closed-form conditional marginals, proposes several gradient estimators with \"marginal expectation\" as the preferred proxy, and derives closed-form expected L1/L2 losses for the identity-decoder case. The method is evaluated across VAEs, diffusion models, an MPNN for MOF charge prediction, chemical process surrogates, and stock allocation, with the central claim that the proposed approach guarantees constraint satisfaction and achieves superior generative performance on every benchmark.","tokens_in":23372,"tokens_out":8582,"duration_ms":79734,"significance":"If the central claims held, this would be a practically valuable and architecture-agnostic alternative to post-hoc projection baselines: exact constrained Gaussian sampling and closed-form conditionals are clean and useful ingredients, and the breadth of experiments is a real asset. The paper also provides a Poisson extension in Appendix G, which broadens its scope. However, the theoretical support for the main gradient estimator is incomplete, and the closed-form loss formulas contain algebra/notation problems that affect the synthetic ground truth and several experiments. The significance is therefore contingent on fixing these load-bearing issues.","major_comments":[{"comment":"The proxy ∇θL ≈ ∂θm(θ)∇zℓ is stated without derivation or an error bound, and for the exact sampling procedure it discards all dependence on the covariance of the constrained Gaussian. Writing the exact sample as z = μc + Lcε, the true gradient contains ∂θLc-type terms; for a nonlinear decoder these terms are first-order in θ and are not negligible. The synthetic validation in §4.2 covers only identity decoders with L1/L2 losses, so it cannot justify the use of this estimator with the sigmoid VAE, U-Net, or MPNN decoders used in §6, nor with NLL/ELBO objectives. Please provide either a bias analysis/proof for Eq. (5) or an empirical validation on at least one nonlinear decoder using the actual objectives of the paper.","section":"§4.1, Eq. (5)"},{"comment":"Under the paper's definition of Σ as a covariance matrix, the closed-form L1 and L2 expected losses are not correct as printed. For L2, E[(zi − yi)^2] = μ_i^2 + Σ_{i,i} − 2y_i μ_i + y_i^2, not μ_i^2 + Σ_{i,i}^2 − 2y_i μ_i + y_i^2; for L1, the coefficient of the exponential term must be the standard deviation, i.e., σ√(2/π) with σ = √Σ_{i,i}, not the variance. The proof in H.2 displays a term of the form μ_i + σ_i^4 − 2y_i μ_i + y_i^2, which is not the second moment of a Gaussian. Because these closed forms are used both as the ground truth for the estimator comparison in Figure 3 and as training losses in §6.4 and §6.5, this is load-bearing; the formulas must be corrected or the notation must be redefined consistently and propagated through the proofs and experiments.","section":"§5, Prop. 5.1; Appendix H.2"},{"comment":"The closed-form loss produces NLL > 1e7 in the MOF experiment, and Appendix I shows that this is caused by variance collapse. This is not a harmless numerical artifact: it means the closed-form trainer is effectively optimizing a degenerate distribution whose likelihood is zero, while the explanation notes that MAD is unaffected. The paper should qualify the claim of probabilistic soundness, restrict it to the likelihood objective, or add a remedy such as a variance floor and re-evaluate the affected experiments.","section":"Table 5, Appendix I"}],"minor_comments":[{"comment":"\"archives\" should be \"achieves\", and \"Reults\" should be \"Results\".","section":"Abstract and §6.2"},{"comment":"On CIFAR and LSUN Church, the IS of Ours is lower than that of DDPM (9.223 vs 9.278 and 2.471 vs 2.460, respectively), so the abstract's claim of superiority \"across every benchmark\" should be qualified to the primary metric or to FID.","section":"Table 3"},{"comment":"The expression \"Pn j=1 zn\" should read \"Pn j=1 zj\".","section":"Appendix G, Prop. G.2"},{"comment":"The description of the ground-truth gradients is confusing: \"The ground truth are sampled from N(0,I) satisfying the constraint\" should be clarified to explain how h_gt is computed for Figure 3.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The paper builds directly on the SIMPLE estimator of Ahmed et al. (2023), which it cites; the novelty of the Gaussian adaptation should be assessed carefully. The algebra issues in the closed-form losses are likely fixable, and the experimental breadth is substantial, but the current theoretical gaps and the acknowledged NLL explosion prevent acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest take: the paper has a nice empirical core and a weak theoretical shell. What is new is extending the SIMPLE marginal-expectation estimator to continuous Gaussian latents with general linear equality constraints, plus a broad benchmark suite. The Gaussian conditioning math (Prop 4.1, H.1) is correct and standard; the closed-form L1/L2 expected losses in Prop 5.1 are correctly derived and genuinely useful for identity-decoder cases. The Poisson extension is a thoughtful bonus. Credit where due: the authors run VAEs, diffusion models, and GNNs across five datasets and three scientific applications, and they explicitly acknowledge the NLL explosion of the closed-form L1 loss in Appendix I. That honesty is real. The soft spots are also real. Equation (5) is a proxy gradient that discards all covariance derivatives of the constrained Gaussian. The stress-test note is on target: for nonlinear decoders (sigmoid VAE, U-Net, MPNN) and for NLL/ELBO losses, the omitted covariance derivatives are first-order, so the estimator is biased. The §4.2 synthetic validation uses an identity decoder and pointwise L1/L2 losses, so it cannot support the 'probabilistically sound' claim made in the abstract. The closed-form L1 loss's variance collapse is not just a numerical curiosity: reporting MAD while NLL > 1e7 in Table 5 paints a partial picture. The diffusion comparison lacks non-trivial baselines (e.g., constrained layer inside diffusion), and the portfolio experiment allows shorting while the softmax baseline does not, tilting the comparison. No code or data is provided, so the empirical claims are not independently checkable. None of this kills the paper. The central idea—transform the distribution, not the samples—is sound and the experiments suggest it works in practice. But the current abstract and Section 6 overstate the theoretical status. A serious referee should ask for either a bias bound on Eq. (5) (even for a restricted class) or a reframed contribution as a well-tested heuristic with strong empirical support. I would also want the missing diffusion baselines, a fairer portfolio comparison, and a code release. Verdict: send to peer review. The paper deserves referee time; the revision required is substantial but tractable. I'd bring it to reading group and would cite the closed-form loss results, though I'd hedge the estimator claim.","headline":"A broad empirical study of a simple idea—conditioning the Gaussian on the constraint—with a theoretically under-supported gradient estimator that needs reframing or a bias bound.","tokens_in":23875,"tokens_out":3443,"would_cite":true,"duration_ms":34166,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Conditioning a generative model's latent distribution on a linear equality constraint satisfies it exactly and keeps or improves data fidelity.","keywords":["constrained generative modeling","hard linear equality constraints","Gaussian conditional distributions","gradient estimators","neuro-symbolic learning","variational autoencoders","diffusion models","scientific machine learning"],"falsifier":"Run the brightness-constrained MNIST VAE with the paper's marginal-expectation proxy replaced by a score-function (REINFORCE) estimator of the exact constrained gradient, holding everything else fixed; if the two training runs converge to materially different latents or the score-function run achieves higher held-out likelihood, the proxy is biased and the reported gains are estimator-specific rather than inherent to distribution-level constraint enforcement.","tokens_in":22849,"feed_emoji":"🎯","tokens_out":18691,"duration_ms":156734,"temperature":0.7,"pith_summary":"The paper claims that deep generative models can be made to respect hard linear equality constraints—charge neutrality in molecules, mass balance in chemical plants, budget allocation in portfolios—by conditioning the model's distribution on the constraint rather than by sampling freely and then patching the samples. It documents that ordinary generative models almost surely violate such constraints even when trained on compliant data, and that existing projection-based fixes restore compliance at the cost of data fidelity. The proposed approach instead trains the constrained distribution $z \\sim p_\\theta(z \\mid Az = k)$ end-to-end, using a gradient estimator that differentiates through the conditional marginal expectation as a differentiable proxy for the latent. Across VAEs, diffusion models, and graph neural networks on five image datasets and three scientific applications, the method attains zero constraint violations while matching or beating unconstrained models and every baseline on fidelity and accuracy metrics. If the paper is right, any Gaussian-latent generative model can absorb linear domain knowledge exactly, with no post-hoc correction layer and no fidelity penalty.","feed_headline":"Constrain the distribution, not the sample, for zero violations","feed_subtitle":"Conditioning Gaussian latents on linear equalities makes models obey domain rules without hurting realism.","key_machinery":"The central object is the constrained Gaussian $z \\sim p_\\theta(z \\mid Az = k)$: the distribution of a Gaussian latent conditioned on a linear equality, which is itself Gaussian with closed-form parameters, mean $\\mu + \\Sigma A^\\top (A\\Sigma A^\\top)^{-1}(k - A\\mu)$ and covariance $\\Sigma - \\Sigma A^\\top (A\\Sigma A^\\top)^{-1} A \\Sigma$. This identity converts constraint satisfaction from a post-hoc projection step into an exact sampling step, usable anywhere in a generative model. The load-bearing training mechanism is the gradient approximation $\\nabla_\\theta L(x,y;\\omega) \\approx \\partial_\\theta m(\\theta)\\, \\nabla_z \\ell(x,y;\\omega)$, with $m(\\theta)$ chosen as the conditional marginal expectation $\\mathbb{E}[z_i \\mid Az = k]$; this proxy is differentiable, carries constraint information, and is shown empirically to dominate constrained marginals, a constrained reparameterization trick, and Constrained Layer projections on bias, variance, and error. A second mechanism, the closed-form expected loss for identity decoders with $\\ell^1$ or $\\ell^2$ losses, removes the need for gradient estimation in regression-style settings and powers the paper's strongest results on scientific applications.","core_discovery":"The paper's central claim is that a hard linear equality constraint $Az = k$ should be enforced at the level of the distribution, not the sample. Given an encoder that outputs Gaussian parameters $\\theta = (\\mu, \\Sigma)$, the model conditions the latent directly: $z \\sim p_\\theta(z \\mid Az = k)$, a Gaussian whose mean and covariance have closed forms, so exact sampling from the feasible set costs no extra machinery. Training targets the constrained expected loss $\\mathbb{E}[\\ell(f_u(z), y) \\mid Az = k]$, and since the gradient of that expectation has no closed form in general, the paper backpropagates through the proxy $\\nabla_\\theta L \\approx \\partial_\\theta m(\\theta)\\, \\nabla_z \\ell(f_u(z), \\hat{y})$ with $m(\\theta) = \\mathbb{E}[z_i \\mid Az = k]$, the conditional marginal expectation. A controlled comparison on a synthetic Gaussian task shows this proxy to have far lower bias and average error than alternatives, including conditional marginal densities, a variance-weighted constrained reparameterization, and projection-based Constrained Layers. When the decoder is the identity map and the loss is $\\ell^1$ or $\\ell^2$, the constrained expected loss is available in closed form, eliminating the estimator altogether. The paper then shows, on brightness-constrained MNIST, CIFAR-10, CelebA, and LSUN images and on MOF charge prediction, chemical-process surrogates, and stock allocation, that unconstrained models violate the constraints with probability close to one while the constrained-distribution method achieves zero violations and better or equal fidelity scores in every reported setting.","pith_inferences":["A stress test the paper does not run: how the marginal-expectation proxy behaves with large or block-structured constraint matrices, since the synthetic validation used small Gaussian instances and the estimator's bias scaling is unexamined.","The conditioning identity suggests a general recipe — any latent family with tractable $p(z \\mid Az = k)$ and tractable conditional marginals could inherit the same end-to-end training; the paper's Poisson and multinomial results are a first step toward that generalization.","An implicit consequence the paper leaves unanalyzed: conditioning on $a$ linear equalities restricts the latent to an $(n-a)$-dimensional affine subspace, so very tight constraint systems shrink effective latent dimensionality, which could matter when the constraint count approaches the latent dimension.","Since the proxy-fidelity evidence uses an identity decoder, a decisive follow-up would compare the proxy against a score-function estimator on a nonlinear decoder; divergent gradient directions would re-attribute the reported gains to the constrained-distribution formulation rather than to the estimator itself."],"forward_implications":["Any deep generative model with a Gaussian latent or output layer can satisfy a linear equality exactly at train and test time, replacing the projection step with a closed-form mean-and-covariance update that adds under a second per epoch in the VAE experiments.","Diffusion models can insert this exact constrained sampling at selected backward steps; on CIFAR-10, CelebA, LSUN Church, and LSUN Cat, the constrained DDPM and DDIM variants drive the violation rate from roughly 0.999 to 0 while matching or improving FID and Inception Score.","In identity-decoder regression settings, the closed-form Gaussian expected loss trains without Monte Carlo noise and beats projection baselines on MOF charge prediction, chemical-process surrogates (CSTR scaled MSE 4.31 versus 9.28), and Sharpe-ratio portfolio allocation (1.9041 versus 1.8092).","Unconstrained generative models do not absorb linear constraints from compliant training data; the paper measures violation rates near 1.0 for plain VAEs and diffusion models, so explicit distribution-level enforcement is necessary rather than optional.","Across estimator designs, the conditional marginal expectation beats the conditional marginal density as a continuous-domain proxy, a divergence the paper links to the discrete k-subset setting where the two coincide."],"supporting_citations":[{"why":"Supplies the reparameterization trick, the standard differentiation-through-sampling technique the paper must replace, and the Vanilla VAE baseline.","marker":"Kingma & Welling (2013)"},{"why":"Provides the Constrained Layer projection baseline used throughout the comparisons, which enforces constraints but degrades generative quality.","marker":"Stoian et al. (2024)"},{"why":"Supplies the L2-projection (Euc/KKThPINN) baseline and the chemical-engineering benchmark suite (CSTR, plant, distillation) reused in Section 6.5.","marker":"Chen et al. (2024)"},{"why":"Introduces the conditional-marginal gradient-estimator idea for discrete k-subset sampling that the paper adapts to continuous Gaussian constraints.","marker":"Ahmed et al. (2023)"},{"why":"Defines the DDPM whose backward diffusion process receives the paper's exact constrained sampling at selected timesteps.","marker":"Ho et al. (2020)"},{"why":"Provides the DDIM sampler and the step-schedule mechanism the paper tunes to place constrained steps for FID/IS balance.","marker":"Song et al. (2021)"},{"why":"Source of the MOF partial-charge task and the observation that models almost surely violate charge neutrality, motivating the work.","marker":"Raza et al. (2020)"}],"fun_headline_variants":["Condition the latent, not the sample, for zero violations","Distribution-level constraints: exact and realistic generation","A closed-form fix for hard linear constraints in DGMs","Forget projection: learn the constrained distribution"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the gradient of the constrained expected loss is faithfully approximated by differentiating the conditional marginal expectation (the paper's Equation 5); this proxy is validated empirically on a synthetic Gaussian task with an identity decoder and $\\ell^1$ or $\\ell^2$ losses, but it is not derived or proven unbiased for nonlinear decoders, general loss functions, or large constraint matrices.","fun_headline_variants_meta":{"raw":{"variants":["Condition the latent, not the sample, for zero violations","Distribution-level constraints: exact and realistic generation","A closed-form fix for hard linear constraints in DGMs","Forget projection: learn the constrained distribution"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000583,"raw_usage":{"total_tokens":2788,"prompt_tokens":1037,"completion_tokens":1751,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":653,"completion_tokens_details":{"reasoning_tokens":1690}},"tokens_in":653,"tokens_out":1751,"duration_ms":18151,"temperature":1.0,"reasoning_tokens":1690,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T19:25:17.894668+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the brightness-constrained MNIST VAE with the paper's marginal-expectation proxy replaced by a score-function (REINFORCE) estimator of the exact constrained gradient, holding everything else fixed; if the two training runs converge to materially different latents or the score-function run achieves higher held-out likelihood, the proxy is biased and the reported gains are estimator-specific rather than inherent to distribution-level constraint enforcement.","supporting_citations":[{"cited_title":"C., Dyrmishi, S., Cordy, M., Lukasiewicz, T., and Giunchiglia, E","cited_arxiv_id":null,"evidence_quote":"Provides the Constrained Layer projection baseline used throughout the comparisons, which enforces constraints but degrades generative quality."},{"cited_title":"Simple: A gradient estimator for k-subset sampling","cited_arxiv_id":null,"evidence_quote":"Introduces the conditional-marginal gradient-estimator idea for discrete k-subset sampling that the paper adapts to continuous Gaussian constraints."},{"cited_title":"M., and Fern, X","cited_arxiv_id":null,"evidence_quote":"Source of the MOF partial-charge task and the observation that models almost surely violate charge neutrality, motivating the work."}],"review_version":1}