{"id":"ef1776ff-c37f-4dad-8519-15a8745ed6a5","arxiv_id":"2506.14607","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A likelihood-based distribution matching method uses a score-based prior trained by denoising score matching and a Gromov-Wasserstein semantic-space regularizer, improving fairness, domain adaptation, and domain translation.","lead":"The authors train a variational autoencoder to match two data distributions by replacing the fixed prior with a flexible score function learned through denoising score matching, avoiding the unstable Jacobian terms of other diffusion-based priors. A Gromov-Wasserstein regularizer computed in CLIP semantic space is added to preserve structure, and the paper reports improvements on fairness, domain adaptation, and domain translation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SFS 'unbiased' gradient claim is conditional on the DSM score at σmin matching the clean prior score; the paper gives no error bound and its own Fig. 1 shows both methods degrade at σmin = 0.001.","rationale":"The central contribution is the Score Function Substitution trick, which claims exact encoder gradients using only the prior's score. The proof in Appendix A has sign and notation inconsistencies, but the final identity is correct provided the score is the true clean score of the prior. The load-bearing condition is therefore the quality of the DSM approximation at the smallest noise level: if Sψ(z, σmin) is not close to ∇_z log Qψ(z), then the SFS gradient is biased, and optimizing LSAUB does not coincide with optimizing LVAUB. This is the same weakest assumption the reader identified, and the paper's own Fig. 1 corroborates that the approximation is fragile at very small σmin. The CLIP confound in the domain adaptation results is an empirical concern, but it does not bear on the correctness of the core gradient-substitution mechanism; the σmin dependence does. The paper does provide a plausible alternating algorithm, reproducible code, and an honest stability experiment, which support a conditional rather than a rejecting verdict. Since the reader already assigned CONDITIONAL with the same underlying concern, no verdict adjustment is needed.","tokens_in":20268,"tokens_out":9115,"duration_ms":101660,"concrete_test":"On the synthetic Gaussian-mixture prior used in Section 3.3 (where the true density is known), train the DSM score model at σmin ∈ {0.001, 0.01, 0.1}, then on a fixed minibatch of encoder samples compare the SFS gradient vector E[-Sψ(z + σmin ε)^T ∂z/∂θ] with the exact reparameterized gradient E[-∇_z log Q(z)^T ∂z/∂θ]. Report relative L2 error and cosine similarity for each σmin and across training checkpoints. If the relative error exceeds about 10% at the σmin used in the main experiments, the 'unbiased' gradient claim fails in the implemented regime and the results should be re-interpreted as relying on a small-σ approximation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Proposition 3.1 (Eq. 6) and the SAUB objective (Eq. 7) are gradient-equivalent to the VAUB cross-entropy term only when the score network Sψ(z*, σmin) equals the clean score ∇_z log Qψ(z). In the implemented algorithm (Eq. 8, Alg. 1 line 7), the score is evaluated at z* = z + σmin ε on samples from the σmin-perturbed distribution, and Sψ is trained by DSM (Eq. 9) to approximate the score of the noisy marginal posterior, not the clean score. The resulting gradient is biased by an amount that depends on σmin (O(σmin²) for smooth densities under standard asymptotics) plus DSM approximation error; no bound or bias correction is given. The paper's own stability study (Section 3.3, Fig. 1) shows that at σmin = 0.001 both SAUB and LSGM degrade, and at σmin = 0.01 SFS is stable but still uses a non-negligible noise level; the experiment demonstrates stability, not unbiasedness. Because the lower-level DSM objective in Eq. 9 only makes the VAUB bound tight in the limit of perfect score matching and σmin → 0, any finite-σmin bias means the encoder is not actually optimizing the claimed VAUB bound, and the SFS surrogate's optimization path may depart from the intended objective. The main experimental section does not report the σmin used in Tables 1–2 or a sensitivity analysis over σmin, so it is unclear whether the deployed regime is one where the clean-score approximation is accurate.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes Score Function Substitution (SFS) for likelihood-based distribution matching in the VAUB framework. Instead of modeling the domain-invariant prior density explicitly, the prior is represented by its score function, trained by denoising score matching. The authors claim (Prop. 3.1) that the encoder-gradient of the cross-entropy term can be expressed using a detached score evaluation, so encoder/decoder updates need not backpropagate through the diffusion score network. A bilevel alternating algorithm (SAUB) alternates VAUB-style encoder/decoder updates with DSM prior updates. A Gromov-Wasserstein regularizer, computed in Euclidean or CLIP semantic space, is added to preserve geometry. Experiments cover synthetic D-shape separation, Adult fairness, MNIST-USPS domain adaptation, and CelebA domain translation, comparing with Gaussian/MoG/VampPrior/LSGM priors and VAUB/adversarial baselines.","tokens_in":20527,"tokens_out":8075,"duration_ms":80623,"significance":"If the core identity and empirical results hold, SFS is a useful contribution: it sidesteps density normalization and diffusion Jacobians, making expressive learned priors practical for distribution matching, and the CLIP-space GW regularizer gives a sensible way to inject semantics. The paper ships code, includes a stability comparison against LSGM, reports computational efficiency, and uses separate CLIP models for training and evaluation in the retrieval metric. However, the exactness/unbiasedness claim is stronger than what is demonstrated: with a finite sigma_min and approximate DSM scores, the SAUB gradient is biased relative to the VAUB objective, and the paper's own stability experiment shows sensitivity at small sigma_min. The missing error bars and anomalous VAUB baseline in Table 1 also weaken the empirical conclusions as printed.","major_comments":[{"comment":"The SFS identity in Proposition 3.1 is exact only for the clean score, i.e., S_psi(z) = grad_z log Q_psi(z); the implemented method evaluates S_psi at z* = z + sigma_min * epsilon on detached samples (Alg. 1, line 7) and trains S_psi with the DSM objective (Eq. 9) on the noisy marginal posterior. For any sigma_min > 0 and any imperfect DSM fit, the SAUB gradient is a biased estimate of the VAUB cross-entropy gradient, and no bias bound or bias correction is provided. The paper's own Fig. 1 shows that both SAUB and LSGM degrade at sigma_min = 0.001, so the choice of sigma_min is empirically consequential, yet Tables 1 and 2 do not report the sigma_min used. Moreover, the stability study in Section 3.3 freezes the prior and trains only the encoder/decoder, so it does not test the full alternating SAUB loop. Please add a quantitative bias analysis (at minimum an O(sigma_min^2) statement with explicit smoothness assumptions), report sigma_min for every experiment, and revise the claims of exactness/unbiasedness to approximate guarantees.","section":"Sections 3.1-3.2, Eqs. (6)-(9), Alg. 1"},{"comment":"As printed, the proof of Proposition 3.1 has a sign inconsistency. Eq. (19) states E_epsilon grad_theta [ (grad_zbar log Q_psi(zbar)|_{zbar=g_theta(epsilon)})^T g_theta(epsilon) ], while Eq. (20) states grad_theta E_epsilon [ -(grad_zbar log Q_psi(zbar)|_{zbar=g_theta(epsilon)})^T g_theta(epsilon) ]. If the score is treated as constant with respect to theta, the derivative of -a^T g_theta is -J_g^T a, so the two displayed expressions are negatives of each other unless the score is zero. The intended identity is correct by a standard pathwise-derivative argument, but the proof as written must be corrected (either the missing minus sign in Eq. (19) or an explicit statement that the score is frozen before differentiation). Since Proposition 3.1 is the central theoretical claim, this is load-bearing despite being local.","section":"Appendix A, Eqs. (19)-(20)"},{"comment":"Table 1 reports no error bars or number of seeds, and the VAUB baseline entries (40.7% for MNIST to USPS, 45.3% for USPS to MNIST) are far below what one would expect from a VAE-based distribution-matching method on this task, including the cited VAUB work. If these numbers are accurate, the architecture and hyperparameter choices that produce them should be described; if they are not, the comparison against the proposed method is not meaningful. Please add multi-seed standard deviations for all methods and reconcile the VAUB numbers with the prior literature or explain the discrepancy (e.g., different encoder, latent dimension, or training budget).","section":"Table 1"}],"minor_comments":[{"comment":"Equation (5) contains LaTeX artifacts such as '/bracehtipupleft/bracehtipdownright' that should be cleaned before publication.","section":"Eq. (5)"},{"comment":"In Section 3.2, 'it's density' should be 'its density'.","section":"Section 3.2"},{"comment":"The notation z* is used inconsistently: earlier z* = z + sigma_min * epsilon, but Eq. (25) writes z* = z. Please align the definitions.","section":"Appendix F, Eq. (25)"},{"comment":"Table 2 reports standard deviations for retrieval metrics but not for SSIM or LPIPS; please report repeated-run variability for all metrics or state explicitly why it is omitted.","section":"Table 2"},{"comment":"Figure 4 shows a fairness-accuracy trade-off without error bars or the number of runs; please add them or state that the curves are single runs.","section":"Fig. 4"},{"comment":"The sentence 'our method not only retains the advantages of the SAUB method' is confusing because SAUB is introduced in this paper; this should read 'VAUB' or be rephrased.","section":"Section 5.2"},{"comment":"References Song et al. 2021a and 2021b are duplicate entries for the same paper; consolidate them.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the scope of a machine-learning venue, and the code availability is a plus. The main issues are the overclaimed exactness of SFS, the sign typo in the appendix proof, and the empirical reporting gaps; I believe these are addressable in a major revision rather than grounds for rejection. I would also ask the editor to verify the VAUB baseline numbers in Table 1, since they are far outside published ranges for this task."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You'll want to know two things about this one. First, the core idea is real: train a score-based prior with denoising score matching, then use only the score (not the density) to compute encoder gradients inside the VAUB objective. That is a natural and practical extension of LSGM to non-adversarial distribution matching, and the detached-score trick does avoid the Jacobian instability that plagues LSGM at low noise levels. The CLIP-based Gromov-Wasserstein regularizer is also a pragmatic way to inject semantic structure, and the paper shows it helps in several tasks. Second, the paper's central claim of \"exact\" or \"unbiased\" gradients is over the top. The implemented estimator evaluates the score at z + σmin ε and detaches the score network; it is an unbiased gradient estimator only if the score network equals the clean score at that point. In practice, DSM trains toward the score of the noisy marginal, so there is an O(σmin²) bias plus score approximation error, and the paper gives no bound or correction. Its own stability experiment (Fig. 1) shows both SAUB and LSGM degrade at σmin = 0.001, and the main experiments do not report which σmin was used. That is a genuine soft spot, but not a fatal one: for small enough σmin the bias is likely negligible, and the method clearly stabilizes training compared to LSGM. The other soft spots are smaller. Table 1 has no error bars and the VAUB baseline (40.7%) looks far below what one would expect, making the gains look larger than they are. The GW-SP results, especially on MNIST-USPS, rely on powerful CLIP features, so comparisons with non-CLIP baselines are not apples-to-apples; that should be stated more honestly. Appendix A's proof also has sign inconsistencies and mixes exact equality with the detached estimator; a careful revision would clean that up. The self-referential prior-posterior loop is standard alternating optimization, not circular reasoning, so the circularity burden is low. This is a paper for people working on domain adaptation, fairness, and score-based latent models. I would send it to peer review — it clearly deserves a serious referee — but I would ask for the empirical gaps to be filled: error bars, the σmin setting for every result, a bias analysis or at least a sensitivity sweep, and a discussion of what CLIP is contributing. With those, the conditional acceptance becomes a straightforward accept.","headline":"A useful, sensible idea for non-adversarial distribution matching with score-based priors, but the unbiasedness claim is stronger than the implemented estimator actually supports.","tokens_in":21128,"tokens_out":4592,"would_cite":true,"duration_ms":50074,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Distribution matching only needs the prior's score function, not its density—and a CLIP-space Gromov-Wasserstein regularizer preserves geometry.","keywords":["score-based priors","distribution matching","denoising score matching","score function substitution","variational alignment upper bound","Gromov-Wasserstein regularization","semantic preservation","domain adaptation"],"falsifier":"Pick a tractable prior and posterior, such as Gaussian mixtures, train a score model to convergence at several noise levels, and compare the exact cross-entropy gradient $\\nabla_\\theta \\mathbb{E}[-\\log Q(z)]$ against the SFS gradient computed with the same samples; if the relative error does not shrink toward zero as $\\sigma_{\\min}$ decreases, or if SAUB's objective drifts from VAUB's as $\\sigma_{\\min} \\to 0$, the unbiasedness claim collapses.","tokens_in":20001,"feed_emoji":"🎯","tokens_out":7729,"duration_ms":71145,"temperature":0.7,"pith_summary":"The paper claims that likelihood-based distribution matching can be trained using only the score function of the prior—the gradient of its log-density—instead of the prior density itself. It introduces the Score Function Substitution (SFS) trick, which rewrites the cross-entropy term in the variational alignment upper bound so that encoder and decoder gradients are computed from a detached score evaluation and remain unbiased. The prior is then updated by denoising score matching in alternation, which tightens the bound without requiring sampling, normalization, or a tractable density. The authors argue this both avoids the bias of fixed priors like Gaussians in VAEs and the instability of adversarial or diffusion-based prior training like LSGM, while enabling geometry-preserving Gromov-Wasserstein regularization in a CLIP semantic space. If correct, expressive priors become practical for fair classification, domain adaptation, and domain translation without adversarial training.","feed_headline":"Score-only priors stabilize distribution matching","feed_subtitle":"Detached score gradients replace log-density terms; CLIP-space Gromov-Wasserstein keeps geometry intact.","key_machinery":"The machinery is the Score Function Substitution (SFS) trick: the identity in Proposition 3.1 that replaces the log-prior evaluation in the cross-entropy term with a dot product against the prior's score, with the score's input detached from the encoder's computational graph. This single substitution is load-bearing because it keeps the encoder/decoder gradient exact while eliminating density evaluation, normalization, prior sampling, and the Jacobian of the diffusion network; it is also what licenses the alternating optimization, since the prior's own parameters are updated separately by denoising score matching (Eq. 9). Around this core sit the VAUB objective being optimized, the score network parameterizing the prior, and the Gromov-Wasserstein loss (GW-EP with Euclidean distances, GW-SP with CLIP semantic distances) that injects geometry preservation.","core_discovery":"On its own terms, the paper establishes that gradient-based distribution matching through the variational alignment upper bound (VAUB) needs only the prior's score, not its density. Its Proposition 3.1 gives the SFS identity: $\\nabla_\\theta \\mathbb{E}_{z_\\theta \\sim q_\\theta(z|x)}[-\\log Q_\\psi(z_\\theta)] = \\nabla_\\theta \\mathbb{E}[- (\\nabla_{\\bar z} \\log Q_\\psi(\\bar z)|_{\\bar z=z_\\theta})^\\top z_\\theta]$, where the score evaluation is treated as constant with respect to $\\theta$. This yields the SAUB objective, whose encoder and decoder gradients coincide with VAUB's but which never calls the density or backpropagates through a diffusion network. Because the substitution does not preserve prior-parameter gradients, the paper alternates: it updates the encoder and decoder with SAUB, then updates the score-based prior with denoising score matching on the encoder's marginal posterior, making the bound tight when the lower-level problem is solved. It further claims that adding a Gromov-Wasserstein structural-preservation loss with CLIP embeddings as the latent metric—GW-SP—improves semantic preservation and downstream performance, reporting accuracy gains on MNIST-to-USPS adaptation, better accuracy at equal demographic parity on Adult, and better image retrieval on CelebA translation.","pith_inferences":["The SFS identity is essentially a pathwise gradient estimator for a log-prior cost, so it should port to any latent-variable objective with a cross-entropy-style term—hierarchical VAEs, regularized autoencoders, or causal representation learning—whenever only the score is available.","Because the paper updates the prior only through denoising score matching, it never optimizes the prior's likelihood directly; a testable extension would be a stochastic estimator for the prior parameters that closes the remaining gap between SAUB and the original VAUB.","Replacing CLIP with another pretrained distance, such as a self-supervised image feature or a medical embedding, is a direct recipe for semantic preservation in domains where CLIP is weak.","If the clean-score approximation at the minimum noise level is the bottleneck, an annealed noise schedule that starts large and decreases $\\sigma_{\\min}$ during training should further tighten the bound while avoiding the $\\sigma_{\\min}=0.001$ instability the paper documents."],"forward_implications":["Expressive, non-normalized priors become usable in VAE-style distribution matching, so the latent prior can be as complex as the encoder's marginal posterior without flow constraints or adversarial training.","Small-noise score models—which give better clean-score approximations—no longer destabilize training the way diffusion-Jacobian backpropagation does in LSGM, so practitioners can safely operate near $\\sigma_{\\min} \\approx 0.01$.","Geometry-preserving regularization moves from pixel and Euclidean spaces to semantic embedding spaces, which should transfer to any task where a pretrained embedding defines meaningful distances.","Alternating denoising score matching makes the variational upper bound tight when the score model converges, giving likelihood-based distribution matching a principled non-adversarial way to track distribution shift.","Memory and time savings from omitting the diffusion Jacobian grow with latent dimension, making score-based priors more favorable for high-dimensional representations."],"supporting_citations":[{"why":"Supplies the VAUB objective and the fixed-prior distribution matching formulation that the paper generalizes.","marker":"Gong et al. (2024)"},{"why":"Proves denoising score matching learns the score of the perturbed distribution, grounding the lower-level prior update.","marker":"Vincent (2011)"},{"why":"Introduces score-based generative modeling and the denoising score matching setup used to parameterize the prior.","marker":"Song & Ermon (2019)"},{"why":"Defines LSGM, the latent score-based generative baseline whose Jacobian backpropagation instability SFS avoids.","marker":"Vahdat et al. (2021)"},{"why":"Gives the Gromov-Wasserstein autoencoder loss adopted as the geometry-preserving regularizer.","marker":"Nakagawa et al. (2023)"},{"why":"Provides the CLIP semantic embedding space used as the metric for GW-SP.","marker":"Radford et al. (2021)"},{"why":"Documents unstable gradients of diffusion-model Jacobians at low noise, cited to explain LSGM's divergence.","marker":"Poole et al. (2022)"},{"why":"Shows that matching the prior to the aggregated posterior tightens the distribution-matching variational bound.","marker":"Cho et al. (2022a)"}],"fun_headline_variants":["Score-only priors replace densities for stable distribution matching","Geometry-preserving score priors boost distribution matching stability","No density needed: score-based priors stabilize distribution matching","Score-only priors make distribution matching stable and efficient"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The encoder and decoder gradients are unbiased only if the denoising score model at the smallest chosen noise level accurately approximates the clean score of the prior; the paper's stability experiment shows both SAUB and LSGM degrade at $\\sigma_{\\min}=0.001$, so this approximation quality is the load-bearing premise.","fun_headline_variants_meta":{"raw":{"variants":["Score-only priors replace densities for stable distribution matching","Geometry-preserving score priors boost distribution matching stability","No density needed: score-based priors stabilize distribution matching","Score-only priors make distribution matching stable and efficient"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000215,"raw_usage":{"total_tokens":1480,"prompt_tokens":1046,"completion_tokens":434,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":662,"completion_tokens_details":{"reasoning_tokens":370}},"tokens_in":662,"tokens_out":434,"duration_ms":4633,"temperature":1.0,"reasoning_tokens":370,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:51:34.819714+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Pick a tractable prior and posterior, such as Gaussian mixtures, train a score model to convergence at several noise levels, and compare the exact cross-entropy gradient $\\nabla_\\theta \\mathbb{E}[-\\log Q(z)]$ against the SFS gradient computed with the same samples; if the relative error does not shrink toward zero as $\\sigma_{\\min}$ decreases, or if SAUB's objective drifts from VAUB's as $\\sigma_{\\min} \\to 0$, the unbiasedness claim collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the VAUB objective and the fixed-prior distribution matching formulation that the paper generalizes."},{"cited_title":"Score-based generative modeling in latent space","cited_arxiv_id":null,"evidence_quote":"Defines LSGM, the latent score-based generative baseline whose Jacobian backpropagation instability SFS avoids."},{"cited_title":"Gromov-Wasserstein Autoencoders","cited_arxiv_id":"2209.07007","evidence_quote":"Gives the Gromov-Wasserstein autoencoder loss adopted as the geometry-preserving regularizer."}],"review_version":1}