{"id":"70181f78-751a-4d79-8c8a-97fe37b3149f","arxiv_id":"2504.17622","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper introduces EnVAE and FEnVAE, deterministic-decoder VAEs trained by energy score rather than likelihood, and reports consistent FID improvements on image benchmarks.","lead":"A new VAE training objective replaces the usual blurry Gaussian reconstruction loss with an energy score computed from decoded samples, improving image reconstruction and generation quality on standard benchmarks. A fast single-sample variant, FEnVAE, makes the approach cheap enough to plug into existing VAE pipelines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Energy-score propriety is invoked for a conditional reconstruction distribution whose target is the conditioning outcome itself; minimizing it does not constrain the marginal generator g#p(z), so the 'statistically principled' claim is unsupported.","rationale":"The reader's weakest_assumption identifies the same gap I find: the energy score's propriety is transferred from a genuine forecasting setting to an autoencoding setting where the conditioning variable and the scored outcome coincide. I make the gap sharper by showing that the expected score E_x S(Q(x),x) is minimized by the degenerate conditional distribution delta_x, so the score acts as a reconstruction loss and contains no term comparing g#p(z) with the data. The toy example with alpha=0 drives the EnVAE loss to zero while the prior-sample distribution diverges from the data, directly contradicting Appendix 6.1.3. Because the empirical comparisons in Tables 1 and 2 and the FEnVAE efficiency analysis are independent of the propriety claim, the paper's practical contribution stands, but the abstract's 'statistically principled' wording and the Section 3.1 / Appendix 6.1.3 assertions must be qualified or removed. A conditional acceptance remains the right verdict: the authors should either supply a correct theorem under which the combined objective controls g#p(z), or reframe the method as a heuristic regularized autoencoder. I therefore agree with the reader's weakest_assumption and see no reason to change the verdict.","tokens_in":22451,"tokens_out":11381,"duration_ms":119651,"concrete_test":"Verify the following analytical counterexample. Let p(z)=N(0,1), q(z|x)=N(mu_x, sigma^2) with mu_1=-mu_-1=mu, decoder g(z)=(1/mu)z for mu>0, beta>0, and alpha=0. As sigma->0, the per-example energy score for x=1 tends to |g(mu)-1|^beta=0 and for x=-1 tends to |g(-mu)-(-1)|^beta=0, so the EnVAE loss tends to 0. However, the generated distribution g#p(z) is N(0,1/mu^2), whose Wasserstein distance to the data {+-1} grows without bound as mu->0. This contradicts the Appendix 6.1.3 claim that the score's propriety favors the correct data distribution. If the authors dispute the alpha=0 setting, re-run the same toy with small alpha>0 and show that the objective's minimizer makes g#p(z) approach the data; the burden is on the claimed theorem, not on the counterexample.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The weakest load-bearing step is the transfer of energy-score propriety to the autoencoder objective in Eq. (4). In standard scoring-rule theory (Gneiting and Raftery, 2007), propriety means that for a fixed predictive distribution Q and a random outcome Y with law P, E[S(Q,Y)] is minimized by Q=P. In EnVAE, the 'outcome' is the training point x, and the predictive distribution Q_{theta,phi}(x) is constructed by an encoder that receives x itself. The relevant population objective is E_{x~data}[S(Q_{theta,phi}(x), x)], and for each x the optimal conditional predictive distribution is the degenerate delta_x, because the training pair is (X,Y) with Y=X. Minimizing this score therefore enforces accurate reconstruction of each x; it does not by itself constrain the marginal generated distribution g_theta#p(z) used for sampling. Section 3.1 states that 'By minimizing the energy score, we aim to learn a generative model that accurately reflects the underlying data distribution,' and Appendix 6.1.3 asserts that 'the score's propriety ensures that, in expectation, the correct data distribution is favored.' Neither statement is derived, and both do not follow from propriety in this conditional-reconstruction setting. A solution that maps each x to a distinct latent code and decodes those codes back to x can drive the per-example score to zero while the push-forward of the prior under g_theta remains arbitrarily far from the data; the KL term penalizes posterior drift but does not compare prior samples with data. Thus the 'statistically principled' framing in the abstract overstates what the objective guarantees.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EnVAE, a VAE variant with a deterministic decoder g_theta trained by minimizing the energy score between the conditional predictive distribution q_phi(z|x) decoded through g_theta and the observation x, plus a KL regularizer. It also introduces FEnVAE, a single-sample surrogate obtained from a first-order Taylor expansion of the decoder, in which the pairwise dispersion term is replaced by a finite-difference term evaluated at one perturbed latent sample. Experiments on CelebA, CIFAR-10, and LSUN report lower FID for reconstruction and generation than likelihood-based reconstruction losses, and consistent gains when the loss is plugged into several existing VAE baselines. The authors claim that proper scoring-rule theory makes the approach statistically principled.","tokens_in":22741,"tokens_out":8318,"duration_ms":78422,"significance":"If the theoretical framing were valid, the paper would offer an interesting bridge between proper scoring rules and amortized variational inference. The empirical work is a strength: it covers multiple datasets and baselines, integrates the loss into three existing VAE/EBM models, includes efficiency and ablation analyses, and provides a public code repository. However, the central statistical claim is not established. The propriety of the energy score is invoked for a conditional reconstruction distribution whose target is the conditioning observation itself; this does not imply that the marginal generated distribution matches the data. The FEnVAE derivation also contains an unjustified replacement of a Jacobian term by a nonlinear finite difference, and the reported smoothness validation measures the encoder rather than the decoder. As a result, the paper currently reads as an empirical demonstration of an energy-score-inspired reconstruction loss rather than as a principled likelihood-free VAE.","major_comments":[{"comment":"The propriety of the energy score is transferred from standard scoring-rule theory to an autoencoder objective without justification. In Gneiting and Raftery (2007), a proper scoring rule S(Q,y) is minimized in expectation over y~P by Q=P for a fixed predictive distribution Q. In Eq. (4), the predictive distribution Q_{theta,phi}(x) is constructed by an encoder that receives x itself, and the population objective is E_{x~p_data}[S(Q_{theta,phi}(x),x)]. For each x, the optimal conditional predictive distribution is the degenerate measure delta_x, so minimizing this score enforces pointwise reconstruction and does not, by itself, constrain the marginal generator g_theta#p(z). The statements in Section 3.1 (\"we aim to learn a generative model that accurately reflects the underlying data distribution\") and Appendix 6.1.3 (\"the score's propriety ensures that, in expectation, the correct data distribution is favored\") therefore do not follow from propriety. The paper needs either a proof that the combination with the KL term yields marginal matching, or a careful restatement of the theoretical claims.","section":"3.1, Eq. (4)"},{"comment":"The derivation of the FEnVAE objective replaces the term ||J_{mu_z} Sigma_z^{1/2} ehat||^beta in Eq. (11) with ||g_theta(mu_z+Sigma_z^{1/2} ehat)-g_theta(mu_z)||^beta using Eq. (12). But Eq. (12) is itself the first-order Taylor approximation of the decoder. For a nonlinear decoder, the finite-difference term is not equal to the Jacobian term, and the claim that this replacement \"locally restores the nonlinear modeling capability\" is not supported. The final loss in Eq. (14) is therefore a new heuristic surrogate rather than a closed-form evaluation of the energy score under the stated linearity assumption. The paper should state the approximation being made at this step and justify why the finite-difference replacement is preferable to retaining the Jacobian term.","section":"3.2, Eqs. (11)-(13)"},{"comment":"The empirical validation of the local-linearity assumption uses the Lipschitz constant of the encoder f(x), but the Taylor expansion in Section 3.2 is taken with respect to the decoder g_theta(z). The smoothness of the encoder does not directly control the size of the omitted O(||z-mu_z||^2) term in Eq. (8), so the Lip values reported in Tables 3-4 and the surrounding discussion do not validate the assumption that underlies FEnVAE. The paper should either measure a relevant decoder Jacobian or Hessian quantity, or reformulate the validity check to target the decoder.","section":"4.3.2, Appendix 6.5, Eq. (26)"}],"minor_comments":[{"comment":"The summation limits in the second term of Eq. (3) are garbled: the text reads \"sum_{i=j} sum_{j:j neq i}\", which should presumably be \"sum_{i=1}^M sum_{j:j neq i}\". Please correct this typo.","section":"Eq. (3)"},{"comment":"The finite-M calculation for beta=2 is algebraically inconsistent. The pairwise sum in Eq. (3) evaluates to - (1/(M-1)) sum_i ||x_i - x_mu||^2 with the usual unbiased variance convention, not the expression used in Eq. (18). The conclusion that beta=2 reduces to the squared Euclidean distance is correct in the population limit, but the displayed derivation should be repaired.","section":"Appendix 6.3, Eq. (18)"},{"comment":"The figure numbering is confusing and duplicated: Figures 1-3 in Section 3.2 are methodology diagrams, while Figures 2-3 in Section 4 are reconstruction and generation samples. Please renumber all figures and ensure each caption matches its content.","section":"Figures 1-3"},{"comment":"The KL term in Eq. (14) is written as D_KL(q_phi(z|x) || p_theta(z)), but the prior elsewhere in the paper is denoted p(z) and is not parameterized by theta. Please make the notation consistent.","section":"Eq. (14)"},{"comment":"The text says \"beta is the exponent of the frequency domain energy score,\" but the energy score in Eq. (1) is defined in the data domain. The frequency-domain interpretation appears only in the qualitative discussion of spectral plots and should be stated as an interpretation rather than a definition.","section":"6.13"}],"recommendation":"major_revision","confidential_remarks":"The empirical results appear consistent and the code is available, so I do not recommend rejection. However, the paper's central theoretical claim about proper scoring rules is not supported in the conditional-reconstruction setting, and the FEnVAE derivation and validation need to be clarified. If the authors reposition the contribution as an empirical reconstruction loss design with an energy-score-inspired regularizer, and remove or prove the propriety-based claims, the work could become suitable for publication. The novelty relative to Pacchiardi and Dutta's scoring-rule minimization and to deterministic autoencoder literature should also be sharpened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Things to know: the empirical recipe is likely useful, but the theory as written does not support the central claim. The energy score is a proper scoring rule for a fixed predictive distribution scored against a fresh outcome; in Eq. (4) the paper instead scores the conditional predictive distribution Q_{θ,φ}(x) against the very x that went into the encoder. For each x, the per-example optimum is a point mass at x, so the reconstruction term alone rewards memorization. Nothing in the score compares the marginal g_θ#p(z) with the data distribution; the KL term keeps the encoder close to the prior but does not constrain the decoder's push-forward. The assertions in Section 3.1 and Appendix 6.1.3 that propriety ensures the correct data distribution is favored simply do not follow. This is the main flaw, and it is load-bearing for the abstract's 'statistically principled' framing.\n\nWhat is actually new is FEnVAE's single-sample surrogate in Eq. (14): replacing the pairwise energy terms with one extra decoding of a √2-scaled perturbation is a neat, practical contribution that I don't see in the cited scoring-rule literature. The experiments are broad and honest: three datasets, several reconstruction-loss baselines, integration into three EBM-based VAEs, and code is available. Gains are consistent though modest (roughly 5–11% FID), and the authors state plainly that FEnVAE sometimes underperforms EnVAE and that β=2 collapses to MSE. That candor counts for something.\n\nSoft spots beyond the theory: no error bars or seed-level variance, so it is hard to know whether the gains are meaningful. The Lipschitz coefficient defined in Appendix 6.5 is for the encoder, but Section 4.3.2 applies it to the decoder's output smoothness; that is a metric mismatch. The single-sample reparameterization coupling in FEnVAE is heuristic, though Figure 6's correlation analysis is reasonable indirect evidence. The local-linearity assumption is checked empirically, not derived, which is acceptable for a practical variant.\n\nWho this is for: anyone who wants a drop-in reconstruction loss sharper than L2/L1 and is willing to treat the scoring-rule framing as intuition rather than proof. It deserves a serious referee; the method is reproducible and the experiments are solid enough to warrant revision rather than rejection. I would not cite it for the propriety argument.","headline":"The single-sample surrogate is a genuinely useful trick, but the paper's central propriety argument is misapplied and needs serious reworking before the 'statistically principled' claim can be taken literally.","tokens_in":23314,"tokens_out":2725,"would_cite":false,"duration_ms":26920,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes replacing a VAE's Gaussian likelihood with the energy score, a strictly proper scoring rule, and shows that the resulting likelihood-free EnVAE and its single-sample variant FEnVAE improve image reconstruction and…","keywords":["likelihood-free inference","variational autoencoders","energy score","proper scoring rules","deterministic decoder","single-sample approximation","generative modeling","image generation"],"falsifier":"Train EnVAE on synthetic data drawn from a known nonlinear decoder with substantial latent variance, then test the learned sampler against the true data distribution with a two-sample energy-distance test; if the test rejects at the paper's sample sizes, the propriety-based claim fails. Separately, if FEnVAE's single-sample loss diverges from EnVAE's multi-sample loss as latent variance grows or decoder curvature increases, the local-linearity premise is falsified.","tokens_in":22228,"feed_emoji":"🖼️","tokens_out":7982,"duration_ms":71295,"temperature":0.7,"pith_summary":"Standard VAEs blur images because their decoder assumes a simple, tractable likelihood, usually an isotropic Gaussian, and the KL objective penalizes mismatch asymmetrically. This paper proposes EnVAE, which removes the likelihood entirely: the decoder is deterministic and the reconstruction loss is the energy score, a strictly proper scoring rule that compares sampled reconstructions with the observed image. Minimizing this score does not require an explicit density, so the model is likelihood-free at the level of the observation model. The paper also derives FEnVAE, a single-sample approximation based on local decoder linearity and small posterior variance, which makes the added cost nearly negligible. If the approach works as claimed, Gaussian likelihood misspecification is not a necessary price of VAE training, and high-fidelity VAE-like models can be trained without adversarial components.","feed_headline":"Energy-score loss sharpens VAE images, no likelihood needed","feed_subtitle":"Replacing the decoder's Gaussian assumption with a proper scoring rule improves reconstruction and generation on standard benchmarks.","key_machinery":"The central object is the energy score, a strictly proper scoring rule defined as $$S_E(P,x) = \\mathbb{E}_{X\\sim P}\\|X-x\\|_2^\\$\\beta$ - \\tfrac{1}{2}\\mathbb{E}_{X,X'\\sim P}\\|X-X'\\|_2^\\$\\beta$,$$ whose expected value is uniquely minimized when the predicted distribution $P$ equals the true data distribution. In EnVAE, $P$ is the distribution of decoded outputs $g_\\theta(z)$ with $z \\sim q_\\phi(z|x)$, so the score becomes a sample-based reconstruction loss. The key simplification in FEnVAE is a first-order Taylor expansion of the decoder, $g_\\theta(z) \\approx g_\\theta(\\mu_z) + J_{\\mu_z}(z-\\mu_z)$, which rewrites the pairwise dispersion term using the latent covariance and then re-expresses it through decoder outputs alone. That yields the single-sample objective $$L^*(\\$\\theta$,\\phi;x) = \\|g_\\$\\theta$(z^*) - x\\|_2^\\$\\beta$ - \\tfrac{1}{2}\\|g_\\$\\theta$(\\mu_z + \\sqrt{2}(z^*-\\mu_z)) - g_\\$\\theta$(\\mu_z)\\|_2^\\$\\beta$ + \\$\\alpha$ D_{\\mathrm{KL}}(q_\\$\\varphi$(z|x)\\|p(z)).$$","core_discovery":"The paper's central claim is that VAEs can be trained without any parametric likelihood by pairing a deterministic decoder with the energy score as the reconstruction objective. For an input $x$, the encoder produces a Gaussian posterior over latent $z$, samples are decoded, and the loss is the expected distance from decoded samples to $x$ minus half the expected pairwise distance among decoded samples, plus the usual KL regularizer toward the prior. Because the energy score is strictly proper, the paper argues that minimizing it favors the correct output distribution in expectation; because the decoder is deterministic, no density for $p_\\theta(x|z)$ is ever evaluated. The fast variant FEnVAE replaces the pairwise Monte Carlo term with a closed-form surrogate built from a first-order Taylor expansion of the decoder around the posterior mean, reusing a single noise draw in both the mean and dispersion terms. Empirically, EnVAE and FEnVAE improve FID for reconstruction and generation over Gaussian, Laplace, Student-t, and frequency-based baselines on CelebA 64, CIFAR-10, and LSUN Church.","pith_inferences":["If the propriety transfer holds in the autoencoder conditional setting, the same construction could replace likelihood terms in other latent-variable models, such as hierarchical VAEs, normalizing-flow decoders, or time-series forecasting models, not just image autoencoders.","The FEnVAE approximation should degrade as posterior variance grows or decoder curvature increases; the paper's own tables show larger latent variance correlating with larger FEnVAE-vs-EnVAE gaps, suggesting a testable bound relating approximation error to the latent covariance and the Jacobian's Lipschitz constant.","A deeper consequence is that blur in VAEs may be attributed less to the KL term itself and more to the Gaussian observation model; if so, likelihood-free reconstruction could be combined with more aggressive latent regularization to trade off sharpness and coverage."],"forward_implications":["VAE pipelines can retain amortized inference and latent-space regularization while dropping the likelihood model; the same architecture can be trained with a scoring-rule loss at minimal added cost.","The single-sample FEnVAE objective makes the energy score practical at scale, with overhead of roughly two extra decoder passes, so large VAE-based systems could adopt likelihood-free reconstruction without architectural changes.","Reconstruction uncertainty is represented by the spread of decoded samples rather than by a parameterized noise model, which yields sharper pixel-wise variance maps in the paper's visualizations.","Because the loss is a proper score rather than an adversarial objective, training remains stable and no discriminator is needed, potentially capturing some GAN-like sharpness without GAN training dynamics.","The score's $\\beta$ exponent controls sensitivity to large errors; the paper shows $\\beta=1$ works best and $\\beta=2$ degenerates to MSE, so $\\beta$ becomes a practical tuning knob connecting the method to standard losses."],"supporting_citations":[{"why":"Defines strictly proper scoring rules and establishes that a proper score is minimized in expectation only by the true distribution, the theoretical basis for the energy-score objective.","marker":"[33]"},{"why":"Supplies the likelihood-free inference framework that trains generative networks by minimizing scoring rules, which EnVAE adapts to the VAE setting.","marker":"[34]"},{"why":"Introduces energy statistics and distance-based scoring, the mathematical foundation of the energy score.","marker":"[45]"},{"why":"Demonstrates scoring-rule-based training for multivariate ensemble postprocessing, one of the concrete precedents the paper follows.","marker":"[37]"},{"why":"Applies scoring-rule minimization to probabilistic forecasting with generative networks, another precedent for the proposed loss.","marker":"[39]"},{"why":"Uses energy-score-style distributional regression for extrapolation, supporting the use of scoring rules as neural objectives.","marker":"[38]"},{"why":"Provides the reparameterization trick and variational autoencoder formulation that EnVAE builds on and modifies.","marker":"[1]"},{"why":"Supplies the smooth-manifold and Taylor-expansion justification used to derive the FEnVAE local linearity approximation.","marker":"[47]"},{"why":"Supplies the manifold hypothesis that the decoder's output space is locally linear, underpinning the FEnVAE derivation.","marker":"[46]"}],"fun_headline_variants":["No likelihood? Energy score sharpens VAEs","Energy-score loss beats Gaussian likelihood in VAEs","Drop the likelihood, keep the sharpness: EnVAE","Fast single-sample energy-score training for VAEs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's argument stands on the assumption that scoring an encoder's conditional output distribution with the energy score pushes the model's overall output distribution toward the true data distribution, a step the paper asserts but does not prove; FEnVAE additionally assumes the decoder is nearly linear over the small spread of latent samples.","fun_headline_variants_meta":{"raw":{"variants":["No likelihood? Energy score sharpens VAEs","Energy-score loss beats Gaussian likelihood in VAEs","Drop the likelihood, keep the sharpness: EnVAE","Fast single-sample energy-score training for VAEs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0007,"raw_usage":{"total_tokens":3168,"prompt_tokens":958,"completion_tokens":2210,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":574,"completion_tokens_details":{"reasoning_tokens":2148}},"tokens_in":574,"tokens_out":2210,"duration_ms":19397,"temperature":1.0,"reasoning_tokens":2148,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:36:02.706463+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train EnVAE on synthetic data drawn from a known nonlinear decoder with substantial latent variance, then test the learned sampler against the true data distribution with a two-sample energy-distance test; if the test rejects at the paper's sample sizes, the propriety-based claim fails. Separately, if FEnVAE's single-sample loss diverges from EnVAE's multi-sample loss as latent variance grows or decoder curvature increases, the local-linearity premise is falsified.","supporting_citations":[{"cited_title":"Strictly proper scoring rules, prediction, and estimation,","cited_arxiv_id":null,"evidence_quote":"Defines strictly proper scoring rules and establishes that a proper score is minimized in expectation only by the true distribution, the theoretical basis for the energy-score objective."},{"cited_title":"Energy statistics: A class of statistics based on distances,","cited_arxiv_id":null,"evidence_quote":"Introduces energy statistics and distance-based scoring, the mathematical foundation of the energy score."},{"cited_title":"Generative machine learning methods for multi- variate ensemble postprocessing,","cited_arxiv_id":null,"evidence_quote":"Demonstrates scoring-rule-based training for multivariate ensemble postprocessing, one of the concrete precedents the paper follows."},{"cited_title":"Probabilistic forecasting with gener- ative networks via scoring rule minimization,","cited_arxiv_id":null,"evidence_quote":"Applies scoring-rule minimization to probabilistic forecasting with generative networks, another precedent for the proposed loss."},{"cited_title":"Engression: extrapolation through the lens of distributional regression,","cited_arxiv_id":null,"evidence_quote":"Uses energy-score-style distributional regression for extrapolation, supporting the use of scoring rules as neural objectives."},{"cited_title":"Auto-encoding variational bayes,","cited_arxiv_id":null,"evidence_quote":"Provides the reparameterization trick and variational autoencoder formulation that EnVAE builds on and modifies."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the smooth-manifold and Taylor-expansion justification used to derive the FEnVAE local linearity approximation."},{"cited_title":"A global geometric framework for nonlinear dimensionality reduction,","cited_arxiv_id":null,"evidence_quote":"Supplies the manifold hypothesis that the decoder's output space is locally linear, underpinning the FEnVAE derivation."}],"review_version":1}