{"id":"67a1281f-d3d6-4356-8eb4-ea822fb95a45","arxiv_id":"2506.01522","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"FIVE is a VAE with an encoder-Jacobian-induced full covariance posterior that matches or improves test log-likelihood over diagonal and full-covariance baselines on MNIST and CIFAR10.","lead":"This paper introduces FIVE, a variational autoencoder that obtains a full covariance posterior from the Jacobians of its encoder, instead of learning a separate covariance matrix. The result is improved test log-likelihoods over standard diagonal VAEs and full-covariance baselines on MNIST and CIFAR10, with only modest added computation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The FIVE posterior covariance is not the Laplace posterior even in the linear-Gaussian case; Eq. (31)'s substitution f'≈(g')^+ fails at the paper's own linear optimum, so the claimed full-covariance posterior and the ELBO interpretation rest on an unproven limit.","rationale":"The reader's conditional verdict is appropriate, but the weakest point is not only the local-linearity/Laplace assumption. Even granting exact linearity, the covariance substitution in Eq. (31) is unjustified: the paper's own linear solution shows f' is not the pseudoinverse of g', so q(z|x) is not the Laplace posterior. The appendix admits the training loss uses a trace surrogate for the log-det term, making the objective a heuristic rather than a valid ELBO. This undermines the theoretical claim that FIVE provides a posterior 'equivalent to a full Gaussian covariance distribution' and complicates the interpretation of the empirical likelihood gains. However, since the experiments might still be reproducible and useful, and the flaws are addressable by re-deriving the surrogate and releasing code, conditional acceptance remains the appropriate verdict. My read does not change the reader's verdict; it sharpens the specific condition that must be met.","tokens_in":14634,"tokens_out":19285,"duration_ms":218719,"concrete_test":"Analytically evaluate the exact ELBO for the 1D linear model of §A.3 with q(z|x)=N(vx, σ²v²), p(z)=N(0,1), p(x|z)=N(wz,σ²), and q(x)=N(0,λ). Compare the maximizer (v,w) of the exact ELBO, computed with the true log-det KL, against the surrogate-loss optimum given in Lemma 1. If the maximizers differ, the trace surrogate changes the training objective, and the reported likelihoods are not produced by the ELBO FIVE claims to optimize. This analytical check settles whether the posterior-covariance mismatch is real without retraining on image datasets.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central derivation breaks at Eq. (31). Differentiating the mode condition uz(x,f(x))=0 gives, exactly, H(x)^{-1} = σ² f'(x)(g'(f(x)))^+^T (see §A.4). FIVE instead substitutes this with σ² f'(x)f'(x)^T, which requires f'(x) ≈ (g'(f(x)))^+. No term in the FIVE loss enforces this relation. The paper's own linear analysis contradicts it: in Lemma 1, the optimum has f'=v=√λ/(σ²+λ) and g'=w=√λ, so f'g'=λ/(σ²+λ)≠1 and f' is not the pseudoinverse of g'. Consequently q's covariance is σ²v²=σ²λ/(σ²+λ)², while the true Laplace posterior variance is σ²/(σ²+λ); these differ by the factor λ/(σ²+λ). Thus q(z|x) is not the Laplace posterior even in the exactly linear, Gaussian case. The equality only holds in the limits σ→0 (with λ fixed) or λ→∞, not in the finite-σ regime of the experiments. The appendix's own Eq. (68) confirms that the training loss replaces the exact log-det KL term with a trace surrogate, so the optimized objective is not a valid ELBO. The empirical likelihood gains in Table 1 may therefore reflect the FIF-style surrogate objective rather than a principled variational posterior, and the abstraction's claim of a 'full Gaussian covariance posterior' is stronger than what the derivation supports.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"FIVE is a variational autoencoder whose approximate posterior is defined implicitly by the encoder Jacobian: q(z|x)=N(f(x), σ² f'(x)f'(x)^T) (Eq. 32). The paper argues that this yields a full-covariance Gaussian posterior at computational cost near that of a diagonal VAE, provides a differential-geometric argument (Theorem 2) that diagonal-posterior VAEs cannot represent arbitrary curved data manifolds, derives a regularized FIF training objective, and reports held-out test log-likelihoods on MNIST and CIFAR-10 in which FIVE matches or exceeds VAE, full-covariance VAE, and FIF baselines. The theoretical connection to the Laplace posterior is the main basis for the claim.","tokens_in":15026,"tokens_out":8379,"duration_ms":78443,"significance":"The geometric obstruction result (Theorem 2) is a clean and useful observation, and the empirical comparison is careful: test likelihoods are estimated by importance sampling with 100 samples, standard errors are reported over three runs, and two architectures per dataset are considered. If the posterior interpretation were valid, FIVE would be a practically attractive way to obtain full covariance without O(d²) parameters or determinant computations. However, the central derivation equating q(z|x) with the Laplace posterior is not valid in the finite-σ regime used in the experiments, and the theorem offered in support overclaims what is proved. The empirical likelihood results can stand on their own, but the paper's theoretical significance as a principled full-covariance VAE is currently not established.","major_comments":[{"comment":"The substitution f'(x) ≈ (g'(f(x)))^+ underlying Eq. (31) is already contradicted by the paper's own linear analysis. In Lemma 1 the global optimum of the FIVE loss is w* = √λ and v* = √λ/(σ²+λ). Then v* w* = λ/(σ²+λ) ≠ 1 for finite σ, so v* is not the pseudoinverse of w*. The resulting posterior q(z|x) = N(vx, σ²v²) has variance σ²λ/(σ²+λ)², while the Laplace posterior variance from Eq. (12) or (48) is σ²/(σ²+λ). These differ by the factor λ/(σ²+λ). Thus Eq. (31) holds only in the limits σ→0 or λ→∞, and the statement that q(z|x) is the Laplace posterior is unsupported in the finite-σ regime of the experiments.","section":"Section 4.3 / Appendix A.3, Eq. (31) and Lemma 1"},{"comment":"The theorem statement overclaims exact recovery of the data distribution. The proof establishes W*W*^T = Σx and hence g#p(z) = N(0, Σx), but the marginal of the generative model is p(x) = ∫ p(x|z)p(z) dz = N(0, Σx + σ²I), which equals N(0, Σx) only when σ→0. The proof itself acknowledges this ('which is also equal to p(x) in the zero σ limit'), so Theorem 3 as stated for the FIVE loss with σ>0 is false. It should be restated as a σ→0 limit or as a statement about the decoder's pushforward, not about the model marginal.","section":"Section 4.3, Theorem 3"},{"comment":"The objective actually optimized is not the ELBO of Section 4.2. Eq. (68) replaces the log-determinant term of the KL divergence with a stop-gradient trace surrogate, and the main text does not state that the training loss is this surrogate rather than the ELBO (34). In addition, the derivation in A.4 moves from Eq. (93) to Eq. (94) as though (g'(f(x)))^+^T g'(f(x))^T = I; for an injective decoder with d < n the product equals (g'(f(x)) g'(f(x))^+)^T, the transpose of a rank-d column-space projection, not the identity. The gradient formula therefore requires an additional assumption or correction. These issues matter because the paper's central claim is that FIVE optimizes a valid variational objective with a full-covariance posterior.","section":"Appendix A.3, Eq. (68); Appendix A.4, Eq. (93)-(94)"},{"comment":"The empirical conclusion is weaker than the abstract's claim that full covariance 'substantially improves model likelihood.' On CIFAR-10 with the larger network, FIVE (2684.44±9.33) is not statistically distinguishable from the standard VAE (2680.36±13.86), and on MNIST with the larger network, FIF (1116.31±27.29) has numerically higher mean than FIVE (1115.17±7.67). The gains are consistent on small networks but not uniformly significant. This does not undermine the empirical contribution, but the summary claims should be calibrated to the table.","section":"Section 5, Table 1 and Abstract"}],"minor_comments":[{"comment":"There is a duplicated phrase: 'enrich the latent structure enrich the latent structure'.","section":"Section 1"},{"comment":"Theorem numbering is duplicated between the main text and appendix (Theorem 4 restates Theorem 1, Theorem 5 restates Theorem 2, Theorem 6 restates Theorem 3); renumber the appendix theorems.","section":"Appendix A"},{"comment":"The displayed KL expression in Eq. (35) has an unbalanced parenthesis, and the stop-gradient operator SG should be defined before its first use in Eq. (27).","section":"Eq. (35) and Eq. (27)"},{"comment":"The Limitations section lists dataset scope but does not mention that the training objective is a stop-gradient surrogate; this should be disclosed as a limitation.","section":"Limitations section"},{"comment":"The claim that computational cost is 'comparable to diagonal VAEs' should be quantified; the Hutchinson trace estimate and Jacobian-vector products add overhead per iteration.","section":"Section 4 / Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's own linear analysis is sufficient to refute the central identity (31); this is not a matter of disagreement with an external consensus. I recommend that the revision either prove a corrected equivalence (e.g., in an appropriate limit with a bound on the error) or explicitly reframe FIVE as a heuristic model whose empirical likelihood gains are the primary contribution. The empirical evaluation is in good shape and can remain the core if the theory is appropriately demoted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper has a genuinely interesting idea: define a VAE posterior whose covariance is sigma^2 f'(x)f'(x)^T, read off from the encoder Jacobian. That is new, and the differential-geometric argument that diagonal posteriors cannot represent curved manifolds (the involutivity obstruction) is solid and worth keeping. The empirical results are also consistently in FIVE's favor across MNIST and CIFAR10, which suggests something real is happening.\n\nBut the central derivation does not hold up under scrutiny. The key step, Eq. (31), replaces the decoder pseudoinverse (g'(f(x)))^+ with f'(x). The paper's own linear analysis contradicts this. In Lemma 1, the optimum has f' = sqrt(lambda)/(sigma^2+lambda) and g' = sqrt(lambda), so f'g' = lambda/(sigma^2+lambda) < 1. The variational covariance sigma^2 f'^2 then differs from the true Laplace posterior variance sigma^2/(sigma^2+lambda) by a factor lambda/(sigma^2+lambda). Equality only holds in the sigma->0 or lambda->infinity limits, not in the finite-sigma regime used in the experiments. So q(z|x) is not the Laplace posterior even in the exactly linear case.\n\nOn top of that, the loss actually optimized in the appendix (Eq. 68) is not the ELBO stated in Section 4.2. It replaces the log-det term with a stop-gradient trace surrogate. The paper doesn't reconcile these two objectives, so the claim that FIVE is a variational method with a principled full-covariance posterior is not supported. Theorem 3 in the main text overclaims exact recovery; the appendix only proves a sigma->0 limit.\n\nThe paper lacks code, which makes the empirical results hard to check. Still, the geometric analysis alone is a contribution, and the method is novel enough that it deserves a serious referee. My recommendation: send it to review, but the authors should be required to correct the theorem, explain the surrogate loss, and release code. If the empirical results survive that, the method may be useful even if the theoretical framing needs to be rewritten.","headline":"Neat idea, but the central derivation breaks: the claimed full-covariance posterior is not the Laplace posterior even in the linear case, and the optimized loss is a stop-gradient surrogate, not the stated ELBO.","tokens_in":15503,"tokens_out":2291,"would_cite":false,"duration_ms":23355,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a regularized free-form injective flow can serve as a variational autoencoder with a full-covariance Gaussian posterior, and that this posterior costs about as much as a diagonal one while improving model likelihood.","keywords":["variational autoencoder","full covariance","free-form injective flow","Laplace approximation","pull-back metric","involutivity","posterior collapse","image likelihood"],"falsifier":"Take a decoder whose manifold has curvature radius comparable to or smaller than the observation noise $\\sigma$ (for example $z = x^2 + y^2$ with $\\sigma$ not very small) and compute the exact posterior $p(z|x) \\propto \\exp(-u(x,z))$ by dense integration in the low-dimensional latent space; if the KL divergence between $p(z|x)$ and $\\mathcal{N}(f(x), \\sigma^2 f'(x)f'(x)^\\top)$ is large, or if FIVE's learned decoder no longer matches the true manifold's parameterization, the local-linearity premise fails.","tokens_in":14461,"feed_emoji":"📐","tokens_out":9572,"duration_ms":91527,"temperature":0.7,"pith_summary":"The paper claims that the standard diagonal-Gaussian posterior used in variational autoencoders is not just a practical convenience but a genuine geometric limitation: for data manifolds whose induced metric has non-involutive eigen-line fields, no reparameterization of the decoder can make the pull-back metric diagonal. To get past this, it introduces FIVE, a VAE whose posterior is a free-form injective flow regularized so that $q(z|x) = \\mathcal{N}(f(x), \\sigma^2 f'(x)f'(x)^\\top)$. This posterior is a full Gaussian covariance, but it is represented implicitly through the encoder Jacobian, so sampling and the KL term cost about the same as a diagonal VAE. If correct, this makes rich posterior covariances practical at scale and explains why diagonal VAEs distort curved manifolds. On MNIST and CIFAR-10, the paper reports that FIVE matches or surpasses the test log-likelihood of standard VAEs, full-covariance VAEs, and standalone flows.","feed_headline":"Full-covariance VAE posteriors at diagonal-Gaussian cost","feed_subtitle":"FIVE's posterior covariance comes free from encoder Jacobians, beating VAEs and flows on image likelihoods.","key_machinery":"The load-bearing object is the variational posterior $q(z|x) = \\mathcal{N}(f(x), \\sigma^2 f'(x)f'(x)^\\top)$, a full-covariance Gaussian whose covariance is written as the outer product of the encoder Jacobian with itself, so it never needs to be materialized. Its justification is a Laplace approximation: with decoder $g$ and noise $\\sigma$, the posterior Hessian obeys $H(x)^{-1} \\approx \\sigma^2 f'(x)f'(x)^\\top$. The supporting machinery is the pull-back metric $G = g'(z)^\\top g'(z)$ from differential geometry, used to prove the diagonal-covariance obstruction through the Frobenius involutivity condition, plus a stochastic trace estimator and a stop-gradient trace identity that compute gradients of the log-determinant term without evaluating determinants. Sampling from the posterior is just a reparameterized draw $z = f(x) + \\sigma f'(x)v$ with $v \\sim \\mathcal{N}(0,I)$, and the KL divergence to the standard normal prior is evaluated using $\\operatorname{tr}(\\sigma^2 f'(x)f'(x)^\\top)$ and the approximate log-determinant gradient.","core_discovery":"As the paper states it, a VAE that forces $q(z|x)$ to be diagonal Gaussian will at optimum force the decoder's pull-back metric $g'(z)^\\top g'(z)$ to be approximately diagonal (Theorem 1), because the true posterior $p(z|x)$ approximates $\\mathcal{N}(f(x), H(x)^{-1})$ with $H = I + \\sigma^{-2} g'(f(x))^\\top g'(f(x))$. Orthogonal coordinate systems with diagonal metric exist only when the eigen-line fields of this metric are involutive (Theorem 2), and in dimension three or higher curved manifolds can fail this test. FIVE sidesteps the constraint by defining the variational posterior directly from the encoder Jacobian, $q(z|x) = \\mathcal{N}(f(x), \\sigma^2 f'(x)f'(x)^\\top)$, which is the Laplace-approximation form of $H(x)^{-1}$. The paper proves that in the linear case the two opposing Jacobian substitutions used in the derivation are consistent and the model recovers the data covariance exactly (Theorem 3). The empirical claim is that this full-covariance posterior, with no explicit covariance matrix ever formed, matches or exceeds the likelihoods of diagonal VAEs, learned full-covariance VAEs, and standalone free-form injective flows on MNIST and CIFAR-10.","pith_inferences":["If the Laplace assumption holds for real image manifolds, the same Jacobian-covariance construction could be applied to other implicit or explicit encoder-decoder architectures, not only FIF-trained ones, whenever the encoder Jacobian is available.","The geometric obstruction suggests a testable diagnostic: for a trained decoder, compute the commutators $[e_i, e_j]$ of the metric's eigen-frame; datasets whose manifolds violate involutivity should show a systematic gap between diagonal-VAE and FIVE likelihood.","Theorem 3's linear recovery result hints that FIVE may behave like a probabilistic autoencoder that finds a principal-subspace structure; a synthetic experiment with data drawn from a known low-rank Gaussian could directly check whether the learned encoder and decoder match the top-$d$ eigen-directions.","At larger $\\sigma$ or sharper curvature, $q(z|x)$ may remain Gaussian while $p(z|x)$ becomes non-Gaussian; a two-dimensional toy with exactly computed posteriors could measure where the approximation degrades."],"forward_implications":["A VAE can carry a full-covariance Gaussian posterior with sampling cost equal to one extra Jacobian-vector product per latent draw, so the diagonal-Gaussian restriction is no longer forced by computation.","Posterior collapse that appears in large-latent diagonal and explicitly full-covariance VAEs is reported to be much weaker for FIVE, which inherits the collapse resistance of free-form injective flows.","The involutivity criterion gives a principled way to identify data manifolds for which diagonal posteriors are provably insufficient: check whether the eigen-line fields of the decoder metric can be integrated.","Combining the FIF training objective with a KL-style regularizer yields a valid ELBO whose approximate KL term needs no determinant computation.","On MNIST and CIFAR-10, with both small and larger networks, FIVE reports test log-likelihoods that match or exceed standard VAE, FC-VAE, and FIF baselines."],"supporting_citations":[{"why":"Introduces free-form injective flows, whose training objective and trace-gradient estimator FIVE regularizes into a VAE.","marker":"[Sorrenson et al., 2024]"},{"why":"Presents the free-form flow variant that lifts architectural constraints, the method from which the FIVE posterior is derived.","marker":"[Draxler et al., 2024]"},{"why":"Establishes injective normalizing flows for manifold learning and density estimation, the modeling paradigm FIVE extends.","marker":"[Brehmer and Cranmer, 2020]"},{"why":"Introduces rectangular injective flows, providing the injective-flow posterior viewpoint that FIVE builds on.","marker":"[Caterini et al., 2021]"},{"why":"Supplies the Frobenius integrability theorem used to prove the involutivity obstruction to orthogonal coordinates (Theorem 2).","marker":"[Lee, 2013]"},{"why":"Defines the VLAE experimental setup and benchmark comparisons that the MNIST/CIFAR-10 experiments follow.","marker":"[Park et al., 2019]"},{"why":"The original VAE formulation with amortized inference and the diagonal-Gaussian baseline FIVE is compared against.","marker":"[Kingma and Welling, 2014]"},{"why":"The full-covariance Gaussian VAE baseline (FC-VAE) whose Cholesky-style posterior cost FIVE avoids.","marker":"[Kingma et al., 2019]"},{"why":"Provides the learnable observation-noise ($\\sigma$-VAE) calibration used by all models in the experiments.","marker":"[Rybkin et al., 2021]"},{"why":"Supplies the stochastic trace estimator that makes the free-form flow gradient tractable.","marker":"[Hutchinson, 1989]"}],"fun_headline_variants":["VAEs break diagonal Gaussian limit with free-form injective flows","Full covariance in VAEs at diagonal cost via free-form flows","Free-form injective flows unlock full-covariance VAE posteriors","Diagonal VAE no more: free-form flows give full covariance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument rests on the Laplace/local-linearity assumption that the decoder $g$ is nearly linear at the scale of the noise $\\sigma$, so the true posterior $p(z|x)$ is well approximated by a Gaussian whose inverse covariance is $\\sigma^2 f'(x)f'(x)^\\top$; the paper's own Limitations section notes this may break down when the data manifold curves sharply on that scale.","fun_headline_variants_meta":{"raw":{"variants":["VAEs break diagonal Gaussian limit with free-form injective flows","Full covariance in VAEs at diagonal cost via free-form flows","Free-form injective flows unlock full-covariance VAE posteriors","Diagonal VAE no more: free-form flows give full covariance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001028,"raw_usage":{"total_tokens":4343,"prompt_tokens":966,"completion_tokens":3377,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":3304}},"tokens_in":582,"tokens_out":3377,"duration_ms":24776,"temperature":1.0,"reasoning_tokens":3304,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:40:26.525520+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a decoder whose manifold has curvature radius comparable to or smaller than the observation noise $\\sigma$ (for example $z = x^2 + y^2$ with $\\sigma$ not very small) and compute the exact posterior $p(z|x) \\propto \\exp(-u(x,z))$ by dense integration in the low-dimensional latent space; if the KL divergence between $p(z|x)$ and $\\mathcal{N}(f(x), \\sigma^2 f'(x)f'(x)^\\top)$ is large, or if FIVE's learned decoder no longer matches the true manifold's parameterization, the local-linearity premise fails.","supporting_citations":[{"cited_title":"Rectangular flows for manifold learning","cited_arxiv_id":null,"evidence_quote":"Introduces rectangular injective flows, providing the injective-flow posterior viewpoint that FIVE builds on."},{"cited_title":"Variational Laplace autoencoders","cited_arxiv_id":null,"evidence_quote":"Defines the VLAE experimental setup and benchmark comparisons that the MNIST/CIFAR-10 experiments follow."},{"cited_title":"Auto-encoding variational Bayes","cited_arxiv_id":null,"evidence_quote":"The original VAE formulation with amortized inference and the diagonal-Gaussian baseline FIVE is compared against."},{"cited_title":"An introduction to variational autoencoders","cited_arxiv_id":null,"evidence_quote":"The full-covariance Gaussian VAE baseline (FC-VAE) whose Cholesky-style posterior cost FIVE avoids."},{"cited_title":"Simple and effective vae training with calibrated decoders","cited_arxiv_id":null,"evidence_quote":"Provides the learnable observation-noise ($\\sigma$-VAE) calibration used by all models in the experiments."},{"cited_title":"A stochastic estimator of the trace of the influence matrix for Laplacian smoothing splines","cited_arxiv_id":null,"evidence_quote":"Supplies the stochastic trace estimator that makes the free-form flow gradient tractable."}],"review_version":1}