Increasing the Generalisation Capacity of Conditional VAEs

Alexej Klushyn; Botond Cseke; Justin Bayer; Nutan Chen; Patrick van der Smagt

REVIEW 3 major objections 4 minor 21 references

Increasing the Generalisation Capacity of Conditional VAEs

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A conditional VAE whose decoder depends only on the latent variable, paired with a learned multimodal prior, learns one-to-many mappings that generalise to unseen conditions.

desk verdict CVAE variant with decoder-only likelihood and a learnable mixture prior: a plausible idea whose headline claim is undercut by a diversity metric that never checks whether generated targets actually match the condition. read the letter →

arxiv 1908.08750 v2 pith:TYK4LZ7T submitted 2019-08-23 stat.ML cs.LG

Alexej Klushyn , Nutan Chen , Botond Cseke , Justin Bayer , Patrick van der Smagt This is my paper · ORCID

classification stat.MLcs.LG

keywords structuredpredictionconditionalvariationalautoencoderslatentvariablemodelsone-to-manymappingmultimodalpriorgeneralisationcapacityEmpiricalBayesVamp

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a conditional variational autoencoder can generalise much better on one-to-many structured-prediction tasks if two changes are made together. First, the decoder is forced to depend only on the latent variable, $p_\theta(y|x,z)=p_\theta(y|z)$, so that the latent code must carry all information needed to produce a target. Second, the prior over the latent code is made a flexible multimodal mixture, the CDV prior, because a unimodal prior cannot represent genuine gaps between output modes. The authors report that on modified MNIST and Fashion-MNIST the model learns several distinct target classes from training data with one target per condition, and on the Cornell Robot Grasping dataset it produces more high-confidence grasps for unseen objects than a standard CVAE. If correct, this is a practical route to making latent-variable models useful in applications where one input has many valid outputs.

What carries the argument

The load-bearing object is the pair formed by the modified conditional latent-variable model $p_\theta(y|x)=\int p_\theta(y|z)p_\theta(z|x)\,dz$ and the CDV prior $p_\pi(z|x)=\frac{1}{K}\sum_{k=1}^K q_\varphi(z|x,\mu_\theta(\tilde z_k(x)))$. The first is a latent-variable model whose likelihood ignores $x$, forcing the latent code to store all target-relevant structure. The second is a uniform mixture of $K$ encoder posteriors evaluated at pseudo-latent variables $\tilde z_k(x)$ mapped through the decoder mean $\mu_\theta$; because it reuses the same encoder and decoder that define the posterior, every mixture component stays on the manifold of encoded data, avoiding the outlier drift that the authors attribute to a classical Gaussian mixture prior. The companion argument uses the magnification factor $\mathrm{MF}=\sqrt{\det(J^T J)}$ to show that a continuous decoder with a unimodal prior cannot create zero-density regions between modes without infinite Jacobians, which is why the multimodal prior is needed.

What would settle it

Construct a synthetic one-to-many dataset where the target has two parts: a mode determined by a hidden categorical variable recoverable from $z$, and a continuous perturbation determined by $x$ itself. Train the CDV-CVAE and a standard CVAE on the same data. If the CDV-CVAE fails to reproduce the $x$-dependent perturbation on held-out conditions while the standard CVAE succeeds, the conditional-independence assumption is violated and the claimed generalisation gain is limited to datasets whose target information is fully summarised by $z$.

Watch

Extended reading notes

Core claim

The paper's central claim is that the usual CVAE, whose decoder sees both the condition $x$ and a latent code $z$, has no incentive to store information in $z$; everything useful can be read directly from $x$. The authors therefore impose the conditional independence $x \perp y \mid z$, replacing $p_\theta(y|x,z)$ with $p_\theta(y|z)$ and defining the conditional latent-variable model as $p_\theta(y|x)=\int p_\theta(y|z)p_\theta(z|x)\,dz$. This forces the latent variable to carry all target-relevant information. To make that feasible, the prior $p_\theta(z|x)$ must be multimodal: a unimodal Gaussian prior cannot support near-zero-density regions between modes except through infinitely large decoder Jacobians. The paper proposes the conditional decoder-based Vamp (CDV) prior, $p_\pi(z|x)=\frac{1}{K}\sum_{k=1}^K q_\varphi(z|x,\mu_\theta(\tilde z_k(x)))$, a mixture of encoder posteriors evaluated at learned pseudo-latent variables passed through the decoder mean. On the paper's evidence, this combination yields more varied and realistic generations for unseen conditions than a standard CVAE or a Gaussian-mixture prior.

Load-bearing premise

The whole construction rests on the assumption that the condition $x$ and the target $y$ become conditionally independent once a latent sample $z$ is known, so the decoder can safely ignore $x$; if useful information flows directly from $x$ to $y$ without being representable as a sampled $z$, that information is thrown away.

Editorial extensions

If this is right

If the central claim is correct, CVAE-style models can learn genuinely multimodal predictive distributions even when the training set contains only one target per condition, because the latent code is forced to organise the space of possible outputs.
The CDV prior should be preferred over Gaussian-mixture priors in high-dimensional latent spaces, since it avoids mixture components that drift away from the encoded-data manifold and produce poor generations.
Generation becomes simpler and the latent space more interpretable: at test time one samples $z\sim p_\theta(z|x)$ and decodes with a network that never sees the condition $x$.
On the robot-grasping benchmark, the model yields a higher fraction of high-confidence grasps for unseen objects (29% versus 22% of samples above a 0.99 discrimination score), supporting practical use in one-to-many control.
The reported ELBO values on the synthetic one-to-many task improve with the CDV prior ($-0.518$ versus $-1.12$ for the standard CVAE), indicating a better fit to multimodal targets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not test the boundary of its conditional-independence assumption; a direct experiment with an $x$-dependent continuous target component would reveal how much information is lost when the decoder never sees $x$.
The CDV parameterisation suggests a general design principle—reuse the inference network at learned pseudo-latent inputs to keep a prior on the data manifold—which could transfer to sequence or graph latent-variable models.
The magnification-factor argument implies that any continuous-decoder latent model with a unimodal prior will struggle to separate output modes; testing this with a deliberately non-smooth decoder would clarify whether the problem is the prior or the continuity.
A calibration check on held-out one-to-many data—how well the model's probability mass matches observed mode frequencies—would separate genuine generalisation from simply broader sampling.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The contribution is modest but real. The paper proposes a CVAE variant where the decoder depends only on the latent variable, paired with a learnable mixture prior (the CDV prior) whose components are evaluated at pseudo-latents produced by a small network. Removing x from the decoder forces the latent to carry condition-specific information, and the prior is designed to avoid the degeneracy of a plain mixture-of-Gaussians. The toy experiment (Fig. 2) shows the method can represent zero-density regions between modes where a unimodal prior fails; that part is convincing. The CDV prior also looks genuinely new relative to the cited VampPrior work, and the authors give a reasonable story for why pseudo-latents beat pseudo-inputs.

The soft spots are exactly where the stress-test note lands. The central claim — higher generalisation capacity — is measured by variety of classes in generated samples (Fig. 5). That metric counts label diversity, not whether a generated upper two-thirds is a valid completion of the specific lower third used as the condition. Since Eq. (3) deliberately removes x from the likelihood, the only channel from condition to output is the prior pθ(z|x). If that channel is weak, a model can get a high diversity score by spreading prior mass over several common classes while producing targets poorly matched to x. The paper asserts x⊥y|z in Sec. 2.2 but never tests whether a 32-dimensional z actually captures all x→y information on these splits. The grasping result (29% vs 22% discriminator pass) has no error bars, no significance test, and no conditional-compatibility control; 10 samples per test condition is small. The variety metric also excludes unclassifiable samples, which biases comparisons toward models that generate typical-class, possibly blurry, outputs.

That said, the paper is not sloppy. The variational derivation is standard, the low-density argument is clearly motivated, and the failure mode of CMoG priors is demonstrated with the nearest-neighbour distance plots (Fig. 6). The main issues are evaluation design, not mathematical error. I would treat it as a working paper, not a settled result. It deserves a serious referee. Send it out with a request for major revision: add per-condition fidelity checks (e.g., verify that the generated target class is compatible with the condition class), include error bars and significance tests, and discuss the conditional independence assumption explicitly. As is, I would not rely on its headline claim.

Referee Report

3 major / 4 minor

Summary. The paper addresses one-to-many structured prediction with conditional VAEs. It proposes a modified conditional latent variable model in which the likelihood depends only on the latent variable, pθ(y|x,z)=pθ(y|z), justified by the assumption that z determines y completely and hence x⊥y|z. To make this viable, the paper introduces the CDV prior, a conditional decoder-based mixture prior whose components are encoder posteriors evaluated at decoded pseudo latent variables. The model is tested on a synthetic toy dataset, on a modified MNIST/Fashion-MNIST task where the lower third of an image is the condition and the upper two-thirds is the target, and on the Cornell Robot Grasping dataset. The authors claim that the CDV-CVAE has significantly higher generalisation capacity, evidenced by greater variety of generated targets and a higher discriminator pass rate for grasping poses.

Significance. If the central claim were fully established, the paper would make a useful contribution: the derivation of Eq. (3) is a clean way to force informative latents, and the CDV prior is a plausible, parameter-efficient mechanism for representing multimodal posteriors. The low-density argument in Sec. 2.3 is explained carefully and supported by the synthetic toy experiment, which is a strength. The paper also introduces a novel prior with a clear relationship to the VampPrior and gives a reasonable motivation for pseudo latent variables instead of pseudo inputs. However, the headline claim of 'higher generalisation capacity' is not yet substantiated by the evaluation, because the main metric measures label diversity rather than condition compatibility, and the conditional independence assumption underlying Eq. (3) is not validated.

major comments (3)

[Sec. 4.2, Fig. 5] The variety metric used to support the central claim counts, for each test condition, how many different classes are assigned by an MNIST/Fashion-MNIST classifier to 10 generated samples, and explicitly excludes samples that cannot be clearly assigned to a class. This measures label diversity, not whether a generated upper two-thirds is a valid completion of the specific lower third used as the condition. Because the proposed model removes x from the likelihood in Eq. (3), a model that uses the conditioning information weakly could produce diverse but condition-incompatible targets and still score well on this metric. The paper needs a condition-fidelity metric, such as evaluating whether generated completions are consistent with the given condition, and should report the proportion of generated samples excluded as unclassifiable, since excluding them biases the comparison.
[Sec. 2.2, Eq. (3)] The step from 'z determines y completely' to x⊥y|z and pθ(y|x,z)=pθ(y|z) is load-bearing, but the assumption is not validated. On the modified MNIST/Fashion-MNIST tasks, the claim is that a 32-dimensional latent variable captures all information in the lower third of the image that is relevant to generating the upper two-thirds. This is a strong assumption about the data-generation process, and the experiments do not test it. Without a check, e.g., comparing conditional likelihoods or generation accuracy against a model that retains x in the decoder, the reported gains could be an artifact of discarding condition information rather than a genuine increase in generalisation capacity.
[Sec. 4.3] The grasping experiment reports that 29% of CDV-CVAE generated poses exceed a discriminator score of 0.99, versus 22% for the CVAE, but provides no error bars, significance test, or information about how the threshold and the number of generated samples per condition affect the result. Since poses are filtered by the discriminator before computing the pass rate, the comparison depends on the number of samples drawn and on the threshold choice. The paper should report means and variances across conditions or random seeds, and should include a condition-compatibility control, for example measuring whether generated poses are plausible for the specific test object, not merely globally plausible.

minor comments (4)

[Sec. 4.2] The paper uses the phrase 'significantly larger' when describing the variety results, but no significance tests or confidence intervals are reported; the claim should be supported by statistical measures or reworded.
[Sec. 2.4, Eq. (4)] The notation for the CDV prior switches from pθ to pπ without a clear explanation of how π={ψ,θ,φ} relates to the earlier parameterization; a brief clarification would improve readability.
[Sec. 4.2, Fig. 6] The comparison between the CMoG prior, conditional VampPrior, and CDV prior is presented only visually and through qualitative statements; quantitative values such as distances or ELBOs for each model would make the comparison more convincing.
[Sec. 2.3] The derivation of the change-of-variables formula uses pθ(g(z)) = 1/√det(J^T J) pθ(z); for the square Jacobian case this is the standard |det J|^{-1} factor, and the meaning of the transposed form should be clarified if non-square Jacobians are intended.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the model change is an explicit design choice and the evaluations are external; the diversity metric is goal-aligned but not a fitted input renamed as prediction.

full rationale

The paper's central reformulation is Eq. 3, derived from an explicitly stated modelling assumption ("z determines y completely, i.e. the mutual information I(x;y|z)=0"), not from any prior fitted quantity. That assumption is an unvalidated modelling choice, but it is not circular: the subsequent model pθ(y|x)=∫pθ(y|z)pθ(z|x)dz does not presuppose the experimental conclusion that generalisation is increased. The CDV prior (Eq. 4) is a new parameterisation of the VampPrior using pseudo-latents; it is motivated by the aggregated-posterior argument and evaluated by ELBO comparisons. No parameter is fitted to the evaluation metrics: the MNIST/Fashion-MNIST variety measure uses an externally trained classifier and samples from trained models, and the grasping experiment uses an external discriminator. The closest concern is that "variety of classes per condition" is a goal-aligned metric for a model explicitly designed to produce multimodal outputs, and that the diversity count does not by itself verify condition fidelity; but that is a validity/correctness limitation, not a circular reduction of the derivation to its inputs. There are no load-bearing self-citations and no uniqueness theorems are imported from the authors' prior work.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central modelling choice is the conditional independence assumption in Eq. (3), which is introduced ad hoc to motivate the new decoder. The optimisation relies on standard variational inference. The experimental claims depend on the chosen number of mixture components, latent dimensions, KL annealing, and the grasping discriminator threshold, none of which are varied or given uncertainties.

free parameters (4)

Number of prior mixture components K = 32 (MNIST/Fashion-MNIST), unspecified for other experiments
The CDV and CMoG priors are mixtures of K components; K is chosen by hand and the paper's Fig. 6 analysis uses K=32. The diversity and quality of generations likely depend on this choice, but no sensitivity study is provided.
Latent dimensionality = 2 (toy), 32 (MNIST/Fashion-MNIST), 16 (grasping)
Latent dimensions are set per experiment, and the method's capacity to route all condition information through z depends on this choice.
Linear KL annealing schedule = first epoch (not further specified)
The authors state the CDV-CVAE is sensitive to over-regularisation and apply linear annealing for the first epoch (Sec. 4), which affects the learned latent representation.
Grasping discriminator threshold = 0.99
The comparison 29% vs 22% depends on the chosen threshold; no analysis of threshold sensitivity is given.

assumptions (4)

ad hoc to paper Conditional independence x⊥y|z when z determines y completely
Introduced in Sec. 2.2 to justify the decoder pθ(y|z) in Eq. (3). It is the core modeling choice the paper is built on and is not proven or empirically validated.
standard math Amortized variational inference with the ELBO is a valid training objective
The paper maximizes the ELBO in Eq. (2), relying on the standard derivation of variational lower bounds.
standard math The change-of-variables and magnification factor argument (MF=sqrt(det(J^T J)))
Used in Sec. 2.3 to argue unimodal priors cannot represent zero-density regions; requires the decoder map g to be bijective and differentiable on the relevant domain.
domain assumption The decoder and encoder neural networks are expressive enough to represent the required conditional distributions
The improved model only works if a low-dimensional z can encode all x-relevant information and the decoder can map the multimodal prior to multimodal targets. This is assumed throughout the experiments.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Increasing the Generalisation Capacity of Conditional VAEs." pith.science (2026). https://pith.science/paper/TYK4LZ7T

@misc{pith2026190808750,
  author       = {Pith},
  title        = {Pith review of: Increasing the Generalisation Capacity of Conditional VAEs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TYK4LZ7T}},
  note         = {Machine review of arXiv:1908.08750}
}

read the original abstract

We address the problem of one-to-many mappings in supervised learning, where a single instance has many different solutions of possibly equal cost. The framework of conditional variational autoencoders describes a class of methods to tackle such structured-prediction tasks by means of latent variables. We propose to incentivise informative latent representations for increasing the generalisation capacity of conditional variational autoencoders. To this end, we modify the latent variable model by defining the likelihood as a function of the latent variable only and introduce an expressive multimodal prior to enable the model for capturing semantically meaningful features of the data. To validate our approach, we train our model on the Cornell Robot Grasping dataset, and modified versions of MNIST and Fashion-MNIST obtaining results that show a significantly higher generalisation capability.

Figures

Figures reproduced from arXiv: 1908.08750 by the authors.

**Figure 2.** Synthetic toy dataset (a) of one-dimensional one-to-many mappings. The [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Samples from the CDV prior depending on the condition [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Modified MNIST and Fashion-MNIST: the goal is to validate whether [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: Variety of generated targets: for each condition in the test dataset, 10 [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

**Figure 6.** Figure 6: The plots show the number of nearest neighbours (encoded MNIST data [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]

**Figure 7.** Figure 7: Cornell Robot Grasping dataset: (a) objects (conditions) with proposed [PITH_FULL_IMAGE:figures/full_fig_p011_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 20 canonical work pages

[1]

ICML (2018)

Alemi, A.A., Poole, B., Fischer, I., Dillon, J.V., Saurous, R.A., Murphy, K.: Fixing a broken ELBO. ICML (2018)

work page 2018
[2]

Proceedings Workshop on Self-Organizing Maps (1997)

Bishop, C.M., Svens’ en, M., Williams, C.K.I.: Magniﬁcation factors for the SOM and GTM algorithms. Proceedings Workshop on Self-Organizing Maps (1997)

work page 1997
[3]

CoNLL (2016)

Bowman, S.R., Vilnis, L., Vinyals, O., Dai, A.M., Jozefowicz, R., Bengio, S.: Gen- erating sentences from a continuous space. CoNLL (2016)

work page 2016
[4]

CoRR (2016)

Chen, X., Kingma, D.P., Salimans, T., Duan, Y., Dhariwal, P., Schulman, J., Sutskever, I., Abbeel, P.: Variational Lossy Autoencoder. CoRR (2016)

work page 2016
[5]

ICLR (2017)

Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., Lerchner, A.: beta-VAE: Learning basic visual concepts with a constrained variational framework. ICLR (2017)

work page 2017
[6]

CoRR (2013)

Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. CoRR (2013)

work page 2013
[7]

NeurIPS (2012)

Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classiﬁcation with deep con- volutional neural networks. NeurIPS (2012)

work page 2012
[8]

Proceedings of the IEEE (1998)

LeCun, Y., Bottou, L., Bengio, Y., Haﬀner, P., et al.: Gradient-based learning applied to document recognition. Proceedings of the IEEE (1998)

work page 1998

Show all 21 references

[9]

The International Journal of Robotics Research (2015)

Lenz, I., Lee, H., Saxena, A.: Deep learning for detecting robotic grasps. The International Journal of Robotics Research (2015)

2015
[10]

ICLR (2017)

Nalisnick, E., Smyth, P.: Stick-breaking variational autoencoders. ICLR (2017)

2017
[11]

ICRA (2016)

Pinto, L., Gupta, A.: Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. ICRA (2016)

2016
[12]

ICML (2014)

Rezende, D.J., Mohamed, S., Wierstra, D.: Stochastic backpropagation and ap- proximate inference in deep generative models. ICML (2014)

2014
[13]

ICLR (2015)

Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. ICLR (2015)

2015
[14]

NeurIPS (2015)

Sohn, K., Lee, H., Yan, X.: Learning structured output representation using deep conditional generative models. NeurIPS (2015)

2015
[15]

NeurIPS (2016)

Sønderby, C.K., Raiko, T., Maaløe, L., Sønderby, S.K., Winther, O.: Ladder vari- ational autoencoders. NeurIPS (2016)

2016
[16]

NeurIPS (2013)

Tang, Y., Salakhutdinov, R.R.: Learning Stochastic Feedforward Neural Networks. NeurIPS (2013)

2013
[17]

AISTATS (2018)

Tomczak, J., Welling, M.: VAE with a VampPrior. AISTATS (2018)

2018
[18]

IEEE Robotics and Automation Letters (2017)

Veres, M., Moussa, M., Taylor, G.W.: Modeling grasp motor imagery through deep conditional generative models. IEEE Robotics and Automation Letters (2017)

2017
[19]

ECCV (2016)

Walker, J., Doersch, C., Gupta, A., Hebert, M.: An uncertain future: Forecasting from static images using variational autoencoders. ECCV (2016)

2016
[20]

arXiv:1708.07747 (2017)

Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: a novel image dataset for bench- marking machine learning algorithms. arXiv:1708.07747 (2017)

2017 arXiv
[21]

ECCV (2016)

Yan, X., Yang, J., Sohn, K., Lee, H.: Attribute2image: Conditional image genera- tion from visual attributes. ECCV (2016)

2016

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

ICML (2018)

Alemi, A.A., Poole, B., Fischer, I., Dillon, J.V., Saurous, R.A., Murphy, K.: Fixing a broken ELBO. ICML (2018)

work page 2018

[2] [2]

Proceedings Workshop on Self-Organizing Maps (1997)

Bishop, C.M., Svens’ en, M., Williams, C.K.I.: Magniﬁcation factors for the SOM and GTM algorithms. Proceedings Workshop on Self-Organizing Maps (1997)

work page 1997

[3] [3]

CoNLL (2016)

Bowman, S.R., Vilnis, L., Vinyals, O., Dai, A.M., Jozefowicz, R., Bengio, S.: Gen- erating sentences from a continuous space. CoNLL (2016)

work page 2016

[4] [4]

CoRR (2016)

Chen, X., Kingma, D.P., Salimans, T., Duan, Y., Dhariwal, P., Schulman, J., Sutskever, I., Abbeel, P.: Variational Lossy Autoencoder. CoRR (2016)

work page 2016

[5] [5]

ICLR (2017)

Higgins, I., Matthey, L., Pal, A., Burgess, C., Glorot, X., Botvinick, M., Mohamed, S., Lerchner, A.: beta-VAE: Learning basic visual concepts with a constrained variational framework. ICLR (2017)

work page 2017

[6] [6]

CoRR (2013)

Kingma, D.P., Welling, M.: Auto-encoding variational Bayes. CoRR (2013)

work page 2013

[7] [7]

NeurIPS (2012)

Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classiﬁcation with deep con- volutional neural networks. NeurIPS (2012)

work page 2012

[8] [8]

Proceedings of the IEEE (1998)

LeCun, Y., Bottou, L., Bengio, Y., Haﬀner, P., et al.: Gradient-based learning applied to document recognition. Proceedings of the IEEE (1998)

work page 1998