REVIEW 3 major objections 5 minor 1 cited by
Implicit Deep Latent Variable Models for Text Generation
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper claims that replacing Gaussian variational posteriors with neural-network-sampled implicit latent codes gives text VAEs more expressive representations, mitigates posterior collapse, and improves language modeling, style…
desk verdict A credible combination of known ideas for text VAEs, but the headline language-modeling gains rest on a KL estimator that is off by a constant and never validated against ground truth. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the implicit posterior sampler $z_{x,i}=G_\phi(x,\epsilon_i)$, a neural encoder that turns a sentence representation and noise into posterior samples, so the variational family is defined by a sampling mechanism rather than by a density. Because the corresponding KL divergence has no closed form, the paper uses the Fenchel-conjugate dual, written (in the paper's Eq. 7) as $\mathrm{KL}(q\|p)=\max_\nu \mathbb{E}_{q}[\nu(x,z)]-\mathbb{E}_{p}[\exp(\nu(x,z))]$, with $\nu$ realized as an MLP; in iVAE-MI the same dual is applied to the aggregated posterior $q_\phi(z)=\mathbb{E}_x q_\phi(z|x)$, and the appendix shows this penalty equals $I(x,z)-\mathbb{E}_x\mathrm{KL}(q_\phi(z|x)\|p(z))$. The dual network carries the method: it supplies the training gradient and the evaluation metric for the latent space.
What would settle it
Re-run the language-modeling comparison with an unbiased KL estimate—such as an analytic KL for a Gaussian or flow-based posterior, or a dual network that includes the missing constant term—and check whether the large perplexity, active-unit, and mutual-information advantages of iVAE over Gaussian VAEs survive; if they shrink or reverse, the central claim is not supported.
Extended reading notes
Core claim
The central claim is that implicit, sample-based latent representations are a practical and effective substitute for Gaussian posteriors in latent-variable text generation, and that matching the aggregated posterior to the prior—rather than each individual posterior—further alleviates posterior collapse by explicitly rewarding mutual information between $x$ and $z$. The paper names these models iVAE and iVAE-MI. On PTB, Yahoo, and Yelp language modeling, both achieve lower negative ELBO and perplexity and higher KL, mutual-information, and active-unit counts than standard, $\beta$-, semi-amortized, cyclical, and lagging VAEs; the same model family improves unaligned sentiment transfer and open-domain dialogue response generation. The paper interprets these results as evidence that the flexible sample-based posterior is what lets the encoder and decoder actually use the latent space rather than collapse into the prior.
Load-bearing premise
The load-bearing premise is that the learned dual network estimates the KL term accurately enough for both training and evaluation; if that estimate is biased or undertrained—and the paper's Eq. (7) omits a constant term that would make the dual exact—the reported gains may reflect the estimator rather than the implicit representation.
Editorial extensions
If this is right
- Text VAEs no longer need to commit to Gaussian inference; posterior flexibility becomes a design choice, and any sampling-based encoder can be plugged into the same dual-KL training loop.
- Regularizing the aggregated posterior instead of each individual posterior gives a direct way to maximize mutual information, so posterior collapse can be attacked through the objective itself rather than only through annealing schedules or decoder redesigns.
- The same recipe carries over to conditional generation: by conditioning the implicit prior, encoder, dual network, and decoder on context, the model improves unaligned style transfer and dialogue response generation without paired data.
- The computational cost of the added dual network is modest in the reported experiments—about 30% more training time than a standard text VAE—making the flexibility gain cheaper than semi-amortized or lagging inference alternatives.
Reading between the lines
- The causal claim that posterior collapse is partially caused by the Gaussian restriction could be tested directly by running the same dual-KL training with a Gaussian posterior; if the gain largely disappears, the sampler, not the training scheme, is doing the work.
- The aggregated-posterior regularizer is not specific to text; it could apply to any amortized generative model where per-sample KL penalties suppress latent use, such as image or speech VAEs.
- In the style-transfer variant the prior itself is learned, so probing whether the learned prior's geometry separates by sentiment would show how much content preservation comes from mutual-information regularization versus the adversarial classifier.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes two implicit deep latent variable models for text generation, iVAE and iVAEMI. Instead of Gaussian variational posteriors, the encoder defines sample-based implicit posterior distributions, and the KL regularization terms are evaluated through Fenchel-dual auxiliary networks. For iVAE, the KL divergence KL(q(z|x)||p(z)) is estimated with a conditional dual network; for iVAEMI, the KL divergence between the aggregated posterior and the prior is used, which the authors show is equivalent to adding a mutual-information regularizer. The models are evaluated on language modeling (PTB, Yahoo, Yelp), unaligned sentiment transfer (Yelp), and dialog response generation (Switchboard, Dailydialog), with reported improvements in perplexity, KL, mutual information, active units, style-transfer metrics, and dialog metrics. Source code is released.
Significance. If the results hold, the paper offers a practical and relatively simple alternative to Gaussian posterior restrictions in text VAEs and provides evidence that sample-based implicit representations can mitigate posterior collapse. Strengths of the manuscript include the breadth of tasks considered, the correct standard mutual-information decomposition in Appendix A, the release of source code, and the comparison against several recent VAE variants. The main risk is that the language-modeling evidence, which is the primary support for the posterior-collapse claim, is built on a KL estimator that is biased as written and is not validated against any reference. Because the paper itself states that its perplexity and lower-bound evaluations are approximated and not directly comparable to other methods, the headline claims need to be re-examined before the contribution can be fully accepted.
major comments (3)
- [Section 3.1, Eq. (7) (and Eq. (13))] The displayed dual form is missing the additive constant from the Fenchel conjugate. The correct identity is KL(q||p) = max_ν [ E_{q} ν - E_{p} exp(ν) ] + 1; equivalently, the expression E_q[ν] - E_p[exp(ν)] equals KL(q||p) - 1 at the optimum. As a result, the KL estimates reported in Table 2 are biased downward by at least 1 even for an optimal auxiliary network, and by an unknown, model-dependent amount when the finite-capacity network is not optimal. Since the NELBO and PPL values in Table 2 are built from these KL estimates, the language-modeling comparisons are not reliable as reported.
- [Section 5, first paragraph, and Table 2] The paper explicitly states that the perplexity and lower-bound evaluations are approximated and "not directly comparable to other methods," yet Table 2 and the surrounding discussion claim that the proposed iVAEs "achieve much better performance" on exactly these metrics. This is an internal inconsistency that directly affects the abstract's claim of "consistently supreme results." Either the numbers must be made comparable (for example, with annealing importance sampling or a closed-form KL control on a Gaussian VAE), or the claims must be restricted to the metrics that are fair to compare.
- [Section 5.1 and Section 3.1] The accuracy of the dual KL estimator is not established. Section 3.1 justifies the approximation by observing that the MLP auxiliary network "converges faster than LSTM encoder and decoder," which says nothing about the quality of the converged fixed point. Section 5.1 then evaluates KL and MI by training the auxiliary networks on the same data with the same optimizer, so any systematic underfitting or bias transfers directly into the reported KL, MI, active-unit, and PPL numbers. I request a sanity check against a case with a closed-form KL (such as a Gaussian VAE), a comparison with an independent estimator, and an assessment of bias and variance across restarts.
minor comments (5)
- [Figure 1 caption] The caption contains a typo: "piror" should be "prior."
- [Section 5.3, BLEU definition] The sentence "BLEU precision and recall is defin[d] as the average and maximum scores" contains a typo and the terminology is confusing; the paper should clarify which of the sampled responses is used for precision and which for recall.
- [Table 3] The column headings "Re." and "Abs." are not defined in the caption; the caption should state explicitly that "Re." is relative time and "Abs." is absolute time.
- [Throughout] The model name is typeset inconsistently as "iV AEMI" and "iV AE MI" (for example, Tables 7 and 8 and Section 5.1); the notation should be unified.
- [Section 5.1] Table 2 reports no standard deviations or multiple-seed results, which is important for the "consistently" improvement claim given that the training involves an adversarial-style auxiliary network; adding variance information would materially strengthen the empirical case.
Circularity Check
No load-bearing circularity; the approximate KL estimator is a validity caveat, not a circular step.
full rationale
The paper's central derivation is self-contained. The sample-based implicit posterior is defined directly in Eq. (6), and the KL term is handled through the Fenchel dual form attributed to Rockafellar et al. (1966) and Dai et al. (2018), which are external references; no uniqueness claim or ansatz is imported from the authors' own prior work. The mutual-information decomposition in Section 3.2 is proved in Appendix A rather than assumed. The main empirical claims for style transfer and dialog response generation are measured against external baselines with external metrics (BLEU, BOW, Distinct, fasttext, KenLM), so those results do not reduce to the model's fitted parameters. The language-modeling numbers in Table 2 do rely on the dual KL estimator, and the paper itself states that the perplexity/lower-bound evaluations 'are approximated. Thus they are not directly comparable to other methods.' This is an acknowledged validity limitation, and Eq. (7) as written is missing the '+1' constant of the Fenchel dual, which is a correctness risk in the reported KL values; however, it is not a definitional equivalence between the model's inputs and its conclusions. Author-affiliated citations (Fu et al. 2019; Gao et al. 2019) appear only as baselines or generic motivation and are not load-bearing. Accordingly, no specific circular step can be exhibited, and the appropriate circularity score is 0.
Assumptions & free parameters
free parameters (3)
- KL annealing weight schedule =
linearly from 0.1 to 1.0 over 10 epochs
- Number of inner updates for auxiliary network ν =
5 per generator update
- Latent dimension =
32 (language modeling, style transfer); 200 (dialog)
assumptions (4)
- standard math Fenchel-Rockafellar duality provides an exact variational representation of KL divergence
- domain assumption The MLP auxiliary network ν_ψ has enough capacity and is trained close to optimality so its value approximates the true KL
- domain assumption The implicit sampler z = G_φ(x, ε) with Gaussian ε can represent sufficiently expressive posteriors for text
- domain assumption The empirical data distribution q(x) is a good proxy for the true distribution and the aggregated posterior integral can be estimated by ancestral sampling
Cite this review
Pith. "Pith review of Implicit Deep Latent Variable Models for Text Generation." pith.science (2026). https://pith.science/paper/3SF56EWA
@misc{pith2026190811527,
author = {Pith},
title = {Pith review of: Implicit Deep Latent Variable Models for Text Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/3SF56EWA}},
note = {Machine review of arXiv:1908.11527}
}
read the original abstract
Deep latent variable models (LVM) such as variational auto-encoder (VAE) have recently played an important role in text generation. One key factor is the exploitation of smooth latent structures to guide the generation. However, the representation power of VAEs is limited due to two reasons: (1) the Gaussian assumption is often made on the variational posteriors; and meanwhile (2) a notorious "posterior collapse" issue occurs. In this paper, we advocate sample-based representations of variational distributions for natural language, leading to implicit latent features, which can provide flexible representation power compared with Gaussian-based posteriors. We further develop an LVM to directly match the aggregated posterior to the prior. It can be viewed as a natural extension of VAEs with a regularization of maximizing mutual information, mitigating the "posterior collapse" issue. We demonstrate the effectiveness and versatility of our models in various text generation scenarios, including language modeling, unaligned style transfer, and dialog response generation. The source code to reproduce our experimental results is available on GitHub.
Figures
Forward citations
Cited by 1 Pith paper
-
GROOT-2: Weakly Supervised Multi-Modal Instruction Following Agents
A weakly supervised latent-variable agent improves multimodal instruction following by combining VAE self-imitating on unlabeled data with a likelihood-based alignment of labeled and video latents.
Reference graph
Works this paper leans on
-
[1]
URL: " 'urlintro :=
ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Mart \' n Arjovsky, Soumith Chintala, and L \'e on Bottou. 2017. Wasserstein gan. CoRR, abs/1701.07875
arXiv 2017
-
[4]
Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. 2015. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349
arXiv 2015
-
[5]
Changyou Chen, Chunyuan Li, Liqun Chen, Wenlin Wang, Yunchen Pu, and Lawrence Carin. 2018. Continuous-time flows for efficient inference and density estimation. ICML
work page 2018
-
[6]
Chris Cremer, Xuechen Li, and David Duvenaud. 2018. Inference suboptimality in variational autoencoders. arXiv preprint arXiv:1801.03558
arXiv 2018
-
[7]
Bo Dai, Hanjun Dai, Niao He, Weiyang Liu, Zhen Liu, Jianshu Chen, Lin Xiao, and Le Song. 2018. Coupled variational bayes via optimization embedding. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems 31, pages 9713--9723. Curran Associates, Inc
work page 2018
- [8]
Show all 48 references
-
[9]
Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, and Lawrence Carin. 2019. Cyclical annealing schedule: A simple approach to mitigating kl vanishing. NAACL
2019
-
[10]
Jianfeng Gao, Michel Galley, and Lihong Li. 2019. Neural approaches to conversational ai. Foundations and Trends in Information Retrieval , 13(2-3):127--298
2019
-
[11]
John J Godfrey and Edward Holliman. 1997. Switchboard-1 release 2. Linguistic Data Consortium, Philadelphia, 926:927
1997
-
[12]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. http://papers.nips.cc/paper/5423-generative-adversarial-nets.pdf Generative adversarial nets . In Z. Ghahramani, M. Welling, C. Cortes, N. D. ...
2014
-
[13]
Xiaodong Gu, Kyunghyun Cho, Jungwoo Ha, and Sunghun Kim. 2018. Dialogwae: Multimodal response generation with conditional wasserstein auto-encoder. arXiv preprint arXiv:1805.12352
2018 arXiv
-
[14]
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville. 2017. https://papers.nips.cc/paper/7159-improved-training-of-wasserstein-gans Improved training of wasserstein gans . In Advances in Neural Information Processing Systems 30, pages 5769--577...
2017 arXiv
-
[15]
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatrick. 2019. Lagging inference networks and posterior collapse in variational autoencoders. arXiv preprint arXiv:1901.05534
2019 arXiv
-
[16]
Clark, and Philipp Koehn
Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. 2013. https://kheafield.com/papers/edinburgh/estimate\_paper.pdf Scalable modified Kneser-Ney language model estimation . In Proceedings of the 51st Annual Meeting of the Association for Computational Li...
2013
-
[17]
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. beta-vae: Learning basic visual concepts with a constrained variational framework. In International Conference on Learning Representation...
2017
-
[18]
Sepp Hochreiter and J \"u rgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735--1780
1997
-
[19]
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P Xing. 2017. Toward controlled generation of text. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1587--1596. JMLR. org
2017
-
[20]
Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427--431. Ass...
2017
-
[21]
Yoon Kim, Sam Wiseman, Andrew Miller, David Sontag, and Alexander Rush. 2018. Semi-amortized variational autoencoders. In International Conference on Machine Learning, pages 2683--2692
2018
-
[22]
Yoon Kim, Kelly Zhang, Alexander M Rush, Yann LeCun, et al. 2017. Adversarially regularized autoencoders. arXiv preprint arXiv:1706.04223
2017 arXiv
-
[23]
Diederik P Kingma and Max Welling. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114
2013 arXiv
-
[24]
Chunyuan Li, Hao Liu, Changyou Chen, Yuchen Pu, Liqun Chen, Ricardo Henao, and Lawrence Carin. 2017 a . Alice: Towards understanding adversarial learning for joint distribution matching. In Advances in Neural Information Processing Systems, pages 5495--5503
2017
-
[25]
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015. A diversity-promoting objective function for neural conversation models. arXiv preprint arXiv:1510.03055
2015 arXiv
-
[26]
Jiwei Li, Will Monroe, Tianlin Shi, S \'e bastien Jean, Alan Ritter, and Dan Jurafsky. 2017 b . Adversarial learning for neural dialogue generation. arXiv preprint arXiv:1701.06547
2017 arXiv
-
[27]
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. 2017 c . Dailydialog: A manually labelled multi-turn dialogue dataset. arXiv preprint arXiv:1710.03957
2017 arXiv
-
[28]
Yingzhen Li, Richard E Turner, and Qiang Liu. 2017 d . Approximate inference with amortised mcmc. arXiv preprint arXiv:1702.08343
2017 arXiv
-
[29]
Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. arXiv preprint arXiv:1603.08023
2016 arXiv
-
[30]
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. 2015. Adversarial autoencoders. arXiv preprint arXiv:1511.05644
2015 arXiv
-
[31]
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: The penn treebank. Computational Linguistics
1993
-
[32]
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. 2017. Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 2391--2400. JMLR. org
2017
-
[33]
Yishu Miao, Lei Yu, and Phil Blunsom. 2016. Neural variational inference for text processing. In International conference on machine learning, pages 1727--1736
2016
-
[34]
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311--318. Association for Computational Linguistics
2002
-
[35]
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532--1543
2014
-
[36]
Yuchen Pu, Zhe Gan, Ricardo Henao, Chunyuan Li, Shaobo Han, and Lawrence Carin. 2017 a . Vae learning via stein variational gradient descent. In Advances in Neural Information Processing Systems, pages 4236--4245
2017
-
[37]
Yuchen Pu, Weiyao Wang, Ricardo Henao, Liqun Chen, Zhe Gan, Chunyuan Li, and Lawrence Carin. 2017 b . Adversarial symmetric variational autoencoder. In Advances in Neural Information Processing Systems, pages 4330--4339
2017
-
[38]
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning, pages 1278--1286
2014
-
[39]
R Tyrrell Rockafellar et al. 1966. Extension of fenchel'duality theorem for convex functions. Duke mathematical journal, 33(1):81--89
1966
-
[40]
Harshil Shah and David Barber. 2018. http://papers.nips.cc/paper/7409-generative-neural-machine-translation.pdf Generative neural machine translation . In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information...
2018
-
[41]
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In Advances in neural information processing systems, pages 6830--6841
2017
-
[42]
Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg-Kirkpatrick. 2017. Improved variational autoencoders for text modeling using dilated convolutions. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 3881--3890. JMLR. org
2017
-
[43]
Shengjia Zhao, Jiaming Song, and Stefano Ermon. 2017 a . Info VAE : Information maximizing variational autoencoders. arXiv preprint arXiv:1706.02262
2017 arXiv
-
[44]
Tiancheng Zhao, Kyusong Lee, and Maxine Eskenazi. 2018. Unsupervised discrete sentence representation learning for interpretable neural dialog generation. arXiv preprint arXiv:1804.08069
2018 arXiv
-
[45]
Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017 b . Learning discourse-level diversity for neural dialog models using conditional variational autoencoders. arXiv preprint arXiv:1703.10960
2017 arXiv
-
[46]
@esa (Ref
\@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...
-
[47]
\@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...
-
[48]
[0] e.g., [0] cf
@open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.