Pith. sign in

REVIEW 4 major objections 5 minor 68 references

Unpaired Image-to-Speech Synthesis with Multimodal Information Bottleneck

T0 review · 4 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A shared text space can carry image content into speech, so cross-modal translation needs no paired examples.

desk verdict A genuinely new task formulation with a serious evaluation citation problem: the ASR model is mis-cited as WaveNet, so the headline numbers need to be re-grounded before they can be trusted. read the letter →

arxiv 1908.07094 v1 pith:FZ2NKRGG submitted 2019-08-19 cs.CV

Shuang Ma , Daniel McDuff , Yale Song This is my paper · ORCID
classification cs.CV
keywords skip-modalgenerationimage-to-speechsynthesismultimodalinformationbottleneckunpairedcross-modaltranslationsharedmodalitymulti-headattention
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces skip-modal generation: translating between two modalities that have no paired examples by passing through a third modality that is paired with both. The demonstration is image-to-speech synthesis, using an image-text dataset and a text-speech dataset with text as the shared go-between. The proposed model funnels all three modalities through a multimodal information bottleneck, so it learns one compact representation of what is being said rather than surface details of any single modality. The authors report the first image-to-speech results from unpaired data and show the same model also improves image captioning and speech recognition. If the claim holds, new translation tasks become buildable from existing paired datasets instead of requiring new aligned data collection.

What carries the argument

The central mechanism is the multimodal information bottleneck, made of a modality transformer and a memory fusion module. The modality transformer maps image and speech embeddings into a text-induced shared space, trained adversarially against a three-way modality classifier that tries to tell the modalities apart. The memory fusion module then reads from a shared external memory with multi-head scaled dot-product attention, producing a compact vector that is decoded into text, image, and speech. Cross-modal reconstruction losses force the memory to retain only content that transfers across modalities, which is what lets a speech decoder produce language driven by an image embedding it never saw during training.

What would settle it

Listen to the generated audio for an image whose caption contains a word absent from every text-to-speech training transcript; the model must garble or omit that word, so a clean and consistent pronunciation would contradict the claimed text bottleneck.

Watch

Extended reading notes

Core claim

The central claim is that a model can translate an image into speech without ever seeing an image-speech pair, provided a text corpus is paired with both modalities. During training, the model solves two cross-modal tasks through a shared bottleneck, so the bottleneck learns a modality-agnostic representation of the content rather than any single modality's surface detail. At test time, an image embedding is pushed through the same bottleneck and fed to the speech decoder. The paper reports that this produces recognizable speech describing the image, with a word error rate of 3.9 against 4.1 for a chained image-to-text-to-speech pipeline and 12.1 for a direct unpaired-translation baseline, and that removing either bottleneck component degrades every task. The same model also improves the underlying image-to-text and speech-to-text tasks, which the authors interpret as a data-efficiency gain.

Load-bearing premise

The shared text space must contain every meaning that needs to cross between image and speech, so any visual concept with no word in the speech corpus is unreachable.

Editorial extensions

If this is right

  • Any two datasets that share one modality can be fused into a skip-modal translator, so new cross-modal tasks become available without collecting new paired data.
  • The same bottleneck vector can drive multiple decoders, which makes combined synthesis and retrieval possible in one model; the appendix shows cross-modal retrieval examples between the two datasets.
  • Training on a mixture of paired datasets improves the individual image-to-text and speech-to-text tasks compared to training each task alone, suggesting a data-efficiency gain.
  • The quality of synthesized speech is capped by the shared text vocabulary: out-of-vocabulary words are the observed failure mode, so enlarging the shared corpus should directly enlarge what can be spoken.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Unstated but testable: spoken vocabulary should scale with the vocabulary overlap between the two datasets, since the observed failures all involve out-of-vocabulary words.
  • Unstated but plausible: replacing the character-level text space with a phoneme or semantic-embedding space would retain the bottleneck effect while softening the vocabulary-mismatch failure.
  • Not attempted here: swapping the decoders should yield speech-to-image synthesis, because the appendix's cross-modal retrieval shows the bottleneck retains image-relevant content.
  • A quantitative prediction: because the memory dimension is the most sensitive hyperparameter, increasing memory size should disproportionately improve fine-grained attributes such as colors, counts, and spatial relations.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The manuscript introduces skip-modal generation, a setting in which an instance from modality A is translated to modality C without paired A-C data by leveraging two datasets (A-B and B-C) that share an intermediate modality B. The concrete demonstration is image-to-speech synthesis using COCO image-text pairs and an in-house text-speech dataset EMT-4, with text as the shared modality. The proposed architecture consists of modality-specific encoders/decoders and a multimodal information bottleneck (MIB) formed by a modality transformer and a memory fusion module. The model is trained end-to-end with cross-modal reconstruction losses and an adversarial modality classifier. Experiments report qualitative and quantitative results on image-to-speech synthesis, image-to-text, speech-to-text, and text-to-speech, plus ablation studies. The paper claims state-of-the-art or competitive performance on the standard tasks and the first reported image-to-speech synthesis results.

Significance. The problem formulation is genuinely novel and potentially useful: it offers a way to synthesize between modalities without paired data by reusing existing datasets, and the shared-memory architecture is simple and interpretable. The paper ships a code URL and reports a substantial set of experiments, including ablations. If the quantitative claims survive a corrected evaluation, the contribution would be a solid empirical advance in multimodal representation learning. However, the current quantitative evidence for the headline task is not interpretable as written (the cited ASR model is not an ASR), and the human study is too small and insufficiently analyzed. The paper is therefore a conditional contribution: the core idea is defensible, but the evidence must be repaired.

major comments (4)
  1. [§4.1, Table 1; §4.2, Table 3 (right)] The reported evaluations use "a pretrained ASR model based on WaveNet [44]", but reference [44] (van den Oord et al., WaveNet: A generative model for raw audio) is a text-to-speech model, not a speech recognizer, so the BLEU and WER numbers in Tables 1 and 3 cannot be computed from it as described. The authors must identify the actual speech recognizer used, specify its version and pretraining data, validate it on EMT-4 and on the synthesized audio, and rerun the evaluations under that protocol. As published, the quantitative support for the headline image-to-speech comparison and for the text-to-speech comparison is uninterpretable.
  2. [§4.1, human evaluation] The human study uses only seven judges on twenty samples (140 evaluations), reports means with standard deviations between 27.6% and 35.7%, and provides no confidence intervals, per-sample agreement, or significance tests. The claim that "subjects picked ours three times more frequently" is therefore not statistically supported. The authors should report the full per-sample preference matrix, inter-judge agreement, and a paired significance test (e.g., Wilcoxon signed-rank or a bootstrap) against each baseline.
  3. [§4.1, vocabulary overlap] The paper notes that the COCO and EMT-4 vocabularies overlap by only 26%, so many COCO captions contain words unseen by the speech decoder during training, and the qualitative results show pronunciation failures for words such as 'berries' and 'grapes'. This limitation is acknowledged, but the paper still draws an unqualified "image-to-speech synthesis" conclusion. The authors should quantify the fraction of the COCO test set expressible in the shared vocabulary, report results separately for in-vocabulary and out-of-vocabulary content, and make the scope of the claim precise.
  4. [§3.2, Eq. (2)-(7)] The term "information bottleneck" is used as the name of the memory fusion module, but no information-theoretic quantity is optimized or measured: the training objective in Eq. (7) contains reconstruction and adversarial losses, not an IB Lagrangian such as I(X;Z) - beta*I(Z;Y), and Eq. (2) is only an attention-weighted sum of memory vectors. The bottleneck effect is architectural, arising from the fixed memory dimension and the shared memory. The authors should either provide a formal connection to the information bottleneck method or revise the terminology to "bottlenecked multimodal representation" so that the theoretical framing matches what is actually implemented.
minor comments (5)
  1. [§4.2 vs. Appendix §6.1] The text says the image encoder uses a "3-layer CNN" (Section 4.2), but the appendix lists four Conv2D layers before the max-pooling; please reconcile this description.
  2. [References] References [2] and [3] are identical, and [34] and [35] are also identical; consolidate the duplicate entries.
  3. [§4.1, Figure 5] The color legend for Figure 5 is hard to apply to the inline transcripts; please make the colored words explicit in the figure itself and provide the audio files in the supplement so readers can verify the transcriptions.
  4. [§4.1, quantitative evaluation] The paper should report the number of test images, the number of generated speech samples, and the variance of BLEU/WER across samples or across caption annotations; none of these are currently given.
  5. [§3.4, Eq. (7)] The adversarial weight alpha=0.1 is stated in the text, but the paper does not report sensitivity to alpha; a brief ablation or a sentence justifying the fixed value would increase confidence in the results.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the derivation is self-contained; the WaveNet ASR citation is an evaluation-validity defect, not a circular dependency.

full rationale

I walked the claimed derivation chain. The model is trained on reconstruction losses over disjoint datasets (Eqs. 3-6): images are reconstructed from text embeddings, speech from text embeddings, and text from image or speech embeddings. Image-to-speech inference is a compositional generation path that passes an image embedding through the modality transformer, the memory fusion module, and the speech decoder; it is not a prediction that is equivalent to a fitted input. The term 'information bottleneck' is used as an interpretation of the shared memory/attention module, but no information-theoretic quantity is derived from or fitted to the target evaluation, so there is no self-definitional step. The quantitative image-to-speech evaluation does rely on 'a pretrained ASR model based on WaveNet [44]', and WaveNet is a raw-audio text-to-speech model, not an ASR system; this is a serious reproducibility and validity defect that makes Table 1 and the text-to-speech WER numbers uninterpretable, but it is an external evaluation instrument rather than an input to training, so it is not circularity. The only self-citation, reference [27] in the related-work section, is not load-bearing. Therefore the paper receives a circularity score of 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper's central claim rests on the modeling assumption that text is a complete and sufficiently overlapping conduit between the two datasets, and on the heuristic assertion that a shared memory with attention acts as an information bottleneck. The only tuned numbers are standard hyperparameters, not fitted to the target image-speech task. No new physical entities are postulated.

free parameters (3)
  • alpha (adversarial loss weight) = 0.1
    Set by hand in Section 3.4; balances the reconstruction loss and the adversarial alignment loss in L = L_recon + alpha * L_adv.
  • n_k (number of memory basis vectors) = 128
    Selected via sensitivity analysis in Appendix 9.2; controls the capacity of the memory fusion module.
  • d_k (dimension of each memory basis vector) = 256
    Selected via sensitivity analysis in Appendix 9.2; performance degrades at d_k=64 and plateaus at 256.
assumptions (3)
  • domain assumption Text is a sufficient shared modality to bridge image and speech content.
    The authors state in Section 3.2 that image and audio contain richer information than text, yet design the bottleneck around text, assuming the content needed for the target speech is expressible in language.
  • domain assumption The two datasets share enough semantic overlap to learn cross-modal correspondence.
    Section 4.1 reports the COCO and EMT-4 vocabularies overlap by only 26%; the learned alignment depends on this overlap being sufficient for the model to transfer content.
  • ad hoc to paper The memory fusion module implements an information bottleneck as defined by Tishby et al.
    The operation u_j = softmax(z_j K^T / sqrt(d_k)) M is labeled an information bottleneck, but no information-theoretic objective is optimized; the term is used heuristically.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unpaired Image-to-Speech Synthesis with Multimodal Information Bottleneck." pith.science (2026). https://pith.science/paper/FZ2NKRGG

@misc{pith2026190807094,
  author       = {Pith},
  title        = {Pith review of: Unpaired Image-to-Speech Synthesis with Multimodal Information Bottleneck},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FZ2NKRGG}},
  note         = {Machine review of arXiv:1908.07094}
}
read the original abstract

Deep generative models have led to significant advances in cross-modal generation such as text-to-image synthesis. Training these models typically requires paired data with direct correspondence between modalities. We introduce the novel problem of translating instances from one modality to another without paired data by leveraging an intermediate modality shared by the two other modalities. To demonstrate this, we take the problem of translating images to speech. In this case, one could leverage disjoint datasets with one shared modality, e.g., image-text pairs and text-speech pairs, with text as the shared modality. We call this problem "skip-modal generation" because the shared modality is skipped during the generation process. We propose a multimodal information bottleneck approach that learns the correspondence between modalities from unpaired data (image and speech) by leveraging the shared modality (text). We address fundamental challenges of skip-modal generation: 1) learning multimodal representations using a single model, 2) bridging the domain gap between two unrelated datasets, and 3) learning the correspondence between modalities from unpaired data. We show qualitative results on image-to-speech synthesis; this is the first time such results have been reported in the literature. We also show that our approach improves performance on traditional cross-modal generation, suggesting that it improves data efficiency in solving individual tasks.

Figures

Figures reproduced from arXiv: 1908.07094 by the authors.

Figure 1
Figure 1. Cross-modal generation typically requires paired data [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Our model consists of modality-specific encoders [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Architectures of the modality-specific encoders and decoders. We provide parameter settings in the supplementary material. where the mini-max game is defined with two terms LT = − E [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Architecture of multimodal information bottleneck. as a linear combination of basis vectors in M with weights given by the scaled dot-product attention [45], u j = softmax  z jKT / p dk  M (2) Intuitively, z j serves as a query to search the relevant keys to determin…
Figure 5
Figure 5. Figure 5: Image-to-speech synthesis results. For the purpose of presentation, we manually transcribed audio results. Red: incorrect word predictions, green: correct/more fine-grained word predictions compared with the baseline, yellow: incorrect word pronunciation, and blue: cor…
Figure 6
Figure 6. Figure 6: Image-to-speech (I2S) and image-to-text (I2T) synthesis results. I2S results are manually transcribed for presentation. on speech-to-text tasks. Our multimodal information bot￾tleneck is trained on a larger variety of data, which helps them learn more powerful represen…
Figure 7
Figure 7. Figure 7: Image-to-speech synthesis results. Green: Fine-grained and correct instances synthesized by our model. Red: incorrect pronun￾ciation synthesized by the piecewise model. Audio samples are available at https://bit.ly/2U7741S ing: one that samples instances from only one …
Figure 8
Figure 8. Figure 8: Cross-Modal retrieval results. The first column shows queries from each modality. The second and third columns show the top-3 [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

68 extracted references · 59 canonical work pages

  1. [44]

    Wavenet: A generative model for raw audio

    Aron van den Oord, Sander Dieleman, Heiga Zen, Karen Si- monyan, Oriol Vinyals, Alexander Graves, Nal Kalchbren- ner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499, 2016

    arXiv 2016

  2. [1]

    Deep speech 2: End-to-end speech recognition in english and mandarin

    Dario Amodei et al. Deep speech 2: End-to-end speech recognition in english and mandarin. In ICML, 2016

    work page 2016

  3. [3]

    Bottom-up and top-down attention for image captioning and visual question answering

    Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, 2018

    2018

  4. [4]

    Deep voice: Real-time neural text-to-speech

    Sercan ¨Omer Arik, Mike Chrzanowski, Adam Coates, Greg Diamos, Andrew Gibiansky, Yongguo Kang, Xian Li, John Miller, Jonathan Raiman, Shubho Sengupta, and Mohammad Shoeybi. Deep voice: Real-time neural text-to-speech. In ICML, 2017

    work page 2017

  5. [5]

    One-sided unsupervised do- main mapping

    Sagie Benaim and Lior Wolf. One-sided unsupervised do- main mapping. In NIPS, 2017

    work page 2017

  6. [6]

    Lawrence Zitnick

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedan- tam, Saurabh Gupta, Piotr Doll ´ar, and C. Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. CoRR, abs/1504.00325, 2015

    arXiv 2015

  7. [7]

    Weiss, Kanishka Rao, Katya Gonina, Navdeep Jaitly, Bo Li, Jan Chorowski, and Michiel Bacchiani

    Chung-Cheng Chiu, Tara Sainath, Yonghui Wu, Rohit Prab- havalkar, Patrick Nguyen, Zhifeng Chen, Anjuli Kannan, Ron J. Weiss, Kanishka Rao, Katya Gonina, Navdeep Jaitly, Bo Li, Jan Chorowski, and Michiel Bacchiani. State-of-the- art speech recognition with sequence-to-sequence models. In ICASSP, 2018

    work page 2018

  8. [8]

    Learning phrase representations using rnn encoder-decoder for statistical machine translation

    Kyunghyun Cho, Bart Van Merri ¨enboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP, 2014

    work page 2014

Show all 68 references
  1. [9]

    StarGAN: Unified gener- ative adversarial networks for multi-domain image-to-image translation

    Yunjey Choi, Min-Je Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. StarGAN: Unified gener- ative adversarial networks for multi-domain image-to-image translation. In CVPR, 2018

    2018

  2. [10]

    J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei- Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, 2009

    2009

  3. [11]

    Unsupervised domain adaptation by backpropagation

    Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML, 2015

    2015

  4. [12]

    Auditory-visual inte- gration during multimodal object recognition in humans: a behavioral and electrophysiological study

    Marie H Giard and Franck Peronnet. Auditory-visual inte- gration during multimodal object recognition in humans: a behavioral and electrophysiological study. Journal of cogni- tive neuroscience, 11(5):473–490, 1999

    1999

  5. [13]

    Deep voice 2: Multi-speaker neural text-to-speech

    Andrew Gibiansky, Sercan Arik, Gregory Diamos, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, and Yanqi Zhou. Deep voice 2: Multi-speaker neural text-to-speech. In NIPS, 2017

    2017

  6. [14]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS. 2014

    2014

  7. [15]

    Griffin and Jae Lim

    D. Griffin and Jae Lim. Signal estimation from modified short-time fourier transform. IEEE Transactions on Acous- tics, Speech, and Signal Processing, 1984

    1984

  8. [16]

    K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Com- puter Vision and Pattern Recognition (CVPR), 2016

    2016

  9. [17]

    Deep neural networks for acoustic modeling in speech recognition

    Geoffrey Hinton, Li Deng, Dong Yu, George Dahl, Abdel- rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Brian Kingsbury, et al. Deep neural networks for acoustic modeling in speech recognition. Signal processing magazine, 2012

    2012

  10. [18]

    Huang, Z

    G. Huang, Z. Liu, L. v. d. Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017

    2017

  11. [19]

    Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional adver- sarial networks. In CVPR, 2017

    2017

  12. [20]

    Recurrent fusion network for image captioning

    Wenhao Jiang, Lin Ma, Yu-Gang Jiang, Wei Liu, and Tong Zhang. Recurrent fusion network for image captioning. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors,ECCV, 2018

    2018

  13. [21]

    Learning to discover cross-domain relations with generative adversarial networks

    Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning to discover cross-domain relations with generative adversarial networks. In ICML, 2017

    2017

  14. [22]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2014

    2014

  15. [23]

    Auto-encoding varia- tional bayes

    Diederik P Kingma and Max Welling. Auto-encoding varia- tional bayes. In ICLR, 2014

    2014

  16. [24]

    Shamma, Michael S

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. ...

    2017

  17. [25]

    Letter-based speech recognition with gated convnets

    Vitaliy Liptchinsky, Gabriel Synnaeve, and Ronan Collobert. Letter-based speech recognition with gated convnets. CoRR, abs/1712.09444, 2017

    2017 arXiv

  18. [26]

    DA-GAN: instance-level image

    Shuang Ma, Jianlong Fu, Chang Wen Chen, and Tao Mei. DA-GAN: instance-level image. In CVPR, 2018

    2018

  19. [27]

    Neural TTS styl- ization with adversarial and collaborative games

    Shuang Ma, Daniel Mcduff, and Yale Song. Neural TTS styl- ization with adversarial and collaborative games. In ICLR, 2019

    2019

  20. [28]

    Semstyle: Learning to generate stylised image captions using unaligned text

    Alexander Mathews, Lexing Xie, and Xuming He. Semstyle: Learning to generate stylised image captions using unaligned text. In The IEEE Conference on Computer Vision and Pat- tern Recognition (CVPR), June 2018

    2018

  21. [29]

    Deep multi-scale video prediction beyond mean square error

    Michael Mathieu, Camille Couprie, and Yann LeCun. Deep multi-scale video prediction beyond mean square error. In ICLR, 2016

    2016

  22. [30]

    Fitting new speakers based on a short untranscribed sample

    Eliya Nachmani, Adam Polyak, Yaniv Taigman, and Lior Wolf. Fitting new speakers based on a short untranscribed sample. In ICML, 2018

    2018

  23. [31]

    Librispeech: An ASR corpus based on public domain audio books

    Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An ASR corpus based on public domain audio books. In ICASSP , 2015. IEEE

    2015

  24. [32]

    Attend to you: Personalized image captioning with context sequence memory networks

    Cesc Chunseong Park, Byeongchang Kim, and Gunhee Kim. Attend to you: Personalized image captioning with context sequence memory networks. In CVPR, 2017

    2017

  25. [33]

    Beyond sensory im- ages: Object-based representation in the human ventral path- way

    Pietro Pietrini, Maura L Furey, Emiliano Ricciardi, M Ida Gobbini, W-H Carolyn Wu, Leonardo Cohen, Mario Guazzelli, and James V Haxby. Beyond sensory im- ages: Object-based representation in the human ventral path- way. Proceedings of the National Academy of Sciences , 101(15)...

    2004

  26. [35]

    Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller

    Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan O. Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. Deep voice 3: 2000-speaker neural text-to-speech. In ICLR, 2018

    2000

  27. [36]

    Generative ad- versarial text to image synthesis

    Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Lo- geswaran, Bernt Schiele, and Honglak Lee. Generative ad- versarial text to image synthesis. In ICML, 2016

    2016

  28. [37]

    Faster r-cnn: Towards real-time object detection with region proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 91–99. 2015

    2015

  29. [38]

    Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, R

    Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, R. J. Skerry-Ryan, Rif A. Saurous, Yannis Agiomyrgiannakis, and Yonghui Wu. Natural TTS synthe- sis by conditioning wavenet on mel spectrogram predicti...

    2017 arXiv

  30. [39]

    Char2wav: End-to-end speech synthesis

    Jose Sotelo, Soroush Mehri, Kundan Kumar, Joao Felipe Santos, Kyle Kastner, Aaron Courville, and Yoshua Bengio. Char2wav: End-to-end speech synthesis. In ICLR, work- shop, 2017

    2017

  31. [40]

    Szegedy, Wei Liu, Yangqing Jia, P

    C. Szegedy, Wei Liu, Yangqing Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V . Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015

    2015

  32. [41]

    Unsupervised cross-domain image generation

    Yaniv Taigman, Adam Polyak, and Lior Wolf. Unsupervised cross-domain image generation. In ICLR, 2017

    2017

  33. [42]

    V oiceloop: V oice fitting and synthesis via a phonolog- ical loop

    Yaniv Taigman, Lior Wolf, Adam Polyak, and Eliya Nach- mani. V oiceloop: V oice fitting and synthesis via a phonolog- ical loop. In ICLR, 2018

    2018

  34. [43]

    The information bottleneck method

    Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. In Allerton Conference on Communication, Control, and Computing, 1999

    1999

  35. [45]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017

    2017

  36. [46]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NIPS, 2017

    2017

  37. [47]

    Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey E. Hinton. Grammar as a foreign language. NIPS, 2015

    2015

  38. [48]

    Show and tell: A neural image caption gen- erator

    Oriol Vinyals, Alexander Toshev, Samy Bengio, and Du- mitru Erhan. Show and tell: A neural image caption gen- erator. In CVPR, 2015

    2015

  39. [49]

    Tacotron: Towards end- to-end speech synthesis

    Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al. Tacotron: Towards end- to-end speech synthesis. In INTERSPEECH, 2017

    2017

  40. [50]

    Yuxuan Wang, Daisy Stanton, Yu Zhang, R. J. Skerry-Ryan, Eric Battenberg, Joel Shor, Ying Xiao, Fei Ren, Ye Jia, and Rif A. Saurous. Style tokens: Unsupervised style modeling, control and transfer in end-to-end speech synthesis. InICML, 2018

    2018

  41. [51]

    Memory networks

    Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. In ICLR, 2015

    2015

  42. [52]

    Courville, Ruslan Salakhutdinov, Richard S

    Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML, 2015

    2015

  43. [53]

    Attngan: Fine- grained text to image generation with attentional generative adversarial networks

    Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine- grained text to image generation with attentional generative adversarial networks. In CVPR, 2018

    2018

  44. [54]

    Image captioning with semantic attention.CVPR, 2017

    Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. Image captioning with semantic attention.CVPR, 2017

    2017

  45. [55]

    Stackgan: Text to photo-realistic image synthesis with stacked genera- tive adversarial networks

    Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaolei Huang, Xiaogang Wang, and Dimitris Metaxas. Stackgan: Text to photo-realistic image synthesis with stacked genera- tive adversarial networks. In CVPR, 2017

    2017

  46. [56]

    Visual to sound: Generating natural sound for videos in the wild

    Yipin Zhou, Zhaowen Wang, Chen Fang, Trung Bui, and Tamara L Berg. Visual to sound: Generating natural sound for videos in the wild. In CVPR, 2018

    2018

  47. [57]

    Im- proving end-to-end speech recognition with policy learning

    Yingbo Zhou, Caiming Xiong, and Richard Socher. Im- proving end-to-end speech recognition with policy learning. 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018

    2018

  48. [58]

    J. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image- to-image translation using cycle-consistent adversarial net- works. In 2017 IEEE International Conference on Computer Vision (ICCV), Oct 2017

    2017

  49. [59]

    Efros, Oliver Wang, and Eli Shechtman

    Jun-Yan Zhu, Richard Zhang, Deepak Pathak, Trevor Dar- rell, Alexei A. Efros, Oliver Wang, and Eli Shechtman. Toward multimodal image-to-image translation. In NIPS, 2018. Appendix

    2018

  50. [60]

    We encour- age the readers to refer to Figure 3 and Figure 4 of our main paper when reading this section

    Network Architectures and Parameter Set- tings We provide implementation details of our model with the parameter settings used in our experiments. We encour- age the readers to refer to Figure 3 and Figure 4 of our main paper when reading this section. We use the follow- ing n...

  51. [61]

    Define: Memory M ∈ Rnk×dk/nheads, where nk = 128,d k = 256,n heads = 4

  52. [62]

    Query qjzj, Key k Conv1D(256, 1, 1)(tanh(M)), Value v tanh(M)

  53. [63]

    (qj h, kh, vh)SplitHeads(qj, k, v), h = 1,··· ,n heads

  54. [64]

    αj hSoftMax ( qj hkh/√dk ) ,h = 1,··· ,n heads

  55. [65]

    uj hαj h× vh,h = 1,··· ,n heads

  56. [66]

    uj ConcatHeads(uj h) 6.3. Decoders (Figure 3 (right) in the main paper) • Image decoder: utxt→ Conv2D⊺(32, 4, 2) → BN → ReLU→ Conv2D⊺(16, 4, 2) → BN→ ReLU→ Conv2D⊺(8, 4, 2)→ BN→ ReLU→ Conv2D⊺(8, 4, 2)→ tanh→ yimg • Text decoder: uimg→ Dropout(LSTM(128), 0.3)→ Dropout(LSTM(128)...

  57. [68]

    Skip-Modal Synthesis Results Figure 7 shows additional image-to-speech synthesis re- sults; we manually transcribed the synthesized audio out- puts for the purpose of presentation. Consistent with the qualitative results reported in the main paper (Figure 5), we see that our a...

  58. [69]

    dog” and “zebra

    Cross-Modal Retrieval Results Besides the synthesis tasks, another way to evaluate the performance of our model is via cross-modal retrieval. In this section, we show qualitative results of cross-modal re- trieval where we use an instance from either dataset and find the most s...

  59. [70]

    bottleneck

    Additional Ablation Experiments 9.1. Different Batch Sampling Strategies As we trained our model on a combination of two datasets, there comes two ways to perform mini-batch train- Piecewise: A room with blue walls. Ours: A room with walls that are painted blue. Piecewise: A t...

  60. [256]

    → DecoderRNN6(Dropout(LSTM(256), 0.3) → Dropout(LSTM(256), 0.3)) → reshape → ymel−spectrogram → CBHG [49] (80 mels) → FC(1025)→ Griffin-Lim (ylinear)→ yspeech

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.