Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Sequential Latent Spaces for Modeling the Intention During Diverse Image Captioning

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Assigning every word position its own latent variable that mimics a backward summary of the future makes sampled image captions substantially more diverse without losing accuracy.

desk verdict Per-word latent variables with backward-ELMo intention distillation give real diversity gains, but the 'intention' mechanism is only proven for encoder means, not for the samples that matter. read the letter →

arxiv 1908.08529 v1 pith:F3XTOI4B submitted 2019-08-22 cs.CV cs.CLcs.LGstat.ML

Jyoti Aneja , Harsh Agrawal , Dhruv Batra , Alexander Schwing This is my paper · ORCID
classification cs.CVcs.CLcs.LGstat.ML
keywords diverseimagecaptioningsequentiallatentvariablemodelconditionalvariationalautoencoderintentionmodelingbackwardLSTMELMofine-graineddiversityMSCOCO
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that the limited diversity of image-captioning models comes from tying all randomness to a single latent variable that only primes the first word, and that the fix is a latent space at every word position. A temporal 'intention' model is trained to mimic a backward-reading summary of the remaining sentence, so each latent sample knows something about where the sentence is going before the decoder emits the next word. On MSCOCO the reported payoff is substantially higher diversity: more than 4200 of 5000 sampled captions were never seen in training, roughly 3400 for the best runner-up, 2-gram diversity rises to 0.54 from 0.35 at 20 samples, and mutual Bleu-4 drops to 0.52. Accuracy stays comparably high, with oracle CIDEr of 1.448 against 1.468 for the part-of-speech baseline, and unlike those baselines the method uses no object-detector or part-of-speech information during inference.

What carries the argument

The central object is a per-word Gaussian latent variable $z_t$ whose posterior mean is pulled toward a backward representation of the future by the L2 penalty $F(\mu^E_t, x, I) = \lambda \|g(\mu^E_t(z_{t-1}, x, I)) - h^B_t\|_2^2$, with $\lambda = 5\times 10^{-4}$. Here $h^B_t$ is the hidden state of the backward pass of a pretrained deep bidirectional language model, which is not fine-tuned and is used only during training. This penalty is added to the ELBO objective and is what makes the latent mean learn to predict a representation of the words that have not yet been generated. At test time the temporal prior is an LSTM-based intention model that supplies $z_t$ given $z_{<t}$, $x_{<t}$, and the image, so the decoder is driven by samples trained to summarize the future rather than by one global latent.

What would settle it

Run the identical Seq-CVAE training but replace $h^B_t$ in the L2 penalty with a random vector of the same dimension, drawn once and fixed, keeping the rest of the architecture and losses unchanged. If the model still produces more than 4000 novel captions out of 5000 and keeps 2-gram diversity near 0.54, the 'intention' regression is not what causes the diversity gain; if those numbers fall to baseline levels, the future-summary target is confirmed as the load-bearing mechanism.

Watch

Extended reading notes

Core claim

The central claim is that a conditional variational autoencoder with a Gaussian latent variable $z_t$ at every word position, trained to summarize the future, yields fine-grained diversity while keeping accuracy on par with existing approaches. The encoder is two-stage: a forward LSTM represents the past $x_{<t}$, and a backward LSTM represents the future $x_{>t}$, with an MLP combining both into the posterior mean and variance. During training the posterior mean is additionally regressed, through another MLP $g$, to the backward hidden state $h^B_t$ via an L2 penalty, so the latent space is encouraged to encode the 'intention' of how to complete the sentence. At inference time only the intention-model LSTM and the decoder run: at each step a latent sample is drawn from the temporal prior, concatenated with the previous word and the image embedding, and used to emit the next word. The paper reports that this produces more than 4200 novel sentences out of 5000 sampled, lowers mutual Bleu-4 to 0.52, raises 2-gram diversity to 0.54 at 20 samples, and keeps oracle CIDEr at 1.448, which it describes as on par with the part-of-speech baseline.

Load-bearing premise

The load-bearing premise is that a backward-reading summary of the remaining words carries enough information about the future to guide decoding, and that the L2 regression used to push the latent toward that summary transfers that information into usable samples.

Editorial extensions

If this is right

  • A single global latent is not necessary for diversity: sampling a fresh latent at every word position yields more novel sentences than the part-of-speech or object-guided baselines, without extra supervision at inference.
  • Captions can be edited or interpolated word-by-word: linearly interpolating the latent vectors at all positions produces a gradual, meaningful transition between two descriptions, giving position-level control over the emitted words.
  • The diversity gains do not require the stronger backward language model at test time, since that representation is used only during training and a backward RNN trained on the caption data alone gives comparable results.
  • Because mutual Bleu-4 drops while oracle CIDEr stays on par, the sampled captions differ from each other at the word level while each still scores well against ground truth, so diversity is not being bought by collapsing quality.
  • Architecture choices matter: sharing one LSTM across encoder, decoder, and transition model, or using a constant Gaussian prior, reduces the reported accuracy and diversity, supporting the paper's separate-network design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An implication the paper leaves implicit is that replacing $h^B_t$ with a fixed random vector of the same dimension is a clean test of whether the intention regression is the active ingredient; the paper ablates the backward representation entirely but does not isolate the L2 target itself.
  • The recipe should transfer to other conditional generation tasks that want controllable diversity, such as story continuation or question generation, since the mechanism is not image-specific, though the paper only tests captioning.
  • Because a backward RNN trained on captions alone performs near the frozen pretrained model, the essential signal may be the structural direction of reading the future rather than the scale of the pretrained language model; a capacity-matched comparison would test this directly.
  • If the latent at time $t$ truly encodes the future, a probe that decodes $x_{>t}$ from $z_t$ should outperform a probe from the forward hidden state; the paper shows regressions to $h^B_t$ and t-SNE structure but does not measure future-word predictability directly.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Seq-CVAE, a conditional variational autoencoder for diverse image captioning that places a latent variable at every word position. A two-stage encoder (a forward LSTM and a pretrained backward ELMo model) defines a Gaussian posterior per time step, while an LSTM 'intention model' defines a data-dependent prior, and the decoder is conditioned on the sampled latents along with image features. An auxiliary term regresses the encoder mean toward the backward ELMo hidden state, which is intended to summarize the future words of the caption. The method is evaluated on the MSCOCO M-RNN split using oracle CIDEr and several diversity metrics; the authors report substantially more novel and diverse captions than the tested baselines while maintaining oracle CIDEr scores that are on par with POS and AG-CVAE. Ablations vary the latent dimension, the use of ELMo, and the form of the intention model.

Significance. If the central mechanism is valid, the paper makes a useful contribution: it is one of the first captioning models with a latent variable at every word position, it uses a data-dependent transition prior, and it does not rely on object detectors or part-of-speech information at inference. The diversity gains are large and the ablation study is informative, including a non-ELMo backward RNN control and standard deviations for the oracle CIDEr metric. However, the paper's main interpretive claim—that the temporal latent space captures the future-directed 'intention'—is not directly tested, and the accuracy claim rests on oracle CIDEr, which is an upper-bound metric. The significance is therefore conditional: the empirical diversity results are promising, but the mechanism and the unselected-set quality need additional evidence before the headline claims are fully supported.

major comments (3)
  1. [§3.4, Eq. (2)] The regularizer F(μE_t, x, I) = λ||g(μE_t) − hB_t||^2 is evaluated only at the encoder mean μE_t, not at the sampled latent z_t. Since F does not depend on z_t, the factor exp(F) in the definition of qφ cancels when integrating over z_t, so qφ remains exactly Gaussian and Eq. (1) never rewards the sampled z_t for carrying information about x_{>t}. Figure 3(c) plots exactly the mean-based distance ||g(μE_t) − hB_t||, so it cannot support a claim about the latent samples used by the decoder and the intention prior. To substantiate the central 'intention' claim, the authors should provide a direct test, for example a probe that predicts future words from samples z_t (or from samples of the intention model at inference), and compare against a latent that is not trained with the future-regression term.
  2. [§4.1, Table 1] The claim of performing 'on par w.r.t. sentence quality' rests entirely on oracle CIDEr, i.e., the best of K generated captions selected against ground-truth references. This is an upper-bound metric and does not measure the quality of a randomly sampled caption or of the full diverse set; a model can score well on oracle CIDEr while generating many poor captions. The paper should report quality for the unselected sample set (for example, average CIDEr over all K samples, or human evaluation of random samples) before the accuracy claim can be accepted.
  3. [§4.3, Tables 3 and 4] The ablations do not isolate the proposed intention mechanism as the main source of the diversity gains. In Table 3, replacing the constant Gaussian intention prior with the data-dependent LSTM intention model (Seq-CVAE+N vs Seq-CVAE, both with ELMo) changes novel sentences from 4162 to 4266 and Div-2 from .43 to .54, whereas the larger gap to baselines is already present for Seq-CVAE+N with ELMo (Div-2 .43 vs .35 for POS). Table 4 further shows no CIDEr gain from the data-dependent intention at 20 samples and only a 'slight edge' at 100 samples. The paper should either provide stronger evidence that the future-prediction regularizer is what drives diversity (for example, compare against a per-word latent model without the backward-supervision term) or soften the contribution claim.
minor comments (5)
  1. [Abstract; §4.2; Table 2] The number of novel sentences is inconsistent: the abstract says 'more than 4200', the text in §4.2 says 4278, and Table 2 reports 4266; please reconcile these numbers.
  2. [Table 3 caption] The symbols (✓/ˆ) in the ELMo column are not defined in the caption; please add a legend explaining what each symbol denotes.
  3. [Table 4 caption] The column headers C@20 and C@100 are not expanded; please define them as CIDEr with 20 and 100 samples, respectively.
  4. [§4.3, 'Is ELMo the reason for good performance?'] The text states that the BRNN variant is 'comparable' to the ELMo-based model, but Table 3 shows CIDEr 1.348 versus 1.448 and mBleu-4 .65 versus .52; please qualify this statement to reflect the actual differences.
  5. [Figure 3(c)] The curves in Figure 3(c) appear to correspond to a single training run; please state how many runs were used and whether the decreasing trend is stable across seeds.

Circularity Check

1 steps flagged · score 2.0 of 10

Interpretive claim about the latent space capturing intention is self-definitional via Eq. (2), but the headline diversity and accuracy results are independent empirical outcomes.

  1. self definitional [Sec. 3.4 Eq. (2); Sec. 4.4 Fig. 3(c) and surrounding text]
    "To ensure that the mean μE_t(z_{t−1},x,I) more closely resembles the information obtained from the backward pass we choose F(μE_t,x,I)=λ‖g(μE_t(z_{t−1},x,I))−hB_t‖^2_2, (2) ... In Fig. 3(c) we illustrate for different training iterations ... the averaged F(μE_t,x,I) given in Eq. (2) ... Intuitively we observe models at later iterations to better match the ELMo representation hB_t. This indicates that the latent space learned by the encoder at a given time t is trained to better regress to word representations which summarize future words."

    Eq. (2) is the training objective itself: the L2 regression between g(μE_t) and hB_t is minimized during training. Fig. 3(c) plots exactly this objective (the averaged F(μE_t,x,I)) against training iterations. Therefore the observation that later iterations match hB_t better, and the conclusion that the latent space is trained to regress to future-word representations, are restatements of the optimization target rather than independent evidence that the sampled latent variables z_t used at inference carry predictive information about future words. Moreover, F is applied only to the encoder mean μE_t, while Eq. (1) trains the decoder on samples z_t; no term in Eq. (1) or Eq. (2) rewards sampled z_t for predicting x_{>t}.

full rationale

The paper's central results—diversity statistics, oracle CIDEr, n-gram diversity, and ablations—are empirical outcomes of a trained system evaluated against held-out test data and external baselines; they are not fitted predictions of the model's own inputs. The KL objective and Eq. (2) are legitimate training mechanisms, and the comparison against AG-CVAE, POSCap, Z-forcing, and beam-search baselines is standard practice rather than load-bearing self-citation. The only mild circularity is in the interpretation of the latent space: the model is trained to make g(μE_t) approximate the backward ELMo state hB_t, so Fig. 3(c) showing that this distance decreases with training is a plot of the training loss itself. This does not independently validate the stronger claim that the sampled intentions z_t (especially those drawn from the prior at inference) predict the future sentence, as the skeptic notes. That gap is a correctness/evidence concern, not a circularity of the diversity result. Overall, the main derivation is self-contained and the self-referential element is limited to an interpretive claim.

Assumptions & free parameters 3 free parameters · 4 assumptions · 1 invented entities

The method rests on standard VAE machinery plus a modeling choice: a per-word latent space regressed to a backward ELMo representation. The free parameters are hyperparameters selected by validation, not numbers fitted to make a specific prediction. The invented entity is the sequential latent variable itself, which is internal to the model.

free parameters (3)
  • latent dimension = 512
    Chosen as best in Tab. 4 ablation across 128, 256, and 512 dimensions; a hyperparameter selected by validation, not derived.
  • lambda (lambda) = 5e-4
    Set by hand as the weight of the L2 intention regression loss in Eq. (2).
  • ELMo hidden dimension = 512
    Fixed by the pretrained ELMo model; used as the size of the backward representation and the MLP output, chosen by architecture, not derived.
assumptions (4)
  • standard math Caption probability factorizes into word conditionals p_theta(x_t|x_<t,I).
    Standard autoregressive decomposition, stated in Sec. 3.1.
  • standard math The variational lower bound in Eq. (1) is a valid training objective and the smoothing posterior q_phi is tractable.
    Standard VAE derivation; the paper adds an auxiliary loss F to the ELBO.
  • ad hoc to paper The backward ELMo representation hB_t encodes the future of the sentence well enough to serve as the intention target.
    Sec. 3.4 states that word representations from the backward pass at any time t are a good encoding of the future x>t; this is the key modeling premise.
  • domain assumption Pretrained VGG16 features and ELMo transfer to the MSCOCO captioning task.
    The decoder and intention model consume VGG16 fc7 image embeddings and the encoder uses ELMo without fine-tuning.
invented entities (1)
  • Per-word intention latent variable z_t
    purpose: A latent Gaussian variable at each word position that controls how the sentence continues, sampled from the intention model at test time.
    This is a model component, not an empirical entity; it has no falsifiable handle outside the model. Its existence is postulated to reduce the complexity of the next-word conditional.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sequential Latent Spaces for Modeling the Intention During Diverse Image Captioning." pith.science (2026). https://pith.science/paper/F3XTOI4B

@misc{pith2026190808529,
  author       = {Pith},
  title        = {Pith review of: Sequential Latent Spaces for Modeling the Intention During Diverse Image Captioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F3XTOI4B}},
  note         = {Machine review of arXiv:1908.08529}
}
read the original abstract

Diverse and accurate vision+language modeling is an important goal to retain creative freedom and maintain user engagement. However, adequately capturing the intricacies of diversity in language models is challenging. Recent works commonly resort to latent variable models augmented with more or less supervision from object detectors or part-of-speech tags. Common to all those methods is the fact that the latent variable either only initializes the sentence generation process or is identical across the steps of generation. Both methods offer no fine-grained control. To address this concern, we propose Seq-CVAE which learns a latent space for every word position. We encourage this temporal latent space to capture the 'intention' about how to complete the sentence by mimicking a representation which summarizes the future. We illustrate the efficacy of the proposed approach to anticipate the sentence continuation on the challenging MSCOCO dataset, significantly improving diversity metrics compared to baselines while performing on par w.r.t sentence quality.

Figures

Figures reproduced from arXiv: 1908.08529 by the authors.

Figure 1
Figure 1. Meaningful diverse captions generated (blue arrows) for a given image by linearly interpolating from one latent vector (green arrows) to another (red arrows). are overloaded, the crisp picture that we intend to draw blurs rapidly if we don’t use language and associations that read￾ers are familiar with. Without the right language the mes￾sage of the description is diluted, remains hard to access or even inaccessible… view at source ↗
Figure 2
Figure 2. (a): Computation graph for the generation network. The hidden states of the intention model LSTM and the decoder LSTM are h T t and hD t respectively. At a given time step t, the latent sample zt depends on all prior words xăt and all prior latent samples zăt. The sample zt along with all prior words xăt predicts xt. (b): Computation graph for the encoder network. The hidden states of the forward LSTM and the backwa… view at source ↗
Figure 3
Figure 3. (a & b): n-gram diversity across word positions. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Qualitative results illustrating captions obtained from different image captioning methods. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: t-SNE plots of the means µ T t obtained from the intention model, learned with ELMo (a) and without ELMo (b). Notice that with ELMo representation, the model better disentangles the means per word [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Variational Prefix Tuning for Diverse and Accurate Code Summarization Using Pre-trained Language Models

    cs.SE 2025-05 conditional novelty 5.0 of 10

    Variational Prefix Tuning adds a CVAE with sampled latent prefixes to pre-trained code summarization models, improving the accuracy and diversity of generated summary sets.

  2. Image Embedding Sampling Method for Diverse Captioning

    cs.CV 2025-02 conditional novelty 5.0 of 10

    A training-free hierarchical embedding sampling method (HBoP) lets a small BLIP model generate captions as diverse as human ones, beating much larger VLMs on diversity metrics.

Reference graph

Works this paper leans on

42 extracted references · 36 canonical work pages · cited by 2 Pith papers

  1. [1]

    Bottom-up and top-down attention for image captioning and visual question answering

    Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. Proc. CVPR, 2017. 2

    work page 2017

  2. [2]

    Jyoti Aneja, Aditya Deshpande, and Alexander G. Schwing. Convolutional image captioning. In Proc. CVPR, 2018. 2, 6

    work page 2018

  3. [3]

    Blei, and Michael I

    Kobus Barnard, Pinar Duygulu, David Forsyth, Nando De Freitas, David M. Blei, and Michael I. Jordan. Matching words and pictures. JMLR, 2003. 1, 2

    work page 2003

  4. [4]

    Moitreya Chatterjee and Alexander G. Schwing. Diverse and coherent paragraph generation from images. In Proc. ECCV,

    work page

  5. [5]

    One billion word benchmark for measuring progress in statistical language modeling

    Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005, 2013. 6

    arXiv 2013

  6. [6]

    Mind’s eye: A recur- rent visual representation for image caption generation

    Xinlei Chen and C Lawrence Zitnick. Mind’s eye: A recur- rent visual representation for image caption generation. In Proc. CVPR, 2015. 1, 2

    work page 2015

  7. [7]

    De- scribing multimedia content using attention-based encoder- decoder networks

    Kyunghyun Cho, Aaron Courville, and Yoshua Bengio. De- scribing multimedia content using attention-based encoder- decoder networks. IEEE Trans. on Multimedia, 2015. 1, 2

    work page 2015

  8. [8]

    A recurrent la- tent variable model for sequential data

    Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. A recurrent la- tent variable model for sequential data. In Proc. NIPS, 2015. 3

    work page 2015

Show all 42 references
  1. [9]

    To- wards diverse and natural image descriptions via a condi- tional gan

    Bo Dai, Sanja Fidler, Raquel Urtasun, and Dahua Lin. To- wards diverse and natural image descriptions via a condi- tional gan. In Proc. ICCV, 2017. 1, 2

    2017

  2. [10]

    Schwing, and David Forsyth

    Aditya Deshpande, Jyoti Aneja, Liwei Wang, Alexander G. Schwing, and David Forsyth. Fast, diverse and accurate im- age captioning guided by part-of-speech. In Proc. CVPR,

  3. [11]

    Girshick, Mar- garet Mitchell, and C

    Jacob Devlin, Saurabh Gupta, Ross B. Girshick, Mar- garet Mitchell, and C. Lawrence Zitnick. Exploring near- est neighbor approaches for image captioning. CoRR abs/1505.04467, 2015. 7

    2015 arXiv

  4. [12]

    Long-term recurrent convolutional net- works for visual recognition and description

    Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. Long-term recurrent convolutional net- works for visual recognition and description. InProc. CVPR,

  5. [13]

    From captions to vi- sual concepts and back

    Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh K Sri- vastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C Platt, et al. From captions to vi- sual concepts and back. In Proc. CVPR, 2015. 1, 2

    2015

  6. [14]

    Every picture tells a story: Generating sentences from images

    Ali Farhadi, Mohsen Hejrati, Mohammad Amin Sadeghi, Pe- ter Young, Cyrus Rashtchian, Julia Hockenmaier, and David Forsyth. Every picture tells a story: Generating sentences from images. In Proc. ECCV, 2010. 1, 2

    2010

  7. [15]

    Sequential neural models with stochastic lay- ers

    Marco Fraccaro, Søren Kaae Sønderby, Ulrich Paquet, and Ole Winther. Sequential neural models with stochastic lay- ers. In Proc. NIPS, 2016. 3

    2016

  8. [16]

    Generative Adversarial Networks

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative Adversarial Networks. In Proc. NIPS, 2014. 1

    2014

  9. [17]

    Z-forcing: Training stochastic recurrent networks

    Anirudh Goyal ALIAS PARTH Goyal, Alessandro Sordoni, Marc-Alexandre Côté, Nan Rosemary Ke, and Yoshua Ben- gio. Z-forcing: Training stochastic recurrent networks. In Proc. NIPS, 2017. 3, 7

    2017

  10. [18]

    Long short-term memory

    Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Computation, 1997. 1, 2

    1997

  11. [19]

    Unnat Jain, Svetlana Lazebnik, and Alexander G. Schwing. Two can play this game: visual dialog with discriminative question generation and answering. In Proc. CVPR, 2018. 2

    2018

  12. [20]

    Unnat Jain, Ziyu Zhang, and Alexander G. Schwing. Cre- ativity: Generating diverse questions using variational au- toencoders. In Proc. CVPR, 2017. 2

    2017

  13. [21]

    Densecap: Fully convolutional localization networks for dense caption- ing

    Justin Johnson, Andrej Karpathy, and Li Fei-Fei. Densecap: Fully convolutional localization networks for dense caption- ing. In Proc. CVPR, 2016. 1, 2

    2016

  14. [22]

    Deep visual-semantic align- ments for generating image descriptions

    Andrej Karpathy and Li Fei-Fei. Deep visual-semantic align- ments for generating image descriptions. In Proc. CVPR,

  15. [23]

    Semi-supervised learning with deep gen- erative models

    Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep gen- erative models. In Proc. NIPS, 2014. 1

    2014

  16. [24]

    Unifying visual-semantic embeddings with multimodal neu- ral language models

    Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. Unifying visual-semantic embeddings with multimodal neu- ral language models. arXiv preprint arXiv:1411.2539, 2014. 1, 2

    2014 arXiv

  17. [25]

    Babytalk: Understanding and generating simple image descriptions

    Girish Kulkarni, Visruth Premraj, Vicente Ordonez, Sag- nik Dhar, Siming Li, Yejin Choi, Alexander C Berg, and Tamara L Berg. Babytalk: Understanding and generating simple image descriptions. PAMI, 2013. 1, 2

    2013

  18. [26]

    Generating diverse and accurate visual cap- tions by comparative adversarial learning

    Dianqi Li, Xiaodong He, Qiuyuan Huang, Ming-Ting Sun, and Lei Zhang. Generating diverse and accurate visual cap- tions by comparative adversarial learning. arXiv preprint arXiv:1804.00861, 2018. 1, 2

    2018 arXiv

  19. [27]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proc. ECCV, 2014. 2, 6

    2014

  20. [28]

    Improved image captioning via policy gradi- ent optimization of spider.arXiv preprint arXiv:1612.00370,

    Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. Improved image captioning via policy gradi- ent optimization of spider.arXiv preprint arXiv:1612.00370,

    arXiv

  21. [29]

    Visualizing data using t-sne

    Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 2008. 8

    2008

  22. [30]

    Deep Captioning with Multimodal Recur- rent Neural Networks (m-rnn)

    Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, and Alan Yuille. Deep Captioning with Multimodal Recur- rent Neural Networks (m-rnn). In Proc. ICLR, 2015. 1, 2, 6

    2015

  23. [31]

    Peters, Mark Neumann, Mohit Iyyer, Matt Gard- ner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer

    Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gard- ner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proc. NAACL,

  24. [32]

    Faster R-CNN: Towards real-time object detection with re- gion proposal networks

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with re- gion proposal networks. In Proc. NIPS, 2015. 2, 5, 6

    2015

  25. [33]

    Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel

    Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jarret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. Proc. CVPR, 2017. 2

    2017

  26. [34]

    Speaking the same language: Matching machine to human captions by adversarial training

    Rakshith Shetty, Marcus Rohrbach, Lisa Anne Hendricks, Mario Fritz, and Bernt Schiele. Speaking the same language: Matching machine to human captions by adversarial training. In Proc. ICCV, 2017. 1, 2

    2017

  27. [35]

    Very deep convo- lutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convo- lutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. 4, 5

    2014 arXiv

  28. [36]

    Grounded compositional se- mantics for finding and describing images with sentences

    Richard Socher, Andrej Karpathy, Quoc V Le, Christopher D Manning, and Andrew Y Ng. Grounded compositional se- mantics for finding and describing images with sentences. TACL, 2014. 1, 2

    2014

  29. [37]

    Learning structured output representation using deep conditional gen- erative models

    Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional gen- erative models. In Proc. NIPS, 2015. 1

    2015

  30. [38]

    Vijayakumar, Michael Cogswell, Ramprasaath R

    Ashwin K. Vijayakumar, Michael Cogswell, Ramprasaath R. Selvaraju, Qing Sun, Stefan Lee, David J. Crandall, and Dhruv Batra. Diverse beam search for improved description of complex scenes. In Proc. AAAI, 2018. 1, 2, 5, 6

    2018

  31. [39]

    Show and tell: A neural image caption gen- erator

    Oriol Vinyals, Alexander Toshev, Samy Bengio, and Du- mitru Erhan. Show and tell: A neural image caption gen- erator. In Proc. CVPR, 2015. 1, 2

    2015

  32. [40]

    Schwing, and Svetlana Lazebnik

    Liwei Wang, Alexander G. Schwing, and Svetlana Lazebnik. Diverse and accurate image description using a variational auto-encoder with an additive gaussian encoding space. In Proc. NIPS, 2017. 1, 2, 5, 6, 7

    2017

  33. [41]

    Show, attend and tell: Neural image caption gen- eration with visual attention

    Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption gen- eration with visual attention. In Proc. ICML, 2015. 1, 2

    2015

  34. [42]

    Boosting image captioning with attributes

    Ting Yao, Yingwei Pan, Yehao Li, Zhaofan Qiu, and Tao Mei. Boosting image captioning with attributes. In Proc. ICCV, 2017. 2

    2017

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.