REVIEW 5 major objections 5 minor 34 references
Stack-VS: Stacked Visual-Semantic Attention for Image Caption Generation
T0 review · 5 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read A stacked visual-semantic attention decoder reports state-of-the-art image captioning results on MSCOCO, with BLEU-4 0.372 and CIDEr 1.226.
desk verdict Credible but incremental composition of known components; the empirical evidence (tiny point gains, no variance) does not support the 'statistically significant improvements' claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Stack-VS decoder cell, a two-LSTM attention module repeated across decoding stages. In each cell, $\mathrm{LSTM}_V$ and $\mathrm{LSTM}_S$ compute separate hidden states, and their attention logits over visual features $v_k \in V_0$ and semantic embeddings $e_k \in E_0$ are computed by Eqs. (8) and (12), with each logit depending on the current hidden states of both LSTM branches and on the attended vectors from the previous stage. The attended vectors $\hat{v}_t^i$ and $\hat{e}_t^i$ are projected and summed with the two LSTM hidden states to form the input of a language LSTM, whose output predicts the next word. The same cell is stacked multiple times per time step, so each stage re-optimizes the visual-semantic attention weights before passing its result to the next stage.
What would settle it
Run Stack-VS and the two strongest baselines, Up-Down and Stack-Cap, from multiple random seeds on the same MSCOCO split, with the same feature files, vocabulary, and evaluation script, then compare confidence intervals for BLEU-4, CIDEr, and SPICE; if the intervals overlap, the claimed consistent significant improvement is falsified.
Extended reading notes
Core claim
The central claim is that a multi-stage, or stacked, decoder that jointly re-optimizes two attention streams—one over Faster R-CNN visual region features and one over MIL-derived semantic attribute embeddings—produces richer image captions than single-stage decoders or decoders using only one modality. Each stage refines the previous stage's hidden state and attended visual and semantic vectors, and the attention weights are recomputed cell by cell rather than shared across stages, unlike the earlier Stack-Cap model. The paper reports that this architecture consistently outperforms all compared baselines on the MSCOCO offline split, with the CIDEr-optimized variant reaching BLEU-4 0.372, CIDEr 1.226, and SPICE 0.216, and also reports higher BLEU-4, METEOR, ROUGE-L, and CIDEr than Up-Down and Stack-Cap on the MSCOCO online C5 test set.
Load-bearing premise
The claim of consistent improvement rests on treating the baselines' published scores as directly comparable to one unseeded run of Stack-VS; the reported margins are only a few hundredths of a metric point, so run-to-run noise could erase them.
Editorial extensions
If this is right
- If the reported scores hold, a joint stacked visual-semantic decoder is a stronger recipe on MSCOCO than either the visual-only multi-stage Stack-Cap or the visual-only single-stage Up-Down: Stack-VS reports BLEU-4 0.372, CIDEr 1.226, and SPICE 0.216.
- The stage-by-stage reweighting is credited with fixing relational words: semantic attention weights rise for prepositions like 'in' and 'on' while visual attention carries nouns, so the largest gains over single-stage baselines would be in captions involving spatial relations.
- The training scheme is compatible with standard RL fine-tuning: the CIDEr-optimized version, not the cross-entropy version alone, produces the headline numbers, meaning the architecture benefits from self-critical sequence training.
- The online MSCOCO C5 results show the same qualitative ordering over Up-Down and Stack-Cap, suggesting the improvement is not an artifact of the offline split alone.
Reading between the lines
- An ablation that deletes the semantic branch while keeping the stacked decoder, or deletes the stacking while keeping the two LSTM branches, would isolate whether the gain comes from joint attention or simply from added model capacity; the paper does not report such an ablation.
- Because the reported leads over Up-Down and Stack-Cap are close to typical run-to-run variation (for example, +0.025 CIDEr), repeated seeded runs with confidence intervals are needed before treating the new numbers as a stable state of the art.
- The same two-branch stacked attention cell could transfer to visual question answering or video captioning, where an answer or caption often needs a first coarse pass followed by refined attention over objects and attributes.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes Stack-VS, a multi-stage image captioning architecture that combines visual features from Faster R-CNN with semantic attribute embeddings obtained through multiple-instance learning. The decoder is a stack of decoder cells, each containing a visual-semantic attention layer and a language LSTM, with hidden states and attention weights refined across stages. The model is trained with per-stage cross-entropy and then with self-critical sequence training optimizing CIDEr. On the MSCOCO Karpathy test split the authors report BLEU-4 0.372, CIDEr 1.226, and SPICE 0.216, and claim consistent, statistically significant improvements over all baselines. The paper also includes qualitative analyses of visual and semantic attention refinement across stages.
Significance. If the empirical claims were substantiated, Stack-VS would be a useful extension of the coarse-to-fine captioning idea: it jointly models visual and semantic attention in a multi-stage decoder, and the staged refinement is illustrated with detailed qualitative examples. The training protocol is standard and the evaluation covers both the Karpathy split and the MSCOCO online server. However, the central claim of statistically significant and consistent improvement is not supported by the evidence: the paper provides no variance estimates, no significance tests, and Table II actually contradicts the 'consistently outperforms' statement on BLEU-1/2/3. The architecture is also underspecified in several key equations. The underlying idea is potentially valuable, but the current manuscript does not establish its headline claims.
major comments (5)
- [Section IV-D, Table I] The claim that 'the improvements are statistically significant on all metrics' is not supported by any statistical evidence. The paper reports point estimates without variance, number of runs, confidence intervals, or significance tests; the margins over Up-Down [CIDEr-Optimize] on the Karpathy test split are +0.009 BLEU-4, +0.025 CIDEr, and +0.002 SPICE, which are well within typical run-to-run noise for captioning metrics. Please provide multi-run results with standard deviations and an appropriate significance test, or retract the significance claim.
- [Section IV-D, Table II] Table II shows that Stack-VS [CIDEr-Optimize] scores below Up-Down [CIDEr-Optimize] on BLEU-1 (0.790 vs 0.802), BLEU-2 (0.634 vs 0.641), and BLEU-3 (0.489 vs 0.491) on the MSCOCO online server, yet the text states that 'our proposed model Stack-VS consistently outperforms all baseline methods.' This internal contradiction must be resolved by qualifying the claim or by explaining why the online-test comparison is not the relevant one.
- [Section III-C, III-D, Eqs. (4), (5), (15)-(17)] The decoder equations are underspecified. Equation (4) defines h^i_t as the output of a generic LSTM, but Section III-D defines three different LSTM outputs (h^i,V_t, h^i,S_t, h^i,L_t) and never states which, if any, corresponds to h^i_t or to the h^(i-1)_t used in Eq. (5). Equation (17) writes the word probability as a function of h^i,L_k while the conditioning set includes I^(i-1), but Eq. (16) shows no dependence on I^(i-1); the equation does not match the text. In addition, x^i,S_t is used in Eq. (11) but never defined. The architecture section must be rewritten so that each state is defined exactly once with consistent stage and time indices.
- [Section III-C] The number of stages Ns is never specified in the text. Figures 4 and 5 use three stages, but the paper should state the value used in the experiments and whether it was tuned. This number is a key architectural hyperparameter that affects parameter count, training time, and the qualitative behavior of the stacked refinement.
- [Section IV-C and Table I] The comparison with baselines is incomplete. Table I omits several contemporaneous models, such as RFNet, GCN-LSTM, and SGAE, which reported CIDEr scores above 1.226 on the same Karpathy split; the claim of state-of-the-art performance is therefore not established. Moreover, the baseline numbers are taken from published papers without re-running under identical conditions, and no attempt is made to control for different feature extractors, vocabulary sizes, or training schedules, so the reported advantages could reflect implementation differences rather than the proposed architecture.
minor comments (5)
- [Section IV-C] The sentence 'Here, we compare our approach with To evaluate the effectiveness of our proposed method' is incomplete and should be rewritten.
- [Section IV-D] The phrase 'Karpath' in the first sentence should be 'Karpathy'.
- [Eq. (15)] Equation (15) uses \hat v_t and \hat e_t without stage superscripts; they should be \hat v^i_t and \hat e^i_t to match Eqs. (10) and (14).
- [References, Section IV-B] The citation numbers for ROUGE and CIDEr appear to be swapped: the text cites 'CIDEr [30]' and 'ROUGE [32]', but reference [30] is Lin's ROUGE paper and reference [32] is Vedantam et al.'s CIDEr paper.
- [Figures 4-5] The y-axis labels in Figures 4 and 5 contain concatenated words such as 'fielda' and 'stuffeda', which are likely rendering artifacts; please verify the figure source files.
Circularity Check
No significant circularity: the Stack-VS proposal is an empirical architecture built on externally cited components and evaluated on the external MSCOCO Karpathy split.
full rationale
The paper's contribution is architectural (stacked decoder cells with dual visual-semantic attention) rather than a derivation from first principles. The backbone components, Faster-RCNN (Section III-B), MIL attribute extraction (Section III-B), LSTM cells (Section III-D), and SCST training (Equations 19-20), are all external, independently published methods; none is a self-citation of the present authors, and none is defined in terms of the headline metrics. The coarse-to-fine scheme is explicitly attributed to [19] (Stack-Cap), not to the authors' own prior work, so no uniqueness theorem or ansatz is smuggled in via self-citation. The final performance claims in Table I are point estimates on the external Karpathy test split; they are not derived from the training objective. The only self-referential element is that Equation 20 optimizes CIDEr as a reward and CIDEr is among the reported metrics; this is standard reinforcement-learning training for captioning rather than a fitted parameter renamed as a prediction, and it does not make the BLEU, METEOR, ROUGE, or SPICE results definitionally circular. Concerns about missing significance tests and small margins are statistical-evidence issues, not circularity.
Assumptions & free parameters
free parameters (3)
- Number of decoder stages Ns =
3 (inferred from figures; not stated in text)
- Number of semantic attributes Ne =
20
- Dimension of visual features and attribute embeddings =
2048
assumptions (3)
- domain assumption Faster-RCNN bottom-up features are a sufficient fixed visual representation of an image.
- domain assumption MIL-extracted semantic attributes reliably capture image content relevant for captioning.
- domain assumption Cross-entropy loss summed over stages plus SCST on CIDEr provides a valid training signal.
Cite this review
Pith. "Pith review of Stack-VS: Stacked Visual-Semantic Attention for Image Caption Generation." pith.science (2026). https://pith.science/paper/TEJ7OMJZ
@misc{pith2026190902489,
author = {Pith},
title = {Pith review of: Stack-VS: Stacked Visual-Semantic Attention for Image Caption Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/TEJ7OMJZ}},
note = {Machine review of arXiv:1909.02489}
}
read the original abstract
Recently, automatic image caption generation has been an important focus of the work on multimodal translation task. Existing approaches can be roughly categorized into two classes, i.e., top-down and bottom-up, the former transfers the image information (called as visual-level feature) directly into a caption, and the later uses the extracted words (called as semanticlevel attribute) to generate a description. However, previous methods either are typically based one-stage decoder or partially utilize part of visual-level or semantic-level information for image caption generation. In this paper, we address the problem and propose an innovative multi-stage architecture (called as Stack-VS) for rich fine-gained image caption generation, via combining bottom-up and top-down attention models to effectively handle both visual-level and semantic-level information of an input image. Specifically, we also propose a novel well-designed stack decoder model, which is constituted by a sequence of decoder cells, each of which contains two LSTM-layers work interactively to re-optimize attention weights on both visual-level feature vectors and semantic-level attribute embeddings for generating a fine-gained image caption. Extensive experiments on the popular benchmark dataset MSCOCO show the significant improvements on different evaluation metrics, i.e., the improvements on BLEU-4/CIDEr/SPICE scores are 0.372, 1.226 and 0.216, respectively, as compared to the state-of-the-arts.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Grounded compositional semantics for finding and describing images with sentences,
R. Socher, A. Karpathy, Q. V . Le, C. D. Manning, and A. Y . Ng, “Grounded compositional semantics for finding and describing images with sentences,” Transactions of the Association for Computational Linguistics, TACL, vol. 2, pp. 207–218, 2014
work page 2014
-
[2]
Explain images with multimodal recurrent neural networks,
J. Mao, W. Xu, Y . Yang, J. Wang, and A. L. Yuille, “Explain images with multimodal recurrent neural networks,” arXiv preprint arXiv:1410.1090, 2014
arXiv 2014
-
[3]
Deep visual-semantic alignments for generating image descriptions,
A. Karpathy and L. Fei-Fei, “Deep visual-semantic alignments for generating image descriptions,” in CVPR, 2015
work page 2015
-
[4]
Show and tell: A neural image caption generator,
O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural image caption generator,” in CVPR, 2015
2015
-
[5]
What value do explicit high level concepts have in vision to language problems?
Q. Wu, C. Shen, L. Liu, A. Dick, and A. van den Hengel, “What value do explicit high level concepts have in vision to language problems?” in CVPR, 2016
work page 2016
-
[6]
Learning to Guide Decoding for Image Captioning
W. Jiang, L. Ma, X. Chen, and et al, “Learning to guide decoding for image captioning,” arXiv:1804.00887, 2018
work page Pith review arXiv 2018
-
[7]
Boosting image captioning with attributes,
T. Yao, Y . Pan, Y . Li, Z. Qiu, and T. Mei, “Boosting image captioning with attributes,” in ICCV, 2017
work page 2017
-
[8]
Long-term recurrent convolutional networks for visual recognition and description,
J. Donahue, L. Anne Hendricks, S. Guadarrama, and et al, “Long-term recurrent convolutional networks for visual recognition and description,” in CVPR, 2015
work page 2015
Show all 34 references
-
[9]
Show, attend and tell: Neural image caption generation with visual attention,
K. Xu, J. Ba, R. Kiros, and et al, “Show, attend and tell: Neural image caption generation with visual attention,” in ICML, 2015
2015
-
[10]
Composing simple image descriptions using web-scale n-grams,
S. Li, G. Kulkarni, T. L. Berg, A. C. Berg, and Y . Choi, “Composing simple image descriptions using web-scale n-grams,” in CoNLL, 2011
2011
-
[11]
From captions to visual concepts and back,
H. Fang, S. Gupta, F. Iandola, and et al, “From captions to visual concepts and back,” in CVPR, 2015
2015
-
[12]
Image captioning with semantic attention,
Q. You, H. Jin, Z. Wang, C. Fang, and J. Luo, “Image captioning with semantic attention,” in CVPR, 2016
2016
-
[13]
Bottom-up and top-down attention for image captioning and visual question answering,
P. Anderson, X. He, C. Buehler, and et al, “Bottom-up and top-down attention for image captioning and visual question answering,” in CVPR, 2018
2018
-
[14]
Long short-term memory
S. Hochreiter and J. Schmidhuber, “Long short-term memory.”
-
[15]
Long-term recurrent convolutional networks for visual recognition and description,
J. Donahue, L. Anne Hendricks, S. Guadarrama, M. Rohrbach, S. Venu- gopalan, K. Saenko, and T. Darrell, “Long-term recurrent convolutional networks for visual recognition and description,” in CVPR
-
[16]
Mat: A multimodal attentive translator for image captioning,
C. Liu, F. Sun, C. Wang, and et al, “Mat: A multimodal attentive translator for image captioning,”arXiv preprint arXiv:1702.05658, 2017
2017 arXiv
-
[17]
Knowing when to look: Adaptive attention via a visual sentinel for image captioning,
J. Lu, C. Xiong, D. Parikh, and R. Socher, “Knowing when to look: Adaptive attention via a visual sentinel for image captioning,” in CVPR, 2017
2017
-
[18]
Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning,
L. Chen, H. Zhang, J. Xiao, L. Nie, and et al, “Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning,” arXiv preprint arXiv:1611.05594 , 2016
2016 arXiv
-
[19]
Stack-captioning: Coarse-to-fine learning for image captioning,
J. Gu, J. Cai, G. Wang, and T. Chen, “Stack-captioning: Coarse-to-fine learning for image captioning,” in AAAI, 2018
2018
-
[20]
Adam: A method for stochastic optimization,
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” 2014
2014
-
[21]
Self-critical sequence training for image captioning,
S. J. Rennie, E. Marcheret, Y . Mroueh, and et al, “Self-critical sequence training for image captioning,” in CVPR, 2017
2017
-
[22]
Deep captioning with multimodal recurrent neural networks (m-rnn),
J. Mao, W. Xu, Y . Yang, J. Wang, Z. Huang, and A. Yuille, “Deep captioning with multimodal recurrent neural networks (m-rnn),” ICLR, 2014
2014
-
[23]
Review networks for caption generation,
Z. Yang, Y . Yuan, Y . Wu, W. W. Cohen, and R. R. Salakhutdinov, “Review networks for caption generation,” in NIPS, 2016
2016
-
[24]
Gated hierarchical attention for image captioning,
Q. Wang and A. B. Chan, “Gated hierarchical attention for image captioning,” 2018
2018
-
[25]
Faster r-cnn: Towards real-time object detection with region proposal networks,
S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in Advances in neural information processing systems , 2015, pp. 91–99
2015
-
[26]
Sequence level train- ing with recurrent neural networks,
M. Ranzato, S. Chopra, M. Auli, and W. Zaremba, “Sequence level train- ing with recurrent neural networks,” arXiv preprint arXiv:1511.06732 , 2015
2015 arXiv
-
[27]
Optimization of image description metrics using policy gradient methods,
S. Liu, Z. Zhu, N. Ye, S. Guadarrama, and K. Murphy, “Optimization of image description metrics using policy gradient methods,” CoRR, abs/1612.00370, vol. 2, 2016. 11
2016 arXiv
-
[28]
Microsoft coco captions: Data collection and evaluation server,
X. Chen, H. Fang, T.-Y . Lin, R. Vedantam, S. Gupta, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco captions: Data collection and evaluation server,” arXiv preprint arXiv:1504.00325 , 2015
2015 arXiv
-
[29]
Microsoft coco: Common objects in context,
T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in ECCV, 2014
2014
-
[30]
Rouge: A package for automatic evaluation of summaries,
C.-Y . Lin, “Rouge: A package for automatic evaluation of summaries,” Text Summarization Branches Out , 2004
2004
-
[31]
Bleu: a method for automatic evaluation of machine translation,
K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in ACL, 2002
2002
-
[32]
Cider: Consensus- based image description evaluation,
R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “Cider: Consensus- based image description evaluation,” in CVPR, 2015
2015
-
[33]
Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,
S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,” in ACL Workshop, 2005
2005
-
[34]
Spice: Semantic propositional image caption evaluation,
P. Anderson, B. Fernando, M. Johnson, and S. Gould, “Spice: Semantic propositional image caption evaluation,” in ECCV, 2016. PLACE PHOTO HERE Wei Wei received the PhD degree from Huazhong University of Science and Technology, Wuhan, China, in 2012. He is currently an associate...
2016
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.