{"id":"816fb4e8-86c3-4101-8436-1ccb510b9f6e","arxiv_id":"1909.02489","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Stack-VS stacks LSTM decoder cells that jointly attend to visual features and semantic attributes to refine image captions stage by stage, with reported gains over 2018 baselines.","lead":"This paper proposes a new image captioning model that combines two existing ideas: multi-stage sentence refinement and simultaneous attention to visual regions and semantic attributes. The authors report small score gains over older baselines on the MSCOCO benchmark, but omit comparison with the strongest 2019 models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim of statistically significant improvements over baselines is unsupported, as the reported gains of +0.009 BLEU-4 and +0.025 CIDEr are within typical run-to-run noise and no variance or significance test is provided.","rationale":"The paper's value proposition is an empirical state-of-the-art claim. The architecture is plausible, but the evidence for superiority is a few hundredths of a point on metrics known to have substantial seed-level variance. No code, no variance, and no significance test are provided. The reader correctly identified this as the weakest assumption. The single most load-bearing issue is that the central claim of 'statistically significant' improvement is not merely unproven but likely false; the reported deltas are smaller than the typical noise in these benchmarks. This would invalidate the paper's main conclusion regardless of the architecture's novelty. I agree with the reader's assessment, so the verdict should remain REJECT (UNCHANGED).","tokens_in":16964,"tokens_out":9774,"duration_ms":93979,"concrete_test":"Run Stack-VS and Up-Down [CIDEr-Optimize] under identical training conditions for 10 random seeds each (same optimizer, batch size, epochs, and SCST schedule), then compute a two-sided paired test on CIDEr. If the 95% confidence interval of the mean difference includes 0, the 'statistically significant' claim in Section IV-D fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim in Section IV-D is that 'Stack-VS consistently outperforms all baseline methods, and the improvements are statistically significant on all metrics.' This assertion rests solely on point estimates in Table I, where the largest advantages over the strongest baseline (Up-Down [CIDEr-Optimize]) are +0.009 BLEU-4, +0.025 CIDEr, and +0.002 SPICE. The paper reports no variance across seeds, no number of runs, no confidence intervals, and no significance test for any metric. Captioning metrics on the Karpathy split exhibit substantial seed-to-seed variance, so differences of this magnitude are plausibly within noise. Moreover, the baselines are taken from published papers without re-running under identical conditions, and the table omits contemporaneous models (e.g., RFNet, GCN-LSTM, SGAE) that reported higher CIDEr than 1.226 on the same split, undermining the 'state-of-the-art' framing. For the claim of consistent, significant improvement to hold, the small differences would need to be shown as reproducible and statistically significant, which the paper does not do.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes Stack-VS, a multi-stage image captioning architecture that combines visual features from Faster R-CNN with semantic attribute embeddings obtained through multiple-instance learning. The decoder is a stack of decoder cells, each containing a visual-semantic attention layer and a language LSTM, with hidden states and attention weights refined across stages. The model is trained with per-stage cross-entropy and then with self-critical sequence training optimizing CIDEr. On the MSCOCO Karpathy test split the authors report BLEU-4 0.372, CIDEr 1.226, and SPICE 0.216, and claim consistent, statistically significant improvements over all baselines. The paper also includes qualitative analyses of visual and semantic attention refinement across stages.","tokens_in":17210,"tokens_out":7057,"duration_ms":70292,"significance":"If the empirical claims were substantiated, Stack-VS would be a useful extension of the coarse-to-fine captioning idea: it jointly models visual and semantic attention in a multi-stage decoder, and the staged refinement is illustrated with detailed qualitative examples. The training protocol is standard and the evaluation covers both the Karpathy split and the MSCOCO online server. However, the central claim of statistically significant and consistent improvement is not supported by the evidence: the paper provides no variance estimates, no significance tests, and Table II actually contradicts the 'consistently outperforms' statement on BLEU-1/2/3. The architecture is also underspecified in several key equations. The underlying idea is potentially valuable, but the current manuscript does not establish its headline claims.","major_comments":[{"comment":"The claim that 'the improvements are statistically significant on all metrics' is not supported by any statistical evidence. The paper reports point estimates without variance, number of runs, confidence intervals, or significance tests; the margins over Up-Down [CIDEr-Optimize] on the Karpathy test split are +0.009 BLEU-4, +0.025 CIDEr, and +0.002 SPICE, which are well within typical run-to-run noise for captioning metrics. Please provide multi-run results with standard deviations and an appropriate significance test, or retract the significance claim.","section":"Section IV-D, Table I"},{"comment":"Table II shows that Stack-VS [CIDEr-Optimize] scores below Up-Down [CIDEr-Optimize] on BLEU-1 (0.790 vs 0.802), BLEU-2 (0.634 vs 0.641), and BLEU-3 (0.489 vs 0.491) on the MSCOCO online server, yet the text states that 'our proposed model Stack-VS consistently outperforms all baseline methods.' This internal contradiction must be resolved by qualifying the claim or by explaining why the online-test comparison is not the relevant one.","section":"Section IV-D, Table II"},{"comment":"The decoder equations are underspecified. Equation (4) defines h^i_t as the output of a generic LSTM, but Section III-D defines three different LSTM outputs (h^i,V_t, h^i,S_t, h^i,L_t) and never states which, if any, corresponds to h^i_t or to the h^(i-1)_t used in Eq. (5). Equation (17) writes the word probability as a function of h^i,L_k while the conditioning set includes I^(i-1), but Eq. (16) shows no dependence on I^(i-1); the equation does not match the text. In addition, x^i,S_t is used in Eq. (11) but never defined. The architecture section must be rewritten so that each state is defined exactly once with consistent stage and time indices.","section":"Section III-C, III-D, Eqs. (4), (5), (15)-(17)"},{"comment":"The number of stages Ns is never specified in the text. Figures 4 and 5 use three stages, but the paper should state the value used in the experiments and whether it was tuned. This number is a key architectural hyperparameter that affects parameter count, training time, and the qualitative behavior of the stacked refinement.","section":"Section III-C"},{"comment":"The comparison with baselines is incomplete. Table I omits several contemporaneous models, such as RFNet, GCN-LSTM, and SGAE, which reported CIDEr scores above 1.226 on the same Karpathy split; the claim of state-of-the-art performance is therefore not established. Moreover, the baseline numbers are taken from published papers without re-running under identical conditions, and no attempt is made to control for different feature extractors, vocabulary sizes, or training schedules, so the reported advantages could reflect implementation differences rather than the proposed architecture.","section":"Section IV-C and Table I"}],"minor_comments":[{"comment":"The sentence 'Here, we compare our approach with To evaluate the effectiveness of our proposed method' is incomplete and should be rewritten.","section":"Section IV-C"},{"comment":"The phrase 'Karpath' in the first sentence should be 'Karpathy'.","section":"Section IV-D"},{"comment":"Equation (15) uses \\hat v_t and \\hat e_t without stage superscripts; they should be \\hat v^i_t and \\hat e^i_t to match Eqs. (10) and (14).","section":"Eq. (15)"},{"comment":"The citation numbers for ROUGE and CIDEr appear to be swapped: the text cites 'CIDEr [30]' and 'ROUGE [32]', but reference [30] is Lin's ROUGE paper and reference [32] is Vedantam et al.'s CIDEr paper.","section":"References, Section IV-B"},{"comment":"The y-axis labels in Figures 4 and 5 contain concatenated words such as 'fielda' and 'stuffeda', which are likely rendering artifacts; please verify the figure source files.","section":"Figures 4-5"}],"recommendation":"major_revision","confidential_remarks":"The paper does not provide code or released models, which makes independent verification of the reported numbers difficult. The claimed improvement over the state of the art is also not supported by the evidence as presented, and the architecture has several reproducibility gaps. The topic is within the journal's scope, and the idea is plausible, but I would not recommend acceptance until the statistical claims, baseline comparison, and equation-level clarity are substantially revised."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the genuinely new thing here is the specific composition—multi-stage Stack-Cap-style decoding with a joint visual-semantic attention inside every decoder cell. I don't see that exact combination in the cited literature, and the two-LSTM cell (one visual, one semantic, feeding a language LSTM) is a reasonable design. The qualitative figures suggest the model does what it says: semantic attention rises for prepositions, visual attention sharpens across stages. Those are real assets.\n\nThe problems are the empirical ones, and they are exactly what the reader flagged. The headline gains over Up-Down CIDEr-Optimize are +0.009 BLEU-4, +0.025 CIDEr, +0.002 SPICE on the Karpathy split. Captioning numbers at that scale move with random seed, and the paper gives no seeds, no repeats, no confidence intervals, no significance test. The sentence in Section IV-D that the improvements are 'statistically significant on all metrics' is not backed by any statistic. Baselines are copied from published papers rather than re-run, so small differences could be implementation or evaluation-script artifacts. The table also omits contemporaneous models (RFNet, GCN-LSTM, SGAE) that reported higher CIDEr on the same split, so the 'state-of-the-art' framing is misleading.\n\nThere are also presentation issues. The formal definition of the stage-level hidden state in Eq. (4) doesn't line up with the three LSTM outputs in the decoder cell; Eq. (17) writes h^{i,L}_k after the text has defined h^{i,L}_t; and the number of stages Ns is never stated anywhere. These are fixable, but they make the method hard to reproduce. No code is provided.\n\nWould I cite it? No. Would I send it to reviewers? Not as is. The architecture idea is worth one page in someone's related work, and a revised version with variance estimates, contemporary baselines, explicit Ns, and code could be a legitimate workshop or short-conference paper. But the central claim—consistent, statistically significant improvement over the state of the art—is not established by the evidence in this draft.","headline":"Credible but incremental composition of known components; the empirical evidence (tiny point gains, no variance) does not support the 'statistically significant improvements' claim.","tokens_in":17731,"tokens_out":1948,"would_cite":false,"duration_ms":21897,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A stacked visual-semantic attention decoder reports state-of-the-art image captioning results on MSCOCO, with BLEU-4 0.372 and CIDEr 1.226.","keywords":["image captioning","visual-semantic attention","multi-stage decoding","coarse-to-fine","bottom-up and top-down attention","LSTM decoder","MSCOCO","reinforcement learning"],"falsifier":"Run Stack-VS and the two strongest baselines, Up-Down and Stack-Cap, from multiple random seeds on the same MSCOCO split, with the same feature files, vocabulary, and evaluation script, then compare confidence intervals for BLEU-4, CIDEr, and SPICE; if the intervals overlap, the claimed consistent significant improvement is falsified.","tokens_in":16779,"feed_emoji":"🖼️","tokens_out":8746,"duration_ms":90114,"temperature":0.7,"pith_summary":"The paper argues that image captioning improves when the decoder not only attends to both visual regions and semantic attributes but keeps refining those attention weights across several stacked stages. It introduces Stack-VS, a coarse-to-fine decoder made of linked cells, each combining a visual-attention LSTM, a semantic-attention LSTM, and a language LSTM. On the standard 5,000-image MSCOCO test split, after cross-entropy training followed by CIDEr-reward reinforcement learning, the authors report BLEU-4 0.372, CIDEr 1.226, and SPICE 0.216, above the compared baselines including Up-Down and Stack-Cap. If the result holds, the takeaway is that staged joint visual-semantic attention is a viable route to finer-grained captions, not merely a marginal tweak.","feed_headline":"Stacked visual-semantic attention lifts image captions on MSCOCO","feed_subtitle":"Re-weighting image regions and semantic attributes over multiple stages yields BLEU-4 0.372 and CIDEr 1.226.","key_machinery":"The load-bearing object is the Stack-VS decoder cell, a two-LSTM attention module repeated across decoding stages. In each cell, $\\mathrm{LSTM}_V$ and $\\mathrm{LSTM}_S$ compute separate hidden states, and their attention logits over visual features $v_k \\in V_0$ and semantic embeddings $e_k \\in E_0$ are computed by Eqs. (8) and (12), with each logit depending on the current hidden states of both LSTM branches and on the attended vectors from the previous stage. The attended vectors $\\hat{v}_t^i$ and $\\hat{e}_t^i$ are projected and summed with the two LSTM hidden states to form the input of a language LSTM, whose output predicts the next word. The same cell is stacked multiple times per time step, so each stage re-optimizes the visual-semantic attention weights before passing its result to the next stage.","core_discovery":"The central claim is that a multi-stage, or stacked, decoder that jointly re-optimizes two attention streams—one over Faster R-CNN visual region features and one over MIL-derived semantic attribute embeddings—produces richer image captions than single-stage decoders or decoders using only one modality. Each stage refines the previous stage's hidden state and attended visual and semantic vectors, and the attention weights are recomputed cell by cell rather than shared across stages, unlike the earlier Stack-Cap model. The paper reports that this architecture consistently outperforms all compared baselines on the MSCOCO offline split, with the CIDEr-optimized variant reaching BLEU-4 0.372, CIDEr 1.226, and SPICE 0.216, and also reports higher BLEU-4, METEOR, ROUGE-L, and CIDEr than Up-Down and Stack-Cap on the MSCOCO online C5 test set.","pith_inferences":["An ablation that deletes the semantic branch while keeping the stacked decoder, or deletes the stacking while keeping the two LSTM branches, would isolate whether the gain comes from joint attention or simply from added model capacity; the paper does not report such an ablation.","Because the reported leads over Up-Down and Stack-Cap are close to typical run-to-run variation (for example, +0.025 CIDEr), repeated seeded runs with confidence intervals are needed before treating the new numbers as a stable state of the art.","The same two-branch stacked attention cell could transfer to visual question answering or video captioning, where an answer or caption often needs a first coarse pass followed by refined attention over objects and attributes."],"forward_implications":["If the reported scores hold, a joint stacked visual-semantic decoder is a stronger recipe on MSCOCO than either the visual-only multi-stage Stack-Cap or the visual-only single-stage Up-Down: Stack-VS reports BLEU-4 0.372, CIDEr 1.226, and SPICE 0.216.","The stage-by-stage reweighting is credited with fixing relational words: semantic attention weights rise for prepositions like 'in' and 'on' while visual attention carries nouns, so the largest gains over single-stage baselines would be in captions involving spatial relations.","The training scheme is compatible with standard RL fine-tuning: the CIDEr-optimized version, not the cross-entropy version alone, produces the headline numbers, meaning the architecture benefits from self-critical sequence training.","The online MSCOCO C5 results show the same qualitative ordering over Up-Down and Stack-Cap, suggesting the improvement is not an artifact of the offline split alone."],"supporting_citations":[{"why":"Defines the bottom-up and top-down attention baseline (Up-Down) that Stack-VS extends, and supplies the visual feature extraction setup reused here.","marker":"[13]"},{"why":"Introduces the coarse-to-fine stacked decoding structure that Stack-VS adopts, and is the primary visual-only multi-stage baseline.","marker":"[19]"},{"why":"Supplies the weakly-supervised attribute detection method used to obtain the semantic attribute embeddings $E_0$.","marker":"[7]"},{"why":"Semantic attention baseline and a source of the MIL attribute idea the paper builds on.","marker":"[12]"},{"why":"Provides self-critical sequence training, the reinforcement-learning optimizer used in the CIDEr-optimized variant.","marker":"[21]"},{"why":"Faster R-CNN is the detector that produces the visual-level region features $V_0$.","marker":"[25]"},{"why":"From captions to visual concepts and back, cited as the MIL model for learning semantic attributes.","marker":"[11]"}],"fun_headline_variants":["Stacked attention refines captions stage by stage","Two-stream attention stack beats single-stage captioners","Multi-stage decoder re-weights visual and semantic cues","Stack-VS boosts caption scores with stacked attention"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim of consistent improvement rests on treating the baselines' published scores as directly comparable to one unseeded run of Stack-VS; the reported margins are only a few hundredths of a metric point, so run-to-run noise could erase them.","fun_headline_variants_meta":{"raw":{"variants":["Stacked attention refines captions stage by stage","Two-stream attention stack beats single-stage captioners","Multi-stage decoder re-weights visual and semantic cues","Stack-VS boosts caption scores with stacked attention"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000207,"raw_usage":{"total_tokens":1428,"prompt_tokens":1003,"completion_tokens":425,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":364}},"tokens_in":619,"tokens_out":425,"duration_ms":5379,"temperature":1.0,"reasoning_tokens":364,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:48:40.014602+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Stack-VS and the two strongest baselines, Up-Down and Stack-Cap, from multiple random seeds on the same MSCOCO split, with the same feature files, vocabulary, and evaluation script, then compare confidence intervals for BLEU-4, CIDEr, and SPICE; if the intervals overlap, the claimed consistent significant improvement is falsified.","supporting_citations":[{"cited_title":"Bottom-up and top-down attention for image captioning and visual question answering,","cited_arxiv_id":null,"evidence_quote":"Defines the bottom-up and top-down attention baseline (Up-Down) that Stack-VS extends, and supplies the visual feature extraction setup reused here."},{"cited_title":"Stack-captioning: Coarse-to-fine learning for image captioning,","cited_arxiv_id":null,"evidence_quote":"Introduces the coarse-to-fine stacked decoding structure that Stack-VS adopts, and is the primary visual-only multi-stage baseline."},{"cited_title":"Boosting image captioning with attributes,","cited_arxiv_id":null,"evidence_quote":"Supplies the weakly-supervised attribute detection method used to obtain the semantic attribute embeddings $E_0$."},{"cited_title":"Image captioning with semantic attention,","cited_arxiv_id":null,"evidence_quote":"Semantic attention baseline and a source of the MIL attribute idea the paper builds on."},{"cited_title":"Self-critical sequence training for image captioning,","cited_arxiv_id":null,"evidence_quote":"Provides self-critical sequence training, the reinforcement-learning optimizer used in the CIDEr-optimized variant."},{"cited_title":"From captions to visual concepts and back,","cited_arxiv_id":null,"evidence_quote":"From captions to visual concepts and back, cited as the MIL model for learning semantic attributes."}],"review_version":1}