{"id":"d734cdf5-e0c6-4219-9114-0974af6affb5","arxiv_id":"2501.14828","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"An ensemble of eight CNN feature extractors plus a transformer reports high BLEU scores on Flickr8k and Flickr30k, but the final caption is selected using the test references, invalidating the evaluation.","lead":"The paper combines eight pretrained image networks with a transformer caption generator and selects the final caption by its BLEU score. The reported benchmark gains on Flickr8k and Flickr30k are not trustworthy because that BLEU-based choice uses the ground-truth captions that the scores are measured against.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'voting' step selects the caption with the highest BLEU-1 score, which requires test reference captions at selection time; the reported SOTA scores are oracle-selected and not deployment-realizable.","rationale":"The reader's weakest assumption is the same as the key concern: the voting step presupposes reference captions. I agree. This is the single most load-bearing issue because the paper's entire contribution is the ensemble-voting framework, and the reported state-of-the-art numbers are the evidence for it. If selection uses test references, the metric is circular, so the central claim fails regardless of other ambiguities. The undefined 'voting-on', the contradiction between eight CNNs and eight transformers, and the METEOR/CIDEr table mismatch all reinforce the assessment but are not needed to reject. Given the paper supplies no code or training details, an independent reimplementation is the only way to verify; a reference-free selection variant is the decisive experiment.","tokens_in":21451,"tokens_out":3098,"duration_ms":32230,"concrete_test":"Implement the selection step without using any reference caption: for each test image, generate the eight candidate captions and choose by (a) the highest mean log-likelihood from each model's decoder, or (b) majority vote over the top beam-search candidates, then compute BLEU-1 through BLEU-4 on Flickr30k. If BLEU-1 falls below the reported 0.798 (and the other BLEU scores similarly), the published numbers depend on oracle access to references. If the authors instead can provide a selection rule that never calls the reference captions and still reproduces 0.798, the objection is answered.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on the ensemble vote in Section 3.3.5 and Algorithm 1 Steps 23–26, where 'the prediction result will be accepted from the model that gains the highest BLEU score.' BLEU is computed against human reference captions. At inference on a new image no such references exist. Therefore the method as specified cannot select a caption without peeking at the ground truth used later to compute the reported metrics. The scores in Tables 2 and 3 (e.g., BLEU-1 0.798 on Flickr30k) are thus maxima over the candidate set with respect to the evaluation references, not the output of a deployable ensemble. The paper never discusses this reference requirement or offers a reference-free proxy; 'voting-on' is undefined, and the description alternates between ensembling eight CNN feature extractors and eight transformer models. Even if those ambiguities were resolved, the oracle-selection protocol alone invalidates the headline comparison with prior single-model results, because no competing method is allowed access to test references during generation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an image captioning system that combines eight CNN feature extractors with a transformer encoder-decoder and an ensemble voting step. The final caption is selected as the one with the highest BLEU-1 score among the eight models' outputs, and the model is evaluated on Flickr8k and Flickr30k, where it reports state-of-the-art BLEU, METEOR, CIDEr, and SPICE scores. The central technical content is a survey-style exposition of transformers, attention mechanisms, and ensemble learning, followed by experiments with several CNN backbones.","tokens_in":21643,"tokens_out":3955,"duration_ms":38202,"significance":"If valid, the proposed ensemble and the reported gains over prior work would be a useful contribution to image captioning. The paper does provide a reasonably broad review of related work, an ablation study, and qualitative examples, and it reports a range of metrics. However, the evaluation protocol is fundamentally circular: the final caption is chosen by comparing candidate captions against the same reference captions that are later used to compute the reported BLEU scores. This makes the headline numbers oracle-selected maxima rather than the outputs of a deployable system, and the comparison with prior methods is therefore not meaningful. The absence of a reference-free selection rule means the central claim of state-of-the-art performance is not established.","major_comments":[{"comment":"The ensemble voting step selects the caption with the highest BLEU-1 score computed against the reference captions, and the same references are then used to compute the BLEU scores reported in Tables 2 and 3. At inference on a new image, reference captions are not available, so the method as specified cannot select a caption. The reported numbers are therefore a maximum over the candidate set with respect to the evaluation references, not the output of a deployable ensemble, and the comparison with prior methods is invalid because no competing method is allowed access to test references during generation. The paper offers no reference-free proxy for the voting step, so this is a load-bearing flaw in the central claim.","section":"§3.3.5, Algorithm 1 Steps 23–26"},{"comment":"The ensemble architecture is specified inconsistently: Algorithm 1 (line 12) lists eight CNN feature extractors (ResNet50, ResNet101, EfficientNetV2, VGG16, VGG19, EfficientNetB4, ResNet152, RegNetX120), while §3.3.5 states that the voting model combines the results of 'each of the eight transformer models,' and §4.6 says the model 'combines eight CNN models via a voting process.' Without a unique statement of what is being ensembled, the method is not reproducible and the attribution of the reported gains is unclear.","section":"§3.3.5 vs. §4.6"},{"comment":"The quantitative presentation contains a direct contradiction: the text in §4.3 says 'Our model obtained the highest result of the METEOR score, 0.604,' but Table 2 reports METEOR=0.235 and CIDEr=0.604, and the same paragraph later lists 'ROUGE L (0.432) and CIDEr (0.604)' as competitive results. This misreporting of the metric values undermines confidence in the numerical claims, although it is secondary to the circularity issue above.","section":"Tables 2 and 3, §4.3"},{"comment":"The ablation study does not cleanly isolate the contribution of the proposed ensemble. The ablation baselines include MobileNetV2, which is absent from the full ensemble model in Algorithm 1 (line 12), and no explanation is given for how the full model relates to the baselines. Consequently, the apparent monotonic improvement from Baseline 1 to the full ensemble cannot be attributed to the specific CNN set or to the voting mechanism.","section":"§4.5, Table 4"}],"minor_comments":[{"comment":"The dataset name is spelled inconsistently as both 'Flickr8K' and 'Flicker8k'; the latter spelling appears in several places, including Tables 2 and 4.","section":"Throughout"},{"comment":"The row for the proposed model lists '0.2690.443' without a separator, making it difficult to distinguish the BLEU-4 value from the ROUGE-L value.","section":"Table 3"},{"comment":"The phrase 'voting-on' in Algorithm 1 (line 25) is never defined; the surrounding text describes BLEU-based selection, but the algorithm and the prose should use a single, precise term.","section":"§3.3.5"},{"comment":"The claim that paired t-tests between the full ensemble and each ablation variant were statistically significant (p < 0.05) is not accompanied by the test statistic, the number of samples, or the exact p-values, so the claim cannot be verified.","section":"§4.6"}],"recommendation":"reject","confidential_remarks":"The paper's central evaluation is circular because the ensemble selects captions by BLEU-1 against test references and then reports BLEU on those same references. This is a fundamental flaw that invalidates the headline state-of-the-art claim, and the architecture ambiguity (eight CNNs versus eight transformers) makes the method irreproducible. Addressing these issues would require redesigning the selection rule and re-running all experiments, which is beyond the scope of a revision. I recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the headline numbers don't survive contact with the method. The ensemble 'voting' step picks the caption with the highest BLEU-1 score (Section 3.3.5, Algorithm 1), which requires the test reference captions at selection time. Those same references are then used to compute the reported BLEU values. So the 0.798 on Flickr30k is an oracle-selected maximum over eight candidate captions, not the output of any deployable model. That makes the central comparison with prior single-model results invalid.\n\nThe paper isn't without merit. The background on attention and transformer-based captioning is clear and readable, and the related work section covers ensemble methods reasonably well. The ablation study on Flickr8k is a sensible idea, and the monotonic improvement as more CNN backbones are added is at least internally consistent. Reporting multiple metrics (BLEU, ROUGE, METEOR, CIDEr, SPICE) is good practice.\n\nThe soft spots beyond the circularity: the method description is inconsistent about whether the ensemble consists of eight CNN feature extractors or eight transformer models. 'Voting-on' is never defined. The paired t-test is mentioned with no details. The text and tables disagree on METEOR and CIDEr values (0.604 appears as both). No code, model sizes, or training details that would allow reproduction are given.\n\nThe novelty is modest: ensemble image captioning exists in the cited literature (Katpally et al. 2020; Zhong et al. 2022), and adding a transformer decoder to eight named CNN backbones is a routine variation. The only new empirical results are the specific benchmark numbers, and those are tainted by the oracle-selection protocol.\n\nWho is this for? Possibly as a cautionary example in a discussion of evaluation pitfalls. Not as a method paper. My recommendation to the editor: reject. If the authors replace the BLEU-based voting with a reference-free combination (e.g., average log-probability, or a trained fusion layer) and re-run the benchmarks, the approach might warrant another look. As it stands, the core claim collapses under inspection.","headline":"The headline numbers are an artifact of oracle selection: the ensemble picks the caption with the highest BLEU against test references and then reports that BLEU.","tokens_in":22197,"tokens_out":2908,"would_cite":false,"duration_ms":29068,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that an ensemble of eight pre-trained CNN feature extractors feeding an attention-based transformer decoder, with voting that selects the highest-BLEU caption, outperforms published image-captioning results on Flickr8k…","keywords":["Image captioning","Ensemble learning","Transformer","Attention mechanism","Voting","BLEU","Flickr8k","Flickr30k"],"falsifier":"Run the same eight CNN/transformer configurations on Flickr30k with the voting step replaced by a fixed aggregator or by a scorer that does not see the reference captions. If BLEU-1 and BLEU-4 drop to the level of the best single model, the reported gain depends on knowing the ground-truth caption during selection.","tokens_in":21217,"feed_emoji":"🖼️","tokens_out":9303,"duration_ms":82349,"temperature":0.7,"pith_summary":"The paper argues that image captioning improves when several pre-trained CNN feature extractors each feed a transformer encoder-decoder and an ensemble-voting step chooses the single best candidate caption. If true, this matters because it offers a modular recipe for captioning systems that avoids recurrent language models and attributes the gain to diversity among visual encoders. The authors report BLEU-1 through BLEU-3 scores of 0.728, 0.495, and 0.323 on Flickr8K and BLEU-1 through BLEU-4 scores of 0.798, 0.561, 0.387, and 0.269 on Flickr30K, claiming these exceed the latest published methods on those benchmarks.","feed_headline":"Captioning ensemble beats single models on Flickr8k and Flickr30k","feed_subtitle":"Eight pretrained CNNs feed an attention transformer, and a BLEU-1 vote picks the best caption.","key_machinery":"The load-bearing machinery is the ensemble-voting pipeline: eight pre-trained CNN feature extractors each produce a visual feature map, a transformer encoder-decoder with multi-head scaled dot-product attention turns each feature map into a caption using beam search of width 10, and a voting module compares the candidate captions against reference captions and keeps the one with the highest BLEU-1 score. The attention mechanism is what aligns image regions with each generated word, and the voting step is what the paper credits for the final gain over individual models.","core_discovery":"On the paper's own terms, the discovery is that combining many CNN visual encoders before a transformer decoder, then selecting the candidate caption with the highest BLEU-1 score, makes generated captions richer and more accurate than any single model in the ensemble. On Flickr8K the proposed model reports BLEU-1/2/3 of 0.728, 0.495, and 0.323; on Flickr30K it reports BLEU-1 through BLEU-4 of 0.798, 0.561, 0.387, and 0.269, plus SPICE of 0.387, and states these outperform the latest methods. The ablation study shows the full ensemble beating every reduced ensemble on all reported metrics, with paired t-tests at p < 0.05 separating the configurations.","pith_inferences":["If the voting step indeed compares candidates with ground-truth captions, the reported numbers are an upper bound; a deployed system would need a reference-free selector, leaving the practical gain over a single model unmeasured.","The ablation's monotone improvements suggest diversity among encoders drives the result; a fair comparison would hold computation constant and test a single transformer trained longer.","A testable extension is to replace BLEU-1 selection with a semantic scorer such as SPICE, which may reward a different winner and change the ensemble's reported trade-offs."],"forward_implications":["A captioning system can be built without recurrent language models: pretrained CNNs plus an attention-based transformer produce the reported gains.","Adding more diverse CNN encoders monotonically raises BLEU and SPICE scores on Flickr8k, according to the ablation study.","Voting by BLEU-1 outperforms every single base model in the ensemble, so ensemble aggregation is the reported source of the improvement.","The reported Flickr30k BLEU-1 through BLEU-4 scores (0.798, 0.561, 0.387, 0.269) exceed all cited comparison methods on those four n-gram levels."],"supporting_citations":[{"why":"It supplies the transformer encoder-decoder and scaled dot-product/multi-head attention used for caption generation.","marker":"[59]"},{"why":"It provides the Flickr8K image-caption dataset used for training and evaluation.","marker":"[42]"},{"why":"It provides the Flickr30K dataset where the model reports its best BLEU scores.","marker":"[43]"},{"why":"It defines the BLEU metric used both for reporting scores and as the voting criterion in the ensemble.","marker":"[76]"},{"why":"It defines the SPICE scene-graph metric used to report semantic caption quality.","marker":"[80]"},{"why":"It supplies the ensemble-learning rationale and voting background the framework builds on.","marker":"[23]"},{"why":"It supports beam search as the decoding method that generates the candidate caption set.","marker":"[68]"}],"fun_headline_variants":["Ensemble CNN-transformer with BLEU vote tops Flickr8k and 30k","Vote by BLEU-1: multi-CNN transformer wins caption benchmarks","Multi-candidate BLEU-1 vote yields richer image captions","Flickr8k/30k ensemble transformer beats single-model captions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The voting step assumes that at selection time the system can compare each candidate caption with human-written reference captions and pick the one with the highest BLEU-1 score; without such references, the selection procedure cannot run as described.","fun_headline_variants_meta":{"raw":{"variants":["Ensemble CNN-transformer with BLEU vote tops Flickr8k and 30k","Vote by BLEU-1: multi-CNN transformer wins caption benchmarks","Multi-candidate BLEU-1 vote yields richer image captions","Flickr8k/30k ensemble transformer beats single-model captions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001362,"raw_usage":{"total_tokens":5567,"prompt_tokens":1028,"completion_tokens":4539,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":4454}},"tokens_in":644,"tokens_out":4539,"duration_ms":35241,"temperature":1.0,"reasoning_tokens":4454,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:44:17.428404+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same eight CNN/transformer configurations on Flickr30k with the voting step replaced by a fixed aggregator or by a scorer that does not see the reference captions. If BLEU-1 and BLEU-4 drop to the level of the best single model, the reported gain depends on knowing the ground-truth caption during selection.","supporting_citations":[{"cited_title":"In: Advances in Neural Information Processing Systems, pp","cited_arxiv_id":null,"evidence_quote":"It supplies the transformer encoder-decoder and scaled dot-product/multi-head attention used for caption generation."},{"cited_title":"Journal of Artificial Intelligence Research 47, 853–899 (2013)","cited_arxiv_id":null,"evidence_quote":"It provides the Flickr8K image-caption dataset used for training and evaluation."},{"cited_title":"Transactions of the Association for Computational Linguistics2, 67–78 (2014)","cited_arxiv_id":null,"evidence_quote":"It provides the Flickr30K dataset where the model reports its best BLEU scores."},{"cited_title":"In: European Conference on Computer Vision, pp","cited_arxiv_id":null,"evidence_quote":"It defines the SPICE scene-graph metric used to report semantic caption quality."},{"cited_title":"Frontiers of Computer Science14, 241–258 (2020)","cited_arxiv_id":null,"evidence_quote":"It supplies the ensemble-learning rationale and voting background the framework builds on."}],"review_version":1}