{"id":"4cf6d5ec-b9bb-4176-a892-e23714a21c10","arxiv_id":"2501.09155","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"VCRScore, a gradient-boosting regressor on ViLT, CLIP, and lexical precision/recall features, achieves the highest Spearman correlation with human caption-quality scores among the metrics tested.","lead":"The authors train a machine-learning model, VCRScore, to score how well a caption describes an image, using vision-language transformer and object-detection features, and report it ranks caption quality more like human raters than BLEU, CIDEr, or CLIPScore. Better image-caption metrics would make model comparisons more reliable, but the reported advantage depends on data-filtering choices that need closer scrutiny.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Image-level train/test leakage is the load-bearing risk: the §4.1 splits are not stratified by image, so VCRScore may memorize seen images and inflate Table 11.","rationale":"The reader and I both locate the central risk in the data-processing choices, so I partially agree with the reader's weakest assumption. The removal of 21,974 zero-score samples from the external datasets is a real distributional bias, but I consider the unstated split construction more directly threatening: the own dataset is only 100 images with six captions each, and a random pair-level 30/70 split leaves captions from nearly every image in both train and test. Because VCRScore is a supervised regressor consuming image-text features, this creates an image-memorization path that the zero-shot baselines do not have. The same issue can affect the external 70/30 split when multiple captions share an image. The paper never states that splits were grouped by image, and the released repository should make this checkable. The paper's own Section 4.4 admission that VCRScore 'did not reach the expected performance' further supports caution. This concern is concrete and fixable: an image-stratified split is the standard way to measure generalization. If the authors show that their splits were already image-grouped, the concern is resolved; if not, the Table 11 advantage may be an artifact. I keep the verdict at CONDITIONAL because the issue does not force rejection; it forces a re-run and a clear reporting of the split procedure.","tokens_in":16870,"tokens_out":5667,"duration_ms":60238,"concrete_test":"Recompute the Table 11 correlations with image-stratified splits: assign all six captions of each of the 100 own-dataset images to the same fold (e.g., 70 images for training and 30 for testing), and group external pairs by image before applying the 70/30 split. If VCRScore's Spearman on held-out images falls below MCIPScore_ref (0.620 own, 0.716 EData) or BertGrammar (0.711 EData), the central 'highest correlation' claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing unstated condition is that the train/test splits in §4.1 are not described as stratified by image. The own dataset is 600 pairs over only 100 images (six captions per image: five model outputs plus one human reference), split 30/70. If the split is at pair level, captions of the same image appear in both training and test. VCRScore's feature stack includes ViLT and CLIP/MCIP image-text similarities, so a gradient-boosting regressor can memorize image-specific visual patterns and transfer them to held-out captions of the same image. The external 'EData' split carries the same ambiguity for datasets with multiple captions per image. Under image-level leakage, the Table 11 Spearman values (0.649 own, 0.743 EData) measure within-image memorization rather than generalization to new images, which is what a captioning metric must do. The paper reports no grouping by image, no image-overlap check, and no cross-dataset validation. The zero-score removal (§4.1) is an additional distributional bias, but the unstratified split is the more direct threat to the central claim. If the released code reveals that splits were already image-grouped, this concern is resolved.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VCRScore, a learned image-captioning metric that combines ViLT similarity, CLIP/MCIP variants (MCIPScore, MCIPScore ref), and precision/recall features computed against a pool of reference words and object-detection labels, into a Gradient Boosting Regressor trained to predict human ratings. The authors construct a new human-labeled dataset of 600 image-caption pairs from 100 COCO images, score them with four taggers on a five-point scale, and also use external datasets (VICR, Flickr8k-Expert, Flickr8k-CF, Composite). After normalizing scores and excluding all zero-score external samples, they train on 70% of external data and 30% of their own data, and test on the remaining splits. They report Spearman correlations with human scores (Table 11: 0.649 on their own test set and 0.743 on the external test set) and conclude that VCRScore is the most human-correlated metric among those compared.","tokens_in":17103,"tokens_out":5834,"duration_ms":52170,"significance":"If the reported correlations were robust, VCRScore would be a useful addition to the image-captioning evaluation toolbox, and the new human-labeled dataset could be a resource for the community. The paper is transparent about its data-processing choices, compares a wide range of lexical and semantic metrics, and provides code via GitHub. However, three load-bearing limitations prevent the paper from establishing its central claim: (1) the exclusion of all zero-score samples truncates the evaluation distribution; (2) the train/test splits are not shown to be image-disjoint, creating a risk of image-level leakage; and (3) the main comparison pits a supervised regressor trained in-domain against zero-shot baselines, so the reported advantage is partly a measure of fit to the annotation distribution. These issues are fixable with additional experiments, but the current evidence is insufficient.","major_comments":[{"comment":"The paper removes all 21,974 samples with human score 0 from the external datasets before splitting into train/test, reducing the data from 81,289 to 59,315. The stated reason ('not helpful to our model generation') is not a valid justification for excluding the low-quality end of the rating scale from the test set. Because the test set in Table 11 is therefore composed entirely of captions with positive human scores, the reported Spearman correlations measure agreement only over the upper part of the quality range. This is precisely the range where lexical and embedding metrics are most competitive; the metrics' known failure mode is on low-quality captions. To support the central claim, the authors must report correlations with zero-score samples included (at least in the test set), or justify why those samples should be excluded from evaluation for all metrics.","section":"Section 4.1, Table 8"},{"comment":"The split is not described as stratified by image. The authors' own dataset consists of 600 image-caption pairs from 100 images (six captions per image: five model outputs plus the COCO reference). A 30/70 pair-level split will almost certainly place different captions of the same image in both train and test. Since VCRScore's feature stack includes ViLT and CLIP/MCIP image-text similarities, a gradient-boosting regressor can memorize image-specific visual patterns and transfer them to held-out captions of the same image, inflating the 0.649 Spearman correlation on the own test set. The external dataset split raises the same concern for datasets with multiple captions per image. The authors should run an image-disjoint split (no image appears in both train and test) and report the resulting correlations; if they have already done this, the paper needs to state it.","section":"Section 4.1, Table 8"},{"comment":"The headline comparison is between a supervised regressor (VCRScore) trained on the human-score distribution of the test datasets and zero-shot baselines (BLEU, ROUGE, METEOR, CIDEr, SPICE, BertScore, CLIPScore, MCIPScore, ViLT) that have not been trained on those labels. The regressor's objective is to predict the very scores against which it is later evaluated; the held-out split mitigates but does not remove the in-domain training advantage. The reported correlations therefore do not establish that VCRScore is a better metric; they establish that a regressor fit to a dataset can rank that dataset's labels. To make the comparison meaningful, the authors should include (a) cross-dataset evaluation (train on one or more external datasets, test on others), and (b) a trained baseline that uses the same regression algorithm on the baseline metric features (e.g., GBR trained to predict human scores from CLIPScore, BertScore, etc.). Without these controls, the claimed superiority in Table 11 is not interpretable.","section":"Section 4.3, Table 11"},{"comment":"The precision and recall features depend on the construction of the word 'pool' from the references and object detection outputs. The paper does not specify which YOLOv3/DETR versions and confidence thresholds were used, how object labels are mapped to words (e.g., singular/plural, synonyms), or how stopwords are handled in the caption tokens. It also does not report an ablation that quantifies the contribution of precision and recall to the final VCRScore correlation, despite these features being a stated part of the proposal. Without this information, the proposed metric is not fully reproducible and the role of the precision-recall component is unverified.","section":"Section 3.2, Eqs. (4)-(6)"}],"minor_comments":[{"comment":"The sentence 'we have a set of 600 scores produced twice by the same tagger' is confusing because there are four taggers; it should say 'each tagger produced 600 scores in each of two stages.'","section":"Section 2.1"},{"comment":"The text says 'Both values are between 0 and 1' for precision and recall, but Equations (5) and (6) multiply by 100, so the values are percentages between 0 and 100.","section":"Section 3.2"},{"comment":"The table reports only point estimates of Spearman correlations. With 360 and 17,794 test samples, the differences between adjacent entries (e.g., 0.743 vs. 0.716) may not be statistically significant; the authors should provide confidence intervals or a significance test.","section":"Table 11"},{"comment":"The asymmetric split (30/70 for the own dataset, 70/30 for the external data) is justified only by 'because of its size.' This is not an explanation; the authors should state the rationale or use a consistent split strategy.","section":"Section 4.1"},{"comment":"The caption 'Correlations on positions' is vague; it should specify that the values are Spearman correlations between the model rankings produced by each metric and the human-generated ranking.","section":"Figure 9"},{"comment":"References [22] and [23] are duplicates of the same BLIP-2 paper, and references [33] and [34] both cite the CLIP paper; one of each pair should be removed or consolidated.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important problem and has a sensible overall architecture, but the current empirical evaluation does not yet support the central claim. The image-level leakage risk and the zero-score removal are the most serious concerns, and the in-domain supervised comparison against zero-shot baselines needs additional controls. I would encourage the editor to send a revised version back to the same reviewers rather than reject at this stage, since the issues are methodologically fixable and the paper includes a potentially useful new dataset."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on VCRScore. The paper is an incremental but genuine attempt to build a learned image-captioning metric, and the authors do some things well. The new 600-pair human dataset with four taggers and two annotation rounds is a real, if small, resource, and the agreement analysis (Krippendorff's alpha, Kendall's tau) is careful. The feature combination—ViLT cosine, MCIP/CLIP similarities, lexical precision/recall with an object-detection pool, plugged into a gradient boosting regressor—is new as a whole, even though every component is published. The comparisons against a wide range of classic and embedding metrics are thorough, and the histograms in Figures 6–8 help show where metrics fail.\n\nThe central claim, though, is not supported as stated. Two problems stand out. First, the train/test splits are not stratified by image. Their own dataset is 600 pairs from 100 images (six captions each), and a 30/70 random split means the same image almost certainly appears in both training and test. Since VCRScore uses image-text similarity features (ViLT, CLIP/MCIP), the regressor can memorize image-specific patterns and transfer them to held-out captions of the same image. The external dataset split has the same issue for multi-caption images. That makes the reported Spearman values (0.649 and 0.743) measure within-image memorization as much as generalization to new images. The paper doesn't report any image-overlap check or cross-dataset validation. If the released code shows image-grouped splits, this concern goes away, but as written it's a load-bearing flaw.\n\nSecond, removing all 21,974 zero-score samples from the external datasets (Section 4.1) changes the label distribution in both training and testing. Zero is the worst score; excluding it means the test set lacks exactly the low-quality captions a metric is supposed to catch. The paper says those samples were \"not helpful,\" but they are the most informative for distinguishing a good metric from a bad one. With the low end truncated, correlations are computed on a restricted range, and the advantage of a supervised regressor over zero-shot baselines may shrink or disappear if those points are included.\n\nThere are smaller issues too: the comparison is asymmetric (VCRScore is trained on the test-domain data while all baselines are zero-shot; no trained baseline like LEIC is included despite being cited), and no confidence intervals or significance tests are reported for the correlations. Given the sample sizes, the gap between VCRScore and, say, MCIPScore ref (0.649 vs 0.620 on 360 pairs) could easily be noise.\n\nThat said, this is not junk. The dataset, the feature analysis, and the honest conclusion—\"higher results were expected\"—are useful for the evaluation-methodology subfield. With image-separated splits, zero-score samples included (or at least analyzed separately), and a trained baseline, the paper could make a solid contribution.\n\nI'd recommend sending it to peer review, because a serious referee could fix the evaluation and salvage the core idea. It's not something I'd cite in its current form, but it's worth a look in a reading group.","headline":"A useful but flawed attempt at a learned captioning metric; the evaluation is compromised by image-level split leakage and zero-score removal, so the central correlation claim is not yet established.","tokens_in":17669,"tokens_out":3394,"would_cite":false,"duration_ms":31569,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that VCRScore, a supervised metric combining CLIP retrieval embeddings, a vision-language transformer distance, and precision-recall features, agrees with human quality ratings better than existing image-captioning…","keywords":["image captioning evaluation","VCRScore","vision-language transformer","CLIP","precision-recall","human evaluation correlation","gradient boosting","Spearman correlation"],"falsifier":"Re-run the same VCRScore training and the Table 11 comparison on the full external dataset without deleting the 21,974 zero-score samples, comparing Spearman correlations on all 81,289 pairs; if VCRScore no longer beats CLIPScore ref and MCIPScore ref, the paper's central claim fails. A second check is to train on some datasets and test on held-out datasets from unseen sources to see whether the advantage survives cross-domain transfer.","tokens_in":16589,"feed_emoji":"🖼️","tokens_out":8185,"duration_ms":75979,"temperature":0.7,"pith_summary":"This paper tries to establish that image-captioning evaluation can be improved by replacing n-gram and raw embedding metrics with a learned regressor trained on human quality scores. The authors built a 600-pair human-tagged dataset from MS COCO images and six caption sources, then trained VCRScore on those scores plus four public datasets, using features from CLIP-based retrieval models, a vision-language transformer, and lexical precision and recall. Reported Spearman correlations with human ratings are 0.649 on 360 held-out pairs and 0.743 on 17,794 external test pairs, higher than every compared metric. If this holds, captioning researchers get a metric that ranks models closer to human judgment than BLEU, CIDEr, BertScore, or CLIPScore.","feed_headline":"A learned caption metric beats BLEU and CLIP at human agreement","feed_subtitle":"VCRScore posts Spearman correlations of 0.649 and 0.743 with human ratings, beating every compared metric.","key_machinery":"The load-bearing object is the VCRScore regressor: a Gradient Boosting Regressor that maps image-caption features into a 0-to-1 human-like score. Its features are cosine similarities from CLIP and retrieval-tuned CLIP between the image, the caption, and references; a distance from the ViLT vision-language transformer; and precision and recall computed as the proportion of caption words found in a pool formed by reference words plus object names detected by YOLOv3 and DETR. Precision and recall supply lexical coverage, CLIP supplies semantic alignment, and ViLT supplies multimodal alignment; the regressor learns how to combine them to match human ratings.","core_discovery":"In the paper's own terms, the discovery is that a trainable metric can be highly correlated with human tagging: VCRScore reaches the highest Spearman correlations among all metrics tested, 0.649 on the authors' held-out 360 image-caption pairs and 0.743 on 17,794 external test pairs. The metric is trained on the mean of eight human ratings per pair, using a Gradient Boosting Regressor whose features are CLIP and retrieval-tuned CLIP similarities, a ViLT image-caption distance, and precision and recall computed against a pool of reference words augmented by object detectors. The authors interpret this as evidence that semantic embedding signals and lexical coverage are complementary, and that a supervised combination of them approximates human judgment better than any single existing metric.","pith_inferences":["If zero-scored pairs were kept in training and testing, the reported gap would likely narrow because low-quality pairs are where lexical and semantic metrics fail; this is a testable prediction, not a paper claim.","Because tagger agreement is only moderate, part of VCRScore's higher correlation may reflect fitting annotator noise; averaging more taggers per pair would test that.","The same feature recipe could be evaluated reference-free by replacing reference words in the pool with detected objects alone, a variant the paper does not report.","A direct extension would swap ViLT for a video-language transformer and object detectors for temporal detections to score video captions; no evidence here supports that transfer."],"forward_implications":["If the reported correlations hold, model rankings from VCRScore are a closer surrogate for human preference than rankings from BLEU, ROUGE, METEOR, SPICE, CIDEr, BertScore, CLIPScore, or MCIPScore.","The result supports combining lexical coverage with semantic and multimodal alignment rather than relying on any single signal.","VCRScore's 0-to-1 output on the human scale makes it usable as a quality filter for caption outputs, unlike unbounded lexical metrics.","The reported advantage is limited to test splits drawn from the same datasets used for training after zero-score samples were removed."],"supporting_citations":[{"why":"Supplies the 100 MS COCO images and human reference captions that anchor the tagging dataset.","marker":"[26]"},{"why":"Provides the helpfulness and correctness caption-quality scheme adapted into the five-point tagging scale.","marker":"[21]"},{"why":"The THUMB rubric inspires the precision and recall formulation and the human-evaluation protocol.","marker":"[15]"},{"why":"CLIP embeddings feed the CLIPScore baselines and the retrieval-based similarities used as VCRScore features.","marker":"[33]"},{"why":"Defines the retrieval-tuned CLIP whose cosine similarities become VCRScore features and comparison baselines.","marker":"[36]"},{"why":"ViLT supplies the vision-language distance feature used by the regressor and is also a comparison metric.","marker":"[17]"},{"why":"The VICR public human-score dataset contributes external training and test pairs.","marker":"[29]"},{"why":"Flickr8k-Expert and Flickr8k-CF supply additional human-rated image-caption pairs for training and testing.","marker":"[14]"},{"why":"The Composite dataset contributes Flickr8k, Flickr30k, and MS COCO relevance and thoroughness ratings.","marker":"[3]"}],"fun_headline_variants":["VCRScore tops human-caption agreement over BLEU and CLIP","VCRScore beats BLEU and CLIP on human-rating correlation","New caption metric VCRScore beats old ones on human agreement","Trainable caption metric targets human scores: VCRScore wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison stands on treating the removal of all 21,974 zero-scored external samples as harmless; if low-quality pairs were kept, the reported correlation advantage could shrink or reverse.","fun_headline_variants_meta":{"raw":{"variants":["VCRScore tops human-caption agreement over BLEU and CLIP","VCRScore beats BLEU and CLIP on human-rating correlation","New caption metric VCRScore beats old ones on human agreement","Trainable caption metric targets human scores: VCRScore wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00071,"raw_usage":{"total_tokens":3190,"prompt_tokens":931,"completion_tokens":2259,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":2182}},"tokens_in":547,"tokens_out":2259,"duration_ms":15227,"temperature":1.0,"reasoning_tokens":2182,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:10:03.282948+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same VCRScore training and the Table 11 comparison on the full external dataset without deleting the 21,974 zero-score samples, comparing Spearman correlations on all 81,289 pairs; if VCRScore no longer beats CLIPScore ref and MCIPScore ref, the paper's central claim fails. A second check is to train on some datasets and test on held-out datasets from unseen sources to see whether the advantage survives cross-domain transfer.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 100 MS COCO images and human reference captions that anchor the tagging dataset."},{"cited_title":"Levinboim, A","cited_arxiv_id":null,"evidence_quote":"Provides the helpfulness and correctness caption-quality scheme adapted into the five-point tagging scale."},{"cited_title":"Kasai, K","cited_arxiv_id":null,"evidence_quote":"The THUMB rubric inspires the precision and recall formulation and the human-evaluation protocol."},{"cited_title":"Schall, K","cited_arxiv_id":null,"evidence_quote":"Defines the retrieval-tuned CLIP whose cosine similarities become VCRScore features and comparison baselines."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ViLT supplies the vision-language distance feature used by the regressor and is also a comparison metric."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The VICR public human-score dataset contributes external training and test pairs."},{"cited_title":"Hodosh, P","cited_arxiv_id":null,"evidence_quote":"Flickr8k-Expert and Flickr8k-CF supply additional human-rated image-caption pairs for training and testing."},{"cited_title":"Aditya, Y","cited_arxiv_id":null,"evidence_quote":"The Composite dataset contributes Flickr8k, Flickr30k, and MS COCO relevance and thoroughness ratings."}],"review_version":1}