Pith. sign in

REVIEW 4 major objections 5 minor 43 references

This paper claims that a specificity-enhanced CLIP model, trained to reward correct added details and penalize incorrect ones, can evaluate long image captions with human-level correlation at a fraction of the cost of LLM-based metrics.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 10:32 UTC pith:ZGE5SN27

load-bearing objection A cheap, practical long-caption metric that mostly delivers, but the specificity claim needs a length-controlled analysis before I'd trust it. the 4 major comments →

arxiv 2509.03897 v2 pith:ZGE5SN27 submitted 2025-09-04 cs.CV cs.CL

SPECS: Specificity-Enhanced CLIP-Score for Long Image Caption Evaluation

Xiaofu Chen , Israfel Salazar , Yova Kementchedjhieva This is my paper
classification cs.CV cs.CL
keywords image caption evaluationrepresentational similarity metricsCLIP fine-tuningspecificitylong image captionsdetail unitshuman correlationefficient evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Long, detailed image captions are poorly served by existing metrics: n-gram metrics miss semantics, and strong LLM-based judges are too expensive for iterative development. This paper argues that the missing ingredient for a cheap metric is specificity—the ability to prefer captions whose added details are true to the image and to penalize ones that are not. It introduces SPECS, a reference-free metric built by fine-tuning a long-context CLIP with a specificity-aware loss on minimal pairs of captions that differ by one detail. On the DECAPBENCH human-correlation benchmark, SPECS raises Pearson correlation over CLIPScore from 0.2183 to 0.5228 and matches or beats the strongest open-source LLM metric FLEUR, while using only about a hundredth of the compute. If correct, this gives captioning researchers a practical, human-aligned metric for routine checkpoint evaluation, at the cost of degraded performance on short captions and general-purpose retrieval.

Core claim

The paper's central discovery is that specificity—defined as the capacity of a vision-language representation to assign higher similarity to a caption when a correct visual detail is added and lower similarity when an incorrect detail is added—can be trained into a CLIP-style model and then used directly as a reference-free caption-evaluation metric. SPECS decomposes captions into cumulative "detail units," builds minimal positive and negative pairs from them, and fine-tunes LongCLIP with three losses: contrastive, positive-detail, and negative-detail. The resulting model achieves a Specificity Rate of 92.87 on sDCI, versus 67.83 for the LongCLIP baseline, and, used as a scoring function, at

What carries the argument

The load-bearing mechanism is the detail-unit decomposition and the specificity-aware training objective built on it. A detail unit is a phrase that contributes at least one new visual detail and fits syntactically and semantically in context; any cumulative subsequence of detail units yields a valid partial caption. This turns each caption into an ordered chain of minimal pairs: adding a positive detail unit should increase cosine similarity between image and text, and adding a negative detail unit should decrease it. The training loss combines standard contrastive loss with hinge losses enforcing these two monotonicities, using a dynamic margin equal to the batch-wise average similarity ga

Load-bearing premise

The load-bearing premise is that GPT-4's rule-based segmentation of captions into detail units reliably isolates minimal, meaningful visual details; if those segmentations are noisy or biased, the learned monotonicity and the specificity-rate numbers may not reflect true detail correctness.

What would settle it

Replace the GPT-4-generated detail-unit segmentation with human-verified segmentation on a random sample of the training captions, retrain SPECS with identical hyperparameters, and measure its DECAPBENCH correlation; if the gain over the LongCLIP* baseline mostly disappears, the segmentation pipeline rather than the specificity objective is carrying the result.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Long-caption evaluation becomes a lightweight, reference-free operation: SPECS needs 2.81e-2 TFLOPs per forward pass with 0.15B parameters, so it can run during training loops where LLM judges are impractical.
  • For captions longer than about 60 tokens, SPECS is the strongest representational-similarity metric on DECAPBENCH (PCC 0.5228, Kendall's tau 0.4078) and surpasses the best open-source LLM judge on Pearson and sample-wise tau; for short captions the authors recommend plain CLIP.
  • The specificity objective is learnable at scale: SPECS raises the average Specificity Rate on sDCI from 67.83 to 92.87, with the largest gain on positive details, showing that monotonic reward for correct detail can be instilled by fine-tuning.
  • Using the resulting model as a general-purpose encoder is not advised: specificity training distorts the embedding space and degrades retrieval and classification, which the paper documents as hubness; the metric itself is the intended product.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The detail-unit segmentation quality is the main hidden variable: because both the training signal and the specificity-rate evaluation flow through GPT-4-generated units, an independent human-verified segmentation study on a sample of captions would reveal how much of the 25-point Specificity Rate gain is attributable to the objective versus the segmentation.
  • The authors leave unexplored whether SPECS generalizes beyond the 100-image DECAPBENCH set; a natural extension is to test it on other long-caption human preference sets or on captions written by humans rather than vision-language models.
  • The strong weight on the positive-detail loss (beta=8 vs gamma=0.8) suggests that most of the benefit comes from learning to prefer more informative captions, which implies the metric may also be usable as a detail-density or informativeness signal, not only a correctness score.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces SPECS, a reference-free representational-similarity metric for long image caption evaluation. SPECS fine-tunes LongCLIP with a specificity objective: given a caption decomposed into detail units, it encourages cosine similarity to increase when a visually grounded detail unit is appended and to decrease when an ungrounded unit is appended. The model is trained on ShareGPT-4V using GPT-4-derived segmentation into detail units. Intrinsic evaluation on sDCI reports a large gain in Specificity Rate (92.87 vs. 67.83 for LongCLIP*). Extrinsic evaluation on DECAPBENCH reports the highest human correlation among RS metrics (PCC 0.5228 vs. 0.2183 for CLIPScore) and higher PCC/Spτ than the open-source FLEUR metric, at roughly 1/260th the inference FLOPs. The paper also documents degraded retrieval/classification and mixed compositionality results, which it presents as limitations of the specificity-oriented objective.

Significance. If the human-correlation result is robust, SPECS is a meaningful practical contribution: it would be the first RS metric to match open-source LLM-based metrics for long captions while remaining usable for iterative development. The paper is transparent about trade-offs, releases code, and compares against a broad set of baselines. However, the central claim currently rests on a 100-image benchmark without confidence intervals, and the proposed specificity mechanism may be confounded with caption length/detail count. These issues need to be resolved before the contribution can be fully assessed.

major comments (4)
  1. [§4.4, Table 2] The headline claim that SPECS matches or outperforms open-source LLM metrics is based on DECAPBENCH, which contains only 100 images. No confidence intervals or paired significance tests are reported; the p-values stated below Table 2 appear to be for individual correlations, not for differences between SPECS and FLEUR/DCScore. Please report bootstrap confidence intervals for PCC and Spτ and a paired test (e.g., Williams's test or bootstrap difference) comparing SPECS with the strongest LLM baselines. Without this, the superiority over FLEUR on PCC (0.5228 vs. 0.4230) could be within sampling noise.
  2. [§3.3, Eq. (3); §4.4, Table 2] The positive detail loss trains the model to assign higher similarity whenever any detail unit from the original caption is appended. Because the ShareGPT-4V captions are synthetically generated and not verified against the image, the model may learn that longer captions (more detail units) are always better, regardless of whether the added details are visually grounded. Negative units are random/shuffled units from other images, so they are easy negatives and do not teach rejection of plausible but incorrect details. This creates a potential length/detail-count confound in the DECAPBENCH correlation. The paper does not report a length-only baseline, partial correlations controlling for caption length/detail count, or any analysis that separates specificity from length. Section 5.3 buckets captions by token count but does not control for length within the correlation. A length-controlled
  3. [§4.3, §5.4, Table 1] The validation set used for hyperparameter tuning is not described. In §5.4, the reported tuning curves show Specificity Rate on what appears to be the same sDCI benchmark that is later presented as the intrinsic evaluation in Table 1. If α, β, γ, learning rate, loss type, and shuffle ratio were selected to maximize SR on sDCI, then Table 1 is a training-selection result rather than an independent evaluation. Please specify the validation split and, if sDCI was used for tuning, report intrinsic results on a held-out benchmark or a cross-validation scheme.
  4. [§4.2, Appendix A] The detail-unit segmentation is generated by GPT-4 from a single manually annotated caption and then applied to the entire dataset. The paper states only that manual inspection found it 'largely effective' but 'somewhat oversegments.' The entire training signal and the Specficity Rate evaluation depend on this segmentation: if the grammar mis-segments, the monotonicity constraint is applied to non-minimal units and the SR numbers become uninterpretable. Please provide quantitative segmentation quality measurements (e.g., agreement with human annotations on a sample, precision/recall on a held-out set of captions) and analyze sensitivity of the learned model to segmentation errors.
minor comments (5)
  1. [Table 2] The column '1 − R² ↓' is confusing because R² is usually higher-is-better; the table reports 1−R² as lower-is-better. Consider renaming to '1 − R²' with an explicit explanation or reporting R² directly.
  2. [§5.2, Appendix B] The dataset is referred to as SugarCREPE++ (SCPP) in §5.2 but 'SCPP++' in Appendix B. Please standardize the name and ensure the abbreviation is consistent.
  3. [Table 6] Typo: 'Throughness' should be 'Thoroughness'.
  4. [Figure 1, §1] The numbers 65.44 and 92.87 in Figure 1 are not defined in the caption; they appear to be Specificity Rates but should be explicitly labeled.
  5. [§4.1] The training data is described as 'synthetically generated by a strong captioning model,' but the possible hallucination rate in this data is not analyzed. A brief discussion of why this is acceptable for the positive/negative triplet construction would be helpful, especially given the concern in Major Comment 2.

Circularity Check

1 steps flagged

Intrinsic specificity result is fitted via hyperparameter selection on sDCI, but central human-correlation claim remains external.

specific steps
  1. fitted input called prediction [Section 5.4 (Hyperparameters) vs. Section 4.4 (Intrinsic Evaluation); Table 5 vs. Table 1]
    "The optimal setting (α=1, β=8, γ=0.8) achieves the highest specificity score of 92.87. ... To evaluate whether our training objective effectively enhances specificity, we compare the specificity rate of SPEC ... on the sDCI benchmark and report results in Table 1."

    Table 5 reports Positive/Negative/Average specificity scores for hyperparameter configurations, and the selected configuration's average (92.87) is exactly the sDCI average reported as SPEC's intrinsic evaluation in Table 1 (95.37/90.37/92.87). Thus the hyperparameters were selected by maximizing the sDCI specificity rate, and the same sDCI specificity rate is then presented as independent evidence that the training objective enhances specificity. The intrinsic headline result is therefore the optimization criterion, not a held-out confirmation. The central human-correlation claim on DECAPBENCH is external and was not used for tuning, so the paper is only partially circular.

full rationale

The paper's main contribution is the claim that SPECS, a reference-free RS metric, matches open-source LLM-based metrics in correlation with human judgments on DECAPBENCH. That result is measured on an external benchmark with baselines and was not used to train or select the model, so it is not circular. The training objective (Eq. 3) directly rewards higher similarity for captions extended with positive detail units, and the intrinsic Specificity Rate (Eq. 1) tests the same inequality; however, evaluating on a held-out dataset like sDCI is a standard generalization check, not a definitional equivalence. The one concrete circular step is that the hyperparameters (α, β, γ) were selected by maximizing the sDCI specificity score (Table 5), and the same sDCI specificity score is then reported as the intrinsic evaluation (Table 1). This makes the intrinsic result partially self-confirmatory, though the improvement over baselines is large even for suboptimal hyperparameters. The skeptical concern about caption-length confounding is a correctness/robustness risk, not a circularity: no equation reduces SPECS to a length heuristic, and the paper's own length-sensitivity analysis and Limitations section acknowledge trade-offs. No self-citation chain or imported uniqueness theorem is present. Overall, the central human-correlation claim has independent content, so the circularity burden is modest.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 1 invented entities

SPECS rests on tuned loss weights and shuffle ratio, an operational definition of 'detail unit' implemented via GPT-4-generated rules, and the assumption that monotonic similarity over these units transfers to human-aligned scoring. These are not externally validated beyond the paper's own benchmarks.

free parameters (5)
  • loss weight α (contrastive) = 1
    Set by hyperparameter tuning on a validation set (Sections 4.3, 5.4).
  • loss weight β (positive detail) = 8
    Tuned; strongly emphasizes positive detail loss.
  • loss weight γ (negative detail) = 0.8
    Tuned in combination with other weights.
  • learning rate = 1e-5
    Tuned; other values degrade performance.
  • shuffle ratio = 90%
    Proportion of negative detail units token-shuffled during training; tuned to 90%.
axioms (4)
  • domain assumption Cosine similarity between image and caption embeddings is an appropriate scoring function for caption quality.
    The metric is defined as clipped cosine similarity (Eq. 5); this presupposes that relative similarity orders captions by quality.
  • domain assumption LongCLIP's long-context embeddings provide a suitable base for specificity fine-tuning.
    The authors start from LongCLIP-B/32 for its 248-token context; this architectural choice is inherited.
  • ad hoc to paper The GPT-4-generated segmentation grammar reliably identifies detail units.
    The entire training data construction depends on this rule-based grammar; only manual inspection is reported.
  • domain assumption ShareGPT-4V provides a sufficiently diverse set of high-quality captions for training.
    Training uses 1.2M synthetically generated captions from COCO, SAM, and LAION; no analysis of distributional coverage is provided.
invented entities (1)
  • detail unit no independent evidence
    purpose: Operational minimal unit of visual information in a caption; used to construct positive and negative triplets for training and evaluation.
    Defined in Section 3.1 and implemented via a GPT-4-generated grammar; it is an ad hoc construct with no external falsifiable predictions.

pith-pipeline@v1.4.0-alltime-deepseek-medium · 14505 in / 10287 out tokens · 92701 ms · 2026-08-05T10:32:42.313658+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of SPECS: Specificity-Enhanced CLIP-Score for Long Image Caption Evaluation." pith.science (2026). https://pith.science/paper/ZGE5SN27

@misc{pith2026250903897,
  author       = {Pith},
  title        = {Pith review of: SPECS: Specificity-Enhanced CLIP-Score for Long Image Caption Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZGE5SN27}},
  note         = {Machine review of arXiv:2509.03897}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

As interest grows in generating long, detailed image captions, standard evaluation metrics become increasingly unreliable. N-gram-based metrics though efficient, fail to capture semantic correctness. Representational Similarity (RS) metrics, designed to address this, initially saw limited use due to high computational costs, while today, despite advances in hardware, they remain unpopular due to low correlation to human judgments. Meanwhile, metrics based on large language models (LLMs) show strong correlation with human judgments, but remain too expensive for iterative use during model development. We introduce SPECS (Specificity-Enhanced CLIPScore), a reference-free RS metric tailored to long image captioning. SPECS modifies CLIP with a new objective that emphasizes specificity: rewarding correct details and penalizing incorrect ones. We show that SPECS matches the performance of open-source LLM-based metrics in correlation to human judgments, while being far more efficient. This makes it a practical alternative for iterative checkpoint evaluation during image captioning model development.Our code can be found at https://github.com/mbzuai-nlp/SPECS.

Figures

Figures reproduced from arXiv: 2509.03897 by Israfel Salazar, Xiaofu Chen, Yova Kementchedjhieva.

Figure 1
Figure 1. Figure 1: Example of specificity in caption evaluation. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Training framework. Given an image and its caption, we produce a base caption, a more positive caption, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Comparison of Pearson Correlation (PCC) across token ranges for different models on the Flickr30k and ImageInWords datasets. end, we divided captions into buckets based on to￾ken counts and measured human correlation within each range. As shown in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

43 extracted references · 17 canonical work pages

  1. [1]

    Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic propositional image caption evaluation. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pages 382--398. Springer

    work page 2016

  2. [2]

    Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65--72

    2005

  3. [3]

    David Chan, Suzanne Petryk, Joseph E Gonzalez, Trevor Darrell, and John Canny. 2023. Clair: Evaluating image captions with large language models. arXiv preprint arXiv:2310.12971

    Pith/arXiv arXiv 2023

  4. [4]

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. 2024. Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision, pages 370--387. Springer

    work page 2024

  5. [5]

    Jaemin Cho, Seunghyun Yoon, Ajinkya Kale, Franck Dernoncourt, Trung Bui, and Mohit Bansal. 2022. https://doi.org/10.18653/v1/2022.findings-naacl.39 Fine-grained image captioning with CLIP reward . In Findings of the Association for Computational Linguistics: NAACL 2022, pages 517--527, Seattle, United States. Association for Computational Linguistics

    work page doi:10.18653/v1/2022.findings-naacl.39 2022

  6. [6]

    Hongyuan Dong, Jiawen Li, Bohong Wu, Jiacong Wang, Yuan Zhang, and Haoyuan Guo. 2024. Benchmarking and improving detail image caption. arXiv preprint arXiv:2405.19092

    Pith/arXiv arXiv 2024

  7. [7]

    Sivan Doveh, Assaf Arbelle, Sivan Harary, Roei Herzig, Donghyun Kim, Paola Cascante-Bonilla, Amit Alfassy, Rameswar Panda, Raja Giryes, Rogerio Feris, and 1 others. 2023. Dense and aligned captions (dac) promote compositional reasoning in vl models. Advances in Neural Information Processing Systems, 36:76137--76150

    work page 2023

  8. [8]

    Sri Harsha Dumpala, Aman Jaiswal, Chandramouli Shama Sastry, Evangelos Milios, Sageev Oore, and Hassan Sajjad. 2024. Sugarcrepe++ dataset: Vision-language model sensitivity to semantic and lexical alterations. Advances in Neural Information Processing Systems, 37:17972--18018

    work page 2024

  9. [9]

    Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian. 2023. Improving clip training with language rewrites. Advances in Neural Information Processing Systems, 36:35544--35575

    work page 2023

  10. [10]

    Roopal Garg, Andrea Burns, Burcu Karagol Ayan, Yonatan Bitton, Ceslee Montgomery, Yasumasa Onoe, Andrew Bunner, Ranjay Krishna, Jason Baldridge, and Radu Soricut. 2024. Imageinwords: Unlocking hyper-detailed image descriptions. arXiv preprint arXiv:2405.02793

    Pith/arXiv arXiv 2024

  11. [11]

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718

    Pith/arXiv arXiv 2021

  12. [12]

    Liqiang Jing, Ruosen Li, Yunmo Chen, Mengzhao Jia, and Xinya Du. 2023. Faithscore: evaluating hallucinations in large vision-language models. corr, abs/2311.01477, 2023. doi: 10.48550. arXiv preprint ARXIV.2311.01477

    Pith/arXiv arXiv 2023

  13. [13]

    Justin Johnson, Andrej Karpathy, and Li Fei-Fei. 2016. Densecap: Fully convolutional localization networks for dense captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4565--4574

    work page 2016

  14. [14]

    Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3128--3137

    2015

  15. [15]

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, and 1 others. 2023. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015--4026

    2023

  16. [16]

    Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. 2009. Cifar-10 and cifar-100 datasets. URl: https://www. cs. toronto. edu/kriz/cifar. html, 6(1):1

    work page 2009

  17. [17]

    Yebin Lee, Imseong Park, and Myungjoo Kang. 2024. Fleur: An explainable reference-free evaluation metric for image captioning using a large multimodal model. arXiv preprint arXiv:2406.06004

    Pith/arXiv arXiv 2024

  18. [18]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730--19742. PMLR

    2023

  19. [19]

    Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694--9705

    2021

  20. [20]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74--81

    2004

  21. [21]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer vision--ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13, pages 740--755. Springer

    2014

  22. [22]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. Advances in neural information processing systems, 36:34892--34916

    2023

  23. [23]

    Shi Liu, Kecheng Zheng, and Wei Chen. 2024. https://arxiv.org/abs/2407.21771 Paying more attention to image: A training-free method for alleviating hallucination in lvlms . Preprint, arXiv:2407.21771

    Pith/arXiv arXiv 2024

  24. [24]

    Mayu Otani, Riku Togashi, Yu Sawai, Ryosuke Ishigami, Yuta Nakashima, Esa Rahtu, Janne Heikkil \"a , and Shin’ichi Satoh. 2023. Toward verifiable and reproducible human evaluation for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14277--14286

    work page 2023

  25. [25]

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

    2002

  26. [26]

    Maitreya Patel, Naga Sai Abhiram Kusumba, Sheng Cheng, Changhoon Kim, Tejas Gokhale, Chitta Baral, and 1 others. 2024. Tripletclip: Improving compositional reasoning of clip via synthetic vision-language negatives. Advances in neural information processing systems, 37:32731--32760

    work page 2024

  27. [27]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748--8763. PmLR

    2021

  28. [28]

    Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, and 1 others. 2015. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211--252

    work page 2015

  29. [29]

    Sara Sarto, Manuele Barraco, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. 2023. Positive-augmented contrastive learning for image and video captioning evaluation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6914--6924

    work page 2023

  30. [30]

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, and 1 others. 2022. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems, 35:25278--25294

    work page 2022

  31. [31]

    Jack Urbanek, Florian Bordes, Pietro Astolfi, Mary Williamson, Vasu Sharma, and Adriana Romero-Soriano. 2024. A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26700--26709

    work page 2024

  32. [32]

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566--4575

    2015

  33. [33]

    Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156--3164

    2015

  34. [34]

    Yixuan Wang, Qingyan Chen, and Duygu Ataman. 2023. Delving into evaluation metrics for generation: A thorough assessment of how metrics generalize to rephrasing across languages. In Proceedings of the 4th Workshop on Evaluation and Comparison of NLP Systems, pages 23--31

    work page 2023

  35. [35]

    Chunyu Xie, Bin Wang, Fanjing Kong, Jincheng Li, Dawei Liang, Gengshen Zhang, Dawei Leng, and Yuhui Yin. 2025. Fg-clip: Fine-grained visual and textual alignment. arXiv preprint arXiv:2505.05071

    Pith/arXiv arXiv 2025

  36. [36]

    Zhenlin Xu, Yi Zhu, Siqi Deng, Abhay Mittal, Yanbei Chen, Manchen Wang, Paolo Favaro, Joseph Tighe, and Davide Modolo. 2024. Benchmarking zero-shot recognition with vision-language models: Challenges on granularity and specificity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1827--1836

    work page 2024

  37. [37]

    Qinghao Ye, Xianhan Zeng, Fu Li, Chunyuan Li, and Haoqi Fan. 2025. Painting with words: Elevating detailed image captioning with benchmark and alignment learning. arXiv preprint arXiv:2503.07906

    Pith/arXiv arXiv 2025

  38. [38]

    Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, and 1 others. 2024. Rlaif-v: Aligning mllms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint arXiv:2405.17220

    arXiv 2024

  39. [39]

    Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. 2022. When and why vision-language models behave like bags-of-words, and what to do about it? arXiv preprint arXiv:2210.01936

    Pith/arXiv arXiv 2022

  40. [40]

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975--11986

    2023

  41. [41]

    Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. 2024. Long-clip: Unlocking the long-text capability of clip. In European Conference on Computer Vision, pages 310--325. Springer

    work page 2024

  42. [42]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  43. [43]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...