Pith. sign in

REVIEW 4 major objections 4 minor 57 references

The paper proposes that a caption's quality be measured by whether the image reconstructed from it can answer the same downstream vision-language questions as the original.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-07-31 23:59 UTC pith:LYKBEJU2

load-bearing objection A transparent proposal whose practical score is not tied to its own equivalence criterion and lacks human validation. the 4 major comments →

arxiv 2607.23235 v1 pith:LYKBEJU2 submitted 2026-07-25 cs.CV

A Reconstruction-Based Framework for Caption Evaluation Beyond Reference Captions

Zhijiang Tang , Jiaxin Qi , Kaihua Tang , Yuhua Zheng , Jianqiang Huang This is my paper
classification cs.CV
keywords image captioning evaluationreference-free metricsemantic equivalencetext-to-image reconstructionvision-language tasksdownstream question answeringcaptioning turing test datasetcaption quality
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes that a caption's quality be judged by whether a text-to-image model, building an image from the caption alone, lets a vision-language judge answer the same downstream questions as the original image. The authors formalize this as semantic equivalence: a caption is as good as its capacity to enable a reconstruction that is semantically equivalent to the original, operationalized by the average downstream-task accuracy of the reconstruction. This yields a reference-free, task-conditioned score that complements human-written reference metrics. To make the idea practical, they introduce the Captioning Turing Test Dataset (CTTD), a compact curated question-answer benchmark that reproduces the broad trends of the full task suite at about ten times lower compute. The paper is explicit that the practical score is not algebraically equivalent to the semantic-equivalence criterion, that a simplified reconstruction can be easier for a judge, and that direct correlation with human caption judgments remains the central validation still required.

Core claim

The central claim is Principle 2: for a fixed image, a caption is better than another if the image reconstructed from it lies closer to the semantic equivalence class of the original. Because semantic equivalence is not directly observable, the paper approximates it with a suite of downstream vision-language tasks: if replacing the original image with the reconstruction lets a fixed judge achieve comparable performance, the two images count as semantically equivalent. The practical score, Eq. (5), is the judge's average accuracy on the reconstructed image, with original-image performance as an internal upper reference. Across thirteen captioning models, the authors find that traditional capt

What carries the argument

The load-bearing object is the semantic equivalence class [I] = {I' : I' ~ I}, with the equivalence relation approximated by Eq. (4): a reconstruction and the original are semantically equivalent when a fixed judge model achieves comparable performance across a suite of vision-language datasets. The practical scoring identity is Eq. (5): S_D(I, C) = (1/n) sum_i J(f_img(C), D_i), the judge's accuracy when conditioned on the caption-generated reconstruction. CTTD is a curated set of about 7,000 images with roughly 15 categorized multiple-choice question-answer pairs per image, built by clustering LVLM-proposed questions into 15 categories and selecting by relevance-diversity; it serves as a lo

Load-bearing premise

The score assumes that a judge's accuracy on a caption-conditioned reconstruction is a valid and roughly monotone proxy for how faithfully the caption preserves image semantics; if a simpler reconstruction inflates accuracy, or if generator and judge errors dominate, the ranking no longer tracks caption quality.

What would settle it

Have humans rank a set of captions for the same image by semantic adequacy and measure the rank correlation with CTTD scores using the default generator and judge; near-zero or negative correlation would refute the central claim. A second check: hold captions fixed, vary the text-to-image generator, and see whether model rankings invert; if they do, the score is measuring the generator at least as much as the caption.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Caption evaluation can proceed without human-written references: the score is computed from the caption, a frozen text-to-image generator, a judge model, and a question bank.
  • The score is task-conditioned: a caption may score high on VQA and low on OCR or grounding, revealing which semantic dimensions survive caption compression.
  • CTTD is a scalable, lower-cost proxy that reproduces the broad model rankings of the full task suite at about ten times less compute.
  • OCR and grounding gaps indicate that fine-grained textual detail and spatial layout are the hardest information to preserve through caption compression.
  • Scores must be interpreted under fixed pipeline components; the paper notes that changing the generator, judge, or question set can change the ranking.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The reconstruction score could serve as a training signal for captioning models: optimizing captions to maximize downstream judge accuracy would directly target semantic preservation without human references, though the paper does not propose this.
  • The same semantic-equivalence test could extend to video captioning, image editing, or instruction evaluation, wherever a text-conditioned reconstruction can be judged by downstream tasks.
  • Different generator-judge pairs effectively operationalize different notions of semantic content, so reporting scores across a matrix of generators and judges would be a natural stress-test of the metric's stability.
  • Because Eq. (5) can favor simplified reconstructions that are easier for the judge, an editorial inference is that scores should be normalized against an image-level baseline (such as original-image or random-guess accuracy) before comparing captions across heterogeneous images.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a reference-free, reconstruction-based framework for image-caption evaluation. It replaces exact pixel reconstruction with a semantic-equivalence criterion: a caption is good if its caption-conditioned reconstruction is semantically interchangeable with the original image, as judged by downstream vision-language task performance (Eq. (4)). Because this is expensive, the score actually used is Eq. (5), the average judge accuracy on the reconstructed image, and a compact surrogate, the Captioning Turing Test Dataset (CTTD), is introduced. The paper reports model rankings for 13 captioners across VQA, STEM, OCR, and grounding tasks, plus CTTD, and includes ablations over generators, judgers, and caption length. The authors explicitly state throughout that these are internal consistency results and that direct correlation with human caption judgments remains the central unmeasured validation.

Significance. If the operational score Eq. (5) were shown to track semantic preservation, the framework would be a useful reference-free, task-conditioned diagnostic for captioning, complementing human preference metrics and reference-based scores. The paper has notable strengths: it releases the CTTD benchmark and code, evaluates across multiple established vision-language datasets, includes generator/judge/length ablations, and is unusually candid about component dependence and the lack of human validation. However, the significance is conditional: the central operational claim is not yet validated, and the paper itself concedes that Eq. (5) is not algebraically equivalent to Eq. (4), that simplified reconstructions may be easier for the judge, and that direct human correlation is still required. These concessions are not merely cosmetic; they affect the interpretation of every reported ranking.

major comments (4)
  1. [§3.2, Eqs. (4)–(5)] The central operational claim is that Eq. (5) realizes Principle 2, but it does not. For fixed I and D_i, Eq. (4) thresholds the mean absolute deviation |J(I,D_i)-J(\hat I,D_i)|, while Eq. (5) ranks by mean J(\hat I,D_i). These rankings coincide only if J(\hat I) is never above J(I) for every caption; the paper concedes 'a simplified reconstruction may be easier for the judge.' A caption that drops OCR text, small objects, or unusual layout can produce a cleaner reconstruction and score above both the original image and a more faithful caption, inverting the intended ordering. No experiment quantifies how often this flip occurs. Tables 1 and 3 use Eq. (5), so the load-bearing monotonicity is unvalidated. Please report per-image or per-caption inversion rates against an Eq. (4)-style original-relative score, with error bars.
  2. [§5 and Appendix A.3 ('Discussion about human correlation')] The paper repeatedly states that direct correlation with human judgments of caption quality 'remains unmeasured and is therefore the central validation still required.' I agree, and this absence is load-bearing for the title's claim of 'caption evaluation.' The Q1/A1 patterns—traditional captioners below LVLMs, within-family scaling—are internal consistency checks, not evidence that the score tracks semantic fidelity as perceived by humans. Without at least system-level and ideally caption-level correlation with human semantic-preservation judgments, the framework is a proposed diagnostic rather than a validated evaluation metric. This is the single most important missing experiment.
  3. [§4.3, Q2 and Table 3] The CTTD surrogate claim rests on 'reproduces broad trends,' but no statistical agreement measure is given; only model-level averages are visually compared. Moreover, CTTD question generation and curation use the same LVLM family as the judge (Qwen2.5-VL-72B) and an automated relevance-diversity filter. The manuscript itself notes this filtering 'may favor questions that the curation components find easier,' but no analysis quantifies the selection bias. Report rank correlation (Spearman or Kendall) between CTTD and the full task suite at the image or caption level, and include a human audit of question validity and answer correctness.
  4. [§4.2/§A.3, Tables 5–7] The ablation tables are used to conclude that CTTD is 'robust and not easily affected by component.' With only two generators, two judgers, and no variance estimates, this conclusion is not supported. Table 7 changes the judge and shifts absolute scores by roughly ten points while preserving rank order; systematic interactions are plausible. Report per-condition correlation coefficients, interaction analyses, and uncertainty intervals, or qualify the robustness claim accordingly.
minor comments (4)
  1. [Throughout] Typos and inconsistent labels: 'Perfermance' (Fig. 3), 'Sence' (Fig. 2), 'judger' vs. 'judge', 'Upperbound', and 'CTDD (ours)' in Fig. 10. Please unify terminology and fix typos.
  2. [§3.3 / §4.2] Eq. (6) uses GMM thresholds tau_low and tau_high but their values are not specified; the relevance-diversity parameter alpha is set to 0.5 with no sensitivity analysis. Please provide the threshold values and a small sensitivity study.
  3. [References] Reference [21] duplicates [11] (both are Radford et al., CLIP). The self-citation [10] (CCCaption) is not load-bearing to the framework; if kept, it should be explicitly related in the related-work discussion.
  4. [Table 2 / Fig. 10] The claim that CTTD shows 'stronger consistency' than CaptionQA/FLEUR is based on visual rank comparison. Report rank-correlation coefficients with the VLT reference ranking and confidence intervals.

Circularity Check

0 steps flagged

No significant circularity: the scoring rule is an explicitly acknowledged proxy evaluated on external benchmarks, not a fitted input or self-referential derivation.

full rationale

The derivation chain is not circular. Principle 2 ('a caption is as good as its capacity to enable a reconstruction that is semantically equivalent to the original') is stipulated as an evaluation criterion, and Eq. (4) formalizes semantic equivalence via judge performance. Eq. (5) is then introduced as a practical score, and the paper explicitly disclaims algebraic equivalence: 'Equation (4) states the intended equivalence criterion, whereas Eq. (5) is a practical score and is not algebraically equivalent to that criterion.' The possible divergence (a simplified reconstruction being easier for the judge) is stated in the same paragraph and again in the conclusion: 'Because it uses reconstructed-image accuracy directly, a simplified or standardized image can occasionally be easier for the judge even when its caption omits details.' This is an acknowledged validity gap, not a hidden reduction: no parameter is fitted to force the model rankings, and the score is computed from frozen components on pre-existing external benchmarks (MMBench, MathVista, RefCOCO, etc.). CTTD is a curated QA bank used as a lower-cost surrogate, not a regression target. The only overlapping-author citation, CCCaption [10], appears in a related-work list on pre-trained VLMs and is not load-bearing for any conclusion. The paper also states plainly that human correlation is unmeasured: 'Direct correlation with detailed human caption assessments remains unmeasured and is therefore the central validation still required.' That missing validation is a correctness/validity concern, which the analysis rules distinguish from circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 1 invented entities

The central claim rests on the assumption that downstream task accuracy on a reconstruction is a valid proxy for caption semantics. The choice of generator, judge, and QA curation are all free choices; the equivalence criterion's tolerance σ is never used in the actual score.

free parameters (3)
  • Relevance-diversity balance α = 0.5
    Eq. (7) in §3.3; chosen by hand to balance relevance and redundancy in CTTD QA curation; affects which questions are retained.
  • GMM density thresholds τ_low, τ_high = not reported
    Eq. (6) in §3.3; hand-set thresholds for image selection; values are not given, so replication requires guessing.
  • Semantic equivalence tolerance σ = not instantiated
    Eq. (4) defines semantic equivalence with tolerance σ, but the practical score Eq. (5) drops σ and the original-image terms; the parameter is never set.
axioms (4)
  • domain assumption Semantic equivalence is approximated by downstream vision-language task performance of a judge model (Eq. 4).
    The framework equates 'semantically equivalent' with judge performance difference ≤ σ on a task suite; this is an unproven proxy for human semantic equivalence.
  • domain assumption A frozen text-to-image generator fimg and judge J are faithful instruments; reconstruction stochasticity does not dominate caption effects.
    The paper acknowledges generator stochasticity and layout shift can confound; ablations only cover two generators and two judges.
  • domain assumption LVLM-generated QA pairs cover caption-relevant semantics without systematic bias.
    CTTD questions are generated by Qwen2.5-VL-72B and curated with LVLM-based filtering; the paper admits automated filtering may favor questions the pipeline finds easier.
  • domain assumption Image selection via GMM on embeddings keeps a representative difficulty range.
    Eq. (6) assumes embedding density correlates with evaluative difficulty; thresholds are unreported.
invented entities (1)
  • Semantic equivalence relation ∼ on image space no independent evidence
    purpose: Defines the reconstruction target [I] and the relaxed evaluation criterion (Eqs. 3–4).
    Introduced as an unobservable construct; no independent evidence that the judge-based approximation matches human semantic judgments, and the paper itself says human correlation is unmeasured.

pith-pipeline@v1.3.0-alltime-deepseek · 19462 in / 10895 out tokens · 97534 ms · 2026-07-31T23:59:18.319916+00:00 · methodology

0 comments
read the original abstract

Image captioning is a primary task in vision--language research, yet assessing how faithfully a caption preserves image semantics without relying on reference captions remains unsettled. Prevailing evaluations rely on human-annotated references, whose content reflects annotator intent and captioning proficiency. In this paper, we study a reconstruction-based principle for caption evaluation: a caption is as good as its capacity to enable reconstruction of the original image. However, because captioning inherently compresses visual information, it is impossible to recover all details, and pixel-wise comparison between reconstructed and source images is neither feasible nor meaningful. Through our in-depth analysis of the nature of captions, whose fundamental purpose is to transmit the semantic content of an image, we propose a revised principle: a caption is as good as its capacity to enable a reconstruction that is semantically equivalent to the original. To assess semantic equivalence, we test whether the reconstruction matches the original image across a suite of downstream vision--language tasks, yielding a reference-free, task-conditioned caption score. We characterize component-dependent limitations and introduce the lower-cost Captioning Turing Test Dataset (CTTD) surrogate.

Figures

Figures reproduced from arXiv: 2607.23235 by Jianqiang Huang, Jiaxin Qi, Kaihua Tang, Yuhua Zheng, Zhijiang Tang.

Figure 1
Figure 1. Figure 1: Illustration of the fundamental role of captions, conveying semantics of an image to others. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of our framework and the Captioning Turing Test Dataset (CTTD) pipeline. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance of various captioning models in our Captioning Turing Test Dataset (CTTD). [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Overview of the Captioning Turing Test Dataset (CTTD). [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Prompts for generating questions of each category. The red portions indicate the parts that [PITH_FULL_IMAGE:figures/full_fig_p015_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Prompt for the captioning model to generate captions for an image. We adjust the caption [PITH_FULL_IMAGE:figures/full_fig_p016_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Prompt for the judge model to answer visual questions by reconstructing the image. The [PITH_FULL_IMAGE:figures/full_fig_p016_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Prompt for generating 15 categories of questions, including Perception, Scene, Spatial, [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Some case examples in CTTD. The red portions indicate the category of the respective [PITH_FULL_IMAGE:figures/full_fig_p020_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Comparison of ranking performance across different evaluation methods. The figure [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

57 extracted references · 14 linked inside Pith

  1. [1]

    Image captioning: Transform- ing objects into words.Advances in neural information processing systems, 32, 2019

    Simao Herdade, Armin Kappeler, Kofi Boakye, and Joao Soares. Image captioning: Transform- ing objects into words.Advances in neural information processing systems, 32, 2019. 9

    2019

  2. [2]

    A comprehen- sive survey of deep learning for image captioning.ACM Computing Surveys (CsUR), 51(6): 1–36, 2019

    MD Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shiratuddin, and Hamid Laga. A comprehen- sive survey of deep learning for image captioning.ACM Computing Surveys (CsUR), 51(6): 1–36, 2019

    2019

  3. [3]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InInternational Conference on Machine Learning, pages 19730–19742. PMLR, 2023

    2023

  4. [4]

    Surveying the landscape of image captioning evaluation: A comprehensive taxonomy and novel ensemble method.arXiv e-prints, pages arXiv–2408, 2024

    Uri Berger, Gabriel Stanovsky, Omri Abend, and Lea Frermann. Surveying the landscape of image captioning evaluation: A comprehensive taxonomy and novel ensemble method.arXiv e-prints, pages arXiv–2408, 2024

    2024

  5. [5]

    Revisit large-scale image-caption data in pre-training multimodal foundation models.arXiv preprint arXiv:2410.02740, 2024

    Zhengfeng Lai, Vasileios Saveris, Chen Chen, Hong-You Chen, Haotian Zhang, Bowen Zhang, Juan Lao Tebar, Wenze Hu, Zhe Gan, Peter Grasch, et al. Revisit large-scale image-caption data in pre-training multimodal foundation models.arXiv preprint arXiv:2410.02740, 2024

    Pith/arXiv arXiv 2024

  6. [6]

    Image captioning evaluation in the age of multimodal llms: Challenges and future perspectives.arXiv preprint arXiv:2503.14604, 2025

    Sara Sarto, Marcella Cornia, and Rita Cucchiara. Image captioning evaluation in the age of multimodal llms: Challenges and future perspectives.arXiv preprint arXiv:2503.14604, 2025

    Pith/arXiv arXiv 2025

  7. [7]

    Show and tell: A neural image caption generator

    Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural image caption generator. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3156–3164, 2015

    2015

  8. [8]

    End-to-end transformer based model for image captioning

    Yiyu Wang, Jungang Xu, and Yingfei Sun. End-to-end transformer based model for image captioning. InProceedings of the AAAI conference on artificial intelligence, volume 36, pages 2585–2594, 2022

    2022

  9. [9]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  10. [10]

    Cccaption: Dual-reward reinforcement learning for complete and correct image captioning.arXiv preprint arXiv:2602.21655, 2026

    Zhijiang Tang, Linhua Wang, Jiaxin Qi, Weihao Jiang, Peng Hou, Anxiang Zeng, and Jian- qiang Huang. Cccaption: Dual-reward reinforcement learning for complete and correct image captioning.arXiv preprint arXiv:2602.21655, 2026

    arXiv 2026

  11. [11]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational Conference on Machine Learning, pages 8748–8763. PmLR, 2021

    2021

  12. [12]

    Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified vision-language understanding and generation. InInternational Conference on Machine Learning, pages 12888–12900. PMLR, 2022

    2022

  13. [13]

    Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  14. [14]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. a...

    Pith/arXiv arXiv 2025

  15. [15]

    Qwen3 technical report, 2025

    Qwen Team. Qwen3 technical report, 2025. URLhttps://arxiv.org/abs/2505.09388

    Pith/arXiv arXiv 2025

  16. [16]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  17. [17]

    Fleur: An explainable reference-free evaluation metric for image captioning using a large multimodal model.arXiv preprint arXiv:2406.06004, 2024

    Yebin Lee, Imseong Park, and Myungjoo Kang. Fleur: An explainable reference-free evaluation metric for image captioning using a large multimodal model.arXiv preprint arXiv:2406.06004, 2024. 10

    Pith/arXiv arXiv 2024

  18. [18]

    Caparena: Benchmarking and analyzing detailed image captioning in the llm era.arXiv preprint arXiv:2503.12329, 2025

    Kanzhi Cheng, Wenpo Song, Jiaxin Fan, Zheng Ma, Qiushi Sun, Fangzhi Xu, Chenyang Yan, Nuo Chen, Jianbing Zhang, and Jiajun Chen. Caparena: Benchmarking and analyzing detailed image captioning in the llm era.arXiv preprint arXiv:2503.12329, 2025

    Pith/arXiv arXiv 2025

  19. [19]

    Prometheus-vision: Vision-language model as a judge for fine-grained evaluation

    Seongyun Lee, Seungone Kim, Sue Park, Geewook Kim, and Minjoon Seo. Prometheus-vision: Vision-language model as a judge for fine-grained evaluation. InFindings of the Association for Computational Linguistics: ACL 2024, pages 11286–11315, 2024

    2024

  20. [20]

    Caprl: Stimulating dense image caption capabilities via reinforcement learning.arXiv preprint arXiv:2509.22647, 2025

    Long Xing, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Jianze Liang, Qidong Huang, Jiaqi Wang, Feng Wu, and Dahua Lin. Caprl: Stimulating dense image caption capabilities via reinforcement learning.arXiv preprint arXiv:2509.22647, 2025

    arXiv 2025

  21. [21]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pages 8748–8763. PmLR, 2021

    2021

  22. [22]

    Clipscore: A reference-free evaluation metric for image captioning.arXiv preprint arXiv:2104.08718, 2021

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning.arXiv preprint arXiv:2104.08718, 2021

    Pith/arXiv arXiv 2021

  23. [23]

    Positive- augmented contrastive learning for image and video captioning evaluation

    Sara Sarto, Manuele Barraco, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Positive- augmented contrastive learning for image and video captioning evaluation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6914–6924, 2023

    2023

  24. [24]

    Evaluating image caption via cycle-consistent text-to-image generation.arXiv preprint arXiv:2501.03567, 2025

    Tianyu Cui, Jinbin Bai, Guo-Hua Wang, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Ye Shi. Evaluating image caption via cycle-consistent text-to-image generation.arXiv preprint arXiv:2501.03567, 2025

    Pith/arXiv arXiv 2025

  25. [25]

    Im- age2text2image: A novel framework for label-free evaluation of image-to-text generation with text-to-image diffusion models

    Jia-Hong Huang, Hongyi Zhu, Yixian Shen, Stevan Rudinac, and Evangelos Kanoulas. Im- age2text2image: A novel framework for label-free evaluation of image-to-text generation with text-to-image diffusion models. InInternational Conference on Multimedia Modeling, pages 413–427. Springer, 2025

    2025

  26. [26]

    Visual fact checker: Enabling high-fidelity detailed caption generation

    Yunhao Ge, Xiaohui Zeng, Jacob Samuel Huffman, Tsung-Yi Lin, Ming-Yu Liu, and Yin Cui. Visual fact checker: Enabling high-fidelity detailed caption generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14033–14042, 2024

    2024

  27. [27]

    Yushi Hu, Benlin Liu, Jungo Kasai, Yizhong Wang, Mari Ostendorf, Ranjay Krishna, and Noah A. Smith. Tifa: Accurate and interpretable text-to-image faithfulness evaluation with question answering.arXiv preprint arXiv:2303.11897, 2023

    Pith/arXiv arXiv 2023

  28. [28]

    Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation

    Jaemin Cho, Abhay Zala, and Mohit Bansal. Davidsonian scene graph: Improving reliability in fine-grained evaluation for text-to-image generation. InInternational Conference on Learning Representations, 2024

    2024

  29. [29]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InEuropean Conference on Computer Vision, pages 740–755. Springer, 2014

    2014

  30. [30]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. InProceedings of the IEEE international conference on computer vision, pages 2425–2433, 2015

    2015

  31. [31]

    Beyond quantity: Distribution- aware labeling for visual grounding.arXiv preprint arXiv:2505.24372, 2025

    Yichi Zhang, Gongwei Chen, Jun Zhu, Jia Wan, and Liqiang Nie. Beyond quantity: Distribution- aware labeling for visual grounding.arXiv preprint arXiv:2505.24372, 2025

    arXiv 2025

  32. [32]

    Mmbench: Is your multi-modal model an all-around player? InEuropean Conference on Computer Vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? InEuropean Conference on Computer Vision, pages 216–233. Springer, 2024

    2024

  33. [33]

    Are we on the right way for evaluating large vision- language models?Advances in Neural Information Processing Systems, 37:27056–27087, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision- language models?Advances in Neural Information Processing Systems, 37:27056–27087, 2024. 11

    2024

  34. [34]

    Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pag...

    2024

  35. [35]

    Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 3, 2023

    Fu Chaoyou, Chen Peixian, Shen Yunhang, Qin Yulei, Zhang Mengdan, Lin Xu, Yang Jinrui, Zheng Xiawu, Li Ke, Sun Xing, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 3, 2023

    Pith/arXiv arXiv 2023

  36. [36]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023

    Pith/arXiv arXiv 2023

  37. [37]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? InEuropean Conference on Computer Vision, pages 169–186

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? InEuropean Conference on Computer Vision, pages 169–186. Springer, 2024

    2024

  38. [38]

    Measuring multimodal mathematical reasoning with math-vision dataset

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37:95095–95169, 2024

    2024

  39. [39]

    Ocrbench: on the hidden mystery of ocr in large multimodal models.Science China Information Sciences, 67(12):220102, 2024

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models.Science China Information Sciences, 67(12):220102, 2024

    2024

  40. [40]

    Ocr-vqa: Visual question answering by reading text in images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In2019 international conference on document analysis and recognition (ICDAR), pages 947–952. IEEE, 2019

    2019

  41. [41]

    Referitgame: Referring to objects in photographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. InProceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787–798, 2014

    2014

  42. [42]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024

    2024

  43. [43]

    Llama-3.2-11b-vision – multimodal large language model (text + image → text)

    Meta AI. Llama-3.2-11b-vision – multimodal large language model (text + image → text). Hugging Face Model Card, 2024. URL https://huggingface.co/meta-llama/Llama-3. 2-11B-Vision. Release date: Sept. 25 2024. Instruction-tuned for image reasoning, captioning and VQA

    2024

  44. [44]

    Internvl3

    Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency.arXiv preprint arXiv:2508.18265, 2025

    Pith/arXiv arXiv 2025

  45. [45]

    Finite mixture models.Annual review of statistics and its application, 6(1):355–378, 2019

    Geoffrey J McLachlan, Sharon X Lee, and Suren I Rathnayake. Finite mixture models.Annual review of statistics and its application, 6(1):355–378, 2019

    2019

  46. [46]

    A density-based algorithm for discovering clusters in large spatial databases with noise

    Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. Inkdd, volume 96, pages 226–231, 1996

    1996

  47. [47]

    Qwen-image technical report, 2025

    Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, Yuxiang Chen, Zecheng Tang, Zekai Zhang, Zhengyi Wang, An Yang, Bowen Yu, Chen Cheng, Dayiheng Liu, Deqing Li, Hang Zhang, Hao Meng, Hu Wei, Jingyuan Ni, Kai Chen, Kuan Cao, Liang Peng, Lin Qu, Minggang Wu, Peng Wang, Shuting Yu, Tingkun...

    Pith/arXiv arXiv 2025

  48. [48]

    Unitbox: An advanced object detection network

    Jiahui Yu, Yuning Jiang, Zhangyang Wang, Zhimin Cao, and Thomas Huang. Unitbox: An advanced object detection network. InProceedings of the 24th ACM International Conference on Multimedia, pages 516–520, 2016

    2016

  49. [49]

    Opensearch-ai / ops-mm-embedding-v1-7b

    Alibaba Cloud OpenSearch-AI Team. Opensearch-ai / ops-mm-embedding-v1-7b. Hug- ging Face Model Hub, 2025. URL https://huggingface.co/OpenSearch-AI/ Ops-MM-embedding-v1-7B. Version v1-7B, fine-tuned from Qwen2-VL

    2025

  50. [50]

    Gpt-5: A new era in language models.OpenAI Technical Report, 2025

    OpenAI. Gpt-5: A new era in language models.OpenAI Technical Report, 2025. URL https://openai.com/gpt5

    2025

  51. [51]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Pith/arXiv arXiv 2001

  52. [52]

    Prism: A framework for decoupling and assessing the capabilities of vlms.Advances in Neural Information Processing Systems, 37:111863–111898, 2024

    Yuxuan Qiao, Haodong Duan, Xinyu Fang, Junming Yang, Lin Chen, Songyang Zhang, Jiaqi Wang, Dahua Lin, and Kai Chen. Prism: A framework for decoupling and assessing the capabilities of vlms.Advances in Neural Information Processing Systems, 37:111863–111898, 2024

    2024

  53. [53]

    High- resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  54. [54]

    question

    Stability AI. Stable diffusion 3.5 large turbo. https://huggingface.co/stabilityai/ stable-diffusion-3.5-large-turbo, Oct 2024. Model available at Hugging Face. 13 A Supplementary Material The following supplementary materials provide additional details to complement the main content. More details for the method:We define the evaluation metrics for variou...

    arXiv 2024

  55. [55]

    [42] [44] [14] [15] Qwen [47] 47.19 57.68 58.64 58.81 59.44 SD3.5 [54] 49.85 58.90 59.30 60.01 60.29 Table 5: Ablation study of different generators on the Captioning Turing Test Dataset. The table compares the performance of two generators, Qwen (Qwen-Image) and SD3.5 (StableDiffusion-3.5- Large-Turbo), across various models: BLIP, LLaVa-V1.6-7B, InternV...

  56. [56]

    gold standard

    [42] [44] [14] [15] 100 48.41 59.97 61.43 61.35 62.28 200 46.62 55.44 56.10 56.38 56.74 300 48.64 60.77 61.65 61.79 62.83 400 48.13 60.02 61.45 61.75 62.75 Table 6: Ablation study of different maximum caption lengths on the Captioning Turing Test Dataset. The models tested are the same as those in Table 5. We conducted extensive ablation experiments. As s...

  57. [57]

    Compares the performance of two judgers, Qwen2.5 (Qwen2.5-VL-3B) and Qwen3 (Qwen3-VL-8B)

    [42] [44] [14] [15] Qwen2.5 [14] 47.42 51.49 51.64 52.14 52.36 Qwen3 [15] 48.68 59.93 61.03 61.26 62.05 Table 7: Ablation study of different judgers on the Captioning Turing Test Dataset. Compares the performance of two judgers, Qwen2.5 (Qwen2.5-VL-3B) and Qwen3 (Qwen3-VL-8B). The models tested are the same as those in Table 5. 22