Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

SC-Captioner: Improving Image Captioning with Self-Correction by Reinforcement Learning

T0 review · 4 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SC-Captioner teaches vision-language models to fix their own captions with a scene-graph-based reward, beating DPO.

desk verdict A genuinely new RL self-correction recipe for captioning with a sensible reward, but the headline gains may be partly an artifact of sharing the same scene-graph parser between reward and metrics. read the letter →

arxiv 2508.06125 v1 pith:ABSSEHFB submitted 2025-08-08 cs.CV

Lin Zhang , Xianfang Zeng , Kangcong Li , Gang Yu , Tao Chen This is my paper
classification cs.CV
keywords imagecaptioningself-correctionreinforcementlearningrewarddesignscene-graphparsinglargevision-languagemodelsdirectpreferenceoptimizationrefinedevaluationmetrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that image-caption models can learn to improve their own first-pass captions if a reinforcement-learning reward is designed around the actual corrections the model makes. The reward decomposes captions into objects, attributes, and relations, then rewards additions and removals that move the caption closer to a reference while punishing those that move it away. Trained this way on a new 6.5K-image human-refined dataset, two 7B vision-language models outperform the same models trained with direct preference optimization on detailed caption metrics, both in natural and self-corrected captions. The paper also proposes a more consistent evaluation metric for detailed captions and a question-answering based relation score.

What carries the argument

The machinery is a correction-based reward computed from scene-graph-parsed set differences. Objects, attributes, and relations are extracted from the initial and self-corrected captions; added and removed elements are identified by set difference and matched against the reference sets. Correct additions and removals earn a bonus; wrong ones incur a punishment. This reward is plugged into a multi-turn policy-gradient objective, with a policy loss on the second-turn self-corrected caption plus a KL term preserving the initial caption, so the model learns when and what to change.

What would settle it

Take a dataset with sparse or short reference captions, train with SC-Captioner, and have human annotators mark which model-added elements are actually visible in the image; if a large fraction of the punished additions are correct, the reward is miscalibrated. A simpler check: compute the correlation between the reward and human judgment of individual edits on a held-out set.

Watch

Extended reading notes

Core claim

The central claim is that self-correction is a learnable behavior and that the right training signal is not a preference over whole captions but a fine-grained reward over caption edits. Using a scene-graph parser, the method represents initial, corrected, and ground-truth captions as sets of objects, attributes, and relations. Set differences isolate what the model added or removed; each such element is matched against the reference sets to give a correctness bonus for accurate refinements and a mistake punishment for wrong insertions and deletions. On DOCCI500 and COCO-LN500, models trained with this reward show higher object and attribute recall and relation QA accuracy than DPO, which te

Load-bearing premise

Training assumes the reference captions are complete enough that an element absent from the reference but present in the image is rare, so punishing such additions is safe.

Editorial extensions

If this is right

  • Applied as a post-training step after supervised fine-tuning, the method lifts CAPTURE, object and attribute recall, and relation QA scores over SFT alone on two 7B vision-language models.
  • The self-corrected captions produced by the trained models score higher than their initial captions, while untrained or SFT-only models' corrections do not.
  • Training on a public dataset such as DOCCI instead of the new RefinedCaps still shows the same advantage over SFT and DPO in same-domain and cross-domain tests.
  • The refined evaluation metrics, including a question-answering based relation score, correlate better with human judgment than the original CAPTURE metric, offering a more reliable reading of detailed caption quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the reward is defined over edit differences rather than full-caption quality, the same formulation could be adapted to other editing tasks (e.g., text-to-image correction or summarization revision) where a reference is available; the paper does not investigate this extension.
  • The dependence on scene-graph parser output means false parses directly corrupt rewards; using a higher-quality parser or combining multiple parses could improve stability further.
  • A direct test of the completeness assumption would be to apply the same training to a dataset with deliberately sparse references and check whether the model's correct additions get penalized; the paper's own caveat in Sec. 3.3 suggests this is the main failure mode.
  • The method could be combined with data augmentation that synthesizes incomplete initial captions, potentially reducing the cost of collecting paired initial/corrected data.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes SC-Captioner, a reinforcement-learning framework for training image-captioning LVLMs to self-correct their own outputs. The reward function parses initial and self-corrected captions into object, attribute, and relation sets using the FACTUAL scene-graph parser, then gives bonuses for additions/removals that match the reference and penalties for those that do not. The paper also introduces RefinedCaps, a 6.5K-image human-refined training set derived from COCO, and proposes refinements to the CAPTURE evaluation metric, including object/relation matching changes and a question-answering relation score. Experiments on LLaVA-1.5-7B and Qwen2-VL-7B compare SFT, SFT+DPO, and SFT+SC-Captioner on DOCCI500 and COCO-LN500, reporting improvements in CAPTURE and the proposed metrics. The central claim is that SC-Captioner significantly outperforms DPO for self-correction and improves caption quality on both initial and self-corrected outputs.

Significance. If the central claim holds, the paper makes a useful contribution: it demonstrates a training signal for self-correction in image captioning, introduces a new human-refined dataset, and proposes a more detailed evaluation protocol. The release of code and the human annotation effort are concrete strengths. However, the headline conclusion depends on the validity of the proposed metrics, and the reward function and those metrics are built from the same scene-graph parsing pipeline. Independent evaluation is therefore essential before the significance of the claimed improvement over DPO can be assessed.

major comments (4)
  1. [§3.3 vs §4.2; Tables 1-2] The reward in §3.3 and the headline metrics in §4.2 share the same FACTUAL scene-graph parser, the same object/attribute/relation decomposition, and similar set-matching logic. Tables 1 and 2 show that SC-Captioner's advantage over SFT+DPO is positive across parser-based columns, but inconsistent on parser-independent BLEU-4/METEOR (e.g., LLaVA DOCCI500: BLEU-4 43.04 vs 42.67, METEOR 23.88 vs 23.47; Qwen COCO-LN500: BLEU-4 35.05 vs 30.34 but METEOR 27.34 vs 26.77). Consequently, the reported improvements may partly reflect the model learning to match FACTUAL's error patterns rather than producing better captions. Please evaluate with reference-free metrics that do not share the parser, and with an independent human preference study, before claiming a general advantage over DPO.
  2. [§9, Table 4] The human-consistency check uses only 100 images and 4 model outputs, and reports Kendall's tau without confidence intervals, inter-annotator agreement, or significance testing. This is too weak to rule out reward hacking of FACTUAL's specific failure modes. A larger evaluation (e.g., 300+ images, pairwise preferences, multiple annotators) is needed to establish that the proposed metric, and the improvements measured with it, align with human judgments of caption quality.
  3. [§5.1, Tables 1-2] All experimental results are reported as single runs without error bars or significance tests. The test sets are small (500 images each), and COCO-LN500 is filtered to captions containing human-related words and at least 60 words. The abstract's 'significantly outperforming' claim is not supported by statistical evidence. Please provide standard deviations over multiple seeds and paired significance tests for the main SFT+DPO vs SFT+Ours comparisons.
  4. [§3.3, §5.2] The reward formulation relies on the premise that reference captions are complete enough that penalizing additions absent from the reference is safe. The paper acknowledges in §3.3 that this may punish correct descriptions, and states that the phenomenon is 'less common' when GT is complete, but no quantitative evidence is provided. The paper's own discussion in §5.2 shows that shorter GT captions in COCO-LN500 cause CAPTURE to penalize correctly identified objects, so the concern is not hypothetical. Please quantify how often the reward punishes image-true but reference-absent additions, or modify the reward to avoid this asymmetry.
minor comments (4)
  1. [Abstract/§1] The test set name is given as COCO-LR500 in the Introduction but COCO-LN500 in Tables 1-2; please make the naming consistent.
  2. [§1, §3.1] There are several typos and garbled formulas, e.g., 'challanges' in the Introduction and the corrupted math rendering in Eq. (1) and Eq. (4). Please proofread the equations and text.
  3. [§4.1] The dataset construction states that 'at least 80% of the objects in the image should appear in the caption' but does not report how this was measured or verified. Clarify the procedure and annotator instructions.
  4. [§11.1, Table 6] The comparison between DOCCI-trained and RefinedCaps-trained models is informative, but the discussion should avoid over-claiming generalization from a single cross-domain test set; the same statistical caveats as the main experiments apply.

Circularity Check

1 steps flagged · score 4.0 of 10

SC-Captioner's headline parser-based gains are partly built into its reward: the FACTUAL scene-graph decomposition and set matching used for training reward are the same components used in the proposed metrics and CAPTURE, so those improvements are partially forced. Independent BLEU/METEOR gains provide some non-circular support, preventing a higher score.

  1. fitted input called prediction [Sec. 3.3 (Reward Formulation) and Sec. 4.2 (Evaluating Metrics); see also Tab. 3]
    "the SOTA scene graph parser FACTUAL [25] is employed to extract objects, attributes and relations of them. Then rewards are calculated based on these three aspects respectively. ... The FACTUAL scene graph parser is employed to extract objects, attributes and relations in the beginning."

    The reward is computed as correctness bonuses and mistake punishments over FACTUAL-parsed object/attribute/relation sets matched against the reference (Eq. 3 and surrounding text). The headline evaluation metrics—CAPTURE and the paper's refined metrics—are computed with the same FACTUAL parser over the same three components, as precision/recall/F1 or QA. The model is therefore trained to maximize a score whose components are essentially the same as the measured outcome. The paper's own ablation confirms this: 'rewards on objects, attributes and relations can notably improve the performance on these three concepts respectively.' Thus the reported gains on parser-based metrics are partly a re-measurement of the training objective rather than independent evidence of better captions. The indep

full rationale

The main circularity concern is not a self-citation chain but the overlap between the training reward and the evaluation metrics. Sec. 3.3 defines the reward using FACTUAL scene-graph parsing into objects/attributes/relations and set-difference matching against GT; Sec. 4.2 defines the proposed metrics using the same FACTUAL parser and the same three components (with GPT-4o/human expansion for precision and QA for relations). CAPTURE, also used as a headline metric, follows the same FACTUAL-based decomposition. Consequently, improvements in Tables 1-3 on CAPTURE and the refined metrics are partly expected from optimizing the reward; Tab. 3 explicitly shows each reward component lifts the corresponding metric component. This is a partial 'training on the test metric' situation. The paper does retain independent evidence: BLEU-4 and METEOR do not share the parser and mostly improve for self-corrected captions (e.g., Qwen COCO-LN500 BLEU-4 35.05 vs 30.34; METEOR 27.34 vs 26.77), so the central claim is not fully forced. The Sec. 9 human-consistency check (100 images, Kendall 45.99) is too small to rule out parser-reward hacking. I found no load-bearing self-citations; the completeness caveat in Sec. 3.3 is a dataset assumption, not circularity. Overall score 4 reflects partial circularity in the headline parser-based evaluation while acknowledging non-circular support from independent metrics.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The central claim rests on the fidelity of the scene-graph parser, the completeness of the human-refined reference captions, and the reliability of GPT-4o-assisted precision expansion. Two key hyperparameters (similarity threshold, KL coefficient) are unreported. RefinedCaps, the main new resource, is not stated to be released.

free parameters (2)
  • Similarity threshold for hard-score matching in reward
    In Sec. 3.3, hard scores reward additions/removals only above an unspecified similarity threshold, likely tuned on a validation set.
  • KL coefficient in RL objective (Eq. 1)
    The policy-gradient objective includes a KL penalty whose coefficient is not reported, though it shapes the trade-off between reward and distributional stability.
assumptions (4)
  • domain assumption FACTUAL scene-graph parser accurately extracts objects, attributes, and relations from captions
    Both the reward function (Sec. 3.3) and the proposed metrics (Sec. 4.2) depend on parser outputs; parsing errors propagate to training signal and evaluation.
  • domain assumption Reference captions in RefinedCaps are complete enough for punishing unmentioned additions to be safe
    Acknowledged in Sec. 3.3; if GT is incomplete, correct additions to real image content are inadvertently punished.
  • domain assumption GPT-4o-generated expanded object and attribute sets used for precision computation are accurate
    In Sec. 4.2, the proposed metric merges GPT-4o outputs 'evaluated by human experts' into reference sets for precision, assuming these extra items truly exist in the image.
  • domain assumption The relation QA evaluation reflects caption quality
    Relations are scored by asking an LM to answer five human-chosen questions per test image (Sec. 4.2); the question-generation and answering pipeline is taken as a reliable proxy for relation correctness.
invented entities (1)
  • RefinedCaps dataset
    purpose: Training data with human-refined, long captions for 6.5K COCO images
    The paper states code is released but does not state that the dataset is publicly released, so there is no external falsifiable handle such as a downloadable benchmark.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SC-Captioner: Improving Image Captioning with Self-Correction by Reinforcement Learning." pith.science (2026). https://pith.science/paper/ABSSEHFB

@misc{pith2026250806125,
  author       = {Pith},
  title        = {Pith review of: SC-Captioner: Improving Image Captioning with Self-Correction by Reinforcement Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ABSSEHFB}},
  note         = {Machine review of arXiv:2508.06125}
}
read the original abstract

We propose SC-Captioner, a reinforcement learning framework that enables the self-correcting capability of image caption models. Our crucial technique lies in the design of the reward function to incentivize accurate caption corrections. Specifically, the predicted and reference captions are decomposed into object, attribute, and relation sets using scene-graph parsing algorithms. We calculate the set difference between sets of initial and self-corrected captions to identify added and removed elements. These elements are matched against the reference sets to calculate correctness bonuses for accurate refinements and mistake punishments for wrong additions and removals, thereby forming the final reward. For image caption quality assessment, we propose a set of metrics refined from CAPTURE that alleviate its incomplete precision evaluation and inefficient relation matching problems. Furthermore, we collect a fine-grained annotated image caption dataset, RefinedCaps, consisting of 6.5K diverse images from COCO dataset. Experiments show that applying SC-Captioner on large visual-language models can generate better image captions across various scenarios, significantly outperforming the direct preference optimization training strategy.

Discussion (0). Sign in to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Modeling Complex Behaviors: Multi-Personality Composition and Dynamic Switching in Vision-Language Models

    cs.CL 2026-06 unverdicted novelty 5.0 of 10

    The work establishes an evaluation framework for personality induction and switching in MLLMs, reporting improved captioning but impaired VQA performance plus balancing and residual effects during multi-trait and dyna...

  2. VCap: Hypergeometric Rewards for Weak-to-Strong Visual Captioning

    cs.CV 2026-05 unverdicted novelty 5.0 of 10

    VCap pairs reference captions as witnesses with visual signals as adjudicators to deliver hypergeometric-precision rewards for RL in visual captioning, enabling an 8B model to outperform SOTA on benchmarks and improve...

Reference graph

Works this paper leans on

63 extracted references · 31 canonical work pages · cited by 2 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ah- mad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 ,

    arXiv

  2. [2]

    Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with hu- man judgments. In Proceedings of the acl workshop on in- trinsic and extrinsic evaluation measures for machine trans- lation and/or summarization, pages 65–72, 2005. 3, 4

    work page 2005

  3. [3]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 2

    arXiv 2023

  4. [4]

    Microsoft coco captions: Data collection and evaluation server

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedan- tam, Saurabh Gupta, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015. 2

    arXiv 2015

  5. [5]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 2

    arXiv 2024

  6. [6]

    Meshed-memory transformer for image cap- tioning

    Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. Meshed-memory transformer for image cap- tioning. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 10578–10587,

  7. [7]

    Cross-domain image captioning with dis- criminative finetuning

    Roberto Dess `ı, Michele Bevilacqua, Eleonora Gualdoni, Nathana¨el Carraz Rakotonirina, Francesca Franzon, and Marco Baroni. Cross-domain image captioning with dis- criminative finetuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6935–6944, 2023. 2

    work page 2023

  8. [8]

    Benchmarking and improv- ing detail image caption

    Hongyuan Dong, Jiawen Li, Bohong Wu, Jiacong Wang, Yuan Zhang, and Haoyuan Guo. Benchmarking and improv- ing detail image caption. arXiv preprint arXiv:2405.19092,

    arXiv

Show all 63 references
  1. [9]

    ����� �: Vila augmented vila

    Yunhao Fang, Ligeng Zhu, Yao Lu, Yan Wang, Pavlo Molchanov, Jang Hyun Cho, Marco Pavone, Song Han, and Hongxu Yin. ����� �: Vila augmented vila. arXiv preprint arXiv:2407.17453, 2024. 2

    2024 arXiv

  2. [10]

    The capacity for moral self-correction in large language models

    Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas I Liao, Kamil ˙e Lukoˇsi¯ut˙e, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. The capacity for moral self-correction in large language models. arXiv preprint arXiv:2302.07459, 2023. 3

    2023 arXiv

  3. [11]

    Glore: When, where, and how to im- prove llm reasoning via global and local refinements

    Alex Havrilla, Sharath Raparthy, Christoforus Nalmpantis, Jane Dwivedi-Yu, Maksym Zhuravinskyi, Eric Hambro, and Roberta Raileanu. Glore: When, where, and how to im- prove llm reasoning via global and local refinements. arXiv preprint arXiv:2402.10963, 2024. 3

    2024 arXiv

  4. [12]

    Self-correction is more than refinement: A learning frame- work for visual and language reasoning tasks

    Jiayi He, Hehai Lin, Qingyun Wang, Yi Fung, and Heng Ji. Self-correction is more than refinement: A learning frame- work for visual and language reasoning tasks. arXiv preprint arXiv:2410.04055, 2024. 3

    2024 arXiv

  5. [13]

    A topic-level self-correctional ap- proach to mitigate hallucinations in mllms

    Lehan He, Zeren Chen, Zhelun Shi, Tianyu Yu, Jing Shao, and Lu Sheng. A topic-level self-correctional ap- proach to mitigate hallucinations in mllms. arXiv preprint arXiv:2411.17265, 2024. 3

    2024

  6. [14]

    Clipscore: A reference-free evaluation met- ric for image captioning

    Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation met- ric for image captioning. arXiv preprint arXiv:2104.08718,

    arXiv

  7. [15]

    Graph-based captioning: Enhancing visual descrip- tions by interconnecting region captions

    Yu-Guan Hsieh, Cheng-Yu Hsieh, Shih-Ying Yeh, Louis B´ethune, Hadi Pour Ansari, Pavan Kumar Anasosalu Vasu, Chun-Liang Li, Ranjay Krishna, Oncel Tuzel, and Marco Cuturi. Graph-based captioning: Enhancing visual descrip- tions by interconnecting region captions. arXiv preprint ...

    2024 arXiv

  8. [16]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022. 7

    2022

  9. [17]

    Large language models cannot self-correct reasoning yet

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798, 2023. 2, 3

    2023 arXiv

  10. [18]

    Attention on attention for image captioning

    Lun Huang, Wenmin Wang, Jie Chen, and Xiao-Yong Wei. Attention on attention for image captioning. In Proceedings of the IEEE/CVF international conference on computer vi- sion, pages 4634–4643, 2019. 1, 2

    2019

  11. [19]

    When can llms actually correct their own mistakes? a critical survey of self-correction of llms

    Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. When can llms actually correct their own mistakes? a critical survey of self-correction of llms. arXiv preprint arXiv:2406.01297, 2024. 3

    2024 arXiv

  12. [20]

    Training language models to self-correct via reinforcement learning

    Aviral Kumar, Vincent Zhuang, Rishabh Agarwal, Yi Su, John D Co-Reyes, Avi Singh, Kate Baumli, Shariq Iqbal, Colton Bishop, Rebecca Roelofs, et al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:2409.12917, 2024. 3, 8

    2024 arXiv

  13. [21]

    Qace: Asking questions to evaluate an image caption.arXiv preprint arXiv:2108.12560,

    Hwanhee Lee, Thomas Scialom, Seunghyun Yoon, Franck Dernoncourt, and Kyomin Jung. Qace: Asking questions to evaluate an image caption.arXiv preprint arXiv:2108.12560,

    arXiv

  14. [22]

    Fleur: An explainable reference-free evaluation metric for image cap- tioning using a large multimodal model

    Yebin Lee, Imseong Park, and Myungjoo Kang. Fleur: An explainable reference-free evaluation metric for image cap- tioning using a large multimodal model. In Proceedings of the 62nd Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers) , page...

    2024

  15. [23]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Interna- tional conference on machine learning, pages 12888–12900. PMLR, 2022. 2

    2022

  16. [24]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 1, 2

    2023

  17. [25]

    Factual: A benchmark for faithful and consistent tex- tual scene graph parsing

    Zhuang Li, Yuyang Chai, Terry Yue Zhuo, Lizhen Qu, Gholamreza Haffari, Fei Li, Donghong Ji, and Quan Hung Tran. Factual: A benchmark for faithful and consistent tex- tual scene graph parsing. arXiv preprint arXiv:2305.17497,

    arXiv

  18. [26]

    Mon- key: Image resolution and text label are important things for large multi-modal models

    Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Mon- key: Image resolution and text label are important things for large multi-modal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog...

    2024

  19. [27]

    Playground v3: Improving text-to- image alignment with deep-fusion large language models

    Bingchen Liu, Ehsan Akhgari, Alexander Visheratin, Aleks Kamko, Linmiao Xu, Shivam Shrirao, Joao Souza, Suhail Doshi, and Daiqing Li. Playground v3: Improving text-to- image alignment with deep-fusion large language models. arXiv preprint arXiv:2409.10695, 2024. 3

    2024 arXiv

  20. [28]

    Attention correctness in neural image captioning

    Chenxi Liu, Junhua Mao, Fei Sha, and Alan Yuille. Attention correctness in neural image captioning. InProceedings of the AAAI conference on artificial intelligence, 2017. 1, 2

    2017

  21. [29]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 1, 2, 6

    2024

  22. [30]

    Improved image captioning via policy gra- dient optimization of spider

    Siqi Liu, Zhenhai Zhu, Ning Ye, Sergio Guadarrama, and Kevin Murphy. Improved image captioning via policy gra- dient optimization of spider. In Proceedings of the IEEE in- ternational conference on computer vision , pages 873–881,

  23. [31]

    Omnicaptioner: One captioner to rule them all

    Yiting Lu, Jiakang Yuan, Zhen Li, Shitian Zhao, Qi Qin, Xinyue Li, Le Zhuo, Licheng Wen, Dongyang Liu, Yuewen Cao, et al. Omnicaptioner: One captioner to rule them all. arXiv preprint arXiv:2504.07089, 2025. 2

    2025 arXiv

  24. [32]

    Self-refine: It- erative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hal- linan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: It- erative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024. 3

    2024

  25. [33]

    Docci: Descriptions of connected and contrasting images

    Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, et al. Docci: Descriptions of connected and contrasting images. arXiv preprint arXiv:2404.19753, 2024. 2, 6, 1

    2024 arXiv

  26. [34]

    Automat- ically correcting large language models: Surveying the land- scape of diverse automated correction strategies

    Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. Automat- ically correcting large language models: Surveying the land- scape of diverse automated correction strategies. Transac- tions of the Association for Computational Linguistics , ...

    2024

  27. [35]

    Bleu: a method for automatic evaluation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pages 311–318,

  28. [36]

    Connecting vision and lan- guage with localized narratives

    Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connecting vision and lan- guage with localized narratives. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23– 28, 2020, Proceedings, Part V 16, pages 647–66...

    2020

  29. [37]

    Is moral self-correction an innate capability of large language models? a mechanistic analysis to self-correction

    Zimo Qi, Guangliang Liu, Kristen Marie Johnson, and Lu Chen. Is moral self-correction an innate capability of large language models? a mechanistic analysis to self-correction. arXiv preprint arXiv:2410.20513, 2024. 2, 3

    2024

  30. [38]

    Recursive introspection: Teaching language model agents how to self-improve

    Yuxiao Qu, Tianjun Zhang, Naman Garg, and Aviral Kumar. Recursive introspection: Teaching language model agents how to self-improve. arXiv preprint arXiv:2407.18219 ,

    arXiv

  31. [39]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  32. [40]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741, 2023. 2

    2023

  33. [41]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christo- pher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024. 3

    2024

  34. [42]

    Glamm: Pixel grounding large multimodal model

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdel- rahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Patter...

    2024

  35. [43]

    Positive-augmented contrastive learning for image and video captioning evaluation

    Sara Sarto, Manuele Barraco, Marcella Cornia, Lorenzo Baraldi, and Rita Cucchiara. Positive-augmented contrastive learning for image and video captioning evaluation. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6914–6924, 2023. 3

    2023

  36. [44]

    Bridge: Bridging gaps in image captioning evalua- tion with stronger visual cues

    Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cuc- chiara. Bridge: Bridging gaps in image captioning evalua- tion with stronger visual cues. In European Conference on Computer Vision, pages 70–87. Springer, 2025. 3

    2025

  37. [45]

    Novelseek: When agent becomes the scientist–building closed-loop system from hy- pothesis to verification

    NovelSeek Team, Bo Zhang, Shiyang Feng, Xiangchao Yan, Jiakang Yuan, Zhiyin Yu, Xiaohan He, Songtao Huang, Shaowei Hou, Zheng Nie, et al. Novelseek: When agent becomes the scientist–building closed-loop system from hy- pothesis to verification. arXiv preprint arXiv:2505.16938 ,

    arXiv

  38. [46]

    A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions

    Jack Urbanek, Florian Bordes, Pietro Astolfi, Mary Williamson, Vasu Sharma, and Adriana Romero-Soriano. A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni...

    2024

  39. [47]

    Cider: Consensus-based image description evalua- tion

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evalua- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015. 3

    2015

  40. [48]

    Show and tell: A neural image caption gen- erator

    Oriol Vinyals, Alexander Toshev, Samy Bengio, and Du- mitru Erhan. Show and tell: A neural image caption gen- erator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015. 1, 2

    2015

  41. [49]

    Git: A generative image-to-text transformer for vision and language

    Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.14100, 2022. 1, 2

    2022 arXiv

  42. [50]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 1, 2, 6

    2024 arXiv

  43. [51]

    Caption anything: Interactive image description with diverse multi- modal controls

    Teng Wang, Jinrui Zhang, Junjie Fei, Hao Zheng, Yunlong Tang, Zhe Li, Mingqi Gao, and Shanshan Zhao. Caption anything: Interactive image description with diverse multi- modal controls. arXiv preprint arXiv:2305.02677, 2023. 2

    2023 arXiv

  44. [52]

    Cogvlm: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079, 2023. 2

    2023 arXiv

  45. [53]

    Gener- ating sequences by learning to self-correct

    Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Gener- ating sequences by learning to self-correct. arXiv preprint arXiv:2211.00053, 2022. 3

    2022 arXiv

  46. [54]

    Painting with words: Elevating detailed image cap- tioning with benchmark and alignment learning

    Qinghao Ye, Xianhan Zeng, Fu Li, Chunyuan Li, and Haoqi Fan. Painting with words: Elevating detailed image cap- tioning with benchmark and alignment learning. In The Thirteenth International Conference on Learning Represen- tations, 2025. 3

    2025

  47. [55]

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hocken- maier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descrip- tions. Transactions of the Association for Computational Linguistics, 2:67–78, 2014. 2

    2014

  48. [56]

    Dolphin: Moving towards closed-loop auto- research through thinking, practice, and feedback

    Jiakang Yuan, Xiangchao Yan, Shiyang Feng, Bo Zhang, Tao Chen, Botian Shi, Wanli Ouyang, Yu Qiao, Lei Bai, and Bowen Zhou. Dolphin: Moving towards closed-loop auto- research through thinking, practice, and feedback. arXiv preprint arXiv:2501.03916, 2025. 3

    2025 arXiv

  49. [57]

    Llamafac- tory: Unified efficient fine-tuning of 100+ language mod- els

    Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafac- tory: Unified efficient fine-tuning of 100+ language mod- els. In Proceedings of the 62nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 3: S...

    2024

  50. [58]

    For relation evaluation, we prompt open-source language models to answer the given questions based on the candi- date captions

    Prompt Templates We follow the official instruction of each LVLM and adopt simple prompt for image captioning and self-correction. For relation evaluation, we prompt open-source language models to answer the given questions based on the candi- date captions. The Prompts used a...

  51. [59]

    We randomly select 100 images in DOCCI500 and ask 4 hu- man annotators to sort the captions provided by 4 differ- ent models, while considering both precision and recall

    Human Consistency of Proposed Metric We conducted an extra experiment to investigate how well our proposed metric aligns with human judgement. We randomly select 100 images in DOCCI500 and ask 4 hu- man annotators to sort the captions provided by 4 differ- ent models, while co...

  52. [60]

    Statistics of Captions We have made some analyses on different datasets includ- ing RefinedCaps, DOCCI, DCI and Localized Narratives in Tab. 5. As shown in the table, captions in our proposed dataset are relatively long and have more densely packed descriptions about objects, ...

  53. [61]

    Same-Domain

    Additional Experiments 11.1. Results of Using Public Dataset for Training We also use the training set of DOCCI [33] which con- sists of 9.7K image-caption pairs as the training set for su- pervised fine-tuning and self-correction training of Qwen2- VL. Metrics for both the in...

  54. [62]

    Visualization Examples 12.1. Annotated examples from RefinedCaps To better illustrate the annotation process and the quality of the RefinedCaps dataset, we present sample cases with cap- tions before and after human refinement in Fig. 7. These examples show that human annotato...

  55. [63]

    Because the training process includ- ing generating annotations for two rounds, the training time is relatively long

    Limitations and Future Work The proposed framework requires ground-truth captions in the whole training process, adding to the data collection and annotation cost. Because the training process includ- ing generating annotations for two rounds, the training time is relatively l...

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.