Pith. sign in

REVIEW 5 major objections 8 minor 29 references

OPCap:Object-aware Prompting Captioning

T0 review · 5 major / 8 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that feeding a captioning decoder a list of detected objects and their predicted attributes reduces object hallucination in generated captions.

desk verdict A plausible but thinly evidenced extension of object-conditioned captioning; the CHAIR gains are partly by construction, and the quality claim fails on the strongest backbone. read the letter →

arxiv 2412.00095 v2 pith:KFST5HWO submitted 2024-11-27 cs.CV

Feiyang Huang This is my paper · ORCID
classification cs.CV
keywords imagecaptioningobjecthallucinationdetectionattributepredictiontarget-awarepromptingtransformerdecoderCOCOnocaps
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that image captioning models invent objects because the decoder is never told which objects are actually present. OPCap adds a target-aware prompting stage: an object detector returns labels and bounding boxes, the boxes are cropped, and an attribute predictor labels each crop with properties such as color, shape, and state. Those labels and attributes are written as delimited token sequences and fed into a Transformer decoder together with the image features. The paper reports lower CHAIRs and CHAIRi hallucination scores when OPCap is added to EffNet, CLIP-base, and CLIP-large captioners, and higher traditional caption metrics for the weaker EffNet encoder. The broader claim is that explicit object-level grounding can reduce hallucination without extra pretrained language models or large-scale data.

What carries the argument

The load-bearing mechanism is the object-aware prompt: a sequence of special tokens ([OBJ] label, [ATTR] attribute) produced by a YOLOS-tiny detector and a CLIP-plus-MLP attribute predictor trained on the VAW dataset. These tokens are embedded into the same d_model space as the image features and prepended to them, giving the Transformer decoder an explicit, current inventory of objects and their properties. The mechanism supplies the decoder with information the image encoder may have compressed or lost, so caption generation is conditioned on a direct statement of what is present.

What would settle it

Take a set of COCO images whose ground-truth captions mention objects that YOLOS-tiny is known to miss, generate captions with and without OPCap, and count whether the missed objects vanish from captions and whether new false objects appear. As a cleaner test, erase all detector outputs on a validation batch (feeding OPCap an empty object list): if hallucination stays at the base model's level, the detector's object list is the carrier of the effect.

Watch

Extended reading notes

Core claim

The central claim is that a target-aware prompting strategy, OPCap, reduces object hallucination in image captioning by injecting detected object labels and their predicted attributes into the decoding process. Given an image, a pretrained object detector returns labels and boxes; the boxes crop regions that an attribute predictor labels with properties such as color, shape, and state. These are formatted as [OBJ] ... [ATTR] ... token sequences, embedded, and concatenated with the image encoder's features to form the Transformer decoder's cross-attention memory. Training freezes the image encoder and requires only the language-model captioning loss. The paper reports lower CHAIRs and CHAIRi scores after adding OPCap across the EffNet, CLIP-base, and CLIP-large decoders, and higher CIDEr and SPICE for the weaker EffNet encoder; it interprets the overall pattern as evidence that explicit object grounding without extra language models makes captions describe what is actually in the image.

Load-bearing premise

If the object detector misses an object or misclassifies it, OPCap cannot correct the error: the omitted object will not appear in the caption, and a wrong label can inject an attribute from the wrong region.

Editorial extensions

If this is right

  • Any existing encoder-decoder captioning model can be retrofitted with OPCap by adding the detector and attribute predictor, freezing the encoder, and training only the decoder with the standard language-model loss.
  • Resource-constrained deployments can attack hallucination without loading a large language model, because the added components are lightweight (YOLOS-tiny and a small CLIP-based classifier).
  • The paper observes that adding OPCap to CLIP-large lowers CIDEr while improving CHAIR, so evaluations aimed at grounded captions should report hallucination metrics alongside traditional caption metrics.
  • On nocaps out-of-domain images, the weak EffNet encoder improves on all reported metrics with OPCap, suggesting explicit object grounding helps describe unseen categories when the base encoder is limited.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the benefit concentrates on weaker image encoders, so a natural extension is to make the prompt adaptive, injecting object tokens only when the encoder's own features are uncertain rather than always.
  • Editorial inference: because the attribute predictor is trained on the VAW vocabulary, its label set may not cover attributes that captions actually need; comparing OPCap under different attribute vocabularies would test whether the vocabulary is the bottleneck.
  • Editorial inference: the discrete token prompt is a lossy bottleneck; a dense region-feature conditioning variant might keep the hallucination reduction while avoiding the traditional-metric regression seen on CLIP-large.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 8 minor

Summary. OPCap is a prompting strategy for image captioning that augments a frozen image encoder and a Transformer decoder with object-level information: a YOLOS-tiny detector supplies object labels and boxes, a CLIP-based attribute predictor supplies per-object attributes from cropped regions, and the resulting [OBJ]/[ATTR] token sequence is concatenated with image features as the decoder's key/value input. The authors claim that this reduces object hallucination and significantly improves caption quality on COCO and nocaps, that the method applies to any image captioning model, and that it requires no additional pretrained models or large-scale datasets. The experiments compare EffNet-, CLIP-base-, and CLIP-large-based captioners with and without the 'OP' intervention on CIDEr, BLEU-4, METEOR, ROUGE-L, SPICE, CHAIRs, CHAIRi, and a newly proposed CLIP Vote protocol. The clearest positive result is for the weak EffNet encoder, where all quality and hallucination metrics improve; the results for the CLIP-based models are mixed or negative.

Significance. If the claims survive revision, the contribution is useful: a lightweight, interpretable intervention (detected objects plus attributes as decoder prompts) that produces large gains for a weak encoder (EffNet: CIDEr 55.95 to 68.09, CHAIRs 40.84 to 32.78) and consistent CHAIR improvements across all tested models. The method is deliberately simple, needs no language model, and the failure mode on CLIP encoders is acknowledged in Section 4.3, which is honest. The paper is not publishable in its present form, however: the abstract's quality claim is contradicted by Table 2b, the claimed ablations are absent, the hallucination gains are confounded by vocabulary sharing between YOLOS-tiny and the CHAIR metric, and the implementation details are insufficient for reproducibility. These problems are fixable in revision, so a major revision is the appropriate outcome.

major comments (5)
  1. [Abstract / Section 4.3 / Table 2b] The abstract's claim that OPCap 'significantly improves the quality of generated captions' is contradicted by the paper's own Table 2b: for CLIP-large+OP, BLEU-4 (36.64 to 35.96), METEOR (28.21 to 27.47), ROUGE-L (56.93 to 56.52), CIDEr (117.39 to 115.59), and SPICE (21.12 to 20.80) all decrease, and Table 2a shows large out-of-domain drops on nocaps (CIDEr 57.14 to 47.02). Section 4.3 itself states that 'models based on CLIP show the opposite trend,' so the unqualified statements in the abstract and conclusion should be replaced by a model-regime-specific claim, with the strong-encoder degradation discussed explicitly as a limitation.
  2. [Section 3.1 / Table 2b / Section 4.2 / Table 1] The hallucination-mitigation evidence is confounded and partially contradicted. CHAIR (Ref [13]) penalizes object words that are absent from the COCO ground-truth object list, and the detector YOLOS-tiny is trained on the same COCO 80-class vocabulary; injecting detector labels into the decoder therefore moves CHAIR in the expected direction partly by construction, even when the injected labels are wrong for the image. Table 1's CLIP-large+OP captions contain new artifacts ('a statue of a statue on a counter', 'a plane flying over an airplane in a desert') that are precisely the hallucination type the paper claims to reduce, and Section 4.2 itself concedes that the strategy is partially dependent on detector performance. A concrete test is needed, for example evaluating hallucination against a vocabulary disjoint from the detector's training classes, or analyzing CHAIR gains conditioned on detector correctness, to show that the gains are not an artifact of vocabulary alignment.
  3. [Section 1 / Section 5] The introduction states 'We conducted experiments and ablation studies,' and the conclusion attributes the gains to 'the target-aware strategy and the fusion module,' but no ablation experiment appears anywhere in Section 4. In particular, there is no experiment isolating the attribute predictor (object labels alone versus labels plus attributes) or the caption token dropout, so the paper's central design claim, that attribute prediction contributes to the reported gains, is not demonstrated.
  4. [Section 3.1 / Section 4.1 / Section 4.3] The method is not reproducible as specified: the top-k in Section 3.1, the sigmoid threshold for attribute selection, the caption token dropout rate, and the object detector confidence threshold are never given values; no training hyperparameters (learning rate, epochs, batch size, optimizer, decoder initialization) are reported; the COCO split description in Section 4.1 ('118,287 images... 5,000 images set aside for validation and testing') does not identify the split used for the captioning evaluation; and the nocaps protocol (validation set versus evaluation server) is unspecified. In addition, no error bars or significance tests accompany the comparisons, several of which are small, such as the CLIP-large CHAIRi change from 4.47 to 4.27.
  5. [Table 3 / Section 4.3] Table 3 and its surrounding text are internally inconsistent. The text claims that 'models incorporating the OPCap method consistently received more votes,' but CLIP-large+OP (1093) receives fewer votes than CLIP-large (1250); the sentence claiming that 'the base version of CLIP received the highest number of votes' is also contradicted by the table, in which CLIP-base+OP (1391) is highest. Moreover, since CLIP is used both as the scoring model and inside the OPCap attribute predictor, and two of the baselines are themselves CLIP-based captioners, this no-reference metric is biased toward the method's own components and does not independently corroborate the quality claim.
minor comments (8)
  1. [Section 2.2 / Section 4.1] The heading 'Hallucnation in models' in Section 2.2 and the heading 'Evualation Metrics' in Section 4.1 contain typos.
  2. [Table 2b] The rows for AoA and UpDown appear misaligned: the values 27.4 and 26.9 sit in the ROUGE-L column but are in the plausible range of METEOR (compare 27.09 for ViTGPT2 and 26.72 for CLIP-base), which would leave ROUGE-L unreported for these baselines.
  3. [Section 4.3] The passage 'This aligns with the findings in []' contains an empty citation that should be filled or removed.
  4. [References] Reference hygiene is incomplete: Refs [2] and [25], Refs [4] and [26], and Refs [19] and [27] are the same papers listed twice, and Refs [1] and [13] are also the same work with different years.
  5. [Section 1 / Section 3.1] The contribution bullet in Section 1 claims the method works 'without requiring additional pre-trained models,' which is contradicted by Section 3.1's use of the pre-trained YOLOS-tiny detector and a pre-trained CLIP encoder inside the attribute predictor.
  6. [Equation (4) / Section 3.1] Equation (4) states that the object and attribute tokens map to 2 x (o + k) feature vectors, but the Section 3.1 example with k = 2 attributes per object produces o object tokens plus 2o attribute tokens plus special tokens, which does not equal 2 x (o + k); the token accounting and the role of the [OBJ] and [ATTR] special tokens need to be clarified.
  7. [Section 2.3] The claim that 'both the object detector and attribute extractor are trained or fine-tuned on the COCO dataset' conflicts with Section 3.1, which states that the attribute predictor is trained on the VAW dataset [18].
  8. [Section 3.2 / Section 4.3] The related-work discussion implies that OPCap 'can be applied to any image captioning model,' but only EffNet and CLIP-based variants are evaluated with +OP, and Section 3.2 restricts the decoder to a Transformer because 'LSTM has difficulty handling the joint features,' so the universality claim is neither tested nor supported by the design.

Circularity Check

1 steps flagged · score 4.0 of 10

CHAIR hallucination gains are partly by construction because the detector prompt vocabulary and CHAIR's scoring vocabulary are both the COCO object set.

  1. self definitional [Sec. 3.1 (YOLOS-tiny object detector), Sec. 4.1 (CHAIR metrics), Sec. 2.3 (COCO-trained detector/attribute predictor)]
    "we employ yolos-tiny [17] as the object detection model ... both the object detector and attribute extractor are trained or fine-tuned on the COCO dataset ... to assess improvements in mitigating hallucination, we employ the hallucination metrics CHAIRs and CHAIRi [13], which evaluate hallucinations at the sentence and object levels, respectively."

    CHAIRs/CHAIRi determine object hallucination by checking whether generated object words appear in the image's COCO ground-truth objects. OPCap feeds the decoder object labels produced by YOLOS-tiny, a detector trained on the COCO object vocabulary, so the prompt vocabulary and the metric's 'present object' vocabulary are the same COCO category set by construction. The model is effectively told which COCO labels to emit, and the metric rewards emitting COCO labels that are in the ground truth. The reported CHAIR gains (e.g., CLIP-large+OP: CHAIRs 6.34->5.80, CHAIRi 4.47->4.27) are therefore partly forced by the alignment between the detector's output space and the metric's scoring space, not independent evidence of reduced hallucination.

full rationale

The paper is an empirical method paper, not a formal derivation, and most components are externally grounded: the baselines are published models, and the detector/attribute modules are pretrained external tools. The one load-bearing circular coupling is the hallucination evaluation. OPCap prompts the decoder with labels from YOLOS-tiny, which is trained on the COCO vocabulary; CHAIRs/CHAIRi score hallucination by whether generated object words are absent from the image's COCO ground-truth objects. Because both use the same COCO object vocabulary, the reported CHAIR improvements are partly by construction rather than independent evidence of grounding. The paper itself concedes in Sec. 4.2 that OPCap's effectiveness 'is partially dependent on the performance of the object detector', and a sample shows a missed crowd leading to omitted objects. The quality metrics are not circular: they compare to reference captions and can go either way, and indeed CLIP-large+OP degrades CIDEr and SPICE in Table 2b. The empty citation '[]' in Sec. 4.3 is a missing reference but is not load-bearing. No self-citation chain is involved. The circularity is therefore partial and isolated to the central hallucination claim, warranting a moderate score rather than a high one.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the accuracy of the object detector and attribute predictor, on a frozen image encoder, and on the COCO evaluation protocol. The paper provides no artifacts to audit these assumptions; it also leaves an empty citation in Section 4.3.

free parameters (4)
  • top-k attributes per object = k=2 shown in Figure 2; no value or ablation reported in experiments
    The number of attributes fed into the decoder is chosen by hand and directly affects prompt length and content; no study of k is provided.
  • attribute prediction sigmoid threshold = not stated
    The paper says 'select the top-k attributes after applying sigmoid activation' but does not state the threshold for declaring an attribute present.
  • caption token dropout rate = not stated
    Random dropout is applied to caption tokens before the causal attention layer (Section 3.2) but the rate is unspecified.
  • object detector confidence threshold = not stated
    YOLOS-tiny is used without reporting the confidence cutoff, which determines the set of objects and boxes.
assumptions (4)
  • domain assumption YOLOS-tiny, pretrained on COCO, produces accurate and complete object labels and bounding boxes.
    The method relies on detected objects as ground truth for caption grounding; the authors acknowledge in Section 4.2 that detection failures cause caption omissions.
  • domain assumption The attribute predictor, a CLIP encoder plus MLP trained on VAW, predicts useful attributes for cropped object regions.
    The attribute branch is a new module whose training details and accuracy are not reported; the caption quality depends on its predictions being correct.
  • domain assumption The COCO 2017 data split used for evaluation follows a standard captioning protocol.
    The paper says 5,000 images are 'set aside for validation and testing' without specifying the test split; baseline numbers suggest Karpathy split, but this is not stated.
  • standard math CHAIR is a valid and sufficient measure of object hallucination.
    The paper uses CHAIRs and CHAIRi as the sole hallucination metrics; this is a standard choice, but the intervention's mechanism is directly aligned with the metric.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OPCap:Object-aware Prompting Captioning." pith.science (2026). https://pith.science/paper/KFST5HWO

@misc{pith2026241200095,
  author       = {Pith},
  title        = {Pith review of: OPCap:Object-aware Prompting Captioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KFST5HWO}},
  note         = {Machine review of arXiv:2412.00095}
}
read the original abstract

In the field of image captioning, the phenomenon where missing or nonexistent objects are used to explain an image is referred to as object bias (or hallucination). To mitigate this issue, we propose a target-aware prompting strategy. This method first extracts object labels and their spatial information from the image using an object detector. Then, an attribute predictor further refines the semantic features of the objects. These refined features are subsequently integrated and fed into the decoder, enhancing the model's understanding of the image context. Experimental results on the COCO and nocaps datasets demonstrate that OPCap effectively mitigates hallucination and significantly improves the quality of generated captions.

Figures

Figures reproduced from arXiv: 2412.00095 by the authors.

Figure 1
Figure 1. Examples of hallucination objects: cat, tie With hallucination: a close up of a person wearing a tie. Without hallucination: a close up of a cat wear￾ing a tie objects: bicycle With hallucination: a group of people sitting on a street. Without hallucination: a bicycle parked on a bench in front of a build￾ing. prominent being object hallucination [1]. As shown in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. OPCap Architecture: The architecture consists of three modules, including the image encoder, object detector [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 25 canonical work pages

  1. [15]

    Let there be a clock on the beach: Reducing object hallucination in image cap- tioning

    Ali Furkan Biten, Llu ´ıs G ´omez i Bigorda, and Di- mosthenis Karatzas. Let there be a clock on the beach: Reducing object hallucination in image cap- tioning. 2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , pages 2473–2482, 2021. 3

    work page 2022

  2. [13]

    Object hallucination in image captioning

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. arXiv preprint arXiv:1809.02156, 2018. 3, 5

    arXiv 2018

  3. [1]

    Object hallucination in image captioning, 2019

    Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning, 2019. 1

    2019

  4. [2]

    Bottom-up and top-down attention for image captioning and visual question answering

    Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018. 2

    work page 2018

  5. [3]

    Look back and predict forward in image cap- tioning

    Yu Qin, Jiajun Du, Yonghua Zhang, and Hongtao Lu. Look back and predict forward in image cap- tioning. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), June 2019. 2

    work page 2019

  6. [4]

    Attention on attention for image cap- tioning

    Lun Huang, Wenmin Wang, Jie Chen, and Xiao- Yong Wei. Attention on attention for image cap- tioning. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), Oc- tober 2019. 2

    work page 2019

  7. [5]

    Ron Mokady, Amir Hertz, and Amit H. Bermano. Clipcap: Clip prefix for image captioning, 2021. 2

    work page 2021

  8. [6]

    Show and tell: A neural im- age caption generator

    Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. Show and tell: A neural im- age caption generator. In CVPR, pages 3156–3164. IEEE Computer Society, 2015. 2

    work page 2015

Show all 29 references
  1. [7]

    Long short-term memory

    Sepp Hochreiter and J ¨urgen Schmidhuber. Long short-term memory. Neural Comput. , 9(8):1735–1780, November 1997. 2

    1997

  2. [8]

    Show, attend and tell: Neural image caption generation with visual at- tention

    Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual at- tention. In Francis Bach and David Blei, editors, Proceedings of the 32nd International C...

    2015

  3. [9]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Sys- tems, NIPS’17, page 6000–6010, Red...

    2017

  4. [10]

    Adaptive path selection for dynamic image captioning

    Tiantao Xian, Zhixin Li, Zhenjun Tang, and Huifang Ma. Adaptive path selection for dynamic image captioning. IEEE Transactions on Circuits and Systems for Video Technology, 32:5762–5775,

  5. [11]

    Vision-enhanced and consensus-aware transformer for image captioning

    Shan Cao, Gaoyun An, Zhenxing Zheng, and Zhiy- ong Wang. Vision-enhanced and consensus-aware transformer for image captioning. IEEE Transac- tions on Circuits and Systems for Video Technology, 32:7005–7018, 2022. 2

    2022

  6. [12]

    Remote sensing image caption generation via transformer and reinforcement learning

    Xiangqing Shen, Bing Liu, Yong Zhou, and Jiaqi Zhao. Remote sensing image caption generation via transformer and reinforcement learning. Multi- media Tools and Applications , 79:26661 – 26682,

  7. [14]

    Self-critical n-step train- ing for image captioning

    Junlong Gao, Shiqi Wang, Shanshe Wang, Si- wei Ma, and Wen Gao. Self-critical n-step train- ing for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019. 3

    2019

  8. [16]

    Mitigating fine-grained hallucina- tion by fine-tuning large vision-language models with caption rewrites

    Lei Wang, Jiabang He, Shenshen Li, Ning Liu, and Ee-Peng Lim. Mitigating fine-grained hallucina- tion by fine-tuning large vision-language models with caption rewrites. In International Conference on Multimedia Modeling , pages 32–45. Springer,

  9. [17]

    You only look at one sequence: Re- thinking transformer in vision through object de- tection

    Yuxin Fang, Bencheng Liao, Xinggang Wang, Jiemin Fang, Jiyang Qi, Rui Wu, Jianwei Niu, and Wenyu Liu. You only look at one sequence: Re- thinking transformer in vision through object de- tection. CoRR, abs/2106.00666, 2021. 4

    2021 arXiv

  10. [18]

    Learning to predict visual attributes in the wild

    Khoi Pham, Kushal Kafle, Zhe Lin, Zhihong Ding, Scott Cohen, Quan Tran, and Abhinav Shrivastava. Learning to predict visual attributes in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13018–13028, June 2021. 4

    2021

  11. [19]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and ...

    2021

  12. [20]

    Lawrence Zit- nick, and Piotr Doll ´ar

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zit- nick, and Piotr Doll ´ar. Microsoft coco: Common objects in context, 2015. 5

    2015

  13. [21]

    Cider: Consensus-based image de- scription evaluation

    Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image de- scription evaluation. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 4566–4575, 2015. 5

    2015

  14. [22]

    Bleu: a method for automatic evalu- ation of machine translation

    Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th annual meeting of the Association for Compu- tational Linguistics, pages 311–318, 2002. 5

    2002

  15. [23]

    Meteor: An automatic metric for mt evaluation with improved correlation with human judgments

    Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceed- ings of the acl workshop on intrinsic and extrin- sic evaluation measures for machine translation and/or summarization, pages 65–72, 2005. 5

    2005

  16. [24]

    Spice: Semantic propositional image caption evaluation

    Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In Computer Vision– ECCV 2016: 14th European Conference, Ams- terdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pages 382–398. Springer,

    2016

  17. [25]

    Bottom-up and top-down attention for image captioning and visual question answering

    Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6...

  18. [26]

    Attention on attention for image cap- tioning

    Lun Huang, Wenmin Wang, Jie Chen, and Xiao- Yong Wei. Attention on attention for image cap- tioning. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, pages 4634– 4643, 2019. 5

    2019

  19. [27]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. 5

    2021

  20. [28]

    no- caps: novel object captioning at scale

    Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. no- caps: novel object captioning at scale. In Proceed- ings of the IEEE International Conference on Com- puter Vision, pages 8948–8957, 2019. 6

    2019

  21. [29]

    Fusecap: Leveraging large language models for enriched fused image captions

    Noam Rotstein, David Bensa¨ıd, Shaked Brody, Roy Ganz, and Ron Kimmel. Fusecap: Leveraging large language models for enriched fused image captions. In Proceedings of the IEEE/CVF Win- ter Conference on Applications of Computer Vision (WACV), pages 5689–5700, January 2024. 7

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.