{"paper":{"title":"CLIPScore: A Reference-free Evaluation Metric for Image Captioning","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"CLIP embeddings can score how well a generated caption matches its image without any human reference captions and match human judgments better than metrics that require them.","cross_cats":["cs.CL"],"primary_cat":"cs.CV","authors_text":"Ari Holtzman, Jack Hessel, Maxwell Forbes, Ronan Le Bras, Yejin Choi","submitted_at":"2021-04-18T05:00:29Z","abstract_excerpt":"Image captioning has conventionally relied on reference-based automatic evaluations, where machine captions are compared against captions written by humans. This is in contrast to the reference-free manner in which humans assess caption quality.\n  In this paper, we report the surprising empirical finding that CLIP (Radford et al., 2021), a cross-modal model pretrained on 400M image+caption pairs from the web, can be used for robust automatic evaluation of image captioning without the need for references. Experiments spanning several corpora demonstrate that our new reference-free metric, CLIPS"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"CLIPScore achieves the highest correlation with human judgements, outperforming existing reference-based metrics like CIDEr and SPICE.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That CLIP's representations pretrained on web data provide a robust, general signal of caption quality that transfers across domains without needing task-specific adaptation or references.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"CLIPScore uses a web-pretrained CLIP model to evaluate image captions without references and achieves higher human correlation than CIDEr or SPICE.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"CLIP embeddings can score how well a generated caption matches its image without any human reference captions and match human judgments better than metrics that require them.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"25c8b47e90ba3b8006754fbdb5d4d6377cf057ff3a96bb095bfb2f4cce773d21"},"source":{"id":"2104.08718","kind":"arxiv","version":3},"verdict":{"id":"e9258c23-c1fd-418b-84cf-9cb4b2aedcaf","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-12T22:19:02.195685Z","strongest_claim":"CLIPScore achieves the highest correlation with human judgements, outperforming existing reference-based metrics like CIDEr and SPICE.","one_line_summary":"CLIPScore uses a web-pretrained CLIP model to evaluate image captions without references and achieves higher human correlation than CIDEr or SPICE.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That CLIP's representations pretrained on web data provide a robust, general signal of caption quality that transfers across domains without needing task-specific adaptation or references.","pith_extraction_headline":"CLIP embeddings can score how well a generated caption matches its image without any human reference captions and match human judgments better than metrics that require them."},"integrity":{"clean":true,"summary":{"advisory":0,"critical":0,"by_detector":{},"informational":0},"endpoint":"/pith/2104.08718/integrity.json","findings":[],"available":true,"detectors_run":[],"snapshot_sha256":"c28c3603d3b5d939e8dc4c7e95fa8dfce3d595e45f758748cecf8e644a296938"},"references":{"count":70,"sample":[{"doi":"","year":2015,"title":"From Images to Sentences through Scene Description Graphs using Commonsense Reasoning and Knowledge","work_id":"dd4c908a-9217-463d-852e-1034998769f6","ref_index":1,"cited_arxiv_id":"1511.03292","is_internal_anchor":false},{"doi":"","year":2021,"title":"Evaluating clip: towards characterization of broader capabilities and downstream implications.arXiv preprint arXiv:2108.02818","work_id":"7161697e-26f5-4f20-b765-eaa45dfbbc04","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2016,"title":"Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic propositional image caption evaluation. In ECCV. Springer","work_id":"0ea2ce8e-367d-446e-a1e5-00dc7c7a43a8","ref_index":3,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2019,"title":"Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. TACL, 7:597--610","work_id":"04b2b247-a772-418c-b7dc-0ec31f152fff","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2005,"title":"Satanjeev Banerjee and Alon Lavie. 2005. METEOR: an automatic metric for mt evaluation with improved correlation with human judgments. In ACL workshop on Evaluation Measures for MT and Summarization","work_id":"41abb198-7a92-44c0-80ee-e7010f8bb6e1","ref_index":5,"cited_arxiv_id":"","is_internal_anchor":false}],"resolved_work":70,"snapshot_sha256":"8cb28f3cb0c8c4cbcfa45256280a5950952845475368a3c2946e74df34d751e5","internal_anchors":1},"formal_canon":{"evidence_count":2,"snapshot_sha256":"e70a7474b2ab98fc43167f394859d7b551a6f4479c663187747cd01fe74d0fad"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"}