Pith. sign in

REVIEW 3 major objections 4 minor 51 references

9.6 CIDEr gain from scoring image-text fit at three checkpoints

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 15:52 UTC pith:HEP7P34O

load-bearing objection A carefully controlled empirical paper that plausibly breaks the strict-ZIC plateau with multi-checkpoint alignment scoring, though the COCO headline gain is partly conditional on COCO-tuned components that the paper honestly flags. the 3 major comments →

arxiv 2607.28986 v1 pith:HEP7P34O submitted 2026-07-31 cs.CV cs.AIcs.CLcs.MA

Adjudicated Captioning: Multi-Agent Alignment Scoring and Consensus-Distilled Beam Arbitration for Strict Zero-Shot Image Captioning

Duy Tran Thanh , Thien-Phuc Doan , Long Nguyen-Vu , Ngo Tan Vu Khanh This is my paper
classification cs.CV cs.AIcs.CLcs.MA
keywords zero-shot image captioningretrieval-augmented generationbeam rerankingmulti-checkpoint alignment scoringBorda-consensus distillationcross-attention verificationself-supervised learninghallucination reduction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper says the zero-shot captioning field has stalled not because of a language-model ceiling but because everyone scores image-text agreement exactly once, at retrieval, and never checks again. It builds an inference-time stack on top of the existing IFCap captioner: a stronger frozen retrieval encoder, a frozen cross-attention verifier that re-ranks the top nine retrieved captions to five, and a small learned beam reranker at the output. The reranker is trained with no reference captions, by distilling a Borda-consensus ranking from three frozen scorers. The full pipeline reaches 117.6 CIDEr on COCO Karpathy versus 108.0 for the prior strict method, with +8.1 on Flickr30k and +5.7 on NoCaps, without retraining the captioner. A no-learned-components version at 115.8 shows that most of the gain is the architectural change of scoring alignment at multiple checkpoints.

Core claim

The central claim is that image-text alignment should be scored at multiple checkpoints rather than once: a stronger dual encoder at retrieval (the largest single lift, +3.6 CIDEr), a frozen cross-attention verifier between retrieval and decoding (BLIP-ITM; its ranking correlates only about 0.4 with retrieval cosine, so it supplies partly orthogonal signal), and at the output a learned arbiter over the captioner's 20-wide beam. The two learned heads—TriFuse, a 113-parameter MLP, and MemAttend, a 17.4K-parameter memory-attended transformer—are trained self-supervised by listwise cross-entropy against a Borda-consensus pseudo-label over three frozen scorers (language-model log-probability, ret

What carries the argument

The mechanism is multi-checkpoint alignment scoring: every stage re-scores image-text fit. A frozen dual encoder retrieves the top-9 corpus captions; a frozen cross-attention verifier (BLIP-ITM) re-ranks them to top-5, with the entity-filter threshold rescaled from K=5 over 9 to K=3 over 5 to keep the same admission fraction; then two self-supervised beam rerankers (TriFuse MLP and MemAttend transformer) fuse three frozen-scorer signals per beam candidate and pick the final caption. The training target is Borda-consensus distillation: the three scorers' per-image rankings are averaged into a soft pseudo-label, so no reference captions or paired supervision enter the loop.

Load-bearing premise

The headline gain rests on the assumption that the +9.6 CIDEr reflects the multi-checkpoint alignment architecture rather than mostly the verifier's COCO fine-tuning plus the COCO-style retrieval corpus; the paper's ablations mitigate this confound but do not fully remove it.

What would settle it

Run the full pipeline with the retrieval corpus swapped to a non-COCO source (e.g., Conceptual Captions or a LAION-COCO subset) while keeping the captioner and all frozen scorers fixed; if the CIDEr gain over the baseline collapses or reverses, the central architectural claim is not the driver. Likewise, an instance-mask CHAIR evaluation would test whether the hallucination reduction survives the reference-proxy assumptions.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Any strict zero-shot captioner that emits a beam can be capped with an alignment-scoring stack and improved without retraining, if the captioner's beam contains captions the frozen scorers can rank.
  • Because the training-free fixed fusion already reaches 115.8, the architectural placement of scorers—not the learned head—is the main cause of the gain.
  • The recipe transfers off-COCO (Flickr30k +8.1, NoCaps +5.7 CIDEr), so the effect is not confined to the benchmark the captioner saw.
  • Reference-proxy hallucination drops about 41% relative (CHAIR s from 13.28% to 7.84%), suggesting the reranked captions are measurably better grounded, not just more fluent.
  • The inductive and transductive protocols match within 0.1 on all metrics, so the learned rerankers generalize without any adaptation to test images.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper's own principle predicts a monotone scaling law: any future stronger frozen scorer placed at the retrieval or verification checkpoint should push CIDEr further; this is directly testable by a substrate sweep.
  • The same 'score alignment again after retrieval' pattern should transfer to other retrieval-augmented generators (open-domain QA, dialogue, code completion) that currently commit their beam under LM probability alone; the paper leaves this as future work.
  • The biggest residual confound—the verifier's COCO fine-tuning and the COCO-style retrieval corpus—is mitigated but not eliminated by the reported ablations; the paper explicitly notes that a retrieval-corpus swap to a non-COCO source remains undone, and that is the cleanest test of whether the gain is architectural or exposure-driven.
  • The Borda-consensus distillation recipe suggests a broader principle: when no ground-truth labels exist, agreement among heterogeneous frozen scorers can serve as a training signal for output-side selection; applying this to other text-only training regimes seems promising.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes Adjudicated Captioning, an inference-time pipeline over the frozen IFCap captioner that scores image–text alignment at three checkpoints: a stronger frozen OpenCLIP ViT-bigG/14 retrieval encoder, a frozen BLIP-ITM Cross-Attention Verifier re-ranking the top-9 retrievals to top-5, and an output-beam reranker (TriFuse, a 113-parameter MLP, plus MemAttend, a 17.4K-parameter transformer). The rerankers are trained by listwise distillation into a Borda-consensus pseudo-label formed from three frozen scorers (GPT-2 LM log-probability, retrieval cosine, verifier score), without paired image–caption labels or reference captions. On COCO Karpathy, the inductive protocol (heads fit on the disjoint validation beam dump) reaches CIDEr 117.6 / SPICE 21.9 versus IFCap's 108.0/20.3; a training-free fixed-α z-mix reaches 115.8, attributing +7.8 to the non-learned architectural intervention and +1.8 to the learned heads. Off-COCO, the recipe reports +8.1 CIDEr on Flickr30k and +5.7 on NoCaps overall, with benchmark-transductive head fitting. The paper includes factorial ablations (Table VII), three evaluation protocols (Table IV), a Verifier-substrate ablation (Table V), byte-exact IFCap reproduction, determinism quantification, and reference-proxy CHAIR/CLIPScore checks.

Significance. If the causal attribution holds, this is a substantial contribution: it identifies a structural explanation for the strict-ZIC plateau (single-checkpoint alignment scoring) and demonstrates a remedy with a clean numerical decomposition. The experimental discipline is a genuine strength: IFCap reproduces byte-identically (Table I); ablations isolate each component (Table VII); inductive/transductive/training-free protocols are explicitly separated (Table IV); determinism noise (~0.1 CIDEr, Sec. V-H) is quantified; and the learned rerankers are tiny and trained without paired labels. The main caveat is that the central causal claim is not fully isolated from COCO-domain exposure: the retrieval index is the COCO Karpathy training-caption corpus, the headline Verifier is COCO-finetuned, and the corpus-swap test the paper itself identifies as decisive is explicitly left undone (Sec. VI-0d). The paper is unusually transparent about these limitations, and the off-COCO numbers and Verifier-substrate swap mitigate the concern substantially, but the strength of the architectural attribution remains conditional pending the missing experiment.

major comments (3)
  1. [Sec. VI-0d; Tables V and VII] The central causal claim—that multi-checkpoint alignment scoring, not COCO-domain exposure, produces the +9.6 CIDEr gain—is not yet fully isolated. The ablation decomposition (Table VII) is measured against a retrieval index built from COCO Karpathy training captions and a COCO-finetuned BLIP-ITM Verifier. The paper itself states in Sec. VI-0d that a retrieval-corpus swap to CC3M or LAION-COCO is the 'most direct test' and 'remains to be performed.' The Flickr30k-finetuned Verifier swap (Table V: 114.0, +6.0) and the off-COCO transfer (Table VI) mitigate but do not close the gap: the residual +3.6 CIDEr in Table V is explicitly COCO-Verifier-conditional, and the cross-domain rows use benchmark-transductive head fitting, so the frozen-stage-only cross-domain contribution is not decomposed. Because the entire novelty claim rests on the gain being caused by the architecture rather than by t
  2. [Table VI vs. Sec. VI-0d; Abstract] There is an internal inconsistency about what was actually run for the cross-domain results, and the abstract overstates them. The Table VI caption says all rows are 'benchmark-transductive ... not fully frozen cross-domain transfer,' and Sec. V-F confirms the rerankers are re-fit per benchmark on that benchmark's beam-20 dump. Yet Sec. VI-0d claims 'Table VI reports cross-benchmark inductive transfer on Flickr30k,' which contradicts the caption, and the supporting Flickr30k cross-benchmark-inductive number is not shown in any table. Meanwhile the abstract reports '+8.1 on Flickr30k Karpathy and +5.7 on NoCaps overall' as 'the same recipe transfers off-COCO' without the transductive-fitting qualification. Readers cannot tell from the abstract or the table alone how much of the off-COCO gain comes from the frozen multi-checkpoint stages versus benchmark-adapted heads. This should be corre
  3. [Sec. III-C/III-D; Eqs. (5) and (8)-(10)] The Borda-consensus pseudo-label q_y(I) in Eq. (8) is a deterministic function of ranks under the same three scorers whose z-scored values form the head input f_y(I) in Eq. (5). The learned heads are therefore trained to reproduce a softmax over rankings of their own input signals; this is self-distillation, not supervision from an independent source. The +1.8 CIDEr contribution of the learned stage (Table VII, rows 7→8) is well controlled—same beam dump, byte-identical captioner outputs—and the inductive/transductive agreement (Table IV) argues against image-level overfitting, but the paper should state explicitly that the consensus target carries no information beyond the three frozen-scorer signals. The gain comes from learned non-linear/contextual aggregation, not from a new signal source, and this bounds the interpretation of the 'first learned Beam Rerankers' claim. This does not u
minor comments (4)
  1. [Sec. III-B vs. Sec. III-C, Eq. (8)] The symbol K is used for two different quantities: the entity-filter admission threshold (K=3, Sec. III-B) and the beam width (K=20 in Eq. (8) and Sec. III-C's 'K-wide beam'). This collision makes the method harder to read; a distinct symbol for beam width (e.g., B) is recommended.
  2. [Sec. V-H, Sec. VI-0d] The ~0.1 CIDEr determinism floor is useful, but both the +7.8 and +1.8 headline deltas are single-seed values (seed 42). The large deltas are far above the floor, but a few seeds for the +1.8 learned-head delta would make the sub-claim tighter, as the paper itself acknowledges in Sec. VI-0d.
  3. [Table III, Sec. VI-0d] The hallucination reduction is measured with the reference-proxy CHAIR variant, and the abstract correctly says 'reference-proxy.' Since an object present in the image but absent from all five references is counted as hallucinated, and an instance-mask run is only 'queued,' the conclusion in Sec. VII should carry the same proxy qualifier when citing the −41% reduction.
  4. [Table II, Qwen2.5-VL-3B row] The Qwen2.5-VL-3B row is a non-regime-comparable MLLM baseline fed the paper's retrievals as in-context exemplars; its unusual pattern (low CIDEr 77.1 but high SPICE 23.3) deserves a sentence of interpretation so readers know what to take from it, rather than a bare number in the context block.

Circularity Check

0 steps flagged

No significant circularity: the learned heads are trained by consensus distillation over the same scorers used as features, but the headline result is measured on held-out human references, so the derivation is not constructionally circular.

full rationale

After walking the derivation chain, I find no constructional circularity. The learned TriFuse/MemAttend heads are fit to Borda-consensus pseudo-labels computed from the same three frozen scorers whose z-scores are their input features (Eqs. 5, 8-10). That is self-distillation rather than independent supervision, and Sec. V-Hc states the pseudo-label is a deterministic function of the frozen-scorer signals. But the headline claim is a CIDEr/SPICE measurement against held-out COCO references; no reference caption or test CIDEr enters the training objective (Sec. IV-D, V-Hc), and hyperparameters (α, β, K, h, layers) are chosen by pseudo-label loss on validation, not by reference CIDEr (Sec. IV-E). The +9.6 CIDEr decomposition is supported by a genuine factorial ablation (Table VII), and the fixed-fusion row isolates the architectural intervention from the learned heads. The acknowledged limitations in Sec. VI-0d — COCO retrieval-corpus style overlap, COCO-finetuned Verifier, and the missing CC3M/LAION-COCO corpus swap — are domain-exposure confounds, not circular reductions; the paper discloses them and reports off-COCO transfer. There are no load-bearing self-citations and no imported uniqueness theorem. The self-distillation point lowers the interpretability of the learned-head contribution but does not make the derivation equivalent to its inputs.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The paper has no invented entities. Its claims rest on the validity of self-supervised consensus distillation and on the operational regime definition that permits frozen paired-pretrained scorers; the latter is explicitly disclosed in Sec. II-D.

free parameters (5)
  • alpha (fixed-fusion weight) = 0.48
    Selected by Borda-consensus pseudo-label loss on a held-out beam dump, not by reference CIDEr; used in the training-free fixed-fusion baseline.
  • beta (ensemble weight) = 0.75
    Selected by Borda-consensus pseudo-label loss on validation; controls the convex combination of TriFuse and MemAttend scores.
  • K (entity-filter threshold) = 3
    Chosen algebraically to approximate the 5/9 admission fraction after the Verifier shrinks the retrieval pool from 9 to 5; not swept.
  • Verifier top-k = 5
    The Verifier re-ranks top-9 to top-5; this pool size is a design choice and a free parameter.
  • TriFuse/MemAttend hyperparameters (h, d_model, layers, epochs, lr) = h=8, d=32, layers=2, epochs=100/200, lr=1e-2/3e-3
    Selected by Borda-consensus pseudo-label loss on validation, not by reference CIDEr; they affect the learned reranker contribution.
axioms (5)
  • domain assumption The three frozen scorers (GPT-2 LM logprob, CLIP cosine, BLIP-ITM matching score) provide alignment signals that correlate with human caption quality.
    The whole reranking scheme relies on these signals; if they were uncorrelated with human judgment, Borda consensus would not improve CIDEr. The paper's results give empirical support, but it is an initial premise.
  • domain assumption The operational definition of strict ZIC permits using frozen paired-pretrained scorers at inference.
    Sec. II-D explicitly states this definition; the reader must accept it to interpret the claim as 'strict zero-shot'.
  • domain assumption The IFCap released checkpoint is a valid, unmodified Captioner whose beam contains candidates that can be improved by reranking.
    The pipeline keeps IFCap frozen; the reranking gain assumes the beam's top-1 by LM is not always the best candidate.
  • domain assumption The Borda-consensus pseudo-label is a valid training target for the rerankers.
    The learned heads are trained to imitate the consensus; this assumes consensus among the three scorers identifies better captions.
  • domain assumption BLIP-ITM's cross-attention score provides partly orthogonal information to CLIP cosine.
    Measured on the data (Spearman rho ~0.4), but the architecture assumes this orthogonality is useful for reranking.

pith-pipeline@v1.3.0-daily-deepseek · 26260 in / 16106 out tokens · 151820 ms · 2026-08-03T15:52:17.760478+00:00 · methodology

0 comments
read the original abstract

Zero-shot image captioning (ZIC) describes images without paired image-caption supervision during captioner training, relying on text-only corpora and frozen pretrained image-text scorers. Existing retrieval-augmented methods score image-text alignment once, at retrieval, then commit the captioner's autoregressive beam under language-model probability alone, leaving the decoder without further visual grounding feedback. Progress has stalled, with no method improving on the strict-regime best since 2024. We propose Adjudicated Captioning, an inference-time multi-agent framework that restores grounding feedback at multiple checkpoints over an unchanged IFCap captioner. First, we install a stronger frozen Retrieval Encoder at the input. Second, between retrieval and decoding we insert a frozen Cross-Attention Verifier that re-ranks the top-9 retrievals to top-5. Third, at the output beam we attach a learned Reranker pairing TriFuse, a multilayer perceptron, with MemAttend, a memory-attended transformer, the pipeline's only learned components; both are trained self-supervised by Borda-consensus distillation across the three frozen scorers, using no paired image-caption labels and no reference captions. Under the inductive headline protocol, with rerankers fit on the disjoint COCO Karpathy validation beam and applied frozen to test, the framework reaches CIDEr 117.6 and SPICE 21.9 on COCO Karpathy, up from 108.0 and 20.3 for IFCap, a +9.6 CIDEr gain, and +7.7 above NES, the strongest synthetic-image-augmented method at 109.9, without retraining the captioner. A training-free fixed-fusion baseline reaches 115.8 CIDEr, so +7.8 of the +9.6 gain comes from the non-learned architectural intervention and the remaining +1.8 from the learned rerankers. The same recipe transfers off-COCO without captioner retraining: +8.1 CIDEr on Flickr30k Karpathy and +5.7 on NoCaps overall.

Figures

Figures reproduced from arXiv: 2607.28986 by Duy Tran Thanh, Long Nguyen-Vu, Ngo Tan Vu Khanh, Thien-Phuc Doan.

Figure 1
Figure 1. Figure 1: From single-checkpoint to multi-checkpoint alignment scoring. Prior strict-ZIC pipelines (top) score image–text alignment at exactly one place: the retrieval-time cosine (✓). Our pipeline (bottom) adds three novel alignment-scoring components (NEW ✓): a stronger Retrieval encoder, a Cross-Attention Verifier, and a learned Beam Reranker. The Captioner is left untouched. The three new heads are trained self-… view at source ↗
Figure 2
Figure 2. Figure 2: Concretely, we make three architectural changes [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 2
Figure 2. Figure 2: Full inference pipeline. The four stages run left-to-right; each agent box carries its role and any side-channel input it consumes, and the concrete output of each stage is shown in the dashed card below. The yellow Memory bank cylinder above Stage 1 represents the external corpus of Ctrain = 566K training captions, with sample caption cards shown alongside, from which the Retrieval Encoder draws. Stage 1,… view at source ↗
Figure 3
Figure 3. Figure 3: TriFuse architecture and training (two-panel layout). Left panel, inference pathway ( all frozen): the unmodified Captioner is shown as a vertical stack of its four sub-components: ViT-B/32, mapping network Mθ, entity filter at K=3, and GPT-2 decoder. The image I feeds into ViT-B/32; the prompt R⋆ 5 (I) feeds into Mθ and the entity filter; GPT-2 emits the 20-wide beam Y20(I). For each candidate y, three fr… view at source ↗
Figure 4
Figure 4. Figure 4: MemAttend architecture (Memory-Attended Reranker). Top: 5 memory tokens MEMj (orange, scalars from the Stage-2 Verifier-adjudicated retrievals) and 20 beam tokens BEAMy (gray, scalars from the three frozen scorers). Stack ( blue, the only trainable parameters): a linear projection Win∈R 32×5 with two role embeddings (MEM vs. BEAM) lifts each token to dmodel=32; two transformer encoder layers (4-head self-a… view at source ↗
Figure 5
Figure 5. Figure 5: CIDEr on COCO Karpathy test, grouped by training [PITH_FULL_IMAGE:figures/full_fig_p010_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative comparison on COCO Karpathy test. Eight test images on which Ours (blue) corrects a salient mistake in the IFCap baseline (red). IFCap’s failures are typical of single-encoder, single-checkpoint retrieval: under-counting (top-left, “a” snowboarder vs. two), wrong subject (banana vendors → “smiling woman”), missing relations (pitcher’s mound mis-localised as “running across a field”), and catego… view at source ↗
Figure 7
Figure 7. Figure 7: Cross-domain CIDEr lifts from the same inference-only [PITH_FULL_IMAGE:figures/full_fig_p013_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Qualitative comparison on cross-domain benchmarks: Flickr30k (top row) and NoCaps (bottom row). Same colour scheme as [PITH_FULL_IMAGE:figures/full_fig_p014_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 7 linked inside Pith

  1. [1]

    Clipcap: Clip prefix for image captioning,

    R. Mokady, A. Hertz, and A. H. Bermano, “Clipcap: Clip prefix for image captioning,”arXiv preprint arXiv:2111.09734, 2021

    Pith/arXiv arXiv 2021

  2. [2]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskeveret al., “Language models are unsupervised multitask learners,”OpenAI blog, vol. 1, no. 8, p. 9, 2019

    2019

  3. [3]

    Text-only training for image captioning using noise-injected clip,

    D. Nukrai, R. Mokady, and A. Globerson, “Text-only training for image captioning using noise-injected clip,” inFindings of the Association for Computational Linguistics: EMNLP 2022, 2022, pp. 4055–4063

    2022

  4. [4]

    Decap: Decoding CLIP latents for zero-shot captioning via text-only training,

    W. Li, L. Zhu, L. Wen, and Y . Yang, “Decap: Decoding CLIP latents for zero-shot captioning via text-only training,” inThe Eleventh International Conference on Learning Representations, 2023

    2023

  5. [5]

    Transferable decoding with visual entities for zero-shot image captioning,

    J. Fei, T. Wang, J. Zhang, Z. He, C. Wang, and F. Zheng, “Transferable decoding with visual entities for zero-shot image captioning,” in Proceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 3136–3146

    2023

  6. [6]

    Meacap: Memory-augmented zero-shot image captioning,

    Z. Zeng, Y . Xie, H. Zhang, C. Chen, B. Chen, and Z. Wang, “Meacap: Memory-augmented zero-shot image captioning,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, pp. 14 100–14 110

    2024

  7. [7]

    Ifcap: Image-like retrieval and frequency-based entity filtering for zero-shot captioning,

    S. Lee, S.-W. Kim, T. Kim, and D.-J. Kim, “Ifcap: Image-like retrieval and frequency-based entity filtering for zero-shot captioning,” inProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024, pp. 20 715–20 727

    2024

  8. [8]

    Microsoft coco: Common objects in context,

    T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” inEuropean conference on computer vision. Springer, 2014, pp. 740– 755

    2014

  9. [9]

    Deep visual-semantic alignments for generating image descriptions,

    A. Karpathy and L. Fei-Fei, “Deep visual-semantic alignments for generating image descriptions,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 3128–3137

    2015

  10. [10]

    Negative entity suppression for zero-shot captioning with synthetic images,

    Z. Lu, H. Xu, B. Liu, and K. Wang, “Negative entity suppression for zero-shot captioning with synthetic images,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 9, 2026, pp. 7636–7643

    2026

  11. [11]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” inInternational conference on machine learning. PmLR, 2021, pp. 8748–8763

    2021

  12. [12]

    Reproducible scaling laws for contrastive language-image learning,

    M. Cherti, R. Beaumont, R. Wightman, M. Wortsman, G. Ilharco, C. Gordon, C. Schuhmann, L. Schmidt, and J. Jitsev, “Reproducible scaling laws for contrastive language-image learning,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 2818–2829

    2023

  13. [13]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,

    J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,” inInternational conference on machine learning. PMLR, 2022, pp. 12 888–12 900

    2022

  14. [14]

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions,

    P. Young, A. Lai, M. Hodosh, and J. Hockenmaier, “From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions,”Transactions of the association for computational linguistics, vol. 2, pp. 67–78, 2014

    2014

  15. [15]

    Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models,

    B. A. Plummer, L. Wang, C. M. Cervantes, J. C. Caicedo, J. Hockenmaier, and S. Lazebnik, “Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models,” inProceedings of the IEEE international conference on computer vision, 2015, pp. 2641–2649

    2015

  16. [16]

    Nocaps: Novel object captioning at scale,

    H. Agrawal, K. Desai, Y . Wang, X. Chen, R. Jain, M. Johnson, D. Batra, D. Parikh, S. Lee, and P. Anderson, “Nocaps: Novel object captioning at scale,” inProceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 8948–8957

    2019

  17. [17]

    Sampling-based approximations to minimum bayes risk decoding for neural machine translation,

    B. Eikema and W. Aziz, “Sampling-based approximations to minimum bayes risk decoding for neural machine translation,” inProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 10 978–10 993

    2022

  18. [18]

    Learning to rank: from pairwise approach to listwise approach,

    Z. Cao, T. Qin, T.-Y . Liu, M.-F. Tsai, and H. Li, “Learning to rank: from pairwise approach to listwise approach,” inProceedings of the 24th international conference on Machine learning, 2007, pp. 129–136

    2007

  19. [19]

    Zerocap: Zero-shot image-to-text generation for visual-semantic arithmetic,

    Y . Tewel, Y . Shalev, I. Schwartz, and L. Wolf, “Zerocap: Zero-shot image-to-text generation for visual-semantic arithmetic,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 17 918–17 928

    2022

  20. [20]

    Language models can see: Plugging visual controls in text generation,

    Y . Su, T. Lan, Y . Liu, F. Liu, D. Yogatama, Y . Wang, L. Kong, and N. Collier, “Language models can see: Plugging visual controls in text generation,”arXiv preprint arXiv:2205.02655, 2022

    Pith/arXiv arXiv 2022

  21. [21]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” inInternational conference on machine learning. PMLR, 2023, pp. 19 730–19 742

    2023

  22. [22]

    Instructblip: Towards general-purpose vision-language models with instruction tuning,

    W. Dai, J. Li, D. Li, A. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, and S. Hoi, “Instructblip: Towards general-purpose vision-language models with instruction tuning,”Advances in neural information processing systems, vol. 36, pp. 49 250–49 267, 2023

    2023

  23. [23]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,”Advances in neural information processing systems, vol. 36, pp. 34 892–34 916, 2023

    2023

  24. [24]

    Improving cross-modal alignment with synthetic pairs for text-only image captioning,

    Z. Liu, J. Liu, and F. Ma, “Improving cross-modal alignment with synthetic pairs for text-only image captioning,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 4, 2024, pp. 3864–3872

    2024

  25. [25]

    High- resolution image synthesis with latent diffusion models,

    R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High- resolution image synthesis with latent diffusion models,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 10 684–10 695

    2022

  26. [26]

    Unleashing text-to-image diffusion prior for zero-shot image captioning,

    J. Luo, J. Chen, Y . Li, Y . Pan, J. Feng, H. Chao, and T. Yao, “Unleashing text-to-image diffusion prior for zero-shot image captioning,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 237–254

    2024

  27. [27]

    Clipscore: A reference-free evaluation metric for image captioning,

    J. Hessel, A. Holtzman, M. Forbes, R. Le Bras, and Y . Choi, “Clipscore: A reference-free evaluation metric for image captioning,” inProceedings of the 2021 conference on empirical methods in natural language processing, 2021, pp. 7514–7528

    2021

  28. [28]

    Passage re-ranking with bert,

    R. Nogueira and K. Cho, “Passage re-ranking with bert,”arXiv preprint arXiv:1901.04085, 2019

    Pith/arXiv arXiv 1901

  29. [29]

    Distilling the knowledge in a neural network,

    G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” 2015. [Online]. Available: https://arxiv.org/abs/1503.02531

    Pith/arXiv arXiv 2015

  30. [30]

    Prefix-tuning: Optimizing continuous prompts for generation,

    X. L. Li and P. Liang, “Prefix-tuning: Optimizing continuous prompts for generation,” inProceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2021, pp. 4582–4597

    2021

  31. [31]

    Laion- 5b: An open large-scale dataset for training next generation image-text models,

    C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsmanet al., “Laion- 5b: An open large-scale dataset for training next generation image-text models,”Advances in neural information processing systems, vol. 35, pp. 25 278–25 294, 2022

    2022

  32. [32]

    Gaussian error linear units (gelus),

    D. Hendrycks and K. Gimpel, “Gaussian error linear units (gelus),”arXiv preprint arXiv:1606.08415, 2016

    Pith/arXiv arXiv 2016

  33. [33]

    Rank aggregation methods for the web,

    C. Dwork, R. Kumar, M. Naor, and D. Sivakumar, “Rank aggregation methods for the web,” inProceedings of the 10th international conference on World Wide Web, 2001, pp. 613–622

    2001

  34. [34]

    Adam: A method for stochastic optimization,

    D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014. 16

    Pith/arXiv arXiv 2014

  35. [35]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,”Advances in neural information processing systems, vol. 30, 2017

    2017

  36. [36]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” inProceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311–318

    2002

  37. [37]

    Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,” inProceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, 2005, pp. 65–72

    2005

  38. [38]

    ROUGE: A package for automatic evaluation of summaries,

    C.-Y . Lin, “ROUGE: A package for automatic evaluation of summaries,” inText Summarization Branches Out. Barcelona, Spain: Association for Computational Linguistics, Jul. 2004, pp. 74–81. [Online]. Available: https://aclanthology.org/W04-1013/

    2004

  39. [39]

    Cider: Consensus- based image description evaluation,

    R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “Cider: Consensus- based image description evaluation,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2015, pp. 4566– 4575

    2015

  40. [40]

    Spice: Semantic propositional image caption evaluation,

    P. Anderson, B. Fernando, M. Johnson, and S. Gould, “Spice: Semantic propositional image caption evaluation,” inEuropean conference on computer vision. Springer, 2016, pp. 382–398

    2016

  41. [41]

    Microsoft coco captions: Data collection and evaluation server,

    X. Chen, H. Fang, T.-Y . Lin, R. Vedantam, S. Gupta, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco captions: Data collection and evaluation server,” arXiv preprint arXiv:1504.00325, 2015

    Pith/arXiv arXiv 2015

  42. [42]

    Object hallucination in image captioning,

    A. Rohrbach, L. A. Hendricks, K. Burns, T. Darrell, and K. Saenko, “Object hallucination in image captioning,” inProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, pp. 4035–4045

    2018

  43. [43]

    Qwen2.5-vl technical report,

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y . Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y . Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin, “Qwen2.5-vl technical report,” 2025

    2025

  44. [44]

    Lavis: A one-stop library for language-vision intelligence,

    D. Li, J. Li, H. Le, G. Wang, S. Savarese, and S. C. Hoi, “Lavis: A one-stop library for language-vision intelligence,” inProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), 2023, pp. 31–41

    2023

  45. [45]

    Contrastive decoding: Open-ended text generation as optimization,

    X. L. Li, A. Holtzman, D. Fried, P. Liang, J. Eisner, T. B. Hashimoto, L. Zettlemoyer, and M. Lewis, “Contrastive decoding: Open-ended text generation as optimization,” inProceedings of the 61st annual meeting of the association for computational linguistics (volume 1: Long papers), 2023, pp. 12 286–12 312

    2023

  46. [46]

    Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning,

    P. Sharma, N. Ding, S. Goodman, and R. Soricut, “Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning,” inProceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018, pp. 2556– 2565

    2018

  47. [47]

    Sigmoid loss for language image pre-training,

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid loss for language image pre-training,” inProceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 11 975–11 986

    2023

  48. [48]

    Polos: Multimodal metric learning from human feedback for image captioning,

    Y . Wada, K. Kaneda, D. Saito, and K. Sugiura, “Polos: Multimodal metric learning from human feedback for image captioning,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 13 559–13 568

    2024

  49. [49]

    Aloha: A new measure for hallucination in captioning models,

    S. Petryk, D. Chan, A. Kachinthaya, H. Zou, J. Canny, J. Gonzalez, and T. Darrell, “Aloha: A new measure for hallucination in captioning models,” inProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), 2024, pp. 342–357

    2024

  50. [50]

    Capability: A comprehensive visual caption benchmark for evaluating both correctness and thoroughness,

    Z. Liu, C.-W. Xie, B. Wen, F. Yu, P. Li, B. Zhang, N. Yang, Z. Gao, Y . Zheng, and H. Xie, “Capability: A comprehensive visual caption benchmark for evaluating both correctness and thoroughness,”Advances in Neural Information Processing Systems, vol. 38, 2026

    2026

  51. [51]

    Test-time adaptation with clip reward for zero-shot generalization in vision-language models,

    S. Zhao, X. Wang, L. Zhu, and Y . Yang, “Test-time adaptation with clip reward for zero-shot generalization in vision-language models,” in International Conference on Learning Representations, vol. 2024, 2024, pp. 3597–3613. 17

    2024