Image Embedding Sampling Method for Diverse Captioning

Na Min An; Sania Waheed

REVIEW 5 major objections 6 minor 58 references

Image Embedding Sampling Method for Diverse Captioning

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read HBoP shows that a small, untrained captioner can reach the caption diversity of much larger vision-language models by feeding BLIP with segmentation-selected patch embeddings.

desk verdict A training-free method to make small VLMs generate more diverse captions is promising and mostly works, but the paper's core input-construction description is ambiguous enough that the reported numbers are not yet reproducible. read the letter →

arxiv 2502.10118 v2 pith:ANJ7QSOV submitted 2025-02-14 cs.CV cs.AI

Sania Waheed , Na Min An This is my paper

classification cs.CVcs.AI

keywords imagecaptioningcaptiondiversitysegmentanythingembeddingsamplinghierarchicalrepresentationvision-languagemodeltraining-freeBLIP

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes HBoP, a training-free captioning framework that makes the small vision-language model BLIP produce captions as varied as human annotations and far larger LLM-based models. The core idea is to not caption the whole image once. Instead, segmentation masks pick out global, regional, and fine-grained image regions, and the model generates a separate caption for each region's patch embeddings. Because the embeddings come from the image encoder's last layer, they retain whole-image context while focusing on one region. On MSCOCO, Flickr30k, and Nocaps, HBoP reaches Div-2 scores of 0.735, 0.750, and 0.748, close to gold captions and higher than BLIP with nucleus sampling, while relevance scores stay comparable.

What carries the argument

The load-bearing object is the Hierarchical Composition Module (HCM), which selects which patch embeddings BLIP sees. SAM produces masks; non-maximum suppression and top-k selection give global masks, K-means clustering groups masks into regional bundles, and the leftover masks form fine-grained tags. For each selected mask, the corresponding patches of the ViT's last-layer embedding are concatenated and zero-padded to the original input shape, then captioned by frozen BLIP. This lets a single forward pass's embedding be reused many times, so the method adds no training and little extra captioning cost; the hierarchy is what controls granularity.

What would settle it

A human rating study would settle the central claim: show annotators HBoP captions and BLIP captions for the same image and ask which is more accurate and fluent; if HBoP is not at least as fluent and relevant, the claim that diversity comes without a relevance loss collapses.

Watch

Extended reading notes

Core claim

The central claim is that caption diversity can be decoupled from model scale. HBoP samples patch embeddings from SAM segmentation masks at three granularities and feeds each zero-padded embedding to BLIP's frozen captioner, producing five captions per image (two global, three regional). This yields diversity scores close to gold captions—Div-2 0.735/0.750/0.748 across MSCOCO/Flickr30k/Nocaps—and lowers mBLEU-4 by over 60% while raising Div-2 by over 30% relative to BLIP with nucleus sampling; on several diversity metrics it beats 7B-to-13B LLM-based captioners. Relevance, measured by CLIP-Score and SBERT, stays in the same range as BLIP's, supporting the claim that the added diversity does not come from drifting away from the image.

Load-bearing premise

Everything rests on BLIP's captioning module producing fluent, relevant captions when fed zero-padded concatenations of SAM-selected patch embeddings, an input it never saw in training.

Editorial extensions

If this is right

Small VLMs can match or beat much larger LLM-based captioners on diversity metrics without any fine-tuning.
The framework works with any segmentation model, and swapping SAM for FastSAM cuts segmentation time from 5.43 seconds to 0.18 seconds per image while keeping the diversity benefit.
Caption granularity becomes a controllable resource: global, regional, and fine-grained captions can be produced from one ViT embedding.
Diversity and relevance need not trade off: HBoP's embedding sampling beats cropping, which boosts diversity at the cost of relevance.
The approach sets a baseline for controlling intermediate image embeddings in captioning, potentially useful beyond captioning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension beyond this paper is to plug the same mask-guided embedding sampling into other frozen captioners, which would test whether the diversity gain is specific to BLIP or a general property of input selection.
Because the hierarchy is controlled by mask size and cluster count, one could tune granularity to scene complexity; the paper hints at this by using two levels for simpler images, so a systematic sweep of K is an obvious next experiment.
The fine-grained masks behave like image tags, so HBoP could double as a training-data generator for region-level tasks such as referring expression comprehension without extra annotation.
An open question the reported metrics do not settle is whether humans find the extra diversity informative; a forced-choice study between HBoP and BLIP captions would resolve that.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the diversity improvement is real and the method is simple enough to be useful, but the paper needs to nail down exactly what goes into BLIP and temper the semantic-integrity claim before I'd trust the numbers.

What's new: HBoP uses SAM masks to select patch embeddings from the ViT encoder and feeds hierarchical concatenations (global, regional, fine-grained) into a frozen BLIP captioner. That's a legitimate, training-free way to get more diverse captions from a small model. The evaluation is thorough: three datasets, multiple diversity metrics, comparison against ModeCap, Seq-CV AE, and several LLM-based models, plus a crop-vs-sampling ablation and FastSAM runtime numbers. The crop comparison is especially useful because it shows that sampling from full-image embeddings preserves relevance better than cropping.

Where it gets soft: the description of the input to BLIP is genuinely ambiguous. Section 3.2 writes Egi = EX ⊙ Mgi, which looks like elementwise masking of the full patch grid, while Appendix C.1 says embeddings from each cluster are concatenated and zero-padded to the original shape. Those are different inputs, and neither was seen in BLIP pretraining. Without a precise specification, the experiments aren't reproducible, and every downstream metric inherits whatever artifacts the out-of-distribution input introduces. The stress-test note is right to flag this as load-bearing. A related gap: there's no ablation with random masks or a pure padding baseline, so we can't tell how much of the diversity gain comes from the SAM selection itself versus just the concatenate-and-pad procedure. A small experiment would settle this.

The semantic-integrity claim is also stronger than the data support. Table 4 shows HBoP at 8.56/2.48 on the two LLM judges, below BLIP (9.36/2.97), BLIP-2, and gold on one judge. The abstract says 'comparable,' but that's generous. The diversity numbers themselves are likely fine, though the paper gives no error bars, so we're taking single-run values at face value.

Minor: Section 4 says two global + three regional captions, while the Appendix footnote says one global + five regional. One of these is wrong.

Bottom line: This is a solid empirical contribution that deserves a serious referee. The authors need to clarify the exact embedding construction, add a control for mask selectivity, and soften the semantic-integrity wording. I'd be happy to see it in a workshop or a main conference after that.

Referee Report

5 major / 6 minor

Summary. The paper proposes HBoP, a training-free hierarchical captioning framework built around BLIP and SAM. It selects top-k global masks, K-means regional clusters, and residual fine-grained masks after NMS, extracts corresponding patch embeddings from the last layer of BLIP's ViT encoder, and feeds them to the BLIP caption decoder with zero-padding, producing multiple captions per image. Experiments on MSCOCO, Flickr30k, and Nocaps report diversity metrics (PCD, mBLEU-4, Div-2), relevance metrics (SBERT, CLIP-S), and LLM-based semantic integrity, claiming that a small VLM can reach diversity comparable to much larger LLM-based models while retaining relevance.

Significance. If the mechanism were fully validated, the paper would make a useful contribution: a training-free way to improve the diversity of small-VLM captioning by exploiting structured segmentation. The authors release code, compare against several strong baselines including LLM-based models, include a crop-versus-embedding ablation that supports the design choice, and provide FastSAM runtime measurements. These are concrete strengths. However, the load-bearing inference procedure is ambiguously specified, and the central mechanism is not isolated from out-of-distribution input effects, so the strength of the empirical claim currently exceeds what the experiments establish.

major comments (5)

[§3.2 vs. §C.1, Eq. (2)] The paper defines two incompatible input constructions. In §3.2, Egi = EX ⊙ Mgi indicates elementwise masking of the full patch grid, which preserves the ViT's spatial layout. In §C.1, the text says the embeddings from each cluster are concatenated and zero-padding is applied to preserve the original shape, which describes a shorter, rearranged sequence that is then padded back to length. These are different inputs, and neither was seen during BLIP pretraining. Because this is the central mechanism, the reported numbers are not currently tied to a single reproducible inference procedure. Please specify the exact tensor construction with shapes, provide pseudocode or a pointer to the released code, and state which variant generated Tables 1, 2, and 6.
[§3.3, §C.1, Table 3] The paper does not include a control that isolates the effect of SAM-selected regions from the effect of perturbing the input embeddings. A control using random masks matched in size and count, or captioning from the [CLS] token or global-pooled embeddings, would show whether the diversity gain reflects regional semantic signal or simply BLIP's response to atypical masked or zero-padded inputs. The crop comparison in Table 3 shows that cropping loses relevance, but it does not establish that the particular mask selection, rather than the general input corruption, is responsible for the diversity behavior.
[Tables 1, 2, 6] All main results are reported as single point estimates, yet the captioning module uses stochastic nucleus sampling and the evaluation draws five captions per image. Without confidence intervals, multiple seeds, or significance tests, claims such as outperforming BLIP-2 or Honeybee on several diversity metrics, and the IoU threshold comparison in Table 2, cannot be distinguished from sampling noise. Please add repeated-run variance, statistical tests for the headline comparisons, and a seed sensitivity analysis for the main tables.
[Table 4, §C.3.3] Table 4 shows that HBoP's semantic integrity is materially lower than BLIP's (LLaMA-2: 8.56 vs. 9.36; GPT-4: 2.48 vs. 2.97) and lower than gold captions. This is in tension with the abstract's claim of maintaining semantic integrity and with the characterization in §C.3.3 that HBoP outperforms PnP-VQA. The paper should either qualify the claim to the specific comparison with PnP-VQA or analyze why the regional captioning procedure reduces coherence, and it should include this cost in the diversity-relevance trade-off discussion.
[§3.2, §C.1, Table 2] The method is described as training-free, but several components are chosen per dataset and apparently on the basis of evaluation metrics: the NMS IoU threshold (0.1), top-k (5), K-means K (5), and the number of hierarchy levels used ('we adjust the different hierarchy levels depending on a given dataset'). Only the IoU threshold is ablated, and that ablation is performed on the same test sets used for the main claims. Please provide a validation protocol, ablations for K and top-k, and per-dataset reporting of which hierarchical levels were used, to rule out selection bias.

minor comments (6)

[Table 1 vs. §C.3.1] The number of HBoP captions is inconsistent: Table 1 says 'two global and three regional captions,' while §C.3.1 says 'one global caption and five regional captions.' Please reconcile the description and the evaluation setup.
[Eq. (1)] The summation limits in the PCD equation are unusual; please make the index ranges explicit, for example by writing 1 ≤ j < i ≤ n.
[Author affiliation] The affiliation contains a typo: 'University of Southhampton' should be 'University of Southampton.'
[Figure 3 caption] The phrase 'as comapred to random and PnP-VQA captions' should read 'as compared to.'
[Table 1, §C.1] The parameter count of HBoP (1B) should be broken down into the SAM ViT-L/16 component and the BLIP captioning component, so that the comparison with the 446M BLIP model is transparent.
[§3.2, §6] The paper introduces 'mBLEU-4' without a formal definition; please specify how the metric is averaged over captions and images. Also, §6 mentions bounding-box approximations while §3.2 discusses masks and patch embeddings; clarify whether actual masks or bounding-box approximations are used to select patches.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: HBoP's diversity and relevance numbers are empirical measurements of BLIP outputs, not identities or fitted predictions.

full rationale

The central claim is that feeding SAM-selected patch embeddings to a frozen BLIP captioner yields more diverse captions while retaining relevance. This is an evaluated claim: the PCD, mBLEU-4, and Div-2 scores are computed from actual generated sentences and are not equal by construction to the mask-selection operation. The crop-baseline comparison (Table 3) shows that regional inputs alone do not force the reported scores, so the result has independent empirical content. The paper is training-free, so there is no parameter fitted to the evaluation data and renamed as a prediction. The NMS IoU threshold (0.1) and per-dataset hierarchy choice are hyperparameters, but the ablation in Table 2 shows the qualitative diversity advantage persists across thresholds, so this is a transparent design choice rather than a circular reduction. Self-citations (An et al., 2024, 2025; Waheed et al., 2025; Bayramli et al., 2025) are contextual and not load-bearing: the sentence-embedding metric uses the externally pretrained all-MiniLM-L6-v2, not a model introduced by the authors. The unresolved discrepancy between Section 3.2's elementwise-mask notation and Appendix C.1's concatenate-and-pad description, and the unvalidated assumption that BLIP can caption zero-padded embedding sequences, are correctness and reproducibility risks rather than circularity, because the reported numbers would be invalidated by a wrong implementation but are not entailed by the definitions.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claim rests on domain assumptions about the reliability of SAM masks, the ability of BLIP to handle zero-padded embedding inputs, and the validity of the chosen evaluation metrics. The free parameters are experimental design choices, some of which appear tuned on the evaluation sets.

free parameters (5)

NMS IoU threshold = 0.1
Set in Section 3.2; the ablation in Table 2 shows this value maximizes PCD and Div-2, indicating tuning on the evaluation set.
Top-k global masks = 5
Section C.1; sets the number of global captions per image, aligned with the five-caption evaluation protocol.
K-means clusters K = 5
Section C.1; sets the number of regional captions per image to match the desired five captions.
Global vs regional caption ratio = 2 global + 3 regional (Sec. 4) or 1 global + 5 regional (Appendix C.3.1)
The split is a protocol choice that controls diversity; the paper reports conflicting values.
Hierarchy levels per dataset = Two-tier for MSCOCO/Flickr30k
Appendix C.1 states that hierarchy levels are adjusted depending on dataset complexity, which can be a post hoc choice.

assumptions (6)

domain assumption SAM segmentation masks correspond to semantically meaningful image regions
Used throughout Sections 3.1 and 3.2 to define global, regional, and fine-grained levels.
domain assumption BLIP's captioner can generate coherent captions from zero-padded, concatenated patch embeddings
Section 3.3 uses zero padding before captioning, without validating this out-of-distribution input.
domain assumption Sentence embeddings from all-MiniLM-L6-v2 capture semantic diversity for PCD
Equation 1 defines PCD using this model; the diversity scores depend on this choice.
domain assumption CLIP-S and SBERT are valid measures of image-caption relevance and semantic similarity
Used for relevance evaluation in Section C.3.2.
domain assumption LLM-based semantic integrity scores correlate with human judgment
Section C.3.3 relies on Llama-2 and GPT-4 ratings as proxies for human evaluation.
standard math NMS with IoU threshold and K-means clustering on masks produce non-redundant, meaningful regions
These are standard algorithms applied in the HCM module.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Image Embedding Sampling Method for Diverse Captioning." pith.science (2026). https://pith.science/paper/ANJ7QSOV

@misc{pith2026250210118,
  author       = {Pith},
  title        = {Pith review of: Image Embedding Sampling Method for Diverse Captioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ANJ7QSOV}},
  note         = {Machine review of arXiv:2502.10118}
}

read the original abstract

Image Captioning for state-of-the-art VLMs has significantly improved over time; however, this comes at the cost of increased computational complexity, making them less accessible for resource-constrained applications such as mobile devices and assistive technologies. Alternatively, comparably smaller VLMs prioritize high-level scene descriptions, overlooking finer details that contribute to a richer understanding of an image. In this paper, we introduce a training-free framework that enhances caption diversity and informativeness by explicitly attending to distinct image regions using a comparably small VLM, BLIP, as the backbone. Our approach leverages structured segmentation to produce hierarchical representations that capture both global and localized semantics. Without requiring additional model training, we demonstrate that our method allows smaller VLMs to achieve performance comparable to larger models in terms of image-caption alignment, semantic integrity, and diversity. We evaluate our framework on MSCOCO, Flickr30k, and Nocaps test datasets, achieving a Div-2 score of 0.735, 0.750, and 0.748 for each dataset, respectively, while maintaining strong image-caption relevancy and semantic integrity with the human-annotated captions.

Figures

Figures reproduced from arXiv: 2502.10118 by the authors.

**Figure 2.** The proposed HBoP framework consists of three components: (1) Image Segmentation Module (ISM), [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Correlation of relevance scores between gold captions and model captions. We observe higher correlations [PITH_FULL_IMAGE:figures/full_fig_p015_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Comparison between captions generated using BLIP-2 ( [PITH_FULL_IMAGE:figures/full_fig_p016_4.png]

**Figure 5.** Figure 5: Additional visualizations of GradCAMs across different model captions. [PITH_FULL_IMAGE:figures/full_fig_p017_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

58 extracted references · 25 canonical work pages

[1]

Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. 2019. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8948--8957

2019
[2]

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716--23736

2022
[3]

Na Min An, Eunki Kim, Wan Ju Kang, Sangryul Kim, Hyunjung Shim, and James Thorne. 2025. Can lvlms and automatic metrics capture underlying preferences of blind and low-vision individuals for navigational aid? arXiv preprint arXiv:2502.14883

work page arXiv 2025
[4]

Na Min An, Sania Waheed, and James Thorne. 2024. https://aclanthology.org/2024.findings-eacl.43 Capturing the relationship between sentence triplets for LLM and human-generated texts to enhance sentence embeddings . In Findings of the Association for Computational Linguistics: EACL 2024, pages 624--638, St. Julian ' s, Malta. Association for Computational...

work page 2024
[5]

Jyoti Aneja, Harsh Agrawal, Dhruv Batra, and Alexander Schwing. 2019 a . https://arxiv.org/abs/1908.08529 Sequential latent spaces for modeling the intention during diverse image captioning . Preprint, arXiv:1908.08529

work page Pith review arXiv 2019
[6]

Jyoti Aneja, Harsh Agrawal, Dhruv Batra, and Alexander Schwing. 2019 b . Sequential latent spaces for modeling the intention during diverse image captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4261--4270

work page 2019
[7]

Satanjeev Banerjee and Alon Lavie. 2005. https://aclanthology.org/W05-0909 METEOR : An automatic metric for MT evaluation with improved correlation with human judgments . In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , pages 65--72, Ann Arbor, Michigan. Association for Comput...

2005
[8]

Zahra Bayramli, Ayhan Suleymanzade, Na Min An, Huzama Ahmad, Eunsu Kim, Junyeong Park, James Thorne, and Alice Oh. 2025. https://doi.org/10.18653/v1/2025.acl-long.1503 Diffusion models through a global lens: Are they culturally inclusive? In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa...

work page doi:10.18653/v1/2025.acl-long.1503 2025

Show all 58 references

[9]

Simone Bianco, Luigi Celona, Marco Donzella, and Paolo Napoletano. 2023. https://arxiv.org/abs/2306.11593 Improving image captioning descriptiveness by ranking and llm-based fusion . Preprint, arXiv:2306.11593

2023
[10]

Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. 2023. Honeybee: Locality-enhanced projector for multimodal llm. arXiv preprint arXiv:2312.06742

2023 arXiv
[11]

Delong Chen, Samuel Cahyawijaya, Etsuko Ishii, Ho Shu Chan, Yejin Bang, and Pascale Fung. 2024. https://arxiv.org/abs/2405.00485 What makes for good image captions? Preprint, arXiv:2405.00485

2024 arXiv
[12]

Qi Chen, Chaorui Deng, and Qi Wu. 2022. Learning distinct and representative modes for image captioning. Advances in Neural Information Processing Systems, 35:9472--9485

2022
[13]

Qi Chen, Chaorui Deng, and Qi Wu. 2023. https://arxiv.org/abs/2209.08231 Learning distinct and representative styles for image captioning . Preprint, arXiv:2209.08231

2023 arXiv
[14]

Cheng-Han Chiang and Hung yi Lee. 2023. https://arxiv.org/abs/2305.01937 Can large language models be an alternative to human evaluations? Preprint, arXiv:2305.01937

2023 arXiv
[15]

Marcella Cornia, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. 2020. Meshed-memory transformer for image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10578--10587

2020
[16]

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. https://arxiv.org/abs/2010.11929 An image is worth 16x16 words...

2021 arXiv
[17]

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. In Internation...

2020
[18]

Zhiyuan Fang, Jianfeng Wang, Xiaowei Hu, Lin Liang, Zhe Gan, Lijuan Wang, Yezhou Yang, and Zicheng Liu. 2022. Injecting semantic concepts into end-to-end image captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18009--18019

2022
[19]

Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. https://arxiv.org/abs/2302.04166 Gptscore: Evaluate as you desire . Preprint, arXiv:2302.04166

2023 arXiv
[20]

Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. Clipscore: A reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7514--7528

2021
[21]

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. In International Conference on Learning Representations

2019
[22]

Jan Hosang, Rodrigo Benenson, and Bernt Schiele. 2017. Learning non-maximum suppression. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4507--4515

2017
[23]

Zhong Ji, Kexin Chen, and Haoran Wang. 2021. Step-wise hierarchical alignment network for image-text matching. In IJCAI

2021
[24]

Andrej Karpathy and Li Fei-Fei. 2015. https://arxiv.org/abs/1412.2306 Deep visual-semantic alignments for generating image descriptions . Preprint, arXiv:1412.2306

2015 arXiv
[25]

Berg, Wan-Yen Lo, Piotr Doll \'a r, and Ross Girshick

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Doll \'a r, and Ross Girshick. 2023. Segment anything. arXiv:2304.02643

2023 arXiv
[26]

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597

2023 arXiv
[27]

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022 a . Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888--12900. PMLR

2022
[28]

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022 b . https://arxiv.org/abs/2201.12086 Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation . Preprint, arXiv:2201.12086

2022 arXiv
[29]

Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013 ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

2004
[30]

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer Vision--ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, ...

2014
[31]

Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et al. 2023 a . Llava-plus: Learning to use tools for creating multimodal agents. arXiv preprint arXiv:2311.05437

2023 arXiv
[32]

Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023 b . https://arxiv.org/abs/2303.16634 G-eval: Nlg evaluation using gpt-4 with better human alignment . Preprint, arXiv:2303.16634

2023 arXiv
[33]

Maxime Oquab, Timoth \'e e Darcet, Th \'e o Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. 2023. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193

2023 arXiv
[34]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 a . https://doi.org/10.3115/1073083.1073135 Bleu: A method for automatic evaluation of machine translation . In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, page ...

2002
[35]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002 b . Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311--318

2002
[36]

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021 a . https://arxiv.org/abs/2103.00020 Learning transferable visual models from natural ...

2021 arXiv
[37]

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021 b . Learning transferable visual models from natural language supervision. In International conference on machine lea...

2021
[38]

Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJC...

2019
[39]

Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2017. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference on computer vision, pages 618--626

2017
[40]

Bin Shao, Jianzhuang Liu, Renjing Pei, Songcen Xu, Peng Dai, Juwei Lu, Weimian Li, and Youliang Yan. 2023. Hivlp: Hierarchical interactive video-language pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 13756--13766

2023
[41]

Michal Shlapentokh-Rothman, Ansel Blume, Yao Xiao, Yuqun Wu, Sethuraman T V, Heyi Tao, Jae Yong Lee, Wilfredo Torres, Yu-Xiong Wang, and Derek Hoiem. 2024. https://arxiv.org/abs/2402.02352 Region-based representations revisited . Preprint, arXiv:2402.02352

2024 arXiv
[42]

Mustafa Shukor, Guillaume Couairon, and Matthieu Cord. 2022. Efficient vision-language pretraining with visual concepts and hierarchical alignment. In 33rd British Machine Vision Conference (BMVC)

2022
[43]

Yucheng Suo, Linchao Zhu, and Yi Yang. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.73 Text augmented spatial aware zero-shot referring image segmentation . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1032--1043, Singapore. Associat...

2023 doi
[44]

Anthony Meng Huat Tiong, Junnan Li, Boyang Li, Silvio Savarese, and Steven CH Hoi. 2022. Plug-and-play vqa: Zero-shot vqa by conjoining large pretrained models with zero training. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 951--967

2022
[45]

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

2023 arXiv
[46]

Lawrence Zitnick, and Devi Parikh

Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. https://doi.org/10.1109/CVPR.2015.7299087 Cider: Consensus-based image description evaluation . In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4566--4575

2015
[47]

Sania Waheed, Na Min An, Michael Milford, Sarvapali D Ramchurn, and Shoaib Ehsan. 2025. Vlm-guided visual place recognition for planet-scale geo-localization. arXiv preprint arXiv:2507.17455

2025 arXiv
[49]

Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. 2022 b . Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442

2022 arXiv
[50]

Yuji Wang, Jingchen Ni, Yong Liu, Chun Yuan, and Yansong Tang. 2025. Iterprime: Zero-shot referring image segmentation with iterative grad-cam refinement and primary word emphasis. arXiv preprint arXiv:2503.00936

2025 arXiv
[51]

Guanghui Xu, Shuaicheng Niu, Mingkui Tan, Yucheng Luo, Qing Du, and Qi Wu. 2021. Towards accurate text-based image captioning with content diversity exploration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12637--12646

2021
[52]

Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. 2019. Hierarchy parsing for image captioning. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2621--2629

2019
[53]

Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67--78

2014
[54]

Seonghoon Yu, Paul Hongsuck Seo, and Jeany Son. 2023. Zero-shot referring image segmentation with global-local context features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19456--19465

2023
[55]

Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. 2022. When and why vision-language models behave like bags-of-words, and what to do about it? In The Eleventh International Conference on Learning Representations

2022
[56]

Xuying Zhang, Xiaoshuai Sun, Yunpeng Luo, Jiayi Ji, Yiyi Zhou, Yongjian Wu, Feiyue Huang, and Rongrong Ji. 2021. Rstnet: Captioning with adaptive attention on visual and non-visual words. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pag...

2021
[57]

Xu Zhao, Wenchao Ding, Yongqi An, Yinglong Du, Tao Yu, Min Li, Ming Tang, and Jinqiao Wang. 2023. https://arxiv.org/abs/2306.12156 Fast segment anything . Preprint, arXiv:2306.12156

2023 arXiv
[58]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[59]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. 2019. Nocaps: Novel object captioning at scale. In Proceedings of the IEEE/CVF international conference on computer vision, pages 8948--8957

2019

[2] [2]

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716--23736

2022

[3] [3]

Na Min An, Eunki Kim, Wan Ju Kang, Sangryul Kim, Hyunjung Shim, and James Thorne. 2025. Can lvlms and automatic metrics capture underlying preferences of blind and low-vision individuals for navigational aid? arXiv preprint arXiv:2502.14883

work page arXiv 2025

[4] [4]

Na Min An, Sania Waheed, and James Thorne. 2024. https://aclanthology.org/2024.findings-eacl.43 Capturing the relationship between sentence triplets for LLM and human-generated texts to enhance sentence embeddings . In Findings of the Association for Computational Linguistics: EACL 2024, pages 624--638, St. Julian ' s, Malta. Association for Computational...

work page 2024

[5] [5]

Jyoti Aneja, Harsh Agrawal, Dhruv Batra, and Alexander Schwing. 2019 a . https://arxiv.org/abs/1908.08529 Sequential latent spaces for modeling the intention during diverse image captioning . Preprint, arXiv:1908.08529

work page Pith review arXiv 2019

[6] [6]

Jyoti Aneja, Harsh Agrawal, Dhruv Batra, and Alexander Schwing. 2019 b . Sequential latent spaces for modeling the intention during diverse image captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4261--4270

work page 2019

[7] [7]

Satanjeev Banerjee and Alon Lavie. 2005. https://aclanthology.org/W05-0909 METEOR : An automatic metric for MT evaluation with improved correlation with human judgments . In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , pages 65--72, Ann Arbor, Michigan. Association for Comput...

2005

[8] [8]

Zahra Bayramli, Ayhan Suleymanzade, Na Min An, Huzama Ahmad, Eunsu Kim, Junyeong Park, James Thorne, and Alice Oh. 2025. https://doi.org/10.18653/v1/2025.acl-long.1503 Diffusion models through a global lens: Are they culturally inclusive? In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa...

work page doi:10.18653/v1/2025.acl-long.1503 2025