Visual Semantic Description Generation with MLLMs for Image-Text Matching

Junyu Chen; Mingyong Li; Yihua Gao

REVIEW 3 major objections 5 minor 27 references

Visual Semantic Description Generation with MLLMs for Image-Text Matching

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read AI-written image summaries improve image-text retrieval scores

desk verdict A useful plug-and-play idea—MLLM-generated visual semantic descriptions as fused anchors for image-text matching—with real gains, but the core fusion equation is dimensionally under-specified and the ablation table doesn't match the main table. read the letter →

arxiv 2507.08590 v1 pith:PW2YMRSA submitted 2025-07-11 cs.MM cs.CV

Junyu Chen , Yihua Gao , Mingyong Li This is my paper · ORCID

classification cs.MMcs.CV

keywords image-textmatchingvisualsemanticdescriptionmultimodallargelanguagemodelcross-modalretrievalprototype-levelalignmentzero-shotcross-domaintransfergatedfeaturefusion

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes using a multimodal large language model (MLLM) as a visual semantic parser: given an image and its captions, the model writes a short natural-language visual semantic description of the main scene, key objects, actions, and visual details. The description is encoded with a text-embedding model, fused into the image feature (and the text feature) through learned gates, and used to build prototype-level clusters for cross-modal alignment. On Flickr30K and MSCOCO, adding this module to GPO, HREM, and CLIP raises retrieval recall and rSum, and the same recipe transfers without further training to news and remote-sensing retrieval. If the claim holds, existing image-text matching models can be strengthened by injecting MLLM-generated language anchors rather than redesigning their architectures.

What carries the argument

The load-bearing object is the Visual Semantic Description (VSD): a 15-30 word MLLM-generated summary that integrates visual elements with the associated captions. Around the VSD, the paper builds two mechanisms. Instance-level semantic alignment concatenates the image feature with the mean-pooled VSD embedding, computes a gating weight with a sigmoid, and interpolates the two vectors (with an analogous gate fusing the original text feature and an encoded copy of the text), then applies a hard-negative triplet loss. Prototype-level semantic alignment runs K-means on all VSD embeddings to form k prototypes, soft-assigns image and text features to those prototypes, refines the assignments with the Sinkhorn algorithm, and trains each modality to predict the other modality's assignment distribution.

What would settle it

Inspect the released code's forward pass for a projection that aligns the 1024-dimensional description embedding with the image feature dimension before Equation (4); if no such projection exists, the fused vector cannot be computed as written and the reported rankings cannot come from the described architecture.

Watch

Extended reading notes

Core claim

The central claim is that a short MLLM-generated visual semantic description (VSD) of each image provides a linguistic bridge across the modality gap. The paper embeds each VSD with a text-retrieval encoder and uses the embedding twice: at instance level, a sigmoid gate interpolates it with the raw image feature (and an encoded copy of the text with the original text feature), and the fused features are trained with a hard-negative triplet loss; at prototype level, VSD embeddings are K-means clustered into prototypes, image and text features are soft-assigned to these prototypes via the Sinkhorn algorithm, and a symmetric classification loss aligns the assignments. The paper reports that this combination substantially improves R@1 and rSum for GPO, HREM, and CLIP on Flickr30K and MSCOCO, that MLLM-generated descriptions outperform descriptions from a conventional image-captioning model, and that the enhanced models show zero-shot gains on the N24News and RSITMD datasets.

Load-bearing premise

The gated fusion in Equation (4) assumes the image feature vector and the encoded visual semantic description vector have the same length so they can be concatenated and interpolated; the paper does not describe how the 1024-dimensional description embedding is aligned with the 512-dimensional image features.

Editorial extensions

If this is right

Existing dual-stream image-text matching models can be upgraded by adding the VSD module and two losses without retraining the MLLM generator.
The choice of description generator matters: MLLM-generated descriptions (MiniCPM-V) give larger gains than a conventional captioning model (Florence-2), so richer semantic parsing is the active ingredient.
The gains persist in zero-shot cross-dataset transfer from MSCOCO to Flickr30K and in cross-domain transfer to news and remote-sensing retrieval.
Prototype-level alignment alone improves the GPO baseline, so clustering descriptions into semantic prototypes contributes independently of instance-level fusion.
The text side also benefits from the auxiliary encoder: fusing original text features with an encoded copy of the same text improves text-to-image retrieval.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The method may work because VSD translates the image into the discrete, structured linguistic space in which the text encoder was pretrained, effectively shrinking the representation gap rather than learning deeper geometric alignment.
If the released implementation contains an unstated projection that aligns the 1024-dimensional description embedding with 512-dimensional image features, the published description of Equation (4) is incomplete, and portability to new base models depends on that hidden alignment choice.
A testable extension separating description value from training value: add the two losses with randomly sampled captions in place of VSD; if scores stay comparable, the benefit is not specific to MLLM semantic parsing.
The prototype count k is set per dataset (896 for Flickr30K and 2560 for MSCOCO), so applying the method to a new domain likely requires re-selecting k rather than using a fixed plug-in default.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take on arXiv:2507.08590. The paper does something genuinely useful: instead of using MLLMs just to rewrite captions for dataset augmentation, it uses MLLM-generated visual semantic descriptions as a fused semantic anchor for image and text features, with instance-level gating and prototype-level Sinkhorn alignment. That combination is new to me, and the empirical story is consistent—three different baselines (GPO, HREM, CLIP) all improve on Flickr30K and MSCOCO, and cross-dataset/cross-domain numbers are reported on two additional sets. The gains are large but not crazy for adding a strong text encoder as an auxiliary signal.

Credit where due: the two-stage generation strategy for test-time (when ground-truth captions are unavailable) is an honest attempt to fix a train/test mismatch, and the authors properly cite the caption-rewriting prior work (LaCLIP, Nguyen et al., Liu et al.).

Now the soft spots, in order of size.

First, Eq. (4) is dimensionally ill-defined as written. BGE-large-v1.5 outputs 1024-d embeddings; OpenCLIP ViT-B/32 image features are 512-d; GPO/HREM use their own pre-extracted features, often 1024-d or higher. The gating fusion concatenates v and d_vsd and mixes them elementwise, but no projection layer is described anywhere. Either the code contains a projection the paper omits, or the described mechanism cannot run as written. This needs to be fixed in a revision—either add the projection to the equations or state the dimensions explicitly.

Second, the ablation table, Tab. IV, does not match the main results. GPO's Flickr30K R@1 is 81.7 in Tab. I but 68.0 in Tab. IV. The 68.0 value matches the cross-dataset transfer protocol in Tab. II (train on MSCOCO, test on Flickr30K), but the text says the ablation is on the test sets without specifying that protocol. The MSCOCO row shows 50.0, which is also not the Tab. I value (58.3). The ablation needs to state its protocol and reconcile the numbers. This is a reporting flaw that makes the component contributions hard to verify.

Third, the training caption procedure is underspecified: the generation prompt uses "associated captions," which suggests ground-truth captions at training time, but the two-stage strategy is only described for testing. If training uses ground-truth captions and testing uses generated captions, the distribution shift should be discussed or an experiment run using only generated captions at training.

The novelty is modest but real; the central claim—VSD helps ITM—survives these issues. The paper deserves a serious referee, but the referee will need code and corrected equations.

Recommendation: send to peer review, require a revision that fixes the projection, clarifies the ablation protocol, and states the caption source per phase.

Referee Report

3 major / 5 minor

Summary. The manuscript proposes VSD, a plug-in enhancement for image-text matching. A frozen MLLM (MiniCPM-V-2.6 or Florence-2) generates a 15-30 word visual semantic description from an image and its captions; BGE-large-v1.5 encodes the description; a gating mechanism adaptively fuses image/text features with the encoded description features; and a prototype-level loss clusters the descriptions with K-means and aligns image/text classification scores via Sinkhorn assignments. Experiments report consistent R@K and rSum gains over GPO, HREM, and OpenCLIP ViT-B/32 on Flickr30K and MSCOCO, plus claimed zero-shot gains on N24News and RSITMD.

Significance. If the claimed effects are genuine, VSD would be a useful and relatively lightweight post-hoc module for improving existing ITM models without retraining the base encoders, and the cross-domain results are interesting. The paper's strengths are the breadth of base models (GPO, HREM, CLIP), the two benchmark datasets, the cross-dataset and cross-domain evaluations, and the public code and checkpoint availability. However, the manuscript currently has a dimension-consistency gap in the core fusion equations and an ambiguity about whether training-time VSDs are generated from ground-truth captions; these issues must be resolved before the empirical claims can be accepted.

major comments (3)

[II-B, Eq. (4)-(5), Eq. (7)] The central fusion operation in Eq. (4) requires v and \tilde d_vsd to have the same dimensionality, and Eq. (5) requires t and \tilde t to do the same, but the paper never states or constructs such a shared space. BGE-large-v1.5 outputs 1024-dimensional embeddings, while OpenCLIP ViT-B/32 produces 512-dimensional features and GPO/HREM use their own pre-extracted feature spaces. No projection layer or dimension-alignment step is described in Section II or Section III-B, so Eq. (4), (5), and (7) are dimensionally ill-defined as written. The released code may implement an undocumented projection; the manuscript should describe it explicitly or the equations must be revised.
[II-B (two-stage generation), II-C, Tables I-III] The training-time VSD generation uses the ground-truth caption set C in Eq. (1), whereas the two-stage strategy described for the test phase first generates captions \hat C from the image and then uses those instead. The text does not state that the same image-only pipeline is used during training. This is load-bearing because \tilde d_vsd is fused into the image representation \hat v; if ground-truth captions are seen at training, the image side inherits lexical and semantic information from the text annotations it will be matched against, and the train/test distribution of VSD inputs differs. The authors should specify the exact procedure used for training VSD generation and, ideally, ablate ground-truth captions versus generated captions at training to quantify the effect on Tables I-III.
[Table IV, III-E] The ablation table is inconsistent with the main results. The GPO row in Table IV reports 68.0/89.2/93.7 (I2T) and 50.0/77.0/84.9 (T2I), which are the cross-dataset numbers from Table II (GPO trained on MSCOCO and tested on Flickr30K), not the in-domain GPO results from Table I (81.7/95.4/97.6 and 61.4/85.9/91.5). The caption says 'average results' but the numbers are not averaged over the two retrieval directions. As a result, the per-module contributions described in Section III-E cannot be assessed against the main experimental setup. Please rerun the ablation on the same protocol as Table I, or clearly relabel the table as a cross-dataset ablation.

minor comments (5)

[I] In Section I, 'MLLMs provide semantic anchor' should be 'semantic anchors'; similar grammatical issues appear throughout the manuscript and should be corrected.
[III-D] In Section III-D, 'improved the rSum by 70.3%' should be 'by 70.3 points' or 'by 15.2% relative', since an absolute increase of 70.3 in rSum is not a percentage.
[II-B] The notation in Eq. (2) and Eq. (3) uses \Ovsd and \tilde d_vsd inconsistently; please use consistent bold or non-bold notation for vectors and clarify the mean-pooling operation.
[References] Reference [15] misspells 'Muennighoff' as 'Muennighof', and Reference [14] should be checked for the correct version and venue information.
[III] The paper reports single-run results without variance or multiple seeds; given that MLLM generation is stochastic and the reported gains are often a few R@1 points, reporting standard deviations or seeds would strengthen the empirical claims.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports empirical retrieval gains, and its equations define a learned fusion and alignment pipeline rather than deriving the result from its own inputs by construction.

full rationale

The claimed result is an empirical performance improvement on Flickr30K, MSCOCO, and zero-shot transfer sets, not a mathematical derivation from first principles. Equations (4)-(8) define a gated fusion and prototype-alignment loss using MLLM-generated descriptions, BGE encodings, and learnable parameters; these parameters are trained and then evaluated on held-out test data, so the reported R@K numbers are not equal to the training objective by construction. The paper contains no uniqueness theorem, no load-bearing self-citation chain, and no fitted parameter that is later renamed as a prediction. The strongest adjacent concern is that training-time descriptions in Eq. (1) use ground-truth captions, which could leak caption information into the image representation; however, the paper explicitly designs a two-stage test-time generation strategy to reduce that distributional gap, and leakage is a methodological validity concern rather than definitional circularity. Similarly, the dimensional mismatch between CLIP and BGE embeddings in Eqs. (4) and (7) is an implementation-correctness issue, not a circular-reasoning issue. Because the central claim rests on external benchmarks and the equations do not reduce to their inputs, no circular step is identified.

Assumptions & free parameters 4 free parameters · 2 assumptions · 1 invented entities

The central method depends on an unstated dimensionality match between image features and BGE-encoded descriptions, plus distributional consistency between training and test VSD generation. The prototype count k and temperature tau are hand-chosen hyperparameters, and the triplet margin gamma is not reported.

free parameters (4)

prototype count k = 896 (Flickr30K), 2560 (MSCOCO)
Set by hand in Sec. III.B; no sensitivity analysis, yet it determines the granularity of the prototype-level alignment loss.
PSA temperature tau = 0.1
Chosen in Sec. III.B; no ablation over this hyperparameter is reported.
triplet margin gamma = not reported
The hard-negative triplet loss in Eq. (6) requires a margin, but its value is never stated in the implementation details.
gating parameters Wv, bv, Wt, bt = learned
The gating weights in Eqs. (4)-(5) are learned during training; they are standard trainable parameters but are nonetheless free.

assumptions (2)

domain assumption Image features and BGE-encoded description features have the same dimensionality d and live in a compatible vector space.
Eq. (4) concatenates [v, d_vsd] and applies gated fusion without any projection layer. BGE-large-v1.5 outputs 1024-d while CLIP ViT-B/32 is 512-d, so this compatibility is not automatic.
domain assumption Training-time and test-time visual semantic descriptions are distributionally consistent.
Sec. II.B states a two-stage generation strategy to ensure consistency, but does not clearly state whether training uses ground-truth captions or MLLM-generated captions as the caption input. If training uses ground-truth captions and test uses generated captions, a distribution shift is introduced.

invented entities (1)

None
purpose: No new physical or conceptual entities are introduced.
The prototypes are learned clusters of description vectors, not new entities; the MLLM and BGE are existing external models.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Visual Semantic Description Generation with MLLMs for Image-Text Matching." pith.science (2026). https://pith.science/paper/PW2YMRSA

@misc{pith2026250708590,
  author       = {Pith},
  title        = {Pith review of: Visual Semantic Description Generation with MLLMs for Image-Text Matching},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PW2YMRSA}},
  note         = {Machine review of arXiv:2507.08590}
}

read the original abstract

Image-text matching (ITM) aims to address the fundamental challenge of aligning visual and textual modalities, which inherently differ in their representations, continuous, high-dimensional image features vs. discrete, structured text. We propose a novel framework that bridges the modality gap by leveraging multimodal large language models (MLLMs) as visual semantic parsers. By generating rich Visual Semantic Descriptions (VSD), MLLMs provide semantic anchor that facilitate cross-modal alignment. Our approach combines: (1) Instance-level alignment by fusing visual features with VSD to enhance the linguistic expressiveness of image representations, and (2) Prototype-level alignment through VSD clustering to ensure category-level consistency. These modules can be seamlessly integrated into existing ITM models. Extensive experiments on Flickr30K and MSCOCO demonstrate substantial performance improvements. The approach also exhibits remarkable zero-shot generalization to cross-domain tasks, including news and remote sensing ITM. The code and model checkpoints are available at https://github.com/Image-Text-Matching/VSD.

Figures

Figures reproduced from arXiv: 2507.08590 by the authors.

**Figure 2.** Qualitative comparison results of bidirectional retrieval [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 23 canonical work pages

[1]

Faster r-cnn: Towards real-time object detection with region proposal networks,

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in NeurIPS, 2015, vol. 28

work page 2015
[2]

Stacked cross attention for image-text matching,

Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He, “Stacked cross attention for image-text matching,” in ECCV, 2018, pp. 201–216

work page 2018
[3]

Image- text embedding learning via visual and textual semantic reasoning,

Kunpeng Li, Yulun Zhang, Kai Li, Yuanyuan Li, and Yun Fu, “Image- text embedding learning via visual and textual semantic reasoning,” TPAMI, vol. 45, no. 1, pp. 641–656, 2022

work page 2022
[4]

At- tentive mask clip,

Yifan Yang, Weiquan Huang, Yixuan Wei, Houwen Peng, et al., “At- tentive mask clip,” in ICCV, 2023, pp. 2771–2781

work page 2023
[5]

Composing object relations and attributes for image-text matching,

Khoi Pham, Chuong Huynh, Ser-Nam Lim, and Abhinav Shrivastava, “Composing object relations and attributes for image-text matching,” in CVPR, 2024, pp. 14354–14363

work page 2024
[6]

Learning transferable visual models from natural language supervi- sion,

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, et al., “Learning transferable visual models from natural language supervi- sion,” in ICML. PMLR, 2021, pp. 8748–8763

work page 2021
[7]

Sigmoid loss for language image pre-training,

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer, “Sigmoid loss for language image pre-training,” in ICCV, 2023, pp. 11975–11986

work page 2023
[8]

Regionclip: Region-based language-image pretraining,

Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, et al., “Regionclip: Region-based language-image pretraining,” inCVPR, 2022, pp. 16793–16803

work page 2022

Show all 27 references

[9]

Sclip: Rethinking self-attention for dense vision-language inference,

Feng Wang, Jieru Mei, and Alan Yuille, “Sclip: Rethinking self-attention for dense vision-language inference,” in ECCV. Springer, 2025, pp. 315– 332

2025
[10]

Improving clip training with language rewrites,

Lijie Fan, Dilip Krishnan, Phillip Isola, Dina Katabi, and Yonglong Tian, “Improving clip training with language rewrites,” in NeurIPS, 2024, vol. 36

2024
[11]

Improving multimodal datasets with image captioning,

Thao Nguyen, Samir Yitzhak Gadre, Gabriel Ilharco, Sewoong Oh, and Ludwig Schmidt, “Improving multimodal datasets with image captioning,” in NeurIPS, 2024, vol. 36

2024
[12]

Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” in ICML. PMLR, 2023, pp. 19730–19742

2023
[13]

Mllms-augmented visual- language representation learning,

Yanqing Liu, Kai Wang, Wenqi Shao, Ping Luo, Yu Qiao, Mike Zheng Shou, Kaipeng Zhang, and Yang You, “Mllms-augmented visual- language representation learning,” arXiv preprint arXiv:2311.18765 , 2023

2023 arXiv
[14]

Minicpm-v: A gpt-4v level mllm on your phone,

Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, et al., “Minicpm-v: A gpt-4v level mllm on your phone,” arXiv preprint arXiv:2408.01800, 2024

2024 arXiv
[15]

C- pack: Packaged resources to advance general chinese embedding,

Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighof, “C- pack: Packaged resources to advance general chinese embedding,” arXiv preprint arXiv:2309.07597, 2023

2023 arXiv
[16]

Fine-grained image- text matching by cross-modal hard aligning network,

Zhengxin Pan, Fangyu Wu, and Bailing Zhang, “Fine-grained image- text matching by cross-modal hard aligning network,” in CVPR, 2023, pp. 19275–19284

2023
[17]

Fast, accurate, and lightweight memory-enhanced embedding learning framework for image-text retrieval,

Zhe Li, Lei Zhang, Kun Zhang, Yongdong Zhang, and Zhendong Mao, “Fast, accurate, and lightweight memory-enhanced embedding learning framework for image-text retrieval,” TCSVT, 2024

2024
[18]

Learning the best pooling strategy for visual semantic embedding,

Jiacheng Chen, Hexiang Hu, Hao Wu, Yuning Jiang, and Changhu Wang, “Learning the best pooling strategy for visual semantic embedding,” in CVPR, 2021, pp. 15789–15798

2021
[19]

Learning semantic relationship among instances for image-text matching,

Zheren Fu, Zhendong Mao, Yan Song, and Yongdong Zhang, “Learning semantic relationship among instances for image-text matching,” in CVPR, 2023, pp. 15159–15168

2023
[20]

Sinkhorn distances: Lightspeed computation of optimal transport,

Marco Cuturi, “Sinkhorn distances: Lightspeed computation of optimal transport,” in NeurIPS, 2013, vol. 26

2013
[21]

Deep visual-semantic alignments for generating image descriptions,

Andrej Karpathy and Li Fei-Fei, “Deep visual-semantic alignments for generating image descriptions,” in CVPR, 2015, pp. 3128–3137

2015
[22]

N24news: A new dataset for multimodal news classification,

Zhen Wang, Xu Shan, Xiangxie Zhang, and Jie Yang, “N24news: A new dataset for multimodal news classification,” arXiv preprint arXiv:2108.13327, 2021

2021 arXiv
[23]

Exploring a fine-grained multiscale method for cross-modal remote sensing image retrieval,

Zhiqiang Yuan, Wenkai Zhang, Kun Fu, Xuan Li, Chubo Deng, Hongqi Wang, and Xian Sun, “Exploring a fine-grained multiscale method for cross-modal remote sensing image retrieval,” TGRS, vol. 60, pp. 1–19, 2021

2021
[24]

Florence-2: Advancing a unified representation for a variety of vision tasks,

Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan, “Florence-2: Advancing a unified representation for a variety of vision tasks,” in CVPR, 2024, pp. 4818–4829

2024
[25]

Align before fuse: Vision and language representation learning with momentum distillation,

Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi, “Align before fuse: Vision and language representation learning with momentum distillation,” in NeurIPS, 2021, vol. 34, pp. 9694–9705

2021
[26]

Remote sensing cross-modal text-image retrieval based on global and local information,

Zhiqiang Yuan, Wenkai Zhang, Changyuan Tian, Xuee Rong, Zhengyuan Zhang, Hongqi Wang, Kun Fu, and Xian Sun, “Remote sensing cross-modal text-image retrieval based on global and local information,” TGRS, vol. 60, pp. 1–16, 2022

2022
[27]

Hypersphere-based remote sensing cross-modal text-image retrieval via curriculum learning,

Weihang Zhang, Jihao Li, Shuoke Li, Jialiang Chen, Wenkai Zhang, Xin Gao, and Xian Sun, “Hypersphere-based remote sensing cross-modal text-image retrieval via curriculum learning,” TGRS, 2023

2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Faster r-cnn: Towards real-time object detection with region proposal networks,

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in NeurIPS, 2015, vol. 28

work page 2015

[2] [2]

Stacked cross attention for image-text matching,

Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He, “Stacked cross attention for image-text matching,” in ECCV, 2018, pp. 201–216

work page 2018

[3] [3]

Image- text embedding learning via visual and textual semantic reasoning,

Kunpeng Li, Yulun Zhang, Kai Li, Yuanyuan Li, and Yun Fu, “Image- text embedding learning via visual and textual semantic reasoning,” TPAMI, vol. 45, no. 1, pp. 641–656, 2022

work page 2022

[4] [4]

At- tentive mask clip,

Yifan Yang, Weiquan Huang, Yixuan Wei, Houwen Peng, et al., “At- tentive mask clip,” in ICCV, 2023, pp. 2771–2781

work page 2023

[5] [5]

Composing object relations and attributes for image-text matching,

Khoi Pham, Chuong Huynh, Ser-Nam Lim, and Abhinav Shrivastava, “Composing object relations and attributes for image-text matching,” in CVPR, 2024, pp. 14354–14363

work page 2024

[6] [6]

Learning transferable visual models from natural language supervi- sion,

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, et al., “Learning transferable visual models from natural language supervi- sion,” in ICML. PMLR, 2021, pp. 8748–8763

work page 2021

[7] [7]

Sigmoid loss for language image pre-training,

Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer, “Sigmoid loss for language image pre-training,” in ICCV, 2023, pp. 11975–11986

work page 2023

[8] [8]

Regionclip: Region-based language-image pretraining,

Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, et al., “Regionclip: Region-based language-image pretraining,” inCVPR, 2022, pp. 16793–16803

work page 2022