Pith. sign in

REVIEW 4 major objections 4 minor 45 references

VL-CLIP: Enhancing Multimodal Recommendations via Visual Grounding and LLM-Augmented CLIP Embeddings

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VL-CLIP claims that cropping product images to the object and rewriting descriptions with an LLM improves CLIP-based retrieval and raises Walmart CTR by 18.6%, ATC by 15.5%, and GMV by 4.0%.

desk verdict Real pipeline, real ablation, but inflated headline numbers because baselines likely weren't fine-tuned and the A/B control is undescribed. read the letter →

arxiv 2507.17080 v1 pith:BYROYUKB submitted 2025-07-22 cs.IR cs.AIcs.CV

Ramin Giahi , Kehui Yao , Sriram Kollipara , Kai Zhao , Vahid Mirjalili , Jianpeng Xu , Topojoy Biswas , Evren Korpeoglu
show 1 more author
Kannan Achan
This is my paper
classification cs.IRcs.AIcs.CV
keywords multimodalrecommendationCLIPfine-tuningvisualgroundingLLMtextrefinemente-commerceretrievalcontrastivelearningHNSWindexingonlineA/Btest
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VL-CLIP is a recipe for upgrading CLIP, a standard model that learns shared image-text embeddings, for e-commerce recommendations. The paper claims that two preprocessing changes before contrastive fine-tuning make the embeddings align much better: cropping each product image to the object itself with a visual grounding detector, and rewriting noisy catalog descriptions into short, visually checkable queries with an LLM. Trained on 7 million Walmart fashion and home products, the model beats CLIP, FashionCLIP, and GCL on retrieval metrics and, in a four-week online A/B test, raises click-through rate by 18.6%, add-to-cart rate by 15.5%, and gross merchandise value by 4.0%. If the claims hold, object-level visual focus and text disambiguation, not a new model architecture, are what CLIP needs for commerce.

What carries the argument

Three components carry the argument. Visual grounding uses Grounding DINO, a zero-shot object detector, to select the highest-confidence bounding box for the product type (for example 'dress') and crop to that region, falling back to the full image when confidence is low; this makes the vision encoder attend to product attributes instead of backgrounds. The LLM text agent runs a Summarizer followed by up to five Evaluator/Refiner iterations, editing for attribute consistency, conciseness, and visual discernibility, then embeds the final query with CLIP's text encoder. Symmetric InfoNCE contrastive loss then fine-tunes the CLIP encoders on the cleaned pairs. For deployment, perceptual hashing removes duplicate images and product-type-grouped HNSW indexes provide approximate nearest-neighbor retrieval at low latency.

What would settle it

Re-run the paper's retrieval comparisons on the public Google Shopping benchmark with CLIP, FashionCLIP, and GCL all fine-tuned on the identical 7-million-item Walmart subset with identical queries and candidate pools. If VL-CLIP's reported HITS@5 lead over FashionCLIP (0.6758 versus 0.4428 on the fashion set) shrinks to a small margin, the claimed gains are an artifact of evaluation setup rather than of visual grounding and LLM text refinement.

Watch

Extended reading notes

Core claim

The central discovery is that much of CLIP's failure in e-commerce is an input problem rather than an architecture problem. Global image embeddings dilute fine-grained product detail, and raw product text is inconsistent or full of non-visual claims, so image-text alignment suffers even after fine-tuning. VL-CLIP therefore cleans both sides before contrastive training: Grounding DINO detects the product region from a metadata-derived prompt and crops it for the CLIP vision encoder, and an LLM summarizer-evaluator-refiner loop converts raw metadata into a concise query that keeps only visually discernible attributes. The cleaned image-text pairs are then used to fine-tune a ViT-B/32 CLIP model with symmetric contrastive loss. On the paper's evaluation, this raises HITS@5 from 0.4428 to 0.6758 on the fashion set and from 0.4227 to 0.6692 on the home set, with the online A/B test reporting 18.6% higher CTR, 15.5% higher add-to-cart rate, and 4.0% higher GMV.

Load-bearing premise

The load-bearing premise is that every baseline was given the same chance: CLIP, GCL, and FashionCLIP are assumed to have been fine-tuned on the same 7 million Walmart products, evaluated on identical query pools and retrieval candidates, and compared online against a competitive production model, but the paper does not state these conditions.

Editorial extensions

If this is right

  • Retrieval systems that currently embed full product photos with CLIP can improve fine-grained matching simply by cropping to the detected product region before embedding.
  • LLM-generated structured queries can replace raw titles and descriptions as CLIP text inputs, improving alignment without changing the model architecture or the training data.
  • Because the online gains are large, the results imply that the top few recommended items' visual and textual coherence strongly drives clicks and purchases in real users.
  • The reported zero-shot transfer to Art, Toys, and Google Shopping suggests the pipeline's benefit is not specific to fashion and home categories.
  • The latency reduction from product-type-grouped HNSW indexing (81% versus IVF in the paper) makes the improved embeddings practical for real-time recommendations at catalog scale.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the ablations suggest visual grounding contributes more than LLM query refinement (removing grounding cost 15.34% HITS@5, removing the LLM step cost 7.40% more), so a cheaper deployment that only adds cropping may capture most of the benefit.
  • Editorial extension: the LLM step's value likely depends on catalog metadata quality; platforms with already-structured, concise titles would see smaller text-side gains, while marketplaces with seller-generated noisy listings would see larger ones.
  • Editorial extension: the same input-cleaning recipe could transfer to other image-text domains with background clutter and inconsistent text, such as real-estate listings, auto parts, or second-hand goods, without retraining from scratch.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes VL-CLIP, a pipeline that adapts CLIP to e-commerce multimodal retrieval by (1) using Grounding DINO to crop product-centric image regions, (2) using an LLM-based Summarizer/Evaluator/Refiner loop to rewrite product descriptions into concise, visually grounded queries, and (3) contrastively fine-tuning CLIP on 7 million Walmart.com fashion and home products. The authors report large offline retrieval gains over CLIP, GCL, and FashionCLIP on Walmart data and a Google Shopping dataset, component ablations, zero-shot attribute classification, VLM-as-judge precision evaluations, cross-domain zero-shot results, and a four-week online A/B test with CTR +18.6%, ATC +15.5%, and GMV +4.0%. The central claim is that combining visual grounding and LLM-augmented text embeddings substantially improves retrieval and recommendation quality at industrial scale.

Significance. If the claims are substantiated, the work is practically significant: it demonstrates a deployable integration of visual grounding and LLM-based text enrichment into a CLIP-based retrieval stack at Walmart scale, with real-user engagement metrics. The ablation study is internally consistent in direction (each component improves HITS@5 and MRR), and the cross-domain results on Google Shopping and on Walmart Art/Toys support transferability. The paper is less strong on the scientific side: no code or data is released, the offline baselines are not shown to be comparably trained, the online control is undescribed, and the LLM-as-judge evaluation risks circularity. These gaps currently prevent the advertised improvements from being attributed to the proposed method with confidence.

major comments (4)
  1. [Section 4.3, Table 1 vs Section 4.4, Table 2] The paper does not state whether CLIP, GCL, and FashionCLIP were fine-tuned or adapted on the same 7M Walmart.com products before the Table 1 comparison. The reported CLIP row (0.3080 Fashion / 0.2355 Home HITS@5) is consistent with a frozen pretrained model, whereas VL-CLIP is fine-tuned on 7M domain products. The paper's own 'VL-CLIP w/o GD, LLM' row in Table 2 (0.4484 Fashion / 0.4418 Home HITS@5) is a same-data, same-procedure CLIP fine-tune without the proposed components; against this more appropriate control, the HITS@5 advantage drops from the advertised 119.42%/184.16% to roughly 51% on both categories. The components still help, so this is a reporting/fairness gap rather than a refutation, but as written the headline retrieval claim is supported only against weaker, likely zero-shot baselines. Please state the training protocol for each baseline or re-run Table 1 with comparably fine-tuned baselines.
  2. [Section 4.9, Table 7] The online A/B test compares VL-CLIP against an undescribed 'deployed baseline model.' Without a description of the control model (whether it used CLIP embeddings, what retrieval and ranking stack it used, and whether it was production-tuned), the reported 18.6% CTR, 15.5% ATC, and 4.0% GMV gains cannot be attributed to the VL-CLIP components specifically; they could partly reflect other changes in the pipeline, such as the HNSW index or product-type partitioning described in Section 3.4. Please specify the baseline architecture, its training data, and the exact treatment/control diff.
  3. [Section 4.6 and Appendix E] The VLM-as-judge evaluation is self-referential: the candidate text encoder is trained on LLM-refined queries (Section 3.2), and the evaluation uses LLM-generated queries whose relevance is judged by GPT-4o for visual match. This shared text style and attribute vocabulary can systematically favor VL-CLIP over baselines whose text encoders were not exposed to LLM-style queries. Tables 4, 6, and 13 therefore need a human-annotated subset (with inter-annotator agreement) or an independently sourced query set to establish that the precision gains reflect retrieval quality rather than judge-query alignment.
  4. [Section 4.3 and Tables 1-6] No confidence intervals, standard errors, or significance tests are reported for any offline retrieval result. Given the evaluation sets are 10,000 samples per category, the claimed 'significant improvements' are not statistically supported as reported. Please include confidence intervals or significance tests, especially for the smaller gaps in Table 5 (VL-CLIP vs GCL) and the zero-shot classification results in Table 3 (e.g., GCL pattern accuracy 0.785 vs FashionCLIP 0.934).
minor comments (4)
  1. [Section 4.4, Table 2] The text says removing Visual Grounding causes an average drop of 15.34% in HITS@5 and 11.23% in MRR; recomputing from the table gives approximately 21.5% / 24.2% HITS@5 drops on Fashion and Home, respectively, and larger MRR drops. The follow-up statement that removing LLM refinement 'further reduces performance by 7.40%' also does not match the table. Please correct the arithmetic or define the averaging base explicitly.
  2. [Algorithm 1 in Appendix C] The loop increments i both in the else branch (line 21) and via the for statement, so the loop steps by 2 when refinement continues; this is either a bug or a typo and should be fixed.
  3. [Appendix E.2] The VLM-as-judge evaluation uses GPT-4o, but the exact model version, inference hyperparameters, and prompt response parsing are not reported, which limits reproducibility of the precision numbers in Tables 4, 6, and 13.
  4. [Section 4.7] The sentence 'Retrieval performance, measured by Recall@10 indicating that the model effectively identified relevant items in the top-10 results for 47% of queries' is grammatically incomplete and should be rewritten to state both the exact Recall@10 value and the epoch at which it was measured.

Circularity Check

1 steps flagged · score 4.0 of 10

Partial circularity in the VLM-agent evaluation: the LLM-refined text encoder is tested on LLM-generated queries with LLM-assigned relevance labels, so part of the reported gain measures fit to LLM text style rather than independent visual-text alignment.

  1. other [Section 4.6 and Appendix E.1/E.2]
    "Since the alignment of text and image information is very subjective, we employ a VLM agent for evaluation... The VLM model is asked to assess whether the provided image accurately matches the given query, producing a binary output of 0 (no match) or 1 (match). ... Query generation: We utilize an LLM to generate search queries from extracted attributes. ... Relevance assessment: ... measured by a VLM (GPT-4o), assigning a binary relevance score."

    The pipeline's text encoder is fine-tuned on queries produced by the LLM Summarizer/Refiner (Section 3.2: 'the LLM agent enhances textual features... q_final is embedded by phi_CLIP-text'). The VLM-agent evaluation then generates test queries with an LLM from VLM-extracted attributes and labels matches with GPT-4o. Thus the test query distribution and the ground-truth labels are both outputs of the same model family that generated the training text. Precision@1/3/5 therefore measures, in part, how well the encoder reproduces LLM query style that the LLM judge recognizes as 'matching'; it is not an externally grounded human relevance measurement. This is a self-referential evaluation loop rather than independent validation.

full rationale

The paper contains no mathematical derivation whose output is an input restated, no fitted constant renamed as a prediction, and no load-bearing self-citation: reference [17] is a related-work citation to the authors' earlier Walmart study and is not used to justify any result. Table 1's HITS@5/MRR use dataset ground-truth item matches, so the central retrieval claim is not circular. The circular element is confined to Section 4.6/Appendix E: the text encoder is trained on LLM-refined queries (Section 3.2), while the evaluation queries are generated by an LLM from VLM-extracted attributes and the relevance labels are assigned by GPT-4o. The judged 'match' is therefore constructed by the same class of model whose text distribution the encoder was optimized to imitate, making the Precision@1/3/5 numbers a self-referential measure. Table 1 baselines appear to be zero-shot (e.g., CLIP at 0.3080 Fashion HITS@5) while VL-CLIP is fine-tuned on 7M products; this is a fairness and reporting-opacity concern, not a circularity concern. Overall, the central content is independent, so score 4.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The framework does not introduce new physical or mathematical entities. It relies on several domain assumptions about its off-the-shelf components (Grounding DINO, LLM, CLIP, HNSW) and makes small hand-chosen design choices (thresholds, iteration counts, few-shot examples) that are not varied in a sensitivity analysis.

free parameters (5)
  • Grounding DINO confidence threshold tau_thresh = not specified numerically
    Images whose top box falls below this threshold are left uncropped; the value is chosen by the authors and not reported, and it directly determines how many images are cropped.
  • Maximum LLM refinement iterations i_max = 5
    Set empirically in Section 3.2 to balance convergence and cost; the final query depends on how many refinement steps run.
  • Early stopping epoch = 6
    Selected by validation Recall@10 in Section 4.7; the deployed model checkpoint is defined by this choice.
  • Evaluation product type subset = 17 types (10 fashion, 7 home)
    Offline evaluation is built on a hand-chosen subset of product types with 10,000 samples per category; results may not generalize to the full long-tail catalog.
  • LLM in-context examples = curated set, content not fully shown
    The few-shot examples in t_in-context (Section 3.2) are hand-crafted to steer Summarizer/Evaluator/Refiner behavior; they are part of the method's tuning and affect generated queries.
assumptions (5)
  • domain assumption Grounding DINO localizes the main product using a product-type text prompt with sufficient accuracy across e-commerce images.
    Invoked in Section 3.1; if the top box is wrong or misses the product, the cropped image harms alignment.
  • domain assumption LLM-refined queries preserve the visually relevant information and remove only noise; the summarized text is a better training signal than raw metadata.
    Section 3.2 assumes the Evaluator/Refiner loop improves query quality; if it discards important attributes, the text encoder is trained on incomplete descriptions.
  • domain assumption VLM-as-judge (GPT-4o) binary relevance scores align with human judgments for query-item and item-item similarity.
    Section 4.6 and Appendix E use GPT-4o as ground truth; no human agreement study is reported.
  • domain assumption A symmetric InfoNCE contrastive loss is an appropriate objective for aligning product images and refined text in the e-commerce domain.
    Section 3.3; standard CLIP objective, but its suitability for the noisy long-tail catalog is assumed.
  • domain assumption HNSW indexing with cosine similarity and product-type grouping preserves the ranking quality measured offline.
    Section 3.4; approximate retrieval may change top-k results compared to exact search, and this is not validated against exact search.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VL-CLIP: Enhancing Multimodal Recommendations via Visual Grounding and LLM-Augmented CLIP Embeddings." pith.science (2026). https://pith.science/paper/BYROYUKB

@misc{pith2026250717080,
  author       = {Pith},
  title        = {Pith review of: VL-CLIP: Enhancing Multimodal Recommendations via Visual Grounding and LLM-Augmented CLIP Embeddings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BYROYUKB}},
  note         = {Machine review of arXiv:2507.17080}
}
read the original abstract

Multimodal learning plays a critical role in e-commerce recommendation platforms today, enabling accurate recommendations and product understanding. However, existing vision-language models, such as CLIP, face key challenges in e-commerce recommendation systems: 1) Weak object-level alignment, where global image embeddings fail to capture fine-grained product attributes, leading to suboptimal retrieval performance; 2) Ambiguous textual representations, where product descriptions often lack contextual clarity, affecting cross-modal matching; and 3) Domain mismatch, as generic vision-language models may not generalize well to e-commerce-specific data. To address these limitations, we propose a framework, VL-CLIP, that enhances CLIP embeddings by integrating Visual Grounding for fine-grained visual understanding and an LLM-based agent for generating enriched text embeddings. Visual Grounding refines image representations by localizing key products, while the LLM agent enhances textual features by disambiguating product descriptions. Our approach significantly improves retrieval accuracy, multimodal retrieval effectiveness, and recommendation quality across tens of millions of items on one of the largest e-commerce platforms in the U.S., increasing CTR by 18.6%, ATC by 15.5%, and GMV by 4.0%. Additional experimental results show that our framework outperforms vision-language models, including CLIP, FashionCLIP, and GCL, in both precision and semantic alignment, demonstrating the potential of combining object-aware visual grounding and LLM-enhanced text representation for robust multimodal recommendations.

Figures

Figures reproduced from arXiv: 2507.17080 by the authors.

Figure 1
Figure 1. Illustration of: (a) visual recommendation improvement achieved by our proposed model, VL-CLIP and (b) visual [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. VL-CLIP model architecture approach consists of three main components: Summarization, Eval￾uation, and Refinement. Given a raw textual input consisting of both structured and unstructured product information, we first construct an initial con￾catenated metadata representation as 𝑡concat. 𝑡concat = [𝑡𝑝 ∥ 𝑡𝑔 ∥ 𝑡raw ∥ 𝑡in-context] where 𝑡𝑝 denotes the product type (e.g., “t-shirt,” “handbag”), 𝑡𝑔 rep￾resents age and ge… view at source ↗
Figure 3
Figure 3. Visualization of product summary generator [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: The validation loss and Recall@10 over epochs [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Examples of similar item recommendations for fashion products based on VL-CLIP. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Examples of query-based retrieval for fashion items: the first column is the query, the rest are top 5 recommended [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Examples of query-based retrieval for home items: the first column is the query, the rest are top 5 recommended [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Examples of similar item recommendation for home items: the first column is the anchor item, the rest are top [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Query-based evaluation process using VLM [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 25 canonical work pages

  1. [1]

    Henning, Karun Singh, Omkar Parkhi, and Fedor Borisyuk

    Sean Bell, Yiqun Liu, Sami Alsheikh, Yina Tang, Edward Pizzi, M. Henning, Karun Singh, Omkar Parkhi, and Fedor Borisyuk. 2020. GrokNet: Unified Computer Vision Model Trunk and Embeddings For Commerce. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (Virtual Event, CA, USA) (KDD ’20). Association for Comp...

    arXiv 2020

  2. [2]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901

    2020

  3. [3]

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. UNITER: UNiversal Image-TExt Representa- tion Learning. In Computer Vision – ECCV 2020 , Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer International Publishing, Cham, 104–120

    work page 2020

  4. [4]

    Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev

  5. [5]

    Patrick John Chia, Giuseppe Attanasio, Federico Bianchi, Silvia Terragni, Ana Rita Magalh aes, Diogo Goncalves, Ciro Greco, and Jacopo Tagliabue. 2022. Con- trastive language and Vision Learning of General Fashion Concepts. Scientific Reports 12, 1 (2022), 18958. https://doi.org/10.1038/s41598-022-23052-9

    work page doi:10.1038/s41598-022-23052-9 2022

  6. [6]

    Eden Dolev, Alaa Awad, Denisa Olteanu Roberts, Zahra Ebrahimzadeh, Marcin Mejran, Vaibhav Malpani, and Mahir Yavuz. 2025. Efficient Large-Scale Vi- sual Representation Learning and Evaluation. In Revolutionizing Fashion and Retail, Nima Dokoohaki, Julia Laserre, and Reza Shirvany (Eds.). Springer Nature Switzerland, Cham, 97–111

    work page 2025

  7. [7]

    Rian Dolphin, Barry Smyth, and Ruihai Dong. 2023. A Machine Learning Ap- proach to Industry Classification in Financial Markets. In Artificial Intelligence and Cognitive Science, Luca Longo and Ruairi O’Reilly (Eds.). Springer Nature Switzerland, Cham, 81–94

    work page 2023

  8. [8]

    Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. 2022. AudioCLIP: Extending Clip to Image, Text and Audio. InICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . 976–980. https: //doi.org/10.1109/ICASSP43922.2022.9747631

    arXiv 2022

Show all 45 references
  1. [9]

    Huang, O

    H. Huang, O. Zheng, D. Wang, et al. 2023. ChatGPT for Shaping the Future of Dentistry: The Potential of Multi-modal Large Language Model. International Journal of Oral Science 15 (2023), 29. https://doi.org/10.1038/s41368-023-00239-y

    2023 doi

  2. [10]

    Yang Jin, Yongzhi Li, Zehuan Yuan, and Yadong Mu. 2023. Learning instance- level representation for large-scale multi-modal pretraining in e-commerce. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11060–11069

    2023

  3. [11]

    Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2020. What Does BERT with Vision Look At?. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics , Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (Eds...

    2020 doi

  4. [12]

    Xiang Li, Congcong Wen, Yuan Hu, and Nan Zhou. 2023. RS-CLIP: Zero-shot Re- mote Sensing Scene Classification via Contrastive Cision-Language Supervision. International Journal of Applied Earth Observation and Geoinformation 124 (2023), 103497. https://doi.org/10.1016/j.jag.20...

    2023

  5. [13]

    Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao

  6. [14]

    Dong Liu and Esther Lopez Ramos. 2025. Multimodal Semantic Retrieval for Product Search. In Companion Proceedings of the ACM on Web Conference 2025 (Sydney NSW, Australia) (WWW ’25) . Association for Computing Machinery, New York, NY, USA, 2170–2175. https://doi.org/10.1145/37...

    2025

  7. [15]

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, and Lei Zhang. 2024. Grounding DINO: Marrying DINO with Grounded Pre-training for Open-Set Object Detection. In Computer Vision – ECCV 2024: 18th Euro...

    2024 doi

  8. [16]

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. ViLBERT: Pretrain- ing Task-agnostic Visiolinguistic Representations for Vision-and-Language Tasks . Curran Associates Inc., Red Hook, NY, USA

    2019

  9. [17]

    Luyi Ma, Xiaohan Li, Zezhong Fan, Kai Zhao, Jianpeng Xu, Jason Cho, Praveen Kanumala, Kaushiki Nag, Sushant Kumar, and Kannan Achan. 2024. Triple modality fusion: Aligning visual, textual, and graph data with large language models for multi-behavior recommendations. arXiv prep...

    2024 arXiv

  10. [18]

    Yiwei Ma, Guohai Xu, Xiaoshuai Sun, Ming Yan, Ji Zhang, and Rongrong Ji. 2022. X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval. In Proceedings of the 30th ACM International Conference on Multimedia (Lisboa, Portugal) (MM ’22). Association for Com...

    2022

  11. [19]

    Yu A Malkov and Dmitry A Yashunin. 2018. Efficient and Robust Approximate Nearest Neighbor Search using Hierarchical Navigable Small World Graphs.IEEE transactions on pattern analysis and machine intelligence 42, 4 (2018), 824–836

    2018

  12. [20]

    Bertalan Meskó. 2023. The Impact of Multimodal Large Language Models on Health Care’s Future. Journal of Medical Internet Research 25 (2023), e52865. https://doi.org/10.2196/52865

    2023 doi

  13. [21]

    Ron Mokady, Amir Hertz, and Amit H. Bermano. 2021. ClipCap: CLIP Prefix for Image Captioning. arXiv:2111.09734 [cs.CV] https://arxiv.org/abs/2111.09734

    2021 arXiv

  14. [22]

    Ferda Ofli, Firoj Alam, and Muhammad Imran. 2020. Analysis of Social Media Data using Multimodal Deep Learning for Disaster Response. In Proceedings of the 17th International Conference on Information Systems for Crisis Response and Management (ISCRAM). ISCRAM, 802–811

    2020

  15. [23]

    Alec Radford, Jong Wook Kim, Ceyuan Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pam Mishkin, Jack Clark, et al

  16. [24]

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Lud- wig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. 202...

    2022

  17. [25]

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki

  18. [26]

    Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai

  19. [27]

    Hao Tan and Mohit Bansal. 2019. LXMERT: Learning Cross-Modality Encoder Representations from Transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP...

    2019 doi

  20. [28]

    Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. 2021. VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding. In Proceedings of the 2021 Conference on Empirical Methods in Nat...

    2021

  21. [29]

    arXiv preprint arXiv:2111.02114 (2021)

    Laion-400m: Open Dataset of Clip-filtered 400 Million Image-Text Pairs. arXiv preprint arXiv:2111.02114 (2021)

    2021 arXiv

  22. [30]

    Christoph Zauner. 2010. Implementation and Benchmarking of Perceptual Image Hash Functions. (2010)

    2010

  23. [31]

    In International Conference on Learning Representations (ICLR)

    VL-BERT: Pre-training of Generic Visual-Linguistic Representations. In International Conference on Learning Representations (ICLR) . https://openreview. net/forum?id=SygXPaEYvH

  24. [32]

    Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Li- unian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, and Jianfeng Gao

  25. [33]

    Tianyu Zhu, Myong Chol Jung, and Jesse Clark. 2025. Generalized Contrastive Learning for Multi-Modal Retrieval and Ranking. InCompanion Proceedings of the ACM on Web Conference 2025 (Sydney NSW, Australia) (WWW ’25). Association for Computing Machinery, New York, NY, USA, 661–...

    2025

  26. [34]

    Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. 2022. FILIP: Fine-grained In- teractive Language-Image Pre-Training. In International Conference on Learning Representations. https://openreview.net/forum?...

    2022

  27. [36]

    Andrew Zhai, Hao-Yu Wu, Eric Tzeng, Dong Huk Park, and Charles Rosenberg

  28. [41]

    dress”, “rug

    Xinliang Zhu, Sheng-Wei Huang, Han Ding, Jinyu Yang, Kelvin Chen, Tao Zhou, Tal Neiman, Ouye Xie, Son Tran, Benjamin Yao, et al. 2024. Bringing Multimodal- ity to Amazon Visual Search System. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mini...

    2024

  29. [42]

    productName

    Remove the information of [100% polyester]. <Refiner iteration 1> Sleeveless halter neck floral maxi sundress with elastic waist, black, boho style. <Evaluator iteration 2> Score: 5/5 Justification: The summary is concise and captures key visual elements such as the sleeveless...

    2025

  30. [43]

    Add the information of [extendable bow-end table top]

  31. [44]

    Add the information of [solid oak wood crowns on chair and table bases]. <Refiner iteration 1> Seven-piece dining set with extendable bow-end oak wood edge laminate table, steel base, and desert polyurethane caster chairs with oak-finished arms and solid oak wood crowns. <Eval...

  32. [45]

    color”) and𝑣𝑖 is its value (e.g., “ blue

    Shorten the summary. <Refiner iteration 2> Seven-piece dining set with oak wood edge laminate table, steel base, and desert polyurethane caster chairs with oak-finished arms. <Evaluator iteration 3> Score: 5/5 Justification: The summary is concise and captures key visual eleme...

    2025

  33. [2019]

    InProceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining

    Learning a Unified Embedding for Visual Search at Pinterest. InProceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2412–2420

  34. [2020]

    In Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX (Glasgow, United Kingdom)

    Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks. In Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXX (Glasgow, United Kingdom). Springer-Verlag, Berlin, Heidelberg, 121–137. https://doi.org/10....

    2020 doi

  35. [2021]

    In Proceedings of the International Conference on Machine Learning

    Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the International Conference on Machine Learning

  36. [2022]

    In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    RegionCLIP: Region-based Language-Image Pretraining. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . 16772–16782. https://doi.org/10.1109/CVPR52688.2022.01629

    2022

  37. [2023]

    In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    Reproducible Scaling Laws for Contrastive Language-Image Learning. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, 2818–2829. https://doi.org/10.1109/cvpr52729.2023.00276

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.