Pith. sign in

REVIEW 3 major objections 7 minor 47 references

Fine-grained context training, done in two separate stages, turns a 4B MLLM into a stronger zero-shot image retriever than larger models on complex queries.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 22:09 UTC pith:LV6EE4P2

load-bearing objection Solid subfield paper: fine-grained synthetic CIR quintuples plus two-stage MLLM tuning deliver real zero-shot gains on complex retrieval with a 4B backbone; synthetic-mod fidelity is the open hinge, not a collapse. the 3 major comments →

arxiv 2607.27959 v1 pith:LV6EE4P2 submitted 2026-07-30 cs.CV cs.IR

FiRE: Enhancing MLLMs with Fine-Grained Context Learning for Complex Image Retrieval

Bohan Hou , Haoqiang Lin , Xuemeng Song , Haokun Wen , Meng Liu , Yupeng Hu , Xiangyu Zhao This is my paper
classification cs.CV cs.IR
keywords Multimodal Large Language ModelImage RetrievalComplex Image RetrievalFine-grained Context ModelingComposed Image RetrievalZero-shot RetrievalTwo-stage Fine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Multimodal large language models can act as universal image retrievers, but prior fine-tuning either used coarse data or mixed context-learning and ranking objectives in one step, which hurts performance on hard queries such as composed image retrieval, long captions, and visual dialog. This paper builds an automatic pipeline that turns unlabeled images into fine-grained quintuples—reference image, detailed caption, human-like modification text, target image, detailed target caption—and then fine-tunes an MLLM in two clean stages: first teach it to reason about the modified scene by generating the fine-grained target caption, then teach it to rank the right target with contrastive and recall-surrogate losses. The resulting single checkpoint, run on a lightweight 4B backbone, beats both specialized zero-shot CIR systems and larger universal MLLM retrievers across multiple complex retrieval benchmarks while staying competitive on ordinary short-text retrieval. A sympathetic reader cares because the same model, without task-specific heads or extra checkpoints, handles the messy multimodal queries people actually issue.

Core claim

Separating fine-grained context reasoning from retrieval alignment, and training both on automatically built fine-grained multimodal quintuples, produces a single zero-shot MLLM checkpoint that outperforms larger entangled or coarse-tuned retrievers on composed, long-text, and dialog image retrieval.

What carries the argument

FiRE: a two-stage fine-tuning recipe on the FiGMaQ quintuple dataset. Stage 1 uses generative loss so the model writes the fine-grained target caption from reference image plus modification; Stage 2 aligns multimodal query and target embeddings with InfoNCE plus Recall@k surrogate losses.

Load-bearing premise

The automatically written modification texts and captions are close enough to real human intent that training on them transfers cleanly to human-annotated test sets.

What would settle it

Train the identical two-stage recipe on the same backbone but replace FiGMaQ modifications with either purely human-written CIR triplets or deliberately degraded synthetic text; if the zero-shot gains on CIRR, CIRCO, Visual Dialog and Urban1K disappear or reverse, the central claim fails.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • One lightweight MLLM checkpoint can serve as a drop-in universal retriever for composed, long-caption, dialog, and short-text image search without per-task fine-tuning.
  • Future universal retrievers should prefer staged objectives (context first, ranking second) over simultaneous multi-task losses.
  • High-quality synthetic fine-grained quintuples can substitute for expensive human CIR annotation at far smaller scale than prior auto-generated triplet sets.
  • Treating the reference image as a long token sequence, rather than a single global embedding, improves transfer to long-text and dialog queries.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same staged recipe could be applied to video or 3D retrieval where queries are likewise long multimodal contexts.
  • If the vagueness-guided modification generator is the main quality lever, cheaper open LLMs with the same prompt may close most of the gap to the 70B teacher.
  • Recall-surrogate losses may be worth adding to any MLLM embedding fine-tune, not only CIR-style setups.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes FiRE, a two-stage fine-tuning strategy that first trains an MLLM for fine-grained multimodal context reasoning (generate fine-grained target captions from reference image + modification) and then for retrieval (InfoNCE + Recall@k surrogate losses aligning multimodal query and target features), together with an automated pipeline that builds FiGMaQ (~87K quintuples: reference/target images, fine-grained CoT captions, and vagueness-guided modification texts) from unlabeled ImageNet-1K images. Using a single BLIP-3-4B checkpoint fine-tuned only on FiGMaQ, the authors report zero-shot gains over dedicated ZS-CIR methods and larger MLLM universal retrievers (E5-V 8B, MCL up to 7B) on CIRR, CIRCO, FashionIQ, Visual Dialog, and Urban1K, while remaining competitive on COCO/Flickr short-text retrieval. Supporting evidence includes multi-dataset tables, component ablations (Table 5), a Combiner transfer check of FiGMaQ vs MMC/LaSCo (Table 6), and qualitative cases.

Significance. If the results hold under stronger validation of the synthetic data, the work is a solid and practically useful contribution to universal image retrieval with MLLMs: it shows that a lightweight 4B backbone can outperform larger MLLM retrievers and specialized ZS-CIR systems in a one-checkpoint zero-shot setting across several complex query types. The disentangled two-stage objective, the fine-grained quintuple construction pipeline, and the public-scale FiGMaQ resource are concrete assets for follow-on work. Strengths include broad multi-task evaluation, explicit ablations of stage/caption/loss choices, and an external Combiner transfer experiment that partially stress-tests dataset quality beyond the authors’ own training loop.

major comments (3)
  1. [§3, §5.4–5.5, Table 6, Fig. 4] §3–§5 and Table 6: The central zero-shot SOTA claim rests on fine-tuning exclusively on 87K auto-generated FiGMaQ quintuples (CoT captions, MLLM caption-similarity pair filter with θ_l=0.6, θ_h=0.83 in Eqs. 1–3, and LLaMA-3.1-70B modifications under the vagueness-guided few-shot prompt in Fig. 3b). There is no quantitative human evaluation of modification faithfulness, naturalness, difficulty, or distributional match to CIRR/FashionIQ/CIRCO edit styles—only one qualitative example (Fig. 4) and an indirect Combiner transfer result (Table 6). Table 5 ablations vary objectives and caption granularity but hold FiGMaQ fixed, so they cannot separate genuine fine-grained context learning from train–test stylistic alignment with LLM-mediated diffs. A load-bearing addition is needed: human ratings (or side-by-side preference vs real CIR annotations) on a stratified sample of FiGMaQ mods, and/or a
  2. [§5.2–5.3, Tables 1–4] §5.2–5.3, Tables 1–4: Several strong baselines (MagicLens, MCL variants, Long-CLIP, CIReVL/LDRE) are cited from original papers with heterogeneous training data scales, visual encoders, and LLM sizes, while E5-V is reproduced. The headline “lightweight 4B beats 8B/7B” claim would be more convincing with (i) a controlled backbone-matched comparison (same BLIP-3-4B under E5-V-style and MCL-style objectives on the same compute) and (ii) explicit reporting of whether any baseline used FashionIQ category prompts or other inference tricks that FiRE also uses. Without that, part of the margin may be attributable to backbone/data/protocol mismatch rather than FiRE alone.
  3. [§4.2, Eqs. (5)–(7), Table 5] §4.2, Eqs. (5)–(7) and Table 5 (w/o-RecallLoss, w/-OneStage): The paper argues that disentangling reasoning then retrieval is essential, and that Recall@k surrogate losses improve discriminative alignment. The ablations support directional gains, but there is no analysis of sensitivity to β_k, τ1/τ2, or of whether a carefully weighted single-stage multi-task objective (generation + InfoNCE + recall) can close the gap to two-stage FiRE under the same total steps/compute. A short sensitivity or matched-budget single-stage sweep would make the “disentangled objectives” claim load-bearing rather than recipe-specific.
minor comments (7)
  1. [Abstract, §5.1, §6] Abstract says “five datasets”; §5.1 and the conclusion refer to seven (CIRR, CIRCO, FashionIQ, Urban1K, Visual Dialog, COCO, Flickr). Align the counts and the “complex vs simple” split everywhere.
  2. [Title, headers] Title/branding uses “FiREὒ5” / “FiRE” inconsistently (title, headers, body). Pick one spelling and use it uniformly.
  3. [§3.1] §3.1: ImageNet-1K unlabeled test split is used as the image pool. Briefly discuss domain shift relative to fashion (FashionIQ) and dialog (VisDial) benchmarks, and whether any near-duplicate filtering vs evaluation sets was performed.
  4. [Figure 2] Figure 2 is dense; the three panels (pipeline / FiRE / inference) would be clearer with larger fonts and a short caption walk-through of the quintuple fields.
  5. [§3.2, Eq. (1)] Eq. (1): clarify whether the M EOS tokens are appended only at the end of the full multimodal sequence or separately to image and text streams in every stage (pair ID, stage-1, stage-2, inference).
  6. [Table 4, §5.3] Table 4: FiRE underperforms E5-V on Flickr R@1/R@5; the discussion attributes this to backbone size and short-text training, which is plausible—state this limitation more explicitly in the abstract/conclusion so the “universal” claim is scoped.
  7. [Throughout] Minor typos/grammar: e.g., “finE-tuning”, “rop for decoration” in Fig. 5, “optimize for optimization” in §5.1.2, and occasional missing articles. A careful proofread pass is needed.

Circularity Check

0 steps flagged

No significant circularity: synthetic FiGMaQ training and external zero-shot benchmarks are independent.

full rationale

This is an empirical retrieval paper, not a first-principles derivation. FiGMaQ is built from unlabeled ImageNet-1K images via CoT captioning, MLLM similarity filtering (Eqs. 1–3, thresholds [0.6, 0.83]), and LLaMA-3.1-70B vagueness-guided modifications; none of these steps use labels or images from CIRR, CIRCO, FashionIQ, Visual Dialog, Urban1K, COCO, or Flickr. Two-stage fine-tuning (generation loss then InfoNCE + Recall@k surrogate) is performed only on FiGMaQ; evaluation is zero-shot on held-out public benchmarks with a single checkpoint. Reported gains are therefore not algebraically forced by construction, fitted test parameters, or a self-citation uniqueness chain. Table 6’s Combiner transfer and Table 5 ablations further treat the synthetic data and objectives as falsifiable inputs rather than tautologies. Minor reuse of the same MLLM family in data generation and fine-tuning is ordinary pipeline practice, not load-bearing circular reasoning. Score 0.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 2 invented entities

Empirical systems paper. Load-bearing background is standard contrastive retrieval and instruction-tuned MLLMs; free knobs are the usual training and filtering hyperparameters; the main postulated artifacts are the FiGMaQ dataset definition and the FiRE stage split rather than new physical entities.

free parameters (5)
  • pair cosine thresholds θ_l, θ_h = 0.6 and 0.83
    Hand-chosen band [0.6, 0.83] that decides which image pairs enter FiGMaQ; directly controls dataset difficulty and size.
  • InfoNCE / align temperatures τ, τ′ = 0.01
    Set to 0.01 for both data-gen alignment and stage-2 retrieval; scales logits and affects hardness of negatives.
  • Recall surrogate weights β_1, β_5 and temperatures τ1, τ2 = β=[0.4,0.15], τ1=1, τ2=0.01
    β1=0.4, β2=0.15, τ1=1, τ2=0.01 chosen to mix R@1 and R@5 surrogates into total loss; not derived.
  • EOS pool size M = 5
    Number of EOS tokens averaged for sequence embedding (Eq.1); architectural choice affecting representation.
  • LoRA rank / alpha / dropout and stage LRs/epochs = r=64, α=128, lr=1e-4, epochs=1 then 2
    rank 64, alpha 128, dropout 0.1, LR 1e-4, stage-1 1 epoch / stage-2 2 epochs, batch 16; standard PEFT knobs that condition final metrics.
axioms (5)
  • domain assumption CIR is a sufficiently representative complex multimodal task that fine-tuning on CIR-style quintuples transfers zero-shot to long-text and visual-dialog retrieval.
    Stated motivation in §1 and §4; underpins the single-checkpoint universal-retriever claim.
  • domain assumption Averaged last-layer EOS embeddings of an instruction-formatted MLLM are adequate dual-encoder retrieval features (decoder-only retrieval practice).
    Eq.(1) and citations to LLM retrieval work; all ranking metrics depend on this representation choice.
  • ad hoc to paper Fine-grained caption semantic similarity is a better proxy for useful reference–target pairs than visual or coarse caption similarity.
    Core design choice in §3.2; justified by observation but not independently validated beyond downstream gains.
  • standard math Standard InfoNCE and differentiable Recall@k surrogate losses improve ranking when added as in Eqs.(5)–(7).
    Taken from contrastive learning and Patel et al. recall-surrogate literature; used as optimization tools.
  • domain assumption Frozen vision encoder + LoRA on LLM preserves enough multimodal alignment while adapting to FiGMaQ.
    Implementation §5.1.2; common PEFT assumption, untested against full fine-tune here.
invented entities (2)
  • FiGMaQ fine-grained multimodal quintuple dataset no independent evidence
    purpose: Supply long captions and human-like modification text so MLLMs can be trained for fine-grained context and CIR-style retrieval.
    Defined by the three-stage auto pipeline in §3; quality argued via Combiner transfer and ablations, but the corpus itself is a paper-introduced artifact not yet an external standard.
  • FiRE two-stage fine-tuning procedure no independent evidence
    purpose: Sequentially optimize context reasoning then query–target alignment with disentangled objectives.
    Method contribution in §4; validated only by this paper’s ablations and benchmarks.

pith-pipeline@v1.2.0-daily-grok45 · 24237 in / 4127 out tokens · 79220 ms · 2026-07-31T22:09:17.365554+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of FiRE: Enhancing MLLMs with Fine-Grained Context Learning for Complex Image Retrieval." pith.science (2026). https://pith.science/paper/LV6EE4P2

@misc{pith2026260727959,
  author       = {Pith},
  title        = {Pith review of: FiRE: Enhancing MLLMs with Fine-Grained Context Learning for Complex Image Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LV6EE4P2}},
  note         = {Machine review of arXiv:2607.27959}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Due to their strong generalizable multimodal processing and reasoning capabilities, Multimodal Large Language Models (MLLMs) have demonstrated significant potential as universal image retrievers, effectively addressing diverse real-world image retrieval tasks. Nevertheless, pioneering studies, while promising, overlook the potential of fine-grained context modeling and disentangled fine-tuning objectives in enhancing MLLMs' retrieval performance, particularly for complex tasks such as long-text-to-image retrieval, visual dialog retrieval, and composed image retrieval (CIR). Therefore, in this work, we propose an automated fine-grained multimodal quintuple dataset construction pipeline and a novel two-stage fine-grained multimodal fine-tuning strategy. The dataset generation pipeline produces a comprehensive CIR dataset with fine-grained image captions and modification text, facilitating fine-grained context modeling. Beyond the previously entangled fine-tuning paradigm, our approach separates the fine-tuning process into two distinct stages: (1) fine-grained context reasoning-oriented fine-tuning and (2) fine-grained retrieval-oriented fine-tuning. These stages aim to sequentially enhance the model's context understanding and query-target alignment capabilities, thereby improving retrieval performance. Extensive experiments across five datasets encompassing diverse and complex image retrieval tasks demonstrate the remarkable superiority of our method over existing approaches in zero-shot retrieval settings, even with a more lightweight MLLM backbone compared to those methods.

Figures

Figures reproduced from arXiv: 2607.27959 by Bohan Hou, Haokun Wen, Haoqiang Lin, Meng Liu, Xiangyu Zhao, Xuemeng Song, Yupeng Hu.

Figure 1
Figure 1. Figure 1: Illustration of complex image retrieval tasks: (a) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of Our Method. (a) illustrates the data generation pipeline, (b) presents our fine-tuning strategy: FiRE, and [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Illustration of instructions involved in: (a) CoT [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Modification text generated by our method and its [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Illustration of CIR and Long-Text-to-Image Re [PITH_FULL_IMAGE:figures/full_fig_p009_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

47 extracted references · 6 linked inside Pith

  1. [1]

    Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Alberto Del Bimbo. 2023. Zero-Shot Composed Image Retrieval with Textual Inversion. InProceedings of the IEEE/CVF International Conference on Computer Vision. IEEE, 15338–15347

    2023

  2. [2]

    Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Alberto Del Bimbo. 2023. Composed image retrieval using contrastive learning and task-oriented clip- based features.ACM Transactions on Multimedia Computing, Communications and Applications20, 3 (2023), 1–24

    2023

  3. [3]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. InAdvances in neural information processing systems. 1877–1901

    2020

  4. [4]

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2024. Scaling instruction-finetuned language models.Journal of Machine Learning Research25, 70 (2024), 1–53

    2024

  5. [5]

    Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. InProceedings of the IEEE conference on computer vision and pattern recognition. 326–335

    2017

  6. [6]

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition. Ieee, 248–255

    2009

  7. [7]

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh

  8. [8]

    Geonmo Gu, Sanghyuk Chun, Wonjae Kim, Yoohoon Kang, and Sangdoo Yun

  9. [9]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685(2021)

    Pith/arXiv arXiv 2021

  10. [10]

    Surgan Jandial, Pinkesh Badjatiya, Pranit Chawla, Ayush Chopra, Mausoom Sarkar, and Balaji Krishnamurthy. 2022. SAC: Semantic Attention Composition for Text-Conditioned Image Retrieval. InProceedings of the IEEE Winter Conference on Applications of Computer Vision. IEEE, 4021–4030

    2022

  11. [11]

    Ting Jiang, Minghui Song, Zihan Zhang, Haizhen Huang, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang, and Fuzhen Zhuang. 2024. E5-v: Universal embeddings with multimodal large language models.arXiv preprint arXiv:2407.12580(2024)

    Pith/arXiv arXiv 2024

  12. [12]

    Shyamgopal Karthik, Karsten Roth, Massimiliano Mancini, and Zeynep Akata

  13. [13]

    Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. 2023. Grounding lan- guage models to images for multimodal inputs and outputs. InProceedings of the International Conference on Machine Learning. PMLR, 17283–17300

    2023

  14. [14]

    Matan Levy, Rami Ben-Ari, Nir Darshan, and Dani Lischinski. 2024. Data roaming and quality assessment for composed image retrieval. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 38. 2991–2999

    2024

  15. [15]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. InProceedings of the International Conference on Machine Learning. PMLR, 19730–19742

    2023

  16. [16]

    Wei Li, Hehe Fan, Yongkang Wong, Yi Yang, and Mohan Kankanhalli. 2024. Improving Context Understanding in Multimodal Large Language Models via Multimodal Composition Learning. InProceedings of the International Conference on Machine Learning. PMLR, 1–21

    2024

  17. [17]

    Haoqiang Lin, Haokun Wen, Xuemeng Song, Meng Liu, Yupeng Hu, and Liqiang Nie. 2024. Fine-grained Textual Inversion Network for Zero-Shot Composed Image Retrieval. InProceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 240–250

    2024

  18. [18]

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. InProceedings of the European Conference on Computer Vision. Springer, 740–755

    2014

  19. [19]

    Yikun Liu, Jiangchao Yao, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023. Zero- shot composed text-image retrieval.arXiv preprint arXiv:2306.07272(2023)

    Pith/arXiv arXiv 2023

  20. [20]

    Zheyuan Liu, Cristian Rodriguez-Opazo, Damien Teney, and Stephen Gould. 2021. Image retrieval on real-life images with pre-trained vision-and-language models. InProceedings of the IEEE/CVF International Conference on Computer Vision. IEEE, 2125–2134

    2021

  21. [21]

    I Loshchilov. 2017. Decoupled weight decay regularization.arXiv preprint arXiv:1711.05101(2017)

    Pith/arXiv arXiv 2017

  22. [22]

    Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. 2024. Fine- Tuning LLaMA for Multi-Stage Text Retrieval. InProceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval. FiREὒ5: Enhancing MLLMs with Fine-Grained Context Learning for Complex Image Retrieval SIGIR ’25, July 13–18, 2025, Padua, Italy A...

    2024

  23. [23]

    Niklas Muennighoff, Hongjin Su, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Aman- preet Singh, and Douwe Kiela. 2024. Generative representational instruction tuning.arXiv preprint arXiv:2402.09906(2024)

    Pith/arXiv arXiv 2024

  24. [24]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. (2022), 27730–27744

    2022

  25. [25]

    Yash Patel, Giorgos Tolias, and Jirí Matas. 2022. Recall@k Surrogate Loss with Large Batches and Similarity Mixup. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 7492–7501

    2022

  26. [26]

    Plummer, Liwei Wang, Chris M

    Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2016. Flickr30k Entities: Collecting Region- to-Phrase Correspondences for Richer Image-to-Sentence Models. (2016)

    2016

  27. [27]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sand- hini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al

  28. [28]

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2019. ZeRO: Memory Optimization Towards Training A Trillion Parameter Models.arXiv preprint arXiv: 1910.02054(2019)

    Pith/arXiv arXiv 2019

  29. [29]

    Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen-Yu Lee, Kate Saenko, and Tomas Pfister. 2023. Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrieval. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 19305–19314

    2023

  30. [30]

    Xuemeng Song, Fuli Feng, Jinhuan Liu, Zekun Li, Liqiang Nie, and Jun Ma. 2017. Neurostylist: Neural compatibility modeling for clothing matching. InProceedings of the ACM international conference on Multimedia. 753–761

    2017

  31. [31]

    Yuanmin Tang, Jing Yu, Keke Gai, Jiamin Zhuang, Gang Xiong, Yue Hu, and Qi Wu

  32. [32]

    Nam Vo, Lu Jiang, Chen Sun, Kevin Murphy, Li-Jia Li, Li Fei-Fei, and James Hays

  33. [33]

    Junyan Wang, Peng Zhang, Cheng Zhang, and Dawei Song. 2019. Scss-lie: A novel synchronous collaborative search system with a live interactive engine. InPro- ceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 1309–1312

    2019

  34. [34]

    Haokun Wen, Xuemeng Song, Xiaolin Chen, Yinwei Wei, Liqiang Nie, and Tat- Seng Chua. 2024. Simple but Effective Raw-Data Level Multimodal Fusion for Composed Image Retrieval. InProceedings of the International ACM SIGIR Confer- ence on Research and Development in Information Retrieval. ACM, 229–239

    2024

  35. [35]

    Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Rogerio Feris. 2021. Fashion IQ: A New Dataset Towards Retrieving Images by Natural Language Feedback. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 11307–11317

    2021

  36. [36]

    InProceedings of the AAAI Conference on Artificial Intelligence

    Context-I2W: Mapping Images to Context-dependent Words for Accurate Zero-Shot Composed Image Retrieval. InProceedings of the AAAI Conference on Artificial Intelligence. AAAI, 5180–5188

  37. [37]

    Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S. Ryoo, Shrikant Kendre, Jieyu Zhang, Can Qin, Shu Zhang, Chia-Chih Chen, Ning Yu, Juntao Tan, Tulika Manoj Awalgaonkar, Shelby Heinecke, Huan Wang, Yejin Choi, Ludwig Schmidt, Zeyuan Chen, Silvio Savarese, Juan Carlos Niebles, Caiming...

    arXiv 2024

  38. [38]

    Zhenyu Yang, Dizhan Xue, Shengsheng Qian, Weiming Dong, and Changsheng Xu. 2024. LDRE: LLM-based Divergent Reasoning and Ensemble for Zero-Shot Composed Image Retrieval. InProceedings of the International ACM SIGIR Confer- ence on Research and Development in Information Retrieval. ACM, 80–90

    2024

  39. [39]

    Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. 2024. Long-CLIP: Unlocking the Long-Text Capability of CLIP. InProceedings of the European Conference on Computer Vision. Springer, 310–325

    2024

  40. [40]

    Kai Zhang, Yi Luan, Hexiang Hu, Kenton Lee, Siyuan Qiao, Wenhu Chen, Yu Su, and Ming-Wei Chang. 2024. Magiclens: Self-supervised image retrieval with open-ended instructions. InProceedings of the International Conference on Machine Learning. PMLR, 1–18

    2024

  41. [41]

    Liangli Zhen, Peng Hu, Xu Wang, and Dezhong Peng. 2019. Deep supervised cross-modal retrieval. InProceedings of the IEEE conference on computer vision and pattern recognition. IEEE, 10394–10403

    2019

  42. [42]

    Xiaohui Xie, Jiaxin Mao, Yiqun Liu, and Maarten de Rijke. 2020. Modeling user behavior for vertical search: images, apps and products. InProceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 2440–2443

    2020

  43. [2017]

    InProceedings of the IEEE conference on computer vision and pattern recognition

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering. InProceedings of the IEEE conference on computer vision and pattern recognition. IEEE, 6904–6913

  44. [2019]

    InProceedings of the IEEE conference on computer vision and pattern recognition

    Composing Text and Image for Image Retrieval - An Empirical Odyssey. InProceedings of the IEEE conference on computer vision and pattern recognition. IEEE, 6439–6448

  45. [2021]

    InProceedings of the International Conference on Machine Learning

    Learning Transferable Visual Models From Natural Language Supervi- sion. InProceedings of the International Conference on Machine Learning. PMLR, 8748–8763

  46. [2023]

    In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    Vision-by-language for training-free compositional image retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 1–15

  47. [2024]

    InProceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition

    Language-only training of zero-shot composed image retrieval. InProceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 13225–13234