Zero Shot Composed Image Retrieval

Gautama Shastry Bulusu Venkata; Santhosh Kakarla

REVIEW 5 major objections 6 minor 14 references

Zero Shot Composed Image Retrieval

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A lightweight fusion module on frozen BLIP2 roughly doubles zero-shot composed image retrieval accuracy on FashionIQ.

desk verdict A straightforward BLIP2 fine-tune that reports only validation Recall@K, claims SOTA without comparing to published CIR baselines, and mislabels a text-only DPO experiment as composed retrieval – worth peer review for a hard revise, not citable as is. read the letter →

arxiv 2506.06602 v1 pith:FZNEJFF6 submitted 2025-06-07 cs.CV

Santhosh Kakarla , Gautama Shastry Bulusu Venkata This is my paper

classification cs.CV

keywords composedimageretrievalzero-shotBLIP-2Q-FormerfusionFashionIQdirectpreferenceoptimizationcontrastivelearningRecall@K

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that zero-shot composed image retrieval—finding a target image by pairing a reference image with a short textual edit—is bottlenecked by architectures that encode vision and language separately, and that a small fusion layer between frozen backbones can remove most of that bottleneck. It reports that fine-tuning only BLIP2's Q-Former with a contrastive retrieval loss on FashionIQ training data raises average Recall@10 from 21.24% with zero-shot CLIP to 45.36%, and average Recall@50 to 67.57%. It also reports a sharply negative result for a preference-based alternative: Retrieval-DPO, which tunes CLIP's text encoder with a Direct Preference Optimization margin on mined hard negatives, reaches only 0.02% Recall@10. If correct, the paper establishes that lightweight, fusion-based fine-tuning is a practical high-leverage route for composed image retrieval without task-supervised triplets.

What carries the argument

The load-bearing mechanism is the Q-Former fusion module: a fixed set of learnable query vectors that attend over the frozen image encoder's patch grid, are projected into the language-token space, and are concatenated with the caption's token embeddings into a single sequence processed by the frozen language backbone. The hidden state at the first position becomes a fused multimodal prompt embedding, trained with the InfoNCE contrastive loss to align it with the target image embedding and push it away from batch negatives. In the failure arm, Retrieval-DPO replaces this fusion with a Direct Preference Optimization loss on CLIP-scaled cosine similarities between caption-only prompt embeddings and positive versus mined hard-negative images, updating only the CLIP text transformer. The comparison isolates what the fusion module contributes: the difference between 45.36% and 0.02% average Recall@10.

What would settle it

The paper's headline claim can be checked directly: run the same fine-tuned BLIP2 checkpoints on the official FashionIQ test split and compare against published test-split numbers; if test Recall@10 is no better than the 21% zero-shot baseline, the state-of-the-art claim collapses because the paper currently reports only validation numbers. A second check targets the failure diagnosis: if the Retrieval-DPO query is modified to actually include the reference image's visual embedding, and Recall@10 remains near 0.02%, then the paper's attribution of the collapse to missing multimodal fusion is not supported.

Watch

Extended reading notes

Core claim

The paper's central claim is that on FashionIQ, a fine-tuned BLIP2 with a Q-Former fusion head is a state-of-the-art composed image retriever: after five epochs of fine-tuning only 3% of BLIP2's parameters, it reports Recall@10 of 45.58% for shirt, 40.06% for dress, and 50.43% for toptee, with average Recall@50 of 67.57% on the validation split. These numbers roughly double the zero-shot CLIP baseline of 21.24% average Recall@10. The paper further claims that the contrast between this success and Retrieval-DPO's 0.02% Recall@10 is diagnostic: preference optimization applied to separate unimodal encoders fails because there is no joint image-text fusion, the margin objective is not aligned with top-K ranking, negatives mined from a static nearest-neighbor index are inconsistently hard, and the vision backbone stays frozen. The authors present this as evidence that effective composed image retrieval requires explicit multimodal fusion and ranking-aware objectives.

Load-bearing premise

The state-of-the-art claim rests on treating validation-split Recall@K on FashionIQ as final benchmark performance, and the Retrieval-DPO failure analysis rests on treating a query that never receives the reference image as a fair composed-retrieval query.

Editorial extensions

If this is right

Fine-tuned BLIP2 with the Q-Former lifts FashionIQ average Recall@10 from about 21% to about 45%, a doubling over zero-shot CLIP with only 3% of parameters trained.
Average Recall@50 rises from 39.92% zero-shot to 67.57%, so the fused prompt is competitive at larger retrieval depths.
Removing the Q-Former costs roughly 4 percentage points of average Recall@10, and disabling mixed precision costs about 1.5 points, identifying the fusion module as the main driver.
Retrieval-DPO, under the settings tested, stays at essentially random chance (0.02% Recall@10), so margin-based preference loss on separate unimodal encoders is not a viable composed-retrieval strategy in this setup.
Gains are uneven across edit types: broad attribute changes retrieve well, while fine-grained pattern edits such as 'add a floral print' fall below 35% Recall@10.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the Q-Former mechanism is what carries the gain, a natural test is to apply the same frozen-backbone fusion recipe to other composed-retrieval benchmarks and to caption subgroups; a large drop outside FashionIQ would suggest the result is dataset-specific rather than architectural.
Because the paper's DPO query never encodes the reference image, a falsifiable extension is to include visual tokens in the text-side prompt; recovering retrieval in that variant would confirm the missing-fusion diagnosis, while continued near-zero performance would shift blame to the margin objective or negative quality.
The 3%-parameter-update result hints that most of the benefit may come from a carefully placed adapter rather than from full fine-tuning; ablating the position and depth of the fusion layer could tell whether the Q-Former's query count is the right capacity knob.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The paper's central claim – that fine-tuned BLIP2 is state-of-the-art on FashionIQ – rests entirely on validation-split numbers, with no test-set Recall@K and no comparison against LinCIR, iSEARLE, or CompoDiff. The DPO negative result is not actually a CIR experiment: the query is built from the caption tokens alone, never encoding the reference image, so at best it shows text-only retrieval fails. The stress-test note is right on both counts.

What is genuinely useful here is a clean, reproducible recipe: BLIP2 Q-Former fine-tuning with InfoNCE, plus ablations showing the Q-Former and mixed-precision contribute. The code link helps, and the writing is clear. The DPO failure, though mis-designed, does illustrate that a margin loss on frozen unimodal embeddings will not move top-K metrics – a cautionary data point if framed correctly.

But the problems are load-bearing. Validation-only numbers are optimistically biased when early stopping was done on that same split. No error bars. Dataset statistics are internally inconsistent: the abstract says roughly 50,000 pairs per category, while Section 3 lists 17K/16K/18K total; normalization stats switch from CLIP to ImageNet between sections. The title "Zero Shot" is wrong because the method fine-tunes on FashionIQ train. And the DPO failure analysis attributes the result to missing fusion, but since the reference image is never encoded, the experiment does not test that claim. To its credit, the paper honestly lists limitations; that does not fix the assertions.

This paper is for someone in CIR who wants a quick baseline recipe and a reminder that evaluation protocol matters. It is not acceptable in current form, but it deserves a serious referee rather than a desk reject – the main experiment is reproducible, and a major revision (add test numbers, compare to prior supervised and zero-shot baselines, rebuild the DPO query with the reference image) could yield a modest contribution. I would send it out, with clear guidance that revision is mandatory.

Referee Report

5 major / 6 minor

Summary. The paper evaluates three approaches to composed image retrieval (CIR) on FashionIQ: a zero-shot CLIP baseline, fine-tuning of BLIP2 with a Q-Former fusion module, and a 'Retrieval-DPO' variant that fine-tunes CLIP's text transformer under a direct preference optimization loss. The central claim is that fine-tuned BLIP2, updating only a lightweight Q-Former and projection layer on FashionIQ training pairs, achieves state-of-the-art retrieval performance on FashionIQ, with Recall@10 of 45.6% (shirt), 40.1% (dress), and 50.4% (top-tee), and average Recall@50 of 67.6%. The paper further reports that Retrieval-DPO collapses to 0.02% Recall@10 and attributes this failure to the lack of joint multimodal fusion, misalignment between the margin objective and top-K metrics, poor negative quality, and frozen vision and transformer layers.

Significance. If the BLIP2 results were reproduced on the standard FashionIQ test protocol and compared with published CIR methods, they would provide a useful data point showing that a small trainable fusion head on a frozen BLIP2 backbone can substantially outperform CLIP-based zero-shot CIR. The paper is transparent about the Retrieval-DPO failure, and the released code and pretrained checkpoints support reproducibility. However, the significance is currently limited by the absence of test-split numbers for the main method, the absence of comparisons to prior CIR baselines, and the fact that the proposed BLIP2 method is supervised adaptation rather than zero-shot retrieval despite the title.

major comments (5)

[§4.2, Table 1; §3.1] The state-of-the-art claim rests entirely on FashionIQ validation Recall@K. Section 3.1 states that the test split is used only for final performance reporting, but no test-split Recall@K is reported for the BLIP2 method; the only test-split numbers in Section 4.3 belong to the failed Retrieval-DPO baseline. Because hyperparameters were selected and early stopping was performed on validation Recall@10, the validation numbers are optimistically biased. Please report test-split results and compare with published CIR methods (e.g., LinCIR, iSEARLE, CompoDiff) under the standard FashionIQ evaluation protocol.
[Title; Abstract; §3.2; §4.2] The method is not zero-shot: BLIP2 is fine-tuned on FashionIQ training pairs with validation-based early stopping, so the title 'Zero Shot Composed Image Retrieval' and the opening framing are misleading. Either remove the zero-shot terminology or reposition the paper as supervised lightweight fine-tuning and benchmark it against supervised CIR baselines on the same protocol.
[§3.1; §4] Dataset statistics are internally inconsistent. Section 3.1 states per-category training pairs of roughly 17K, 16K, and 18K for shirt, dress, and toptee, with over 50,000 unique edit instructions total, while Section 4 claims approximately 50,000 reference-target image pairs per category. These numbers cannot both be correct. Please correct the dataset description and give exact counts of pairs and captions per split.
[§3.3; §4.3] The Retrieval-DPO baseline does not encode the reference image at all: the query is formed solely from the caption tokens prefixed with <|image|> and compared against image embeddings. This makes the 0.02% Recall@10 failure unsurprising and not informative about preference-based CIR as such. The paper should either incorporate the reference image into the query (e.g., by combining image and text features) or explicitly frame Retrieval-DPO as a caption-only retrieval ablation rather than a CIR baseline.
[§4.2] The ablation claims (removing the Q-Former reduces average Recall@10 by approximately 4 points; disabling mixed precision reduces it by approximately 1.5 points) are reported without a table, error bars, or test-set confirmation. Please provide quantitative support for these ablations and rerun them on the test split.

minor comments (6)

[§3.1; §4] Image preprocessing is inconsistent: Section 3.1 reports CLIP statistics (mean = [0.481, 0.457, 0.408], std = [0.268, 0.261, 0.276]), while Section 4 reports ImageNet statistics (mean = [0.485, 0.456, 0.406], std = [0.229, 0.224, 0.225]). Please clarify which normalization was used and ensure it is identical across all methods.
[§3.2; §4.2; §6] The description of trainable parameters is inconsistent: Section 3.2 says only the Q-Former and its projection are updated, Section 4.2 says all trainable parameters of BLIP2 are optimized, and the Conclusion says only 3% of parameters are updated. Please state exactly which parameters are trainable and report the trainable parameter count.
[§4.2] The BLIP2 model variant is not specified; please state which vision encoder and language backbone are used (e.g., ViT-G/14 and OPT or FlanT5) and the exact number of Q-Former queries, since these details are needed for reproducibility.
[§3; §2] InfoNCE is attributed to Rusak et al. (2024) in Section 3, but the original formulation is due to Oord et al. (2018), which is reference [12]. Please cite the original work for the loss.
[Figure 2] Figure 2 is a JSON dump of metrics and is not informative; consider replacing it with a standard plot of validation curves or a table with error bars.
[§4.3] The paper reports test-split Recall@10/50 for Retrieval-DPO but does not report the corresponding validation numbers or the exact number of train/validation/test queries and gallery images; please provide these details for completeness.

Circularity Check

1 steps flagged · score 6.0 of 10

Reported BLIP2 'state-of-the-art' Recall@K is the validation metric used for early stopping and hyperparameter selection, with no test-split numbers; the claim does not leave the optimization split.

fitted input called prediction [Section 3.1 'Dataset Preparation'; Section 4.2 'Fine Tuned BLIP2 Performance' (Table 1)]
"We use the training split for model fine-tuning, the validation split for hyperparameter selection and early stopping (monitoring Recall@10), and the test split only for final performance reporting. ... After five epochs of fine-tuning, our model attains strong retrieval performance on all three FashionIQ categories. Table 1 summarizes the per-category and average recalls. ... Our results demonstrate that fine-tuned BLIP2 with Q-Former fusion achieves state-of-the-art retrieval on FashionIQ."

The reported headline numbers (R@10 45.6/40.1/50.4, avg R@50 67.6) are Recall@K on the FashionIQ validation split (Table 1). That same split was used for hyperparameter selection and early stopping, with validation R@10 as the monitored criterion (Sections 3.1 and 3.2). The model is therefore selected to maximize the very quantity later presented as the final result; reporting it as 'state-of-the-art' is reporting the fitted objective rather than an independent prediction. The paper reserves the test split 'only for final performance reporting' but provides no test-split Recall@K for BLIP2, so the central performance claim never leaves the split on which it was optimized.

full rationale

No equations in the paper transform a fitted parameter into the reported Recall values, and no load-bearing self-citation chain is present: BLIP2, CLIP, and DPO are cited as external prior work. The Retrieval-DPO failure analysis is empirical and self-consistent. The main circularity is the validation-selection loop: hyperparameters and early stopping are chosen on FashionIQ validation R@10, and the same validation R@10 is then presented as the final evidence of state-of-the-art performance, in contradiction with the paper's own stated protocol that the test split is for final reporting. This is a fitted-input-called-prediction pattern rather than a definitional equivalence, so the score is moderate (6). The absence of comparisons to published CIR methods (LinCIR, iSEARLE, CompoDiff) is a correctness/completeness risk, not a circularity, and is not scored here.

Assumptions & free parameters 6 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical entities. Its main free parameters are training hyperparameters selected on the validation split. The key unproven assumptions are that validation numbers stand in for test performance, that the dataset statistics are reliable, and that the Retrieval-DPO query, which omits the reference image, still represents composed retrieval.

free parameters (6)

BLIP2 learning rate = 1e-4
AdamW learning rate for Q-Former fine-tuning; chosen by the authors, not derived.
BLIP2 weight decay = 0.05
Weight decay used in BLIP2 fine-tuning; affects regularization and final recall.
BLIP2 batch size = 256
Contrastive batch size; negatives in InfoNCE depend on it.
Number of BLIP2 training epochs = 5
Selected by early stopping on validation Recall@10; final number is a validation-chosen value.
DPO scaling factor beta = 0.1
Controls margin sharpness in the DPO loss; tuned but the method still fails.
FAISS top-k for hard negative mining = 50
Number of nearest neighbors searched to select one hard negative; affects training signal.

assumptions (3)

domain assumption FashionIQ official splits prevent image overlap across train, validation, and test.
Invoked in Sections 3.1 and 4, but the later text says 'approximately 50,000 reference-target image pairs per category', which is inconsistent with the earlier 17K/16K/18K counts and makes the overlap claim unverified.
domain assumption Validation Recall@K is a reliable proxy for final benchmark performance.
All BLIP2 results are reported on validation; the paper never provides test-split numbers, so the central claim assumes validation behavior transfers.
domain assumption BLIP2's frozen encoders plus a trained Q-Former produce a unified representation suitable for contrastive CIR.
This is the architectural premise of the method and is not proved; the paper only shows empirical validation numbers.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Zero Shot Composed Image Retrieval." pith.science (2026). https://pith.science/paper/FZNEJFF6

@misc{pith2026250606602,
  author       = {Pith},
  title        = {Pith review of: Zero Shot Composed Image Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FZNEJFF6}},
  note         = {Machine review of arXiv:2506.06602}
}

abstract

Composed image retrieval (CIR) allows a user to locate a target image by applying a fine-grained textual edit (e.g., ``turn the dress blue'' or ``remove stripes'') to a reference image. Zero-shot CIR, which embeds the image and the text with separate pretrained vision-language encoders, reaches only 20-25\% Recall@10 on the FashionIQ benchmark. We improve this by fine-tuning BLIP-2 with a lightweight Q-Former that fuses visual and textual features into a single embedding, raising Recall@10 to 45.6\% (shirt), 40.1\% (dress), and 50.4\% (top-tee) and increasing the average Recall@50 to 67.6\%. We also examine Retrieval-DPO, which fine-tunes CLIP's text encoder with a Direct Preference Optimization loss applied to FAISS-mined hard negatives. Despite extensive tuning of the scaling factor, index, and sampling strategy, Retrieval-DPO attains only 0.02\% Recall@10 -- far below zero-shot and prompt-tuned baselines -- because it (i) lacks joint image-text fusion, (ii) uses a margin objective misaligned with top-$K$ metrics, (iii) relies on low-quality negatives, and (iv) keeps the vision and Transformer layers frozen. Our results show that effective preference-based CIR requires genuine multimodal fusion, ranking-aware objectives, and carefully curated negatives.

Figures

Figures reproduced from arXiv: 2506.06602 by the authors.

**Figure 2.** JSON dump of per-category and average recall metrics on FashionIQ validation. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Validation Recall@10 and Recall@50 for the Retrieval–DPO pipeline across 10 epochs. Metrics remain near [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

14 extracted references · 7 canonical work pages

[1]

Agnolucci, A

L. Agnolucci, A. Baldrati, M. Bertini, and A. Del Bimbo. iSEARLE: Improving textual inversion for zero-shot composed image retrieval. arXiv preprint arXiv:2405.02951, 2024

arXiv 2024
[2]

Rusak, P

E. Rusak, P. Reizinger, A. Juhos, O. Bringmann, R. S. Zimmermann, and W. Brendel. InfoNCE: Identifying the gap between theory and practice. arXiv preprint arXiv:2407.00143, 2024

arXiv 2024
[3]

Radford, J

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark,et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020, 2021

arXiv 2021
[4]

J. Li, D. Li, S. Savarese, and S. Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023

arXiv 2023
[5]

Rafailov, A

R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023

arXiv 2023
[6]

H. Wen, X. Zhang, X. Song, Y . Wei, and L. Nie. Target-guided composed image retrieval. arXiv preprint arXiv:2309.01366, 2023

work page Pith review arXiv 2023
[7]

G. Gu, S. Chun, W. Kim, H. Jun, Y . Kang, and S. Yun. CompoDiff: Versatile composed image retrieval with latent diffusion. Transactions on Machine Learning Research, 2023. arXiv:2303.11916

arXiv 2023
[8]

L. Wang, W. Ao, V . N. Boddeti, and S.-N. Lim. Generative zero-shot composed image retrieval. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , Poster #340, 2025

work page 2025

Show all 14 references

[9]

G. Gu, S. Chun, W. Kim, Y . Kang, and S. Yun. Language-only training of zero-shot composed image retrieval (LinCIR). In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 13221–13232. IEEE, 2024

2024
[10]

M. U. Anwaar, E. Labintcev, and M. Kleinsteuber. Compositional learning of image-text query for image retrieval. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 1137–1147. IEEE, 2021

2021
[11]

Z. Liu, C. Rodriguez-Opazo, D. Teney, and S. Gould. Image retrieval on real-life images with pre-trained vision-and-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2125–2134. IEEE, 2021

2021
[12]

A. v. d. Oord, Y . Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018

2018 arXiv
[13]

T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In Proceedings of the 37th International Conference on Machine Learning (ICML) , pages 1597–1607. PMLR, 2020

2020
[14]

X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer. LiT: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 18123– 18133. IEEE, 2022. 8

2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Agnolucci, A

L. Agnolucci, A. Baldrati, M. Bertini, and A. Del Bimbo. iSEARLE: Improving textual inversion for zero-shot composed image retrieval. arXiv preprint arXiv:2405.02951, 2024

arXiv 2024

[2] [2]

Rusak, P

E. Rusak, P. Reizinger, A. Juhos, O. Bringmann, R. S. Zimmermann, and W. Brendel. InfoNCE: Identifying the gap between theory and practice. arXiv preprint arXiv:2407.00143, 2024

arXiv 2024

[3] [3]

Radford, J

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark,et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020, 2021

arXiv 2021

[4] [4]

J. Li, D. Li, S. Savarese, and S. Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023

arXiv 2023

[5] [5]

Rafailov, A

R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290, 2023

arXiv 2023

[6] [6]

H. Wen, X. Zhang, X. Song, Y . Wei, and L. Nie. Target-guided composed image retrieval. arXiv preprint arXiv:2309.01366, 2023

work page Pith review arXiv 2023

[7] [7]

G. Gu, S. Chun, W. Kim, H. Jun, Y . Kang, and S. Yun. CompoDiff: Versatile composed image retrieval with latent diffusion. Transactions on Machine Learning Research, 2023. arXiv:2303.11916

arXiv 2023

[8] [8]

L. Wang, W. Ao, V . N. Boddeti, and S.-N. Lim. Generative zero-shot composed image retrieval. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , Poster #340, 2025

work page 2025