REVIEW 4 major objections 5 minor 28 references
Triplet Synthesis For Enhancing Composed Image Retrieval via Counterfactual Image Generation
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Automatically synthesized counterfactual image triplets improve composed image retrieval, especially when training data is scarce.
desk verdict Plausible data-augmentation trick for CIR, but the claim that counterfactual editing is what helps is untested without a same-size control. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is a triplet synthesis pipeline built on counterfactual image generation. It begins with BLIP-2 producing a reference caption for a real image; a fine-tuned LLaMA model with LoRA then rewrites the caption into a counterfactual caption by changing one attribute and outputs the corresponding modification text. Stable Diffusion, guided by prompt-to-prompt editing and null-text inversion, generates the target image from the reference image and the counterfactual caption, preserving the original content outside the edited region. The result is a triplet of reference image, modification text, and target image in which the image pair differs locally rather than globally; this local-difference property is what the paper claims makes the synthetic triplets effective for training CIR models.
What would settle it
Use a collection of images with known attribute labels, synthesize triplets with the proposed pipeline, and measure edit fidelity with an attribute classifier that checks whether each target image flips exactly the intended attribute and leaves the other labeled attributes untouched. Then train the same CIR models on only the high-fidelity triplets versus all synthetic triplets; if the two training sets give the same recall, local-edit fidelity is not what drives the gains, and if the gains disappear when fidelity is explicitly enforced, the mechanism is confirmed as the active ingredient.
Extended reading notes
Core claim
The paper's central claim is that synthetic triplets generated by counterfactual image generation consistently improve the accuracy of composed image retrieval models. In the data-scarce evaluation, where the training image pool is reduced to 30% of the original, adding 5,000 synthetic triplets raises the fine-tuned BLIP model's recall on the CIRR test set from 39.66 to 40.75 at R@1, from 68.07 to 69.83 at R@5, from 78.17 to 81.04 at R@10, and from 93.01 to 94.80 at R@50; on FashionIQ the corresponding gains are from 36.34 to 39.13 at R@10 and from 58.27 to 60.61 at R@50. An ablation varying the proportion of original training images shows that the same 5,000 synthetic triplets boost recall at every rank and every proportion, with the largest relative improvements when the original data is smallest. The paper attributes these gains to the counterfactual target images preserving global structure while changing only the attribute named in the modification text, so the CIR model learns the intended edit rather than incidental correlations.
Load-bearing premise
Everything rests on the image-editing step producing a target image that differs from the reference only in the attribute named in the modification text, with the rest of the image unchanged; if that edit fidelity fails, the synthetic triplets contain unintended changes and the reported gains could weaken or reverse.
Editorial extensions
If this is right
- CIR training can be scaled without manual triplet annotation: any image collection can be captioned, perturbed, and edited into new triplets.
- In data-scarce regimes, adding a fixed number of synthetic triplets yields consistent recall gains across R@1, R@5, R@10, and R@50 on both natural-scene and fashion benchmarks.
- The gains appear across two different CIR model families, a fine-tuned BLIP and Combiner, indicating the synthetic data helps the task rather than overfitting a single architecture.
- The ablation shows the benefit persists even at 100% of training images, though the relative improvement is largest when the original image pool is small.
Reading between the lines
- The same pipeline could be used to generate training data for other edit-conditioned vision-language tasks, such as text-guided image editing or visual question answering about attributes; the paper only evaluates retrieval.
- Because the counterfactual caption is produced by a fixed perturbation prompt, the diversity of synthetic triplets is limited by the LLM's rewrites; varying the perturbation instructions or sampling multiple counterfactuals per image could broaden coverage beyond the paper's 5,000 and 3,000 triplet budgets.
- An untested question is whether a model trained purely on synthetic triplets can match one trained on human annotations when the same number of images is available, which would clarify how fully synthetic data can replace manual collection.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an automatic pipeline for synthesizing composed-image-retrieval (CIR) training triplets. Given a reference image, a BLIP-2 caption is generated, a fine-tuned LLM perturbs one attribute to produce a modification text and a counterfactual caption, and prompt-to-prompt editing with null-text inversion in Stable Diffusion generates the target image. The authors evaluate the approach on CIRR and FashionIQ under a 30%-data regime, comparing a BLIP-based CIR model and Combiner with and without the synthetic triplets, and they also report an ablation across training-image proportions. The central claim is that synthetic triplets yield consistent and significant CIR improvements in data-scarce settings.
Significance. If the central claim were established, the contribution would be practically valuable: it offers a way to reduce manual triplet annotation costs and to enrich small CIR datasets with controllable, attribute-localized image pairs. The pipeline is simple and builds on publicly available components (LANCE, prompt-to-prompt, null-text inversion, BLIP-2), which makes it easy to reproduce in principle. However, the current experimental design does not isolate the proposed counterfactual generation mechanism from a pure increase in training-set size, and Table 2 itself contradicts the word 'consistently' for the Combiner baseline. The potential significance is therefore real but not yet demonstrated.
major comments (4)
- [Section 3.2, Table 2] The sentence in Section 3.2 that 'the use of synthetic triplets consistently enhances the performance of the CIR models' is not supported by the reported numbers. For the Combiner baseline, the proposed method decreases CIRR R@1 from 32.65 to 32.58 and FashionIQ R@10 from 31.93 to 31.92, while CIRR R@10 changes only from 75.59 to 75.60. Only the BLIP baseline shows consistent gains. The claim should be weakened or the analysis should explain why these near-zero/decreasing results are still consistent with the stated conclusion.
- [Section 3.2, Table 1] The main comparison is confounded by training-set size. In the 30%-data setting, CIRR has 1,392 original triplets and the method adds 5,000 synthetic triplets (a roughly 4.6x increase), and FashionIQ has 1,487 original triplets with 3,000 synthetic triplets (roughly 3x). The baseline always uses fewer training triplets than the proposed condition, so the observed improvements could be a pure dataset-size effect. The paper needs a same-size control, for example a baseline trained on an equivalently enlarged set of original triplets (or repeated triplets), and an experiment that varies the number/quality of synthetic triplets while holding total training size fixed.
- [Section 3.3, Figure 4] The ablation study does not resolve the confound. It fixes 5,000 synthetic triplets and varies the proportion of original training images, but it still compares 'original triplets only' against 'original triplets plus 5,000 synthetic triplets.' The total number of training triplets therefore differs between the two curves at every point, so the graph cannot attribute the improvement to the counterfactual editing mechanism rather than to additional data. The authors should compare against a same-size baseline or vary the synthetic-triplet composition while keeping the total count constant. In addition, no error bars or multiple-seed results are reported, so the differences in Figure 4 cannot be assessed for significance.
- [Section 2.3] The method's key assumption is that prompt-to-prompt editing with null-text inversion produces a target image that differs from the reference image only in the attribute named by the modification text. The paper provides only qualitative examples (Figure 3) and does not quantitatively evaluate edit fidelity. Unintended changes in the generated target images would make the synthetic triplets noisy and could weaken the training signal. A quantitative evaluation, such as measuring attribute accuracy with a pretrained VQA or CLIP-based metric, or a human study, would support the claim that the synthesized triplets are high-quality.
minor comments (5)
- [Section 3.1] The text says 'We synthesized 5,000 triplets from CIRR dataset, and 3,000 triplets from FashionIQ dataset using 1,500 training images from each dataset, respectively,' but Table 1 reports 5,082 CIRR training images and 13,623 FashionIQ training images in the 30% setting. Please clarify which image set the 1,500 number refers to and how many images were actually used for synthesis.
- [General] Table 2 and Figure 4 would benefit from reporting variance across random seeds or the number of runs; without this, it is unclear whether differences such as 32.58 vs. 32.65 are meaningful.
- [General] There are several typographical issues, including the title line reading 'RETRIEV AL' and 'GENERA TION' on separate lines, which should be corrected.
- [Section 2.2] The perturbation process is described as using a 'fine-tuned LLM' but no details are given about the fine-tuning data, the number of training samples, the LoRA hyperparameters, or the prompt template. Adding this information would improve reproducibility.
- [Section 3] The paper does not include a limitations section or a discussion of failure cases; given that the experimental evidence for the central claim is currently mixed, a short discussion of when synthetic triplets help versus when they do not would be useful.
Circularity Check
No circularity found: the claimed CIR improvement is measured on held-out retrieval splits, and the reused components (LANCE, prompt-to-prompt editing, null-text inversion) are external; the same-group citations are not load-bearing.
full rationale
Walking the derivation chain, the central claim is that adding synthetic triplets improves CIR accuracy under data scarcity (Section 3.2). The synthetic triplets are produced by an external pipeline: BLIP-2 captioning, a fine-tuned LLM perturbation step, Stable Diffusion, prompt-to-prompt cross-attention editing, and null-text inversion (Sections 2.2-2.3). No parameter is fitted to the retrieval test labels, and no quantity is defined in terms of the result it is used to predict. The CIR models are trained on original plus synthetic training triplets and evaluated on held-out CIRR test and FashionIQ validation data (Section 3.2 and Table 1), so the reported R@k gains are not forced by construction. The paper does cite same-group works ([8] and [16]), but neither is load-bearing: [16] appears only in a background list of counterfactual-generation models [15-17], and [8] appears only among prior CIR methods [3-8]. The concern that the baseline always has fewer training triplets than the proposed condition is a real experimental-validity issue (the gain could be a dataset-size effect), but it is a confound, not a circularity: the measured quantity is still an externally held-out retrieval metric rather than a restatement of the method's inputs. No self-definitional step, fitted-input-renamed-as-prediction, imported uniqueness theorem, or ansatz-smuggled-via-citation is present. Therefore no circular step is identified.
Assumptions & free parameters
free parameters (2)
- Number of synthetic triplets =
5,000 for CIRR, 3,000 for FashionIQ
- Reduction proportion of training images =
30%
assumptions (3)
- domain assumption Counterfactual image generation preserves the unmodified content of the reference image while applying the specified attribute change
- domain assumption The fine-tuned LLaMA model produces valid modification text and a counterfactual caption that contains a feasible visual edit
- domain assumption The evaluation protocol of prior work [14] applies to this data-scarce setting
Cite this review
Pith. "Pith review of Triplet Synthesis For Enhancing Composed Image Retrieval via Counterfactual Image Generation." pith.science (2026). https://pith.science/paper/GLX4LUGN
@misc{pith2026250113968,
author = {Pith},
title = {Pith review of: Triplet Synthesis For Enhancing Composed Image Retrieval via Counterfactual Image Generation},
year = {2026},
howpublished = {\url{https://pith.science/paper/GLX4LUGN}},
note = {Machine review of arXiv:2501.13968}
}
read the original abstract
Composed Image Retrieval (CIR) provides an effective way to manage and access large-scale visual data. Construction of the CIR model utilizes triplets that consist of a reference image, modification text describing desired changes, and a target image that reflects these changes. For effectively training CIR models, extensive manual annotation to construct high-quality training datasets, which can be time-consuming and labor-intensive, is required. To deal with this problem, this paper proposes a novel triplet synthesis method by leveraging counterfactual image generation. By controlling visual feature modifications via counterfactual image generation, our approach automatically generates diverse training triplets without any manual intervention. This approach facilitates the creation of larger and more expressive datasets, leading to the improvement of CIR model's performance.
Reference graph
Works this paper leans on
-
[1]
Triplet Synthesis For Enhancing Composed Image Retrieval via Counterfactual Image Generation
INTRODUCTION The explosion of digital content has made the efficient access to and management of vast amounts of visual information increasingly es- sential. As the volume of visual information grows, users are seek- ing more advanced tools to find their desired information. Recently, there has been a growing demand for image retrieval that can inter- pre...
work page Pith review arXiv 2025
-
[2]
a photo of a white sports car driving down a road with mountains in the background
TRIPLET SYNTHESIS USING COUNTERFACTUAL IMAGE GENERA TION 2.1. Overview of our method The overview of our triplet synthesis method is presented in Fig. 1. The objective of this task is to acquire a triplet ⟨Iref , t, Itarget⟩, which serves as the training data for CIR. Here, Iref represents a reference image, t denotes modification text, and Itarget is a t...
-
[3]
EXPERIMENTAL RESULTS 3.1. Settings To verify the effectiveness of our method, we utilized two datasets: Composed Image Retrieval with Reasoning (CIRR) dataset [13], which includes complex query images in natural scenes, and Fash- ionIQ [12] dataset, which focuses on fashion items. Each dataset is intended to verify the ability to handle complex queries. C...
-
[4]
CONCLUSIONS In this paper, we have proposed a method to improve the perfor- mance of CIR through automatic triplet synthesis using counterfac- tual image generation. Our method enhances the ability to generate diverse and high-quality training examples, advancing the capabili- ties of the CIR model. Experimental results using CIRR and Fash- ionIQ datasets...
-
[5]
Training vision transformers for image retrieval,
Alaaeldin El-Nouby, Natalia Neverova, Ivan Laptev, and Herv´e J´egou, “Training vision transformers for image retrieval,” arXiv preprint arXiv:2102.05644, 2021
arXiv 2021
-
[6]
Boosting vision transformers for image re- trieval,
Chull Hwan Song, Jooyoung Yoon, Shunghyun Choi, and Yan- nis Avrithis, “Boosting vision transformers for image re- trieval,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023, pp. 107–117
work page 2023
-
[7]
Pic2Word: Mapping pictures to words for zero-shot composed image re- trieval,
Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen-Yu Lee, Kate Saenko, and Tomas Pfister, “Pic2Word: Mapping pictures to words for zero-shot composed image re- trieval,” in Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, 2023, pp. 19305–19314
work page 2023
-
[8]
Zero-shot composed image retrieval with textual inversion,
Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Al- berto Del Bimbo, “Zero-shot composed image retrieval with textual inversion,” in Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision , 2023, pp. 15338– 15347
work page 2023
Show all 28 references
-
[9]
CompoDiff: Versatile composed image retrieval with latent diffusion,
Geonmo Gu, Sanghyuk Chun, Wonjae Kim, HeeJae Jun, Yoohoon Kang, and Sangdoo Yun, “CompoDiff: Versatile composed image retrieval with latent diffusion,” Transactions on Machine Learning Research, 2024
2024
-
[10]
Language-only training of zero-shot com- posed image retrieval,
Geonmo Gu, Sanghyuk Chun, Wonjae Kim, Yoohoon Kang, and Sangdoo Yun, “Language-only training of zero-shot com- posed image retrieval,” in Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, 2024, pp. 13225–13234
2024
-
[11]
Composed image retrieval using contrastive learning and task-oriented clip-based features,
Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Al- berto Del Bimbo, “Composed image retrieval using contrastive learning and task-oriented clip-based features,” ACM Transac- tions on Multimedia Computing, Communications and Appli- cations, vol. 20, no. 3, pp. 1–24, 2023
2023
-
[12]
Zero-shot composed image re- trieval considering query-target relationship leveraging masked image-text pairs,
Huaying Zhang, Rintaro Yanagi, Ren Togo, Takahiro Ogawa, and Miki Haseyama, “Zero-shot composed image re- trieval considering query-target relationship leveraging masked image-text pairs,” in Proceedings of the International Confer- ence on Image Processing. IEEE, 2024, pp. 2431–2437
2024
-
[13]
SPIRIT: Style-guided patch interaction for fashion image retrieval with text feedback,
Yanzhe Chen, Jiahuan Zhou, and Yuxin Peng, “SPIRIT: Style-guided patch interaction for fashion image retrieval with text feedback,” ACM Transactions on Multimedia Computing, Communications and Applications , vol. 20, no. 6, pp. 1–17, 2024
2024
-
[14]
Real20m: A large-scale e-commerce dataset for cross-domain retrieval,
Yanzhe Chen, Huasong Zhong, Xiangteng He, Yuxin Peng, and Lele Cheng, “Real20m: A large-scale e-commerce dataset for cross-domain retrieval,” in Proceedings of the ACM Interna- tional Conference on Multimedia, 2023, pp. 4939–4948
2023
-
[15]
Fashion retrieval via graph reasoning networks on a similarity pyramid,
Zhanghui Kuang, Yiming Gao, Guanbin Li, Ping Luo, Yimin Chen, Liang Lin, and Wayne Zhang, “Fashion retrieval via graph reasoning networks on a similarity pyramid,” in Pro- ceedings of the IEEE/CVF International Conference on Com- puter Vision, 2019, pp. 3066–3075
2019
-
[16]
Fashion IQ: A new dataset towards retrieving images by natural language feedback,
Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Rogerio Feris, “Fashion IQ: A new dataset towards retrieving images by natural language feedback,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, ...
2021
-
[17]
Image retrieval on real-life images with pre-trained vision-and-language models,
Zheyuan Liu, Cristian Rodriguez-Opazo, Damien Teney, and Stephen Gould, “Image retrieval on real-life images with pre-trained vision-and-language models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 2125–2134
2021
-
[18]
CoVR: Learning composed video retrieval from web video captions,
Lucas Ventura, Antoine Yang, Cordelia Schmid, and G¨ul Varol, “CoVR: Learning composed video retrieval from web video captions,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2024, pp. 5270–5279
2024
-
[19]
LANCE: Stress-testing visual models by generating language-guided counterfactual images,
Viraj Prabhu, Sriram Yenamandra, Prithvijit Chattopadhyay, and Judy Hoffman, “LANCE: Stress-testing visual models by generating language-guided counterfactual images,” in Pro- ceedings of the Neural Information Processing Systems, 2023, pp. 25165–25184
2023
-
[20]
Reinforcing pre-trained models using coun- terfactual images,
Xiang Li, Ren Togo, Keisuke Maeda, Takahiro Ogawa, and Miki Haseyama, “Reinforcing pre-trained models using coun- terfactual images,” in Proceedings of the International Confer- ence on Image Processing. IEEE, 2024, pp. 486–492
2024
-
[21]
COCO- counterfactuals: Automatically constructed counterfactual ex- amples for image-text pairs,
Tiep Le, Vasudev Lal, and Phillip Howard, “COCO- counterfactuals: Automatically constructed counterfactual ex- amples for image-text pairs,” in Advances in Neural Informa- tion Processing Systems, 2023, vol. 36, pp. 71195–71221
2023
-
[22]
BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi, “BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” in Pro- ceedings of the International Conference on Machine Learn- ing, 2023, pp. 19730–19742
2023
-
[23]
High-resolution image synthe- sis with latent diffusion models,
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer, “High-resolution image synthe- sis with latent diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, 2022, pp. 10684–10695
2022
-
[24]
Prompt-to-prompt image edit- ing with cross-attention control,
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or, “Prompt-to-prompt image edit- ing with cross-attention control,” in Proceedings of the Inter- national Conference on Learning Representations, 2023
2023
-
[25]
Null-text inversion for editing real im- ages using guided diffusion models,
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or, “Null-text inversion for editing real im- ages using guided diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, 2023, pp. 6038–6047
2023
-
[26]
A corpus for reasoning about natural language grounded in photographs,
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi, “A corpus for reasoning about natural language grounded in photographs,” in Proceedings of the An- nual Meeting of the Association for Computational Linguistics, 2019, pp. 6418–6428
2019
-
[27]
LLaMA: Open and efficient foundation language models,
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al., “LLaMA: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023
2023 arXiv
-
[28]
LoRA: Low-rank adaptation of large language models,
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen, “LoRA: Low-rank adaptation of large language models,”arXiv preprint arXiv:2106.09685, 2021
2021 arXiv
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.