Pith. sign in

REVIEW 4 major objections 5 minor 28 references

Triplet Synthesis For Enhancing Composed Image Retrieval via Counterfactual Image Generation

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read Automatically synthesized counterfactual image triplets improve composed image retrieval, especially when training data is scarce.

desk verdict Plausible data-augmentation trick for CIR, but the claim that counterfactual editing is what helps is untested without a same-size control. read the letter →

arxiv 2501.13968 v1 pith:GLX4LUGN submitted 2025-01-22 cs.CV cs.LGeess.IV

classification cs.CVcs.LGeess.IV
keywords composedimageretrievalcounterfactualgenerationtripletsynthesisprompt-to-prompteditingnull-textinversiondiffusionmodelsdata-scarcetrainingvision-language
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Composed image retrieval (CIR) requires training triplets of a reference image, a modification text, and a target image that reflects the text; collecting these by hand is expensive. This paper argues that such triplets can instead be synthesized automatically by generating counterfactual images that differ from a reference only in the attribute named by the modification text. The proposed pipeline captions each reference image, perturbs the caption with an LLM to define a counterfactual, and edits the original image with prompt-to-prompt control in Stable Diffusion to produce the target. Experiments on CIRR and FashionIQ show that adding these synthetic triplets raises recall at several ranks for both a fine-tuned BLIP model and Combiner, with the largest gains in data-scarce settings. If this holds, cheap synthetic data can substitute for a substantial share of manual annotation in CIR.

What carries the argument

The mechanism is a triplet synthesis pipeline built on counterfactual image generation. It begins with BLIP-2 producing a reference caption for a real image; a fine-tuned LLaMA model with LoRA then rewrites the caption into a counterfactual caption by changing one attribute and outputs the corresponding modification text. Stable Diffusion, guided by prompt-to-prompt editing and null-text inversion, generates the target image from the reference image and the counterfactual caption, preserving the original content outside the edited region. The result is a triplet of reference image, modification text, and target image in which the image pair differs locally rather than globally; this local-difference property is what the paper claims makes the synthetic triplets effective for training CIR models.

What would settle it

Use a collection of images with known attribute labels, synthesize triplets with the proposed pipeline, and measure edit fidelity with an attribute classifier that checks whether each target image flips exactly the intended attribute and leaves the other labeled attributes untouched. Then train the same CIR models on only the high-fidelity triplets versus all synthetic triplets; if the two training sets give the same recall, local-edit fidelity is not what drives the gains, and if the gains disappear when fidelity is explicitly enforced, the mechanism is confirmed as the active ingredient.

Watch

Extended reading notes

Core claim

The paper's central claim is that synthetic triplets generated by counterfactual image generation consistently improve the accuracy of composed image retrieval models. In the data-scarce evaluation, where the training image pool is reduced to 30% of the original, adding 5,000 synthetic triplets raises the fine-tuned BLIP model's recall on the CIRR test set from 39.66 to 40.75 at R@1, from 68.07 to 69.83 at R@5, from 78.17 to 81.04 at R@10, and from 93.01 to 94.80 at R@50; on FashionIQ the corresponding gains are from 36.34 to 39.13 at R@10 and from 58.27 to 60.61 at R@50. An ablation varying the proportion of original training images shows that the same 5,000 synthetic triplets boost recall at every rank and every proportion, with the largest relative improvements when the original data is smallest. The paper attributes these gains to the counterfactual target images preserving global structure while changing only the attribute named in the modification text, so the CIR model learns the intended edit rather than incidental correlations.

Load-bearing premise

Everything rests on the image-editing step producing a target image that differs from the reference only in the attribute named in the modification text, with the rest of the image unchanged; if that edit fidelity fails, the synthetic triplets contain unintended changes and the reported gains could weaken or reverse.

Editorial extensions

If this is right

  • CIR training can be scaled without manual triplet annotation: any image collection can be captioned, perturbed, and edited into new triplets.
  • In data-scarce regimes, adding a fixed number of synthetic triplets yields consistent recall gains across R@1, R@5, R@10, and R@50 on both natural-scene and fashion benchmarks.
  • The gains appear across two different CIR model families, a fine-tuned BLIP and Combiner, indicating the synthetic data helps the task rather than overfitting a single architecture.
  • The ablation shows the benefit persists even at 100% of training images, though the relative improvement is largest when the original image pool is small.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same pipeline could be used to generate training data for other edit-conditioned vision-language tasks, such as text-guided image editing or visual question answering about attributes; the paper only evaluates retrieval.
  • Because the counterfactual caption is produced by a fixed perturbation prompt, the diversity of synthetic triplets is limited by the LLM's rewrites; varying the perturbation instructions or sampling multiple counterfactuals per image could broaden coverage beyond the paper's 5,000 and 3,000 triplet budgets.
  • An untested question is whether a model trained purely on synthetic triplets can match one trained on human annotations when the same number of images is available, which would clarify how fully synthetic data can replace manual collection.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes an automatic pipeline for synthesizing composed-image-retrieval (CIR) training triplets. Given a reference image, a BLIP-2 caption is generated, a fine-tuned LLM perturbs one attribute to produce a modification text and a counterfactual caption, and prompt-to-prompt editing with null-text inversion in Stable Diffusion generates the target image. The authors evaluate the approach on CIRR and FashionIQ under a 30%-data regime, comparing a BLIP-based CIR model and Combiner with and without the synthetic triplets, and they also report an ablation across training-image proportions. The central claim is that synthetic triplets yield consistent and significant CIR improvements in data-scarce settings.

Significance. If the central claim were established, the contribution would be practically valuable: it offers a way to reduce manual triplet annotation costs and to enrich small CIR datasets with controllable, attribute-localized image pairs. The pipeline is simple and builds on publicly available components (LANCE, prompt-to-prompt, null-text inversion, BLIP-2), which makes it easy to reproduce in principle. However, the current experimental design does not isolate the proposed counterfactual generation mechanism from a pure increase in training-set size, and Table 2 itself contradicts the word 'consistently' for the Combiner baseline. The potential significance is therefore real but not yet demonstrated.

major comments (4)
  1. [Section 3.2, Table 2] The sentence in Section 3.2 that 'the use of synthetic triplets consistently enhances the performance of the CIR models' is not supported by the reported numbers. For the Combiner baseline, the proposed method decreases CIRR R@1 from 32.65 to 32.58 and FashionIQ R@10 from 31.93 to 31.92, while CIRR R@10 changes only from 75.59 to 75.60. Only the BLIP baseline shows consistent gains. The claim should be weakened or the analysis should explain why these near-zero/decreasing results are still consistent with the stated conclusion.
  2. [Section 3.2, Table 1] The main comparison is confounded by training-set size. In the 30%-data setting, CIRR has 1,392 original triplets and the method adds 5,000 synthetic triplets (a roughly 4.6x increase), and FashionIQ has 1,487 original triplets with 3,000 synthetic triplets (roughly 3x). The baseline always uses fewer training triplets than the proposed condition, so the observed improvements could be a pure dataset-size effect. The paper needs a same-size control, for example a baseline trained on an equivalently enlarged set of original triplets (or repeated triplets), and an experiment that varies the number/quality of synthetic triplets while holding total training size fixed.
  3. [Section 3.3, Figure 4] The ablation study does not resolve the confound. It fixes 5,000 synthetic triplets and varies the proportion of original training images, but it still compares 'original triplets only' against 'original triplets plus 5,000 synthetic triplets.' The total number of training triplets therefore differs between the two curves at every point, so the graph cannot attribute the improvement to the counterfactual editing mechanism rather than to additional data. The authors should compare against a same-size baseline or vary the synthetic-triplet composition while keeping the total count constant. In addition, no error bars or multiple-seed results are reported, so the differences in Figure 4 cannot be assessed for significance.
  4. [Section 2.3] The method's key assumption is that prompt-to-prompt editing with null-text inversion produces a target image that differs from the reference image only in the attribute named by the modification text. The paper provides only qualitative examples (Figure 3) and does not quantitatively evaluate edit fidelity. Unintended changes in the generated target images would make the synthetic triplets noisy and could weaken the training signal. A quantitative evaluation, such as measuring attribute accuracy with a pretrained VQA or CLIP-based metric, or a human study, would support the claim that the synthesized triplets are high-quality.
minor comments (5)
  1. [Section 3.1] The text says 'We synthesized 5,000 triplets from CIRR dataset, and 3,000 triplets from FashionIQ dataset using 1,500 training images from each dataset, respectively,' but Table 1 reports 5,082 CIRR training images and 13,623 FashionIQ training images in the 30% setting. Please clarify which image set the 1,500 number refers to and how many images were actually used for synthesis.
  2. [General] Table 2 and Figure 4 would benefit from reporting variance across random seeds or the number of runs; without this, it is unclear whether differences such as 32.58 vs. 32.65 are meaningful.
  3. [General] There are several typographical issues, including the title line reading 'RETRIEV AL' and 'GENERA TION' on separate lines, which should be corrected.
  4. [Section 2.2] The perturbation process is described as using a 'fine-tuned LLM' but no details are given about the fine-tuning data, the number of training samples, the LoRA hyperparameters, or the prompt template. Adding this information would improve reproducibility.
  5. [Section 3] The paper does not include a limitations section or a discussion of failure cases; given that the experimental evidence for the central claim is currently mixed, a short discussion of when synthetic triplets help versus when they do not would be useful.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the claimed CIR improvement is measured on held-out retrieval splits, and the reused components (LANCE, prompt-to-prompt editing, null-text inversion) are external; the same-group citations are not load-bearing.

full rationale

Walking the derivation chain, the central claim is that adding synthetic triplets improves CIR accuracy under data scarcity (Section 3.2). The synthetic triplets are produced by an external pipeline: BLIP-2 captioning, a fine-tuned LLM perturbation step, Stable Diffusion, prompt-to-prompt cross-attention editing, and null-text inversion (Sections 2.2-2.3). No parameter is fitted to the retrieval test labels, and no quantity is defined in terms of the result it is used to predict. The CIR models are trained on original plus synthetic training triplets and evaluated on held-out CIRR test and FashionIQ validation data (Section 3.2 and Table 1), so the reported R@k gains are not forced by construction. The paper does cite same-group works ([8] and [16]), but neither is load-bearing: [16] appears only in a background list of counterfactual-generation models [15-17], and [8] appears only among prior CIR methods [3-8]. The concern that the baseline always has fewer training triplets than the proposed condition is a real experimental-validity issue (the gain could be a dataset-size effect), but it is a confound, not a circularity: the measured quantity is still an externally held-out retrieval metric rather than a restatement of the method's inputs. No self-definitional step, fitted-input-renamed-as-prediction, imported uniqueness theorem, or ansatz-smuggled-via-citation is present. Therefore no circular step is identified.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The method rests on the assumption that existing counterfactual generation and editing tools can create triplets cleanly; no direct validation of triplet quality is provided.

free parameters (2)
  • Number of synthetic triplets = 5,000 for CIRR, 3,000 for FashionIQ
    Chosen by the authors without an ablation on triplet count; the central performance comparison relies on these fixed quantities.
  • Reduction proportion of training images = 30%
    Experimental setting to simulate data-scarcity; not a model parameter but a design choice that affects reported performance.
assumptions (3)
  • domain assumption Counterfactual image generation preserves the unmodified content of the reference image while applying the specified attribute change
    The method in Section 2.3 assumes prompt-to-prompt editing with null-text inversion makes only local changes. If the editing introduces unintended modifications, the synthesized triplets would be low-quality and the training signal would be corrupted.
  • domain assumption The fine-tuned LLaMA model produces valid modification text and a counterfactual caption that contains a feasible visual edit
    Section 2.2 relies on LLM perturbation to generate captions that correspond to realistic image edits; the quality of these captions is not evaluated.
  • domain assumption The evaluation protocol of prior work [14] applies to this data-scarce setting
    The authors use CIRR test and FashionIQ validation to match prior evaluation, assuming the same metrics and data splits are appropriate.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Triplet Synthesis For Enhancing Composed Image Retrieval via Counterfactual Image Generation." pith.science (2026). https://pith.science/paper/GLX4LUGN

@misc{pith2026250113968,
  author       = {Pith},
  title        = {Pith review of: Triplet Synthesis For Enhancing Composed Image Retrieval via Counterfactual Image Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GLX4LUGN}},
  note         = {Machine review of arXiv:2501.13968}
}
read the original abstract

Composed Image Retrieval (CIR) provides an effective way to manage and access large-scale visual data. Construction of the CIR model utilizes triplets that consist of a reference image, modification text describing desired changes, and a target image that reflects these changes. For effectively training CIR models, extensive manual annotation to construct high-quality training datasets, which can be time-consuming and labor-intensive, is required. To deal with this problem, this paper proposes a novel triplet synthesis method by leveraging counterfactual image generation. By controlling visual feature modifications via counterfactual image generation, our approach automatically generates diverse training triplets without any manual intervention. This approach facilitates the creation of larger and more expressive datasets, leading to the improvement of CIR model's performance.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 25 canonical work pages

  1. [1]

    Triplet Synthesis For Enhancing Composed Image Retrieval via Counterfactual Image Generation

    INTRODUCTION The explosion of digital content has made the efficient access to and management of vast amounts of visual information increasingly es- sential. As the volume of visual information grows, users are seek- ing more advanced tools to find their desired information. Recently, there has been a growing demand for image retrieval that can inter- pre...

  2. [2]

    a photo of a white sports car driving down a road with mountains in the background

    TRIPLET SYNTHESIS USING COUNTERFACTUAL IMAGE GENERA TION 2.1. Overview of our method The overview of our triplet synthesis method is presented in Fig. 1. The objective of this task is to acquire a triplet ⟨Iref , t, Itarget⟩, which serves as the training data for CIR. Here, Iref represents a reference image, t denotes modification text, and Itarget is a t...

  3. [3]

    brown” with “white

    EXPERIMENTAL RESULTS 3.1. Settings To verify the effectiveness of our method, we utilized two datasets: Composed Image Retrieval with Reasoning (CIRR) dataset [13], which includes complex query images in natural scenes, and Fash- ionIQ [12] dataset, which focuses on fashion items. Each dataset is intended to verify the ability to handle complex queries. C...

  4. [4]

    Our method enhances the ability to generate diverse and high-quality training examples, advancing the capabili- ties of the CIR model

    CONCLUSIONS In this paper, we have proposed a method to improve the perfor- mance of CIR through automatic triplet synthesis using counterfac- tual image generation. Our method enhances the ability to generate diverse and high-quality training examples, advancing the capabili- ties of the CIR model. Experimental results using CIRR and Fash- ionIQ datasets...

  5. [5]

    Training vision transformers for image retrieval,

    Alaaeldin El-Nouby, Natalia Neverova, Ivan Laptev, and Herv´e J´egou, “Training vision transformers for image retrieval,” arXiv preprint arXiv:2102.05644, 2021

  6. [6]

    Boosting vision transformers for image re- trieval,

    Chull Hwan Song, Jooyoung Yoon, Shunghyun Choi, and Yan- nis Avrithis, “Boosting vision transformers for image re- trieval,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023, pp. 107–117

  7. [7]

    Pic2Word: Mapping pictures to words for zero-shot composed image re- trieval,

    Kuniaki Saito, Kihyuk Sohn, Xiang Zhang, Chun-Liang Li, Chen-Yu Lee, Kate Saenko, and Tomas Pfister, “Pic2Word: Mapping pictures to words for zero-shot composed image re- trieval,” in Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, 2023, pp. 19305–19314

  8. [8]

    Zero-shot composed image retrieval with textual inversion,

    Alberto Baldrati, Lorenzo Agnolucci, Marco Bertini, and Al- berto Del Bimbo, “Zero-shot composed image retrieval with textual inversion,” in Proceedings of the IEEE/CVF Inter- national Conference on Computer Vision , 2023, pp. 15338– 15347

Show all 28 references
  1. [9]

    CompoDiff: Versatile composed image retrieval with latent diffusion,

    Geonmo Gu, Sanghyuk Chun, Wonjae Kim, HeeJae Jun, Yoohoon Kang, and Sangdoo Yun, “CompoDiff: Versatile composed image retrieval with latent diffusion,” Transactions on Machine Learning Research, 2024

  2. [10]

    Language-only training of zero-shot com- posed image retrieval,

    Geonmo Gu, Sanghyuk Chun, Wonjae Kim, Yoohoon Kang, and Sangdoo Yun, “Language-only training of zero-shot com- posed image retrieval,” in Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, 2024, pp. 13225–13234

  3. [11]

    Composed image retrieval using contrastive learning and task-oriented clip-based features,

    Alberto Baldrati, Marco Bertini, Tiberio Uricchio, and Al- berto Del Bimbo, “Composed image retrieval using contrastive learning and task-oriented clip-based features,” ACM Transac- tions on Multimedia Computing, Communications and Appli- cations, vol. 20, no. 3, pp. 1–24, 2023

  4. [12]

    Zero-shot composed image re- trieval considering query-target relationship leveraging masked image-text pairs,

    Huaying Zhang, Rintaro Yanagi, Ren Togo, Takahiro Ogawa, and Miki Haseyama, “Zero-shot composed image re- trieval considering query-target relationship leveraging masked image-text pairs,” in Proceedings of the International Confer- ence on Image Processing. IEEE, 2024, pp. 2431–2437

  5. [13]

    SPIRIT: Style-guided patch interaction for fashion image retrieval with text feedback,

    Yanzhe Chen, Jiahuan Zhou, and Yuxin Peng, “SPIRIT: Style-guided patch interaction for fashion image retrieval with text feedback,” ACM Transactions on Multimedia Computing, Communications and Applications , vol. 20, no. 6, pp. 1–17, 2024

  6. [14]

    Real20m: A large-scale e-commerce dataset for cross-domain retrieval,

    Yanzhe Chen, Huasong Zhong, Xiangteng He, Yuxin Peng, and Lele Cheng, “Real20m: A large-scale e-commerce dataset for cross-domain retrieval,” in Proceedings of the ACM Interna- tional Conference on Multimedia, 2023, pp. 4939–4948

  7. [15]

    Fashion retrieval via graph reasoning networks on a similarity pyramid,

    Zhanghui Kuang, Yiming Gao, Guanbin Li, Ping Luo, Yimin Chen, Liang Lin, and Wayne Zhang, “Fashion retrieval via graph reasoning networks on a similarity pyramid,” in Pro- ceedings of the IEEE/CVF International Conference on Com- puter Vision, 2019, pp. 3066–3075

  8. [16]

    Fashion IQ: A new dataset towards retrieving images by natural language feedback,

    Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Rogerio Feris, “Fashion IQ: A new dataset towards retrieving images by natural language feedback,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2021, ...

  9. [17]

    Image retrieval on real-life images with pre-trained vision-and-language models,

    Zheyuan Liu, Cristian Rodriguez-Opazo, Damien Teney, and Stephen Gould, “Image retrieval on real-life images with pre-trained vision-and-language models,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021, pp. 2125–2134

  10. [18]

    CoVR: Learning composed video retrieval from web video captions,

    Lucas Ventura, Antoine Yang, Cordelia Schmid, and G¨ul Varol, “CoVR: Learning composed video retrieval from web video captions,” in Proceedings of the AAAI Conference on Artificial Intelligence, 2024, pp. 5270–5279

  11. [19]

    LANCE: Stress-testing visual models by generating language-guided counterfactual images,

    Viraj Prabhu, Sriram Yenamandra, Prithvijit Chattopadhyay, and Judy Hoffman, “LANCE: Stress-testing visual models by generating language-guided counterfactual images,” in Pro- ceedings of the Neural Information Processing Systems, 2023, pp. 25165–25184

  12. [20]

    Reinforcing pre-trained models using coun- terfactual images,

    Xiang Li, Ren Togo, Keisuke Maeda, Takahiro Ogawa, and Miki Haseyama, “Reinforcing pre-trained models using coun- terfactual images,” in Proceedings of the International Confer- ence on Image Processing. IEEE, 2024, pp. 486–492

  13. [21]

    COCO- counterfactuals: Automatically constructed counterfactual ex- amples for image-text pairs,

    Tiep Le, Vasudev Lal, and Phillip Howard, “COCO- counterfactuals: Automatically constructed counterfactual ex- amples for image-text pairs,” in Advances in Neural Informa- tion Processing Systems, 2023, vol. 36, pp. 71195–71221

  14. [22]

    BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi, “BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” in Pro- ceedings of the International Conference on Machine Learn- ing, 2023, pp. 19730–19742

  15. [23]

    High-resolution image synthe- sis with latent diffusion models,

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bj ¨orn Ommer, “High-resolution image synthe- sis with latent diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, 2022, pp. 10684–10695

  16. [24]

    Prompt-to-prompt image edit- ing with cross-attention control,

    Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or, “Prompt-to-prompt image edit- ing with cross-attention control,” in Proceedings of the Inter- national Conference on Learning Representations, 2023

  17. [25]

    Null-text inversion for editing real im- ages using guided diffusion models,

    Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or, “Null-text inversion for editing real im- ages using guided diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, 2023, pp. 6038–6047

  18. [26]

    A corpus for reasoning about natural language grounded in photographs,

    Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi, “A corpus for reasoning about natural language grounded in photographs,” in Proceedings of the An- nual Meeting of the Association for Computational Linguistics, 2019, pp. 6418–6428

  19. [27]

    LLaMA: Open and efficient foundation language models,

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al., “LLaMA: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

  20. [28]

    LoRA: Low-rank adaptation of large language models,

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen, “LoRA: Low-rank adaptation of large language models,”arXiv preprint arXiv:2106.09685, 2021

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.