{"id":"ba1921c8-14da-4cd4-b1d3-5fc694472fdd","arxiv_id":"2501.13968","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Counterfactual image generation can synthesize reference-text-target triplets automatically, and the authors report improved CIR recall in data-scarce setups for one of two tested models.","lead":"The paper introduces an automatic pipeline that creates composed image retrieval training triplets by generating counterfactual images with diffusion models. It could reduce manual annotation costs for retrieval systems that need fine-grained text-image understanding.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported gains may be driven by adding thousands of extra training triplets rather than by counterfactual editing; the experiments lack a same-size control that isolates the proposed method.","rationale":"The reader's weakest assumption concerns whether counterfactual generation produces clean, attribute-only edits. That is a real risk, but I see a more directly load-bearing problem: even if every generated triplet were perfectly clean, the current experiments cannot attribute the observed gains to counterfactual editing because the training set size is confounded with the treatment. Adding 5,000 triplets to a 1,392-triplet baseline is a massive increase in data quantity, and no control condition holds quantity fixed while varying triplet quality. Without such a control, a skeptic can explain all reported improvements as 'more data helps.' The reader correctly noticed that the Combiner baseline does not consistently improve, which further weakens the central claim, but the missing size-matched control is the decisive gap. My concrete test would settle the concern by comparing against a non-counterfactual augmentation of equal size. Since the paper remains promising but under-supported, the conditional verdict stands unchanged.","tokens_in":7702,"tokens_out":4017,"duration_ms":41236,"concrete_test":"Re-run the 30%-data CIRR experiment with a same-size control: replace the 5,000 LANCE-generated triplets with 5,000 triplets produced by the same pipeline but with the prompt-to-prompt editing step disabled (i.e., plain Stable Diffusion generation from the counterfactual caption), or with 5,000 randomly selected original-image/caption pairs from the full dataset. If the control yields R@1/R@10/R@50 improvements comparable to the LANCE condition, the central claim is not supported. Report per-seed standard deviations across at least three runs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 3.2: 'the use of synthetic triplets consistently enhances the performance of the CIR models') is not established by the experimental design because the baseline always has fewer training triplets than the proposed condition. In the 30%-data setting (Table 1), CIRR has 1,392 original triplets and the method adds 5,000 synthetic triplets, a roughly 4.6x increase; FashionIQ has 1,487 originals and 3,000 synthetic triplets, a roughly 3x increase. Any observed R@k improvement could therefore be a pure dataset-size effect. The ablation in Section 3.3 varies the number of original images while fixing synthetic triplets at 5,000, but it still compares 'with' versus 'without' those extra 5,000 examples; it never holds training-set size constant or varies synthetic-triplet quality. The claim that counterfactual image generation specifically is responsible is thus untested. A second symptom of the same problem is Table 2 itself: against the Combiner baseline, the method decreases CIRR R@1 (32.65 to 32.58) and FashionIQ R@10 (31.93 to 31.92), so the word 'consistently' is also too strong.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an automatic pipeline for synthesizing composed-image-retrieval (CIR) training triplets. Given a reference image, a BLIP-2 caption is generated, a fine-tuned LLM perturbs one attribute to produce a modification text and a counterfactual caption, and prompt-to-prompt editing with null-text inversion in Stable Diffusion generates the target image. The authors evaluate the approach on CIRR and FashionIQ under a 30%-data regime, comparing a BLIP-based CIR model and Combiner with and without the synthetic triplets, and they also report an ablation across training-image proportions. The central claim is that synthetic triplets yield consistent and significant CIR improvements in data-scarce settings.","tokens_in":7955,"tokens_out":3689,"duration_ms":36319,"significance":"If the central claim were established, the contribution would be practically valuable: it offers a way to reduce manual triplet annotation costs and to enrich small CIR datasets with controllable, attribute-localized image pairs. The pipeline is simple and builds on publicly available components (LANCE, prompt-to-prompt, null-text inversion, BLIP-2), which makes it easy to reproduce in principle. However, the current experimental design does not isolate the proposed counterfactual generation mechanism from a pure increase in training-set size, and Table 2 itself contradicts the word 'consistently' for the Combiner baseline. The potential significance is therefore real but not yet demonstrated.","major_comments":[{"comment":"The sentence in Section 3.2 that 'the use of synthetic triplets consistently enhances the performance of the CIR models' is not supported by the reported numbers. For the Combiner baseline, the proposed method decreases CIRR R@1 from 32.65 to 32.58 and FashionIQ R@10 from 31.93 to 31.92, while CIRR R@10 changes only from 75.59 to 75.60. Only the BLIP baseline shows consistent gains. The claim should be weakened or the analysis should explain why these near-zero/decreasing results are still consistent with the stated conclusion.","section":"Section 3.2, Table 2"},{"comment":"The main comparison is confounded by training-set size. In the 30%-data setting, CIRR has 1,392 original triplets and the method adds 5,000 synthetic triplets (a roughly 4.6x increase), and FashionIQ has 1,487 original triplets with 3,000 synthetic triplets (roughly 3x). The baseline always uses fewer training triplets than the proposed condition, so the observed improvements could be a pure dataset-size effect. The paper needs a same-size control, for example a baseline trained on an equivalently enlarged set of original triplets (or repeated triplets), and an experiment that varies the number/quality of synthetic triplets while holding total training size fixed.","section":"Section 3.2, Table 1"},{"comment":"The ablation study does not resolve the confound. It fixes 5,000 synthetic triplets and varies the proportion of original training images, but it still compares 'original triplets only' against 'original triplets plus 5,000 synthetic triplets.' The total number of training triplets therefore differs between the two curves at every point, so the graph cannot attribute the improvement to the counterfactual editing mechanism rather than to additional data. The authors should compare against a same-size baseline or vary the synthetic-triplet composition while keeping the total count constant. In addition, no error bars or multiple-seed results are reported, so the differences in Figure 4 cannot be assessed for significance.","section":"Section 3.3, Figure 4"},{"comment":"The method's key assumption is that prompt-to-prompt editing with null-text inversion produces a target image that differs from the reference image only in the attribute named by the modification text. The paper provides only qualitative examples (Figure 3) and does not quantitatively evaluate edit fidelity. Unintended changes in the generated target images would make the synthetic triplets noisy and could weaken the training signal. A quantitative evaluation, such as measuring attribute accuracy with a pretrained VQA or CLIP-based metric, or a human study, would support the claim that the synthesized triplets are high-quality.","section":"Section 2.3"}],"minor_comments":[{"comment":"The text says 'We synthesized 5,000 triplets from CIRR dataset, and 3,000 triplets from FashionIQ dataset using 1,500 training images from each dataset, respectively,' but Table 1 reports 5,082 CIRR training images and 13,623 FashionIQ training images in the 30% setting. Please clarify which image set the 1,500 number refers to and how many images were actually used for synthesis.","section":"Section 3.1"},{"comment":"Table 2 and Figure 4 would benefit from reporting variance across random seeds or the number of runs; without this, it is unclear whether differences such as 32.58 vs. 32.65 are meaningful.","section":"General"},{"comment":"There are several typographical issues, including the title line reading 'RETRIEV AL' and 'GENERA TION' on separate lines, which should be corrected.","section":"General"},{"comment":"The perturbation process is described as using a 'fine-tuned LLM' but no details are given about the fine-tuning data, the number of training samples, the LoRA hyperparameters, or the prompt template. Adding this information would improve reproducibility.","section":"Section 2.2"},{"comment":"The paper does not include a limitations section or a discussion of failure cases; given that the experimental evidence for the central claim is currently mixed, a short discussion of when synthetic triplets help versus when they do not would be useful.","section":"Section 3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript reads more like a workshop contribution than a journal paper in its current form. The central claim is plausible but the experiments as designed cannot distinguish the proposed counterfactual-editing mechanism from a simple increase in training data. I would not reject the paper, because the issue is fixable with additional experiments, but the revision needs to be substantial. Also, the paper does not mention code or data release, which would improve reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take on arXiv:2501.13968. The core idea is simple and worth a look: use LANCE-style counterfactual generation (LLM caption perturbation + prompt-to-prompt editing with null-text inversion) to synthesize CIR triplets automatically, removing the manual annotation bottleneck. That's a real problem, and the paper applies existing tools in a sensible way. The BLIP results are decent: on CIRR R@10 goes from 78.17 to 81.04, and FashionIQ R@10 from 36.34 to 39.13. That's not nothing.\n\nBut the central claim -- that counterfactual editing, specifically, is what drives the gain -- is untested. The experiments compare a baseline with 1,392 CIRR triplets against the same baseline plus 5,000 synthetic triplets (roughly 4.6x more data). FashionIQ goes from 1,487 to 4,487. So any improvement could just be more training data. The ablation in Section 3.3 keeps synthetic triplets fixed at 5,000 and varies original data, but it still compares with vs. without those 5,000 extra examples; it never isolates quality. A same-size control would be straightforward: add 5,000 triplets from a simpler source, such as random caption swaps or existing datasets, and see if the counterfactual version does better. Without that, the paper overreaches when it says gains are 'consistent' -- Table 2 shows Combiner R@1 on CIRR drops from 32.65 to 32.58 and FashionIQ R@10 drops from 31.93 to 31.92. Not a disaster, but not consistent.\n\nOther weaknesses: no error bars or significance tests, no code, and the fine-tuning details for the LLM are sketchy. The core assumption that prompt-to-prompt editing produces attribute-only changes is also untested; the qualitative examples look fine, but a few cherry-picked cases don't tell you about failure rate.\n\nWho is this for? People working on CIR with scarce data, and anyone building synthetic training pipelines. It's a reasonable workshop paper or late-breaking abstract, and the idea deserves to be tested properly. I'd send it to peer review (it's probably aimed at ICIP), but I'd ask for the same-size control before accepting. I wouldn't cite it yet.\n\nRecommendation: engage with it, but don't take the numbers at face value.","headline":"Plausible data-augmentation trick for CIR, but the claim that counterfactual editing is what helps is untested without a same-size control.","tokens_in":8491,"tokens_out":2354,"would_cite":false,"duration_ms":22323,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Automatically synthesized counterfactual image triplets improve composed image retrieval, especially when training data is scarce.","keywords":["composed image retrieval","counterfactual image generation","triplet synthesis","prompt-to-prompt editing","null-text inversion","diffusion models","data-scarce training","vision-language retrieval"],"falsifier":"Use a collection of images with known attribute labels, synthesize triplets with the proposed pipeline, and measure edit fidelity with an attribute classifier that checks whether each target image flips exactly the intended attribute and leaves the other labeled attributes untouched. Then train the same CIR models on only the high-fidelity triplets versus all synthetic triplets; if the two training sets give the same recall, local-edit fidelity is not what drives the gains, and if the gains disappear when fidelity is explicitly enforced, the mechanism is confirmed as the active ingredient.","tokens_in":7510,"feed_emoji":"🖼️","tokens_out":10987,"duration_ms":96267,"temperature":0.7,"pith_summary":"Composed image retrieval (CIR) requires training triplets of a reference image, a modification text, and a target image that reflects the text; collecting these by hand is expensive. This paper argues that such triplets can instead be synthesized automatically by generating counterfactual images that differ from a reference only in the attribute named by the modification text. The proposed pipeline captions each reference image, perturbs the caption with an LLM to define a counterfactual, and edits the original image with prompt-to-prompt control in Stable Diffusion to produce the target. Experiments on CIRR and FashionIQ show that adding these synthetic triplets raises recall at several ranks for both a fine-tuned BLIP model and Combiner, with the largest gains in data-scarce settings. If this holds, cheap synthetic data can substitute for a substantial share of manual annotation in CIR.","feed_headline":"Synthetic triplets lift composed image retrieval accuracy","feed_subtitle":"The method generates training triplets automatically, so retrieval models improve even when labeled data is scarce.","key_machinery":"The mechanism is a triplet synthesis pipeline built on counterfactual image generation. It begins with BLIP-2 producing a reference caption for a real image; a fine-tuned LLaMA model with LoRA then rewrites the caption into a counterfactual caption by changing one attribute and outputs the corresponding modification text. Stable Diffusion, guided by prompt-to-prompt editing and null-text inversion, generates the target image from the reference image and the counterfactual caption, preserving the original content outside the edited region. The result is a triplet of reference image, modification text, and target image in which the image pair differs locally rather than globally; this local-difference property is what the paper claims makes the synthetic triplets effective for training CIR models.","core_discovery":"The paper's central claim is that synthetic triplets generated by counterfactual image generation consistently improve the accuracy of composed image retrieval models. In the data-scarce evaluation, where the training image pool is reduced to 30% of the original, adding 5,000 synthetic triplets raises the fine-tuned BLIP model's recall on the CIRR test set from 39.66 to 40.75 at R@1, from 68.07 to 69.83 at R@5, from 78.17 to 81.04 at R@10, and from 93.01 to 94.80 at R@50; on FashionIQ the corresponding gains are from 36.34 to 39.13 at R@10 and from 58.27 to 60.61 at R@50. An ablation varying the proportion of original training images shows that the same 5,000 synthetic triplets boost recall at every rank and every proportion, with the largest relative improvements when the original data is smallest. The paper attributes these gains to the counterfactual target images preserving global structure while changing only the attribute named in the modification text, so the CIR model learns the intended edit rather than incidental correlations.","pith_inferences":["The same pipeline could be used to generate training data for other edit-conditioned vision-language tasks, such as text-guided image editing or visual question answering about attributes; the paper only evaluates retrieval.","Because the counterfactual caption is produced by a fixed perturbation prompt, the diversity of synthetic triplets is limited by the LLM's rewrites; varying the perturbation instructions or sampling multiple counterfactuals per image could broaden coverage beyond the paper's 5,000 and 3,000 triplet budgets.","An untested question is whether a model trained purely on synthetic triplets can match one trained on human annotations when the same number of images is available, which would clarify how fully synthetic data can replace manual collection."],"forward_implications":["CIR training can be scaled without manual triplet annotation: any image collection can be captioned, perturbed, and edited into new triplets.","In data-scarce regimes, adding a fixed number of synthetic triplets yields consistent recall gains across R@1, R@5, R@10, and R@50 on both natural-scene and fashion benchmarks.","The gains appear across two different CIR model families, a fine-tuned BLIP and Combiner, indicating the synthetic data helps the task rather than overfitting a single architecture.","The ablation shows the benefit persists even at 100% of training images, though the relative improvement is largest when the original image pool is small."],"supporting_citations":[{"why":"LANCE; the counterfactual image generation framework this paper extends from model evaluation to triplet synthesis.","marker":"[15]"},{"why":"BLIP-2; produces the reference caption that the perturbation step edits.","marker":"[18]"},{"why":"Stable Diffusion; the diffusion model that renders the target image from the counterfactual caption.","marker":"[19]"},{"why":"Prompt-to-prompt; injects cross-attention maps so only the edited tokens change the generated image.","marker":"[20]"},{"why":"Null-text inversion; reconstructs the original image in latent space so editing can be applied to real photos.","marker":"[21]"},{"why":"LLaMA; the base language model fine-tuned for structured caption perturbation.","marker":"[23]"},{"why":"CoVR; provides the fine-tuned BLIP CIR baseline and the automatic triplet-collection method the paper improves upon.","marker":"[14]"},{"why":"Combiner; the second CIR model used to check that synthetic triplets help beyond a single architecture.","marker":"[7]"},{"why":"FashionIQ; one of the two evaluation datasets, providing fashion-oriented image-text feedback triplets.","marker":"[12]"},{"why":"CIRR; the other evaluation dataset, providing natural-scene triplets with reasoning-based queries.","marker":"[13]"}],"fun_headline_variants":["Counterfactual triplets sharpen image retrieval","Synthetic triplets boost composed image retrieval","Automatic triplet synthesis lifts retrieval accuracy","Data-scarce image retrieval improved by synthetic triplets","Counterfactual images make retrieval models smarter"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the image-editing step producing a target image that differs from the reference only in the attribute named in the modification text, with the rest of the image unchanged; if that edit fidelity fails, the synthetic triplets contain unintended changes and the reported gains could weaken or reverse.","fun_headline_variants_meta":{"raw":{"variants":["Counterfactual triplets sharpen image retrieval","Synthetic triplets boost composed image retrieval","Automatic triplet synthesis lifts retrieval accuracy","Data-scarce image retrieval improved by synthetic triplets","Counterfactual images make retrieval models smarter"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000158,"raw_usage":{"total_tokens":1211,"prompt_tokens":914,"completion_tokens":297,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":229}},"tokens_in":530,"tokens_out":297,"duration_ms":3023,"temperature":1.0,"reasoning_tokens":229,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T16:53:33.457132+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use a collection of images with known attribute labels, synthesize triplets with the proposed pipeline, and measure edit fidelity with an attribute classifier that checks whether each target image flips exactly the intended attribute and leaves the other labeled attributes untouched. Then train the same CIR models on only the high-fidelity triplets versus all synthetic triplets; if the two training sets give the same recall, local-edit fidelity is not what drives the gains, and if the gains disappear when fidelity is explicitly enforced, the mechanism is confirmed as the active ingredient.","supporting_citations":[{"cited_title":"Fashion retrieval via graph reasoning networks on a similarity pyramid,","cited_arxiv_id":null,"evidence_quote":"LANCE; the counterfactual image generation framework this paper extends from model evaluation to triplet synthesis."},{"cited_title":"CoVR: Learning composed video retrieval from web video captions,","cited_arxiv_id":null,"evidence_quote":"BLIP-2; produces the reference caption that the perturbation step edits."},{"cited_title":"LANCE: Stress-testing visual models by generating language-guided counterfactual images,","cited_arxiv_id":null,"evidence_quote":"Stable Diffusion; the diffusion model that renders the target image from the counterfactual caption."},{"cited_title":"Reinforcing pre-trained models using coun- terfactual images,","cited_arxiv_id":null,"evidence_quote":"Prompt-to-prompt; injects cross-attention maps so only the edited tokens change the generated image."},{"cited_title":"COCO- counterfactuals: Automatically constructed counterfactual ex- amples for image-text pairs,","cited_arxiv_id":null,"evidence_quote":"Null-text inversion; reconstructs the original image in latent space so editing can be applied to real photos."},{"cited_title":"High-resolution image synthe- sis with latent diffusion models,","cited_arxiv_id":null,"evidence_quote":"LLaMA; the base language model fine-tuned for structured caption perturbation."},{"cited_title":"Real20m: A large-scale e-commerce dataset for cross-domain retrieval,","cited_arxiv_id":null,"evidence_quote":"CoVR; provides the fine-tuned BLIP CIR baseline and the automatic triplet-collection method the paper improves upon."},{"cited_title":"Pic2Word: Mapping pictures to words for zero-shot composed image re- trieval,","cited_arxiv_id":null,"evidence_quote":"Combiner; the second CIR model used to check that synthetic triplets help beyond a single architecture."},{"cited_title":"Zero-shot composed image re- trieval considering query-target relationship leveraging masked image-text pairs,","cited_arxiv_id":null,"evidence_quote":"FashionIQ; one of the two evaluation datasets, providing fashion-oriented image-text feedback triplets."},{"cited_title":"SPIRIT: Style-guided patch interaction for fashion image retrieval with text feedback,","cited_arxiv_id":null,"evidence_quote":"CIRR; the other evaluation dataset, providing natural-scene triplets with reasoning-based queries."}],"review_version":1}