Pith. sign in

REVIEW 2 major objections 1 minor 12 references

CIRCLED extends three existing datasets into 22,608 consistent multi-turn composed image retrieval sessions across nine subsets and multiple domains.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-29 18:36 UTC pith:BG6ZQMIP

load-bearing objection CIRCLED adds scale and cross-domain reach to multi-turn CIR but the consistency claims rest on an unvalidated pipeline. the 2 major comments →

arxiv 2605.26734 v1 pith:BG6ZQMIP submitted 2026-05-26 cs.CV

CIRCLED: A Multi-turn CIR Dataset with Consistent Dialogues across Domains

Tomohisa Takeda , Yu-Chieh Lin , Yuji Nozawa , Youyang Ng , Osamu Torii , Yusuke Matsui This is my paper
classification cs.CV
keywords multi-turn composed image retrievalMTCIR datasetdialogue consistencyFashionIQCIRRCIRCOdataset constructionprogressive refinement
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper builds CIRCLED to overcome two limits in prior multi-turn composed image retrieval work: restriction to the fashion domain and lack of consistent dialogue history across turns. It starts from FashionIQ, CIRR, and CIRCO, then uses a retrieval pipeline followed by filters on success, length, consistency, and redundancy to produce sessions in which each query progressively narrows toward a target image. The result is a collection more than twice the size of the previous largest resource and spread over nine subsets rather than one domain. A reader would care because the new scale and domain breadth supply a practical test bed for models that must maintain context while refining image searches over several turns.

Core claim

We construct CIRCLED by extending FashionIQ, CIRR, and CIRCO. In CIRCLED, the query at each turn progressively approaches the target image. Data are generated via a CIReVL-based retrieval pipeline and curated with multiple filters on retrieval success, turn length, consistency, and information redundancy to ensure quality. In total, we collect 22,608 multi-turn sessions across nine subsets, substantially exceeding Multi-turn FashionIQ (11,505 sessions) in both scale and generality. We further apply multiple baseline methods and quantitatively assess retrieval accuracy on CIRCLED.

What carries the argument

The CIReVL-based retrieval pipeline plus filters on retrieval success, turn length, consistency, and information redundancy, which produces multi-turn sessions with progressive query refinement and consistent dialogue history.

Load-bearing premise

The pipeline and filters produce high-quality dialogues that accurately reflect progressive refinement without artifacts or domain-specific biases.

What would settle it

Human evaluation of a random sample of sessions that finds frequent inconsistencies in dialogue history or turns that do not progressively approach the stated target image.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Baseline retrieval methods can be run and scored for accuracy on a dataset more than twice as large as prior multi-turn fashion resources.
  • Nine subsets drawn from three source collections enable both within-domain and cross-domain evaluation of multi-turn CIR systems.
  • The progressive-refinement structure supplies a direct test of whether models maintain context across successive queries.
  • Public release of the 22,608 sessions and generation code allows other groups to reproduce or extend the benchmark.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Models trained on CIRCLED may expose domain biases that single-domain collections conceal, because the nine subsets span different visual styles and object types.
  • The consistency filters could be reused as a quality-control step when building multi-turn datasets for text or video retrieval.
  • The size increase supports statistical comparisons between single-turn and multi-turn performance that smaller resources could not sustain.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The paper claims to construct CIRCLED, a multi-turn composed image retrieval (MTCIR) dataset with 22,608 sessions across nine subsets by extending FashionIQ, CIRR, and CIRCO. Sessions are generated via a CIReVL-based retrieval pipeline and curated using filters on retrieval success, turn length, consistency, and information redundancy to ensure progressive refinement toward target images and cross-domain generality; baseline methods are then evaluated for retrieval accuracy, with the dataset and code released publicly.

Significance. If the construction pipeline demonstrably yields consistent, artifact-free multi-turn dialogues, CIRCLED would supply a substantially larger and more domain-general benchmark than Multi-turn FashionIQ, supporting research on progressive query refinement in CIR.

major comments (2)
  1. [Abstract] Abstract: the generation and filtering steps are described but no quantitative evidence, ablation studies, or human validation results are supplied to confirm that the CIReVL pipeline plus filters actually achieve the claimed dialogue-history consistency and absence of artifacts or domain bias.
  2. [Dataset Construction] Dataset construction section: the central claim that the automated pipeline produces high-quality, consistent multi-turn sessions rests entirely on the unvalidated combination of CIReVL retrieval and heuristic filters; without reported consistency metrics, human ratings, or ablation on filter impact, the scale and generality advantage cannot be substantiated.
minor comments (1)
  1. The public release of dataset and code is a positive step for reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback highlighting the need for validation of the dataset construction process. We address each major comment below and agree that additional quantitative evidence and studies will strengthen the claims.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the generation and filtering steps are described but no quantitative evidence, ablation studies, or human validation results are supplied to confirm that the CIReVL pipeline plus filters actually achieve the claimed dialogue-history consistency and absence of artifacts or domain bias.

    Authors: We agree that the abstract (and manuscript) would be improved by including such evidence. The current version describes the filters but does not report metrics or human studies. In the revised manuscript we will add quantitative consistency metrics (e.g., retrieval success rates pre- and post-filtering), an ablation on filter impact, and results from a human validation study on a sampled subset of sessions to confirm progressive refinement, lack of artifacts, and cross-domain generality. revision: yes

  2. Referee: [Dataset Construction] Dataset construction section: the central claim that the automated pipeline produces high-quality, consistent multi-turn sessions rests entirely on the unvalidated combination of CIReVL retrieval and heuristic filters; without reported consistency metrics, human ratings, or ablation on filter impact, the scale and generality advantage cannot be substantiated.

    Authors: We acknowledge that the central claim currently rests on the pipeline description and filter design without explicit supporting metrics or studies. While the filters target the stated properties, we will revise the Dataset Construction section to include consistency metrics, human ratings on dialogue quality, and ablation experiments showing the contribution of each filter. These additions will better substantiate the scale and generality advantages relative to Multi-turn FashionIQ. revision: yes

Circularity Check

0 steps flagged

No circularity: dataset construction is self-contained description of pipeline from external sources

full rationale

The paper's contribution is empirical dataset construction by extending FashionIQ, CIRR, and CIRCO via a described CIReVL-based retrieval pipeline plus heuristic filters. No equations, parameter fitting, or derivations exist. No self-citation chain is load-bearing for a mathematical claim, and the construction steps do not reduce to tautology or rename fitted inputs as predictions. The central claim (scale and generality of 22,608 sessions) is a direct count from the process, not a derived result forced by prior self-work. This matches the default case of a non-circular dataset paper.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

This is a dataset-construction paper; it introduces no free parameters, mathematical axioms, or invented entities. All content rests on existing public datasets and an off-the-shelf retrieval pipeline.

pith-pipeline@v0.9.1-grok · 5717 in / 1069 out tokens · 35429 ms · 2026-06-29T18:36:28.461179+00:00 · methodology

0 comments
read the original abstract

Existing Multi-Turn Composed Image Retrieval (MTCIR) datasets lack dialogue-history consistency and are restricted to the fashion domain. To address these limitations, we construct CIRCLED by extending FashionIQ, CIRR, and CIRCO. In CIRCLED, the query at each turn progressively approaches the target image. Data are generated via a CIReVL-based retrieval pipeline and curated with multiple filters on retrieval success, turn length, consistency, and information redundancy to ensure quality. In total, we collect 22,608 multi-turn sessions across nine subsets, substantially exceeding Multi-turn FashionIQ (11,505 sessions) in both scale and generality. We further apply multiple baseline methods and quantitatively assess retrieval accuracy on CIRCLED. Our work provides a practical, high-quality benchmark to facilitate future research on multi-turn CIR. The dataset and code are publicly available at https://huggingface.co/datasets/tk1441/CIRCLED and https://github.com/mti-lab/circled.

Figures

Figures reproduced from arXiv: 2605.26734 by Osamu Torii, Tomohisa Takeda, Youyang Ng, Yu-Chieh Lin, Yuji Nozawa, Yusuke Matsui.

Figure 1
Figure 1. Figure 1: Example of multi-turn CIR search. When users cannot find the image they want, [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Comparison of multi-turn CIR datasets. Solid arrows indicate the progression [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Word clouds of relative captions. Top row: Fashion domain (FashionIQ, Multi [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Multi-turn image retrieval pipeline. (a) In Turn 1, we merge the caption generated [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Examples rejected by our filtering. (a) Rank-margin filter ( [PITH_FULL_IMAGE:figures/full_fig_p013_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Examples of filtered CIRCLED sessions demonstrating gradual progression toward [PITH_FULL_IMAGE:figures/full_fig_p014_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Comparison of Hits@10 by turn across subsets. [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Final Recall@10 and AUC over Hits@10 across baselines, by subset (codes as in [PITH_FULL_IMAGE:figures/full_fig_p017_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Example results for history-integration strategies. On the dataset extended from [PITH_FULL_IMAGE:figures/full_fig_p022_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

12 extracted references · 3 canonical work pages · 1 internal anchor

  1. [1]

    Datasheets for Datasets

    The license information for the images used in this paper is provided in Appendix F. 18 CIRCLED: Multi-turn CIR Dataset Yanzhe Chen, Zhiwen Yang, Jinglin Xu, and Yuxin Peng. Mai: A multi-turn aggregation- iteration model for composed image retrieval.https://openreview.net/forum?id= gXyWbl71n1, 2025. ICLR 2025 submission. Timnit Gebru, Jamie Morgenstern, B...

    work page Pith review arXiv 2025

  2. [2]

    Self-Preference Bias in LLM-as-a-Judge

    URLhttps://arxiv.org/abs/2410.21819. Hui Wu, Yupeng Gao, Xiaoxiao Guo, Ziad Al-Halah, Steven Rennie, Kristen Grauman, and Rogerio Feris. The fashion iq dataset: Retrieving images by combining side information and relative natural language feedback. InCVPR, pages 11307–11317, 2021. Yifei Yuan and Wai Lam. Conversational fashion image retrieval via multitur...

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  3. [3]

    Be ex tr em el y specific about colors , positions , or actions

  4. [4]

    left " or

    Avoid relative terms like " left " or " right "

  5. [5]

    Do not use quotes or e x p l a n a t o r y text

  6. [6]

    Focus on a single , clear change

  7. [7]

    23 Takeda et al

    Must be di ff er en t from previous s u g g e s t i o n s C.3 Auxiliary Caption Generation The prompt for merging the reference image caption with the relative caption to create an auxiliary caption for retrieval. 23 Takeda et al. A user is p e r f o r m i n g image r et ri ev al . The user provides a r ef er en ce image caption and a m o d i f i c a t i ...

  8. [8]

    SESSION N A T U R A L N E S S : Are the u t t e r a n c e s c o n s i s t e n t l y human - like t h r o u g h o u t the session ?

  9. [9]

    CO HE RE NC E / C O N S I S T E N C Y : Is there logical flow without c o n t r a d i c t i o n s or abrupt changes ?

  10. [10]

    GOAL - DIRECTED PROGRESS : Does each turn move closer to the target image Z ?

  11. [11]

    R E D U N D A N C Y ( higher = better ) : Is there low r e p e t i t i o n of i n f o r m a t i o n ?

  12. [12]

    generated image

    OVERALL : Holistic quality of the entire session Appendix E. Image Generation for FashionIQ Data For licensing reasons, in this paper we use generated images in the figures about Fash- ionIQ dataset. The images are generated by GPT-4o image generation function, using text prompts that describe the images in FashionIQ dataset. Appendix F. Image Licenses Th...

    work page arXiv 2054