{"id":"909b80a7-055e-44f0-90ac-d7b00c371237","arxiv_id":"2506.18246","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"The authors propose REIR (retrieval plus localization of object instances from text queries), construct the REIRCOCO benchmark, and report a dual-stream contrastive baseline, CLARE, that outperforms two-stage TIR+REC baselines.","lead":"This paper defines Referring Expression Instance Retrieval (REIR), a task that asks a model to find and localize an object in a large image gallery from a natural language description. It introduces a large AI-generated benchmark, REIRCOCO, and an end-to-end baseline called CLARE that beats untrained retrieval and grounding baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"REIRCOCO test images may not be disjoint from MSCOCO/RefCOCO images used in Stage 1 pretraining, so the reported state-of-the-art gains could be inflated by image-level leakage.","rationale":"The reader's weakest assumption correctly identifies split hygiene as the most fragile premise. The paper's own description makes the leak plausible: REIRCOCO is constructed from MSCOCO/RefCOCO (Section 3.2), and pretraining uses exactly those datasets (Appendix A.1), yet no explicit exclusion of test images is stated. If the splits overlap, the central comparison collapses because the model could memorize test images and their boxes during pretraining. This is not an accusation of misconduct; it is a missing control that must be documented for the benchmark to support the claimed results. I also note two secondary issues that weaken the SOTA claim but do not change the verdict: the two-stage baselines in Table 1 are not described as trained on REIRCOCO, so the comparison may be between a fine-tuned model and frozen baselines, and Appendix B.2 admits that DeepSeek R1 filters descriptions without seeing images, weakening the claimed uniqueness guarantee. Both are addressable, but the image-overlap question is the first to settle because it affects every reported number. The paper otherwise has real strengths: the task definition is clear, the ablations isolate MORE and CLIA contributions, and the qualitative examples provide supporting evidence. The conditional verdict remains appropriate, and my stress-test does not require changing it.","tokens_in":21645,"tokens_out":5985,"duration_ms":70883,"concrete_test":"Publish exact image-ID lists for the REIRCOCO train/test split and for the Stage 1 pretraining set described in Appendix A.1, then compute the intersection between REIRCOCO test image IDs and the pretraining image IDs (COCO detection plus RefCOCO/RefCOCO+/RefCOCOg). If the intersection is nonempty, rerun Tables 1-3 on a disjoint held-out split and report whether CLARE's margins persist. As a secondary check, fine-tune the strongest two-stage baseline on REIRCOCO train with the same data used for CLARE to test whether the quoted gap is partly an artifact of comparing a tuned end-to-end model against frozen baselines.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline claim in Section 5.3 that CLARE achieves state-of-the-art REIR performance presupposes a clean evaluation. REIRCOCO is built from MSCOCO and RefCOCO instances (Section 3.2), and Stage 1 pretraining uses MSCOCO and RefCOCO for detection and grounding supervision (Appendix A.1). Nowhere does the paper document a train/test split or state that REIRCOCO test images were excluded from pretraining. If test images appeared in pretraining with their ground-truth boxes, CLARE could exploit image-level familiarity rather than instance-level alignment, inflating the BoxRecall numbers in Tables 1 and 3, and also the RefCOCO results in Table 2 if REIRCOCO includes RefCOCO test images. The absence of this split documentation is an omitted support for the central comparison; the state-of-the-art claim is only as strong as the split is clean.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a new visual-language task, Referring Expression Instance Retrieval (REIR), which requires retrieving the correct image from a gallery and localizing the referred object instance given a fine-grained natural-language expression. To support the task, the authors construct a large benchmark, REIRCOCO, by prompting GPT-4o to generate five referring expressions per object instance from MSCOCO and RefCOCO annotations, then filtering with DeepSeek R1. They propose an end-to-end baseline, CLARE, a dual-stream SigLIP + Deformable-DETR architecture trained in two stages (detection/REC pretraining, then REIRCOCO finetuning with a contrastive language-instance alignment loss and a Mix of Relation Experts module). Experiments compare CLARE with two-stage TIR+REC baselines on REIRCOCO, report REC results on RefCOCO/+/g, and TIR results on REIRCOCO, concluding that CLARE achieves state-of-the-art REIR performance and strong generalization.","tokens_in":21893,"tokens_out":3906,"duration_ms":44290,"significance":"The task definition is timely and addresses a real gap between image-level retrieval and within-image localization. The dataset construction pipeline is scalable and the decision to release code and benchmark is a concrete contribution. If the evaluation is sound, CLARE provides a credible end-to-end architecture that avoids per-image cross-modal fusion, which could be useful for gallery-scale instance search. The ablation studies (Tables 4-7) directly test several design choices and are informative. However, the central state-of-the-art claim depends on two unverified assumptions: that REIRCOCO test images are disjoint from pretraining images, and that the comparison protocols against two-stage baselines are fair. These issues, if unresolved, materially weaken the empirical contribution.","major_comments":[{"comment":"The paper does not document whether the REIRCOCO test set images are disjoint from the MSCOCO and RefCOCO images used in Stage 1 pretraining (Appendix A.1 describes pretraining on MSCOCO for detection and RefCOCO/+/g for REC). Because REIRCOCO is constructed from these exact datasets (Section 3.2), the model may have seen the test images and their ground-truth boxes during pretraining, which would inflate BoxRecall numbers in Tables 1 and 3. The authors must state how the REIRCOCO train/test split is defined, verify that no pretraining image appears in the REIRCOCO test set, and report the degree of overlap. Without this, the headline comparison in Section 5.3 is not trustworthy.","section":"Section 3.2 and Appendix A.1"},{"comment":"The two-stage baselines (CLIP/EVA-CLIP/SigLIP combined with DeepSeek-VL2 or SimVG) are used off-the-shelf, with no indication that they were trained or fine-tuned on REIRCOCO. In contrast, CLARE is explicitly fine-tuned on REIRCOCO in Stage 2 (Section A.1, Eq. 6). This is a zero-shot-versus-supervised comparison, and the claim that CLARE 'outperforms all combinations of two-stage baselines' is therefore not a meaningful state-of-the-art result for a newly proposed task. The authors should either fine-tune the two-stage components on REIRCOCO (e.g., fine-tune SimVG on REIRCOCO expressions, or train a contrastive adapter on top of TIR features) and report those results, or evaluate CLARE in a zero-shot setting without REIRCOCO finetuning.","section":"Section 5.3, Table 1"},{"comment":"The paper repeatedly calls CLARE 'state-of-the-art' on REIR, but REIR is a new benchmark introduced in the same paper and the model is trained and tuned on that benchmark. This is a self-referential evaluation loop: the benchmark is authored by the same group, and the baseline is a method they developed and tuned. The authors should temper this claim and clarify that this is a first baseline on a new task, not a claim of superiority on an established benchmark. Independent validation (e.g., by external groups, or by testing CLARE on a held-out set constructed without the authors' pipeline) would be needed to substantiate the 'state-of-the-art' wording.","section":"Sections 1, 5.3, and 6"}],"minor_comments":[{"comment":"The heading 'Ablation study on CILA' uses 'CILA' while the model component is consistently named CLIA elsewhere; please fix this typo.","section":"Table 6 heading"},{"comment":"The definition of BoxRecall@k says 'given a batch of N referring expressions' but the gallery of instance candidates is not explicitly defined. It would be helpful to state whether the gallery is the full test set, a randomly sampled subset, or the batch itself, and to specify how the top-k candidates are formed across images during inference.","section":"Section 5.1, Eq. 9"},{"comment":"The filtering stage uses DeepSeek R1, which does not process images; it evaluates descriptions using only captions and the GPT-generated text. The paper should discuss the possibility that this text-only filter may retain expressions that misdescribe the actual visual content, and quantify the impact on dataset quality (e.g., by reporting a human-annotated quality sample).","section":"Section B.2"},{"comment":"The abstract states 'over 30,000 images and 200,000 uniquely annotated object instances,' while Section 3.2 gives exact numbers '30,106 images and 215,835 object instances' and later '613,548 fine-grained descriptions.' Please align these numbers for consistency.","section":"Abstract and Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The most serious issue is the unverified test/train image disjointness between REIRCOCO and the pretraining data; this must be checked before the claims in Tables 1 and 3 can be accepted. I would suggest the editor ask the authors to provide a clear statement of the dataset split and to confirm that no pretraining images leak into the REIRCOCO test set. The second issue, that the baselines are not trained on REIR, is also critical and should be addressed by retraining or by rephrasing the comparison as zero-shot versus supervised."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the quick take: the REIR task and REIRCOCO are genuinely useful, and CLARE is a reasonable first model. But before I'd trust the SOTA numbers, I need the authors to state that the REIRCOCO test images are disjoint from the COCO/RefCOCO images used in Stage 1 pretraining. They never say that, and given both are built from the same sources, it's a load-bearing omission.\n\nWhat's actually new: the task definition is clean — retrieve and localize a referred instance from a gallery — and REIRCOCO is a large-scale resource with over 600k expressions. The two-stage LLM pipeline (GPT-4o generation, DeepSeek filtering) is thoughtful, with explicit prompts to ensure referential uniqueness. The CLIA loss is a sensible extension of SigLIP to instance-level alignment, and the ablations are informative: pretraining matters, the MORE module gives a modest but consistent gain, and the contrastive loss is essential. The REC results on RefCOCO family are a useful sanity check and show the model is not just tuned to REIRCOCO.\n\nSoft spots, in proportion. The main one is the split documentation. REIRCOCO is built from MSCOCO and RefCOCO, and Stage 1 pretraining uses those same datasets. The paper does not say that REIRCOCO test images were excluded from pretraining, nor does it describe how the REIRCOCO train/test split was constructed beyond a passing mention of a \"5k-query and 5k-image split.\" If test images appeared in pretraining with ground-truth boxes, the BoxRecall numbers in Tables 1 and 3 could be inflated by image-level familiarity rather than instance-level alignment. This is an omitted support for the central comparison, and the stress-test note is right to flag it. The fix is straightforward: document the split and confirm disjointness. Minor issues: the hyperparameter descriptions disagree between Section 5.2, Appendix A.1, and Table 8 (batch sizes of 4 vs 8 per GPU), and \"CILA\" appears in Table 6 where it should be \"CLIA.\" Also, the REIR evaluation protocol — gallery construction, number of queries, distractor statistics — is under-specified; it should be in the main text or a supplementary.\n\nWho this is for: anyone working on retrieval, grounding, or benchmark design. The paper deserves a serious referee: the task and dataset are valuable contributions even if CLARE's margins shrink once the split is clarified. I'd send it to review with the split documentation as a required revision, not a desk reject.\n\nRecommendation: engage with it, conditional on the authors providing a clean train/test split statement and a complete evaluation protocol.","headline":"Useful new benchmark and a plausible baseline, but the missing train/test disjointness statement between REIRCOCO and the pretraining data is a load-bearing omission that needs to be fixed before the SOTA claims can be trusted.","tokens_in":22432,"tokens_out":2881,"would_cite":true,"duration_ms":32046,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces REIR, a task that requires retrieving and localizing a specific object instance from a gallery based on a fine-grained referring expression, and proposes CLARE, an end-to-end dual-stream model that learns…","keywords":["Referring Expression Instance Retrieval","REIRCOCO","contrastive language-instance alignment","referring expression comprehension","text-image retrieval","mixture of experts","end-to-end grounding","vision-language pretraining"],"falsifier":"Rebuild the REIRCOCO evaluation with a split whose test images are guaranteed absent from every pretraining stage, and check whether CLARE's BR@1 at IoU 0.5 stays near 29.5; a large drop would indicate the reported numbers depended on images and boxes seen during pretraining.","tokens_in":21469,"feed_emoji":"🔍","tokens_out":7200,"duration_ms":59505,"temperature":0.7,"pith_summary":"Referring Expression Instance Retrieval (REIR) asks a model to take a fine-grained natural language description of a specific object and return both the gallery image containing that object and the object's bounding box. The paper argues that existing tasks fail this setting: text-image retrieval finds images but not instances, and referring expression comprehension grounds instances but cannot rank across a gallery. To support the task, the authors build REIRCOCO, a dataset of over 30,000 images and 613,548 instance-level expressions generated by prompting vision-language models and filtered for uniqueness. They also present CLARE, an end-to-end dual-stream model that aligns text queries with detected object instances through a contrastive objective and a mixture of relation experts. CLARE is reported to outperform all two-stage TIR-plus-REC baselines on REIRCOCO across IoU thresholds and ranking levels, and to generalize competitively to standard referring expression comprehension and text-image retrieval benchmarks.","feed_headline":"One end-to-end model retrieves and localizes instances from galleries","feed_subtitle":"A new benchmark plus contrastive baseline returns the right image and the exact object box from a fine-grained text query.","key_machinery":"The load-bearing components are the Contrastive Language-Instance Alignment (CLIA) loss and the dual-stream architecture that makes it possible. CLARE uses SigLIP vision and text encoders to produce independent image and text features, a Deformable-DETR-based object extractor to propose up to 900 candidates per image with instance embeddings, and a Mix of Relation Experts (MORE) module that refines the text embedding through one shared expert plus top-2 routing among four specialized experts. CLIA treats each expression-object pair across the whole batch as a binary classification problem with a learnable temperature and bias, so the model learns to separate the true referent from distractors in other images. The same object features feed a position decoder that outputs bounding boxes, so retrieval and localization share one representation and one score.","core_discovery":"The paper's central claim is that instance-level retrieval and localization can be solved jointly by an end-to-end contrastive model rather than by cascading a text-image retriever with a referring-expression comprehender. CLARE encodes the expression into a query embedding, detects object candidates in each gallery image, and scores every candidate by dot-product similarity with the query; the top-scoring bounding box is the answer. A contrastive objective, CLIA, extends the SigLIP sigmoid loss to align expressions with objects across images and batches, while a Mix of Relation Experts (MORE) routes the text embedding through shared and specialized experts to capture spatial and relational cues. On the REIRCOCO benchmark the authors report state-of-the-art BoxRecall@k across IoU thresholds 0.5, 0.7, and 0.9, with the ViT-L variant reaching 29.53 BR@1 at IoU 0.5, and also report strong transfer to REC and TIR benchmarks. The conclusion the authors draw is that cross-image instance-level contrastive alignment is the right inductive bias for grounding fine-grained expressions in open galleries.","pith_inferences":["If the test images in REIRCOCO are not disjoint from the COCO/RefCOCO images used in pretraining, the reported gains could be inflated by the model having seen the exact test images and their ground-truth boxes; a clean split would settle this.","REIRCOCO's expressions are generated and filtered by LLMs, so the benchmark inherits whatever distributional biases GPT-4o and DeepSeek-R1 have; human verification of a sample would test whether the 'unambiguous' property really holds at gallery scale.","The success of instance-level contrastive alignment suggests a natural extension: using REIRCOCO as a pretraining corpus for downstream REC or TIR tasks, or as a training signal for open-vocabulary detection.","A testable extension would be to vary the number of routed experts and the gallery size to measure where relational reasoning saturates, since the paper only ablates one configuration at 4 routed experts with top-2."],"forward_implications":["If CLARE's results hold, gallery-scale instance search becomes feasible without per-image cross-modal fusion: object features are precomputed once, and a query is encoded once and matched by dot product.","The REIRCOCO benchmark gives the community a way to measure retrieval and grounding jointly, so future models can be compared on a single BoxRecall@k metric instead of separate recall and precision scores.","The reported transfer results suggest that instance-level contrastive alignment is a competitive alternative to early-fusion REC models, potentially simplifying training pipelines.","Because CLARE outperforms cascades of strong TIR and REC models, the paper argues that end-to-end training avoids error accumulation in the retrieve-then-localize pipeline.","The staging of pretraining on detection/REC data followed by REIRCOCO fine-tuning is shown to be necessary; removing either stage drops BR@1 from 26.39 to 13.44 or 4.02."],"supporting_citations":[{"why":"Supplies the SigLIP vision and text encoders and the sigmoid contrastive objective that CLIA extends to instance-level alignment.","marker":"[54]"},{"why":"Provides the Deformable DETR encoder-decoder used as the object extractor that proposes candidate instances.","marker":"[58]"},{"why":"Provides the MSCOCO images and object-box annotations from which REIRCOCO is built and on which detection pretraining runs.","marker":"[29]"},{"why":"Provides RefCOCO images and expressions used both in REIRCOCO construction and in grounding pretraining.","marker":"[20]"},{"why":"Provides RefCOCOg data used in the grounding pretraining stage for modeling context in referring expressions.","marker":"[35]"},{"why":"GPT-4o generates the candidate referring expressions in the REIRCOCO pipeline.","marker":"[1]"},{"why":"DeepSeek-R1 filters the generated expressions for accuracy and uniqueness.","marker":"[10]"},{"why":"DeepSeek-VL2 serves as a state-of-the-art REC baseline in the two-stage comparisons.","marker":"[47]"},{"why":"SimVG serves as an REC expert baseline in the two-stage comparisons.","marker":"[9]"},{"why":"CLIP serves as a TIR baseline in the two-stage pipeline and in the TIR generalization experiments.","marker":"[38]"}],"fun_headline_variants":["Instance-level retrieval and localization in one end-to-end pass","From text query to exact object box across many images","CLARE closes the gap between retrieval and grounding","New benchmark and baseline for referring instance retrieval","Fine-grained text finds its object and its image at once"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The REIRCOCO test images are not the same images used in pretraining, a fact the paper does not explicitly state; if the split overlaps with COCO or RefCOCO training data, the reported retrieval-and-localization numbers could be inflated.","fun_headline_variants_meta":{"raw":{"variants":["Instance-level retrieval and localization in one end-to-end pass","From text query to exact object box across many images","CLARE closes the gap between retrieval and grounding","New benchmark and baseline for referring instance retrieval","Fine-grained text finds its object and its image at once"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00064,"raw_usage":{"total_tokens":3017,"prompt_tokens":1086,"completion_tokens":1931,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":702,"completion_tokens_details":{"reasoning_tokens":1856}},"tokens_in":702,"tokens_out":1931,"duration_ms":17149,"temperature":1.0,"reasoning_tokens":1856,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:23:01.037590+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rebuild the REIRCOCO evaluation with a split whose test images are guaranteed absent from every pretraining stage, and check whether CLARE's BR@1 at IoU 0.5 stays near 29.5; a large drop would indicate the reported numbers depended on images and boxes seen during pretraining.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides RefCOCOg data used in the grounding pretraining stage for modeling context in referring expressions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GPT-4o generates the candidate referring expressions in the REIRCOCO pipeline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DeepSeek-R1 filters the generated expressions for accuracy and uniqueness."}],"review_version":1}