{"id":"191c936c-4e3d-4945-a5f7-3e6f7064c5bf","arxiv_id":"2506.22864","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A zero-shot pipeline combining SAM2 masks, Alpha-CLIP region embeddings, and Qwen2.5-VL reranking and grounding improves mask-aware text-to-image retrieval on COCO and D3.","lead":"This paper defines MaTIR, a task that retrieves images from text and also returns the mask of the described object, and proposes a zero-shot two-stage pipeline. It indexes SAM2 masks with Alpha-CLIP region embeddings, then uses an MLLM to rerank and locate objects.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reranking gains depend on unvalidated Qwen2.5-VL True/False logits; a hard-label reranking ablation would show whether Eq. 2 is load-bearing.","rationale":"The central claim is that the two-stage zero-shot pipeline solves MaTIR. The most load-bearing component is the MLLM reranking, because it produces the largest absolute gains (9.4 and 20.3 points). Eq. 2 defines the reranker but the paper does not validate that the model's token logits constitute a monotone relevance score. The reader's 'calibration across images' concern is slightly too strong: ranking only requires monotonicity in z_True - z_False, not calibrated probabilities. But the underlying worry stands. The proposed hard-label ablation is decisive because it changes only the scoring function while keeping the same model and candidates. I also considered pretraining leakage (COCO/D3 images may appear in Alpha-CLIP, SAM2, or Qwen2.5-VL training), which is a real secondary concern for the 'zero-shot' generalization claim, but it does not single out a specific equation and is less directly testable within the paper's framework. Therefore I keep the reader's CONDITIONAL verdict unchanged.","tokens_in":9318,"tokens_out":10902,"duration_ms":129242,"concrete_test":"Re-run Table 1 with a hard-label reranker: parse Qwen2.5-VL's generated answer as True/False (discarding any other response) and sort all candidates with True above False, keeping relative order from Stage 1 within each group; recompute mAP@50 on COCO and D3. If hard-label reranking reproduces most of the 9.4/20.3-point gains, Eq. 2's soft logits are not load-bearing and the concern is settled. If gains collapse, the soft logits matter; then also compute rank correlation between S' and human/GT labels on a sample of 200 candidates to check monotonicity, and report the fraction of queries where z_True and z_False are not the top-two first-token logits.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 (Eq. 2) constructs the reranking score S' = exp(z_True)/(exp(z_True)+exp(z_False)) from Qwen2.5-VL 7B token logits. This score is responsible for the largest reported gains: in Table 1, reranking lifts mAP@50 from 83.54 to 92.97 on COCO and from 40.75 to 61.00 on D3. The paper never validates that z_True - z_False is a reliable relevance signal for the candidate set; no calibration check, no comparison to alternative rerankers, no analysis of tie/abstention behavior. Note that for mere reranking, cross-image calibration is not necessary: any monotone transform of z_True - z_False preserves order, so the load-bearing assumption is narrower than the reader's 'comparable across images' phrasing. What must hold is that the logit difference is monotonically related to the probability that the query object is present in the candidate image. Because Qwen2.5-VL is a generative instruction-tuned model, its first-token logits are not designed for this scoring task; the two-token softmax also ignores probability mass on other tokens. If the logit difference is noisy or biased (e.g., position bias, object-count bias, or hallucination), the reported reranking gains are artifacts of a particular score, not of multimodal reasoning. The paper's own ablation (Table 1 'Ours w/o reranking') shows the pipeline already works without it, but the headline numbers and the interpretation of MLLM understanding depend on this unvalidated scoring function.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Mask-aware Text-to-Image Retrieval (MaTIR), a task in which a text query must retrieve images containing the described object and also output a segmentation mask for that object. The proposed method is a zero-shot, two-stage pipeline: (i) offline SAM 2 mask proposals and Alpha-CLIP region embeddings enable coarse nearest-neighbor retrieval (Eq. 1), and (ii) Qwen2.5-VL reranks the top-N_C candidates via a softmax over True/False token logits (Eq. 2), then generates bounding boxes that are matched to SAM 2 masks by IoU. The paper reports strong gains over CLIP-based retrieval baselines on COCO and D3 (Table 1: mAP@50 92.97 vs 78.07 on COCO; 61.00 vs 29.49 on D3) and over LISA++, SAM4MLLM, and APE on the combined metric mAP@50@50 (Table 2). Ablations in Tables 3 and 4 support the choices of Alpha-CLIP region embeddings over cropping, and of the MLLM grounding stage over stage-1-only mask selection.","tokens_in":9676,"tokens_out":2488,"duration_ms":27764,"significance":"If the reported results are reliable, the paper makes a useful contribution: it proposes a new benchmark task (MaTIR) with a metric (mAP@50@50), demonstrates that a purely zero-shot, training-free composition of SAM 2, Alpha-CLIP, and an MLLM can outperform specialized RES models on the new task, and releases code. The design choices are well motivated and the ablations are consistent with the narrative. However, the central quantitative claim depends on an unvalidated reranking score (Eq. 2), and the evaluation lacks error bars or significance tests, which tempers confidence in the headline gains. The work is incremental but reasonable for a workshop-style/short conference paper.","major_comments":[{"comment":"The reranking score S' = exp(z_True)/(exp(z_True)+exp(z_False)) is the sole component responsible for the substantial gains in Table 1 (COCO: 83.54 to 92.97; D3: 40.75 to 61.00). The paper provides no validation that the Qwen2.5-VL token logits z_True and z_False are monotonically related to the probability that the described object is present in the candidate image. Because the two-token softmax discards probability mass on other tokens and the model is an instruction-tuned generative model whose first-token logits are not designed for scoring, the reranking gains could be an artifact of this particular scoring function. Please add an ablation that reranks using hard True/False labels from the model's decoded response, or another validated relevance score, and report whether the gains persist; also report calibration or agreement statistics for Eq. (2).","section":"§3.2, Eq. (2) and Table 1"},{"comment":"The evaluation reports no error bars, confidence intervals, or significance tests. The D3 benchmark uses only 422 queries and the COCO gallery evaluation uses 80 category queries; the mAP estimates may have substantial variance. The 20.3-point D3 reranking gain is large, but without repeated runs or statistical testing, it is difficult to assess whether the gap is robust. Please provide standard deviations over multiple runs, or at least a paired bootstrap or permutation test for the main comparisons in Tables 1 and 2.","section":"§4 and Tables 1–4"},{"comment":"The RES baselines (LISA++, SAM4MLLM, APE) are evaluated only on the top-N_K images retrieved by the authors' own pipeline, so Table 2 measures segmentation quality conditioned on the same retrieval results. This is a fair isolation of the segmentation stage, but the paper's abstract and conclusion state that the method outperforms previous RES methods on the MaTIR task as a whole. An end-to-end comparison in which each RES method selects its own retrieved set (or the paper explicitly scopes the claim to the segmentation stage) would be more precise.","section":"§4.1, Table 2"}],"minor_comments":[{"comment":"The text describes 'LAVT combines textual features and visual features through a pixel-word attention module' but does not provide a citation or reference list entry; please add the LAVT reference.","section":"§2, Related Work"},{"comment":"The 7 best CLIP prompts used for text ensembling are not specified; please list them or cite the source so the experiments are reproducible.","section":"§4, Implementation Details"},{"comment":"The definition of mAP@50@50 says a mask is correct if it matches a ground-truth mask with IoU 0.5 or higher. Please clarify whether this is mask IoU or box IoU, and how multiple correct masks per image are handled in the average-precision computation.","section":"§3.2, Evaluation Metric"},{"comment":"The COCO gallery uses only 80 category-name queries, which is a very small query set; please mention this limitation explicitly when interpreting the COCO results.","section":"§4.1, Datasets"},{"comment":"The table caption says 'CLIPResNet50x64' and 'CLIPViT-L/14' without a space; consider formatting for readability, and note that the two are different backbone variants.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short ICMR paper with a non-trivial new task and a reasonable pipeline, but the main empirical claim rests on an unvalidated logit-based reranking score. The required additions (calibration analysis or an alternative reranker baseline, plus error bars or significance tests) are feasible within the scope of a revision. I would not reject, but I would not accept without addressing the reranking concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's real contribution is the task: MaTIR, which forces retrieval and referring segmentation into one evaluation, and the mAP@50@50 metric. That's a fair and useful framing, and the zero-shot pipeline (SAM2 masks, Alpha-CLIP region embeddings, Qwen2.5-VL reranking and grounding) is clean and reproducible, with code released. The ablations are honest: Alpha-CLIP beats cropping, and the full pipeline beats stage-1-only localization. I trust the coarse retrieval numbers.\n\nThe soft spot is exactly where the stress-test lands. Equation (2) turns Qwen2.5-VL's True/False logits into a reranking score, and the biggest reported gains (9.4 points on COCO, 20.3 on D3) come from that step. The paper never checks whether the logit difference is monotonically related to object presence; no hard-label ablation, no alternative scoring, no tie analysis. I agree with the stress-test note that cross-image calibration is not needed for reranking, but you do need monotonicity, and that is not demonstrated. A simple ablation swapping the softmax for the argmax token, or asking the MLLM for a confidence score, would settle it. Without that, the headline numbers are hard to interpret.\n\nThe evaluation is also thinner than I'd like. COCO queries are just 80 category names, not referring expressions; D3 has only 422 queries; no error bars or significance tests. The TIR baselines are only CLIP variants from one prior paper, and the RES baselines are applied only after the same retrieval/reranking, so the comparison is fair but narrow. There's also no leakage audit for pretrained models evaluated on COCO, which matters for a new benchmark.\n\nNone of this kills the central claim: the two-stage pipeline is a sensible zero-shot method for a new task, and the coarse retrieval already beats the compared baselines. The unvalidated logit score is addressable, not fatal. As a short conference paper, this is a reasonable first baseline for MaTIR. Readers working on interpretable retrieval or MLLM-based grounding will get value from it, and I'd use it as a citation for the task and metric. It deserves serious peer review, but the reviewer should insist on the reranking ablation before accepting the numbers.\n\nI'd bring it to reading group and would send it to review, but I'd not yet trust the reranking gains as evidence of MLLM understanding.","headline":"New task formulation is solid, but the reranking gains rest on an unvalidated MLLM logit score and the evaluation is thin.","tokens_in":753,"tokens_out":972,"would_cite":true,"duration_ms":39430,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A text query can retrieve both the right images and the exact object masks, and a two-stage zero-shot pipeline achieves it on COCO and D3.","keywords":["text-to-image retrieval","referring expression segmentation","mask grounding","zero-shot retrieval","SAM 2","Alpha-CLIP","multimodal large language model","retrieval interpretability"],"falsifier":"A calibration check would settle it: for queries whose reranking score is near 0.5, measure whether retrieval accuracy is near chance; alternatively, replace the score with the raw 'True' logit or a different MLLM and see whether the reported COCO and D3 gains survive.","tokens_in":9106,"feed_emoji":"🖼️","tokens_out":7867,"duration_ms":68795,"temperature":0.7,"pith_summary":"The paper introduces MaTIR, a task in which a textual query must return both the relevant images and the segmentation mask of the described object in each image. It argues that standard text-to-image retrieval lacks this interpretability and that running referring-expression segmentation on every gallery image is too expensive. To solve it, the paper proposes a zero-shot, two-stage pipeline: SAM 2 proposes object masks and Alpha-CLIP encodes each masked region offline, so online retrieval is a nearest-neighbor search, then Qwen2.5-VL reranks the top candidates and grounds the object with a bounding box that is matched to a SAM 2 mask by IoU. On COCO and D3 the pipeline reports mAP@50 of 92.97 and 61.00 and mAP@50@50 of 71.64 and 49.16, ahead of CLIP-based retrieval and RES baselines. If correct, it shows that mask-level interpretable text-to-image retrieval is achievable without any training.","feed_headline":"Zero-shot pipeline returns images plus the object masks they describe","feed_subtitle":"SAM 2 masks plus Alpha-CLIP embeddings and an MLLM reranker beat trained segmentation baselines on COCO and D3.","key_machinery":"The machinery is a two-stage retrieval cascade. In stage one, SAM 2 (the Segment Anything model for images and videos) proposes object masks for every gallery image, Alpha-CLIP, a CLIP variant with an auxiliary alpha channel for region awareness, encodes each mask together with its image into region-level embeddings, and the per-image score is the maximum cosine similarity over its masks. In stage two, Qwen2.5-VL reranks the top $N_C$ candidates using the softmax of its 'True' versus 'False' logits, keeps the top $N_K$, and outputs a bounding box for the object; the final mask is the SAM 2 mask whose bounding box has the highest IoU with the MLLM's box. This cascade converts a text query into both an image ranking and a mask in one pass, with all heavy feature computation done offline.","core_discovery":"The central claim is that a retrieval system can return, alongside each relevant image, a precise segmentation mask of the described object, and that this can be done zero-shot by combining a segmentation-aware embedder with an MLLM reranker. The paper defines the MaTIR task and the mAP@50@50 metric, then shows that the two-stage pipeline beats both whole-image retrieval baselines and post-retrieval RES models. The key evidence is that coarse Alpha-CLIP retrieval alone gives 83.54 and 40.75 mAP@50 on COCO and D3, reranking with the MLLM relevance score $S' = \\exp(z_{\\mathrm{True}})/(\\exp(z_{\\mathrm{True}})+\\exp(z_{\\mathrm{False}}))$ lifts it to 92.97 and 61.00, and the full grounding pipeline reaches 71.64 and 49.16 mAP@50@50, ahead of LISA++, SAM4MLLM, and APE. The paper attributes the advantage to Alpha-CLIP's region-level embeddings preserving contextual information and to the MLLM's ability to reason about fine-grained descriptions and output bounding boxes.","pith_inferences":["The reranking score assumes cross-image comparability of the MLLM's True/False logits; a calibration or temperature step could change the reported margins, and the D3 gain is the most sensitive test.","The pipeline is bounded by SAM 2's proposal recall: objects SAM 2 never proposes cannot be retrieved or masked, so failures should concentrate on rare or tiny objects even when the MLLM grounds them correctly.","Because the output is a mask rather than a caption, the same query could drive downstream uses such as region-specific image editing, grounded visual question answering, or training-data curation, which the paper does not explore.","The mAP@50@50 metric combines rank and mask IoU at a single threshold; a graded mask similarity or a recall-oriented variant could change the relative ranking of the baselines."],"forward_implications":["Mask-aware retrieval can be evaluated as one task rather than retrieval followed by segmentation, with mAP@50@50 as a shared metric.","A zero-shot, open-vocabulary pipeline can beat dedicated RES models (LISA++, SAM4MLLM, APE) when those models are applied after retrieval.","MLLM reranking adds large gains over segmentation-aware coarse retrieval, raising D3 mAP@50 from 40.75 to 61.00.","Alpha-CLIP region embeddings preserve contextual background information that cropping-based region features lose, improving COCO mAP@50 from 71.06 to 83.54 before reranking.","Offline mask and embedding indexing keeps online query cost low, so searching large galleries does not require segmenting every image at query time."],"supporting_citations":[{"why":"Supplies the object mask proposals for every gallery image, the basis of stage-one segmentation-aware retrieval.","marker":"[26]"},{"why":"Supplies the region-level visual embeddings aligned with text that carry the coarse retrieval stage.","marker":"[28]"},{"why":"The MLLM used for reranking via True/False logits and for bounding-box grounding.","marker":"[2]"},{"why":"Provides the whole-image retrieval baseline and the text encoder and prompt ensembling underlying Alpha-CLIP.","marker":"[25]"},{"why":"Provides the COCO retrieval setup and the Dense-CLIP and Cluster-CLIP baselines.","marker":"[16]"},{"why":"Supplies the D3 dataset with description-based mask annotations and complete annotations.","marker":"[31]"},{"why":"Supplies the COCO dataset used as the 5,000-image gallery with category-level queries.","marker":"[19]"},{"why":"One of the referring-expression-segmentation baselines compared in the object-level retrieval evaluation.","marker":"[33]"},{"why":"One of the referring-expression-segmentation baselines compared in the object-level retrieval evaluation.","marker":"[6]"},{"why":"One of the referring-expression-segmentation baselines compared in the object-level retrieval evaluation.","marker":"[27]"}],"fun_headline_variants":["Mask-aware retrieval: find images and segment objects in one pass","Zero-shot pipeline returns images plus precise object masks","MaTIR: unifying image search and object segmentation","SAM 2, Alpha-CLIP, and MLLM team up for mask-aware TIR","Beyond captions: retrieval that grounds objects with masks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire reranking gain rests on the assumption that the model's confidence in answering 'True' versus 'False' is a fair, comparable relevance score across different images.","fun_headline_variants_meta":{"raw":{"variants":["Mask-aware retrieval: find images and segment objects in one pass","Zero-shot pipeline returns images plus precise object masks","MaTIR: unifying image search and object segmentation","SAM 2, Alpha-CLIP, and MLLM team up for mask-aware TIR","Beyond captions: retrieval that grounds objects with masks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000215,"raw_usage":{"total_tokens":1452,"prompt_tokens":993,"completion_tokens":459,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":372}},"tokens_in":609,"tokens_out":459,"duration_ms":41151,"temperature":1.0,"reasoning_tokens":372,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:55:33.027848+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A calibration check would settle it: for queries whose reranking score is near 0.5, measure whether retrieval accuracy is near chance; alternatively, replace the score with the raw 'True' logit or a different MLLM and see whether the reported COCO and D3 gains survive.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the object mask proposals for every gallery image, the basis of stage-one segmentation-aware retrieval."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the region-level visual embeddings aligned with text that carry the coarse retrieval stage."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the whole-image retrieval baseline and the text encoder and prompt ensembling underlying Alpha-CLIP."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the COCO retrieval setup and the Dense-CLIP and Cluster-CLIP baselines."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the D3 dataset with description-based mask annotations and complete annotations."},{"cited_title":"Lawrence Zitnick","cited_arxiv_id":null,"evidence_quote":"Supplies the COCO dataset used as the 5,000-image gallery with category-level queries."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the referring-expression-segmentation baselines compared in the object-level retrieval evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the referring-expression-segmentation baselines compared in the object-level retrieval evaluation."}],"review_version":1}