{"id":"a8e07f4b-7de5-46ba-ac6e-28c843e201cc","arxiv_id":"2608.10524","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new multi-match surveillance TBIR benchmark and a fine-tuning framework with cross-modal soft labels and intra-modal distillation improve mAP@20 by 7.8 points over standard contrastive tuning.","lead":"This paper introduces a new surveillance image retrieval benchmark in which a single text query can match many images, along with a fine-tuning method that improves retrieval accuracy by about 8 points on average. It matters because current benchmarks assume one match per query, which is unrealistic for security and other domain-specific applications.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Potential train/test overlap between the unreleased internal training set and SecMM-TBIR could inflate the +7.8 mAP@20 gain; the paper asserts decoupling without providing overlap evidence.","rationale":"The most load-bearing condition for the central claim is that SecMM-TBIR is a valid held-out benchmark. The reader focused on label completeness; I agree that matters, but the more directly testable threat is train/test leakage. The paper's only evidence for source separation is the word 'decoupled' in Section 4.1; no dedup statistics, no source audit, no feature-level overlap check. Given that the benchmark data pool (Appendix A) includes self-collected surveillance data and public person/vehicle datasets, and the internal training set is built from real-world surveillance scenarios via Rex-Omni and Qwen3-VL, overlap is plausible. If overlap exists, the fine-tuned models could memorize the test images, inflating the SAFT-vs-ITC gap; the +7.8 point claim would then reflect benchmark contamination, not method quality. This does not imply any intent; it is a standard evaluation-integrity requirement. The proposed test, near-duplicate detection followed by decontaminated re-evaluation, can settle it. Because the data is unreleased, the concern is currently unresolvable by the community, which supports keeping the verdict conditional. I therefore recommend retaining the conditional verdict with the added condition that the authors must provide overlap statistics or release the data.","tokens_in":16954,"tokens_out":7490,"duration_ms":63091,"concrete_test":"After the benchmark and internal training set are released, compute perceptual hashes (e.g., pHash) and embedding cosine similarities between every SecMM-TBIR image and every internal training image, flagging pairs above a near-duplicate threshold. Remove all flagged images from the training set, retrain the ITC baseline and SAFT under the exact protocol of Section 4.1, and re-measure mAP@20 on SecMM-TBIR. If the average +7.8 point gain persists on the decontaminated training set, the concern is resolved; if the gain shrinks substantially or disappears, the central claim must be revised to acknowledge leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"SecMM-TBIR must be a clean held-out benchmark for the central claim to hold. Section 4.1 says the Specialized Internal Dataset is 'decoupled from SecMM-TBIR benchmark', but no overlap or deduplication statistics are reported. The benchmark pool (Appendix A) is sourced from self-collected surveillance data, web crawls, and public benchmarks like CUHK-PEDES, RSTPReid, ICFG-PEDES, SYNTH-PEDES, and CUHK-CompCars. The internal training set is also composed of real-world surveillance pedestrian and vehicle images built via Rex-Omni and Qwen3-VL captions. These sources are not shown to be disjoint. If even a few percent of benchmark images or near-duplicates appear in the internal training set, the fine-tuned models can memorize them; the reported +5.4/+10.3 mAP@20 gains over ITC on pedestrian/vehicle would then reflect dataset contamination rather than algorithmic superiority. Because both the training set and benchmark are unreleased, readers cannot currently check this premise. This concern is distinct from the reader's label-completeness worry: incomplete labels distort absolute scores and can bias any method, but leakage systematically inflates the relative gain claimed for the paper's fine-tuning approach.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses text-based image retrieval in domain-specific, semantically compressed settings such as surveillance. It introduces SecMM-TBIR, a multi-match benchmark of 50k surveillance images and 200 queries, built through a three-phase pipeline that uses LLM/VLM generation, clustering-based sampling, and multi-expert candidate retrieval followed by manual verification. It also proposes SAFT, a fine-tuning framework combining the standard ITC loss with Semantic-Aware Soft-Label Supervision (SASS) and Intra-modal Structural Distillation (ISD), where a frozen universal multi-modal embedder supplies cross-modal and intra-modal soft targets. Experiments on six CLIP-like models report an average mAP@20 gain of 7.8 points over ITC fine-tuning on SecMM-TBIR, with additional gains on Flickr30K, MS-COCO, Fashion200K, and ARO.","tokens_in":17199,"tokens_out":6040,"duration_ms":56451,"significance":"If the benchmark and method are sound, the paper makes a useful contribution: it identifies a real limitation of single-match TBIR evaluation in narrow domains, proposes a concrete data-construction pipeline, and demonstrates a training recipe that consistently improves over ITC across several architectures. The loss definitions in Section 3.3 are clear and the ablation studies, especially the tables showing incremental contributions of SASS and ISD, support the internal consistency of the method. The promised public release of the benchmark would be valuable to the community. However, the central empirical claims rest on a benchmark whose label completeness has not been demonstrated and on an unreleased internal training set whose decoupling from the benchmark is only asserted, so the headline gains cannot currently be independently verified.","major_comments":[{"comment":"The benchmark labels are defined as the union of top-1000 retrieval candidates from four expert embedders, followed by manual verification. Any true positive that is not retrieved by any of the four experts can never receive a positive label, and mAP@20 will penalize a model that retrieves such an unlabeled true positive. Because the paper's headline claim is the +7.8 mAP@20 gain on SecMM-TBIR, label completeness is load-bearing, yet no recall estimate for the candidate-generation step and no inter-annotator agreement statistics are reported. The authors should report the distribution of labeled positives per query, Cohen's kappa or a similar agreement measure for the manual verification, and an independent estimate of the false-negative rate in C(q), for example by annotating a random sample of images outside the candidate union.","section":"Section 3.1, Phase 2 and Phase 3; Eq. (1); Appendix A"},{"comment":"The paper states that the Specialized Internal Dataset is 'decoupled from SecMM-TBIR benchmark', but both the training set and the benchmark are built from real-world surveillance pedestrian and vehicle images, and no overlap or near-duplicate analysis is provided. Since both datasets are unreleased, readers cannot rule out that some benchmark images or near-duplicates appeared in the internal training set, which would inflate the reported +5.4/+10.3 mAP@20 gains over ITC through memorization rather than algorithmic improvement. The authors should provide quantitative overlap statistics (e.g., image hash or embedding-similarity distributions between the training set and SecMM-TBIR) or release both datasets so that the decoupling claim can be checked.","section":"Section 4.1, Specialized Internal Dataset"},{"comment":"All results are reported as point estimates without error bars, confidence intervals, or multiple seeds. The benchmark contains only 200 queries, so the per-query variance may be substantial, and the claimed 7.8-point average gain could in principle be within optimization or sampling noise for some model/domain combinations. The authors should report mean and standard deviation over at least three fine-tuning runs, or an equivalent query-level statistical test, for the main comparison between ITC and SAFT.","section":"Section 4.2, Tables 1 and 3"},{"comment":"There is a preference-alignment risk between the benchmark labels and the teacher model used by SAFT. The candidate set C(q) is generated by four multi-modal embedders, and SAFT distills soft targets from UniME-V2, also a large multi-modal embedder; if these models share a common notion of relevance, the evaluation could reward SAFT for matching the same embedder preferences that determined which labels exist, rather than for better retrieval per se. A concrete test would be to construct a small held-out subset whose labels come from exhaustive manual annotation independent of the four expert retrievers and to re-report SAFT versus ITC on that subset.","section":"Section 3.3 and Section 3.1, Phase 2"}],"minor_comments":[{"comment":"The caption states that MobileCLIP-S1 is allowed to 'outperform 2B pretrained embedders', but this holds only on the vehicle subset: in Table 4 the pedestrian mAP@20 of MobileCLIP-S1 with SAFT is 54.9, below UniME-V2-2B's 57.4. The claim should be qualified.","section":"Figure 1 caption and Table 4"},{"comment":"The table formatting is difficult to read; the '−3.3', '−1.8', etc. values are visually merged with the preceding numbers, and the model order (T-22, T-45, M-S0, M-S1, B/16, B/32) differs from the order in Table 1. This should be cleaned up.","section":"Table 2"},{"comment":"The Fashion200K evaluation uses a modified test set generated with Qwen3-VL and Qwen-3.5 rather than the original split, so the numbers in Table 9 are not directly comparable to published Fashion200K results. The authors should state this clearly in the main text or report results on the original test split.","section":"Appendix C, Table 9"},{"comment":"There is a typo: 'In contrasct' should be 'In contrast'.","section":"Section 2.2"},{"comment":"The definition of the hard-negative threshold alpha uses a margin beta, but the signs and relative scale are not discussed; clarifying why a positive margin excludes false negatives while retaining hard negatives would help readers reproduce the analysis in Table 5.","section":"Section 3.2, Eq. (2)"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the proposed method is plausible, but the two unreleased datasets—the benchmark and the internal training set—make the central empirical claim impossible to verify. I strongly encourage the editor to require, at minimum, quantitative overlap statistics and label-completeness evidence before further consideration. The reading of the benchmark construction also suggests that the authors should consider adding a robustness analysis against incomplete labels, since the candidate-union design is a central design choice."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper is worth your time. It builds a new multi-match retrieval benchmark for surveillance (SecMM-TBIR, 50k images, 200 queries) and a fine-tuning method, SAFT, that consistently improves over standard image-text contrastive learning across six different CLIP-like models. The reported +7.8 mAP@20 gain is not a one-off: it holds on pedestrian and vehicle subsets, and the method also nudges up performance on Flickr30K, MS-COCO, Fashion200K, and ARO. That is a believable, well-executed result.\n\nThe genuinely new pieces are the multi-match label paradigm for a compressed domain and the design of SAFT. Unlike prior soft-label methods that use unimodal teacher distributions, SAFT uses a cross-modal teacher (UniME-V2) to produce soft targets for the image-text matching, plus an intra-modal structural distillation loss. The loss equations are clearly written, the ablations are sensible, and the comparison against CUSA is fair since CUSA also gets the stronger teacher. The authors correctly diagnose why hard negative mining and image self-supervision hurt in this setting.\n\nThe soft spots are real but not fatal. The benchmark's multi-match labels are the union of top-1000 candidates from four expert embedders, then manually verified. No inter-annotator agreement is reported, and the union can miss relevant images that none of the experts retrieve. The paper also reports no error bars or multiple seeds, so the absolute numbers carry unknown noise. My bigger worry is the train/test overlap point. The internal training set is said to be \"decoupled\" from SecMM-TBIR, but no deduplication statistics are shown. Given that both the training set and the benchmark pool draw on real-world surveillance images, leakage could inflate the relative gain. This is a legitimate concern until the authors release overlap evidence.\n\nThe paper would clearly improve if the benchmark and the internal training set are made public, or at least if the authors provide image-level near-duplicate statistics between the two. Right now the central claim rests on artifacts we cannot check. That said, the method itself is sound, the thinking is clear, and the empirical pattern is consistent. I'd send this to peer review, but the referee should make data release and overlap analysis the gating questions.\n\nWho benefits? Researchers in text-based image retrieval, domain-specific fine-tuning, and benchmark design. It deserves a serious referee, and I'll likely cite it once the data is out.\n\nBest,\n[You]","headline":"Solid, useful paper with a new multi-match surveillance benchmark and a soft-label fine-tuning method that consistently beats ITC, but the unreleased data and missing overlap checks make the headline numbers provisional.","tokens_in":17715,"tokens_out":1806,"would_cite":true,"duration_ms":17713,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new benchmark and a soft-label fine-tuning method raise text-based image retrieval in surveillance by an average of 7.8 mAP@20 points over standard contrastive fine-tuning.","keywords":["text-based image retrieval","multi-match benchmark","surveillance retrieval","false negatives","contrastive learning","soft-label supervision","knowledge distillation","CLIP fine-tuning"],"falsifier":"Take a random sample of images outside the top-1000 union for each query and have independent annotators judge relevance; if any nontrivial fraction are relevant, the benchmark's multi-match labels are incomplete and reported mAP gains are partly an artifact of unlabeled positives. The benchmark release should include the full verification protocol so this check can be run.","tokens_in":1647,"feed_emoji":"🔍","tokens_out":2606,"duration_ms":74390,"temperature":0.7,"pith_summary":"Text-based image retrieval benchmarks usually assume each query has exactly one correct image, which is false in surveillance and other compressed domains where one brief query matches many images. The paper builds a data engine, DSMM-TBIR, that generates domain queries with LLMs, curates diverse image pools, and produces verified multi-match labels; this yields the SecMM-TBIR benchmark of 50k surveillance images and 200 queries, which the paper says will be released. It then argues that standard contrastive fine-tuning fails in such domains because semantically similar unpaired images are treated as false negatives. To fix this, it proposes Semantic-Aware Fine-Tuning (SAFT), which distills soft cross-modal alignment targets and intra-modal image-image structure from a strong frozen teacher. Across several CLIP-style models, SAFT raises mean average precision at top 20 (mAP@20) on SecMM-TBIR by an average of 7.8 points over standard image-text contrastive tuning and also improves general-domain retrieval.","feed_headline":"Soft-label tuning lifts domain image retrieval by 7.8 points","feed_subtitle":"A multi-match benchmark plus teacher-distilled soft labels fixes false negatives in security retrieval.","key_machinery":"The central machinery is the pair consisting of the DSMM-TBIR data engine and the SAFT fine-tuning framework. The engine uses Distribution-Aware Prompting (DAP), an LLM prompt that turns statistical text distributions into natural queries, and Centroid-Guided Diversity Sampling (CGDS), which clusters the image pool with K-means in an embedding space and samples within clusters for diversity. Multi-Expert Collaborative Filtering then takes the union of the top-1000 candidates from four different universal embedding models per query, and manual verification converts that union into multi-match labels. SAFT's machinery is a frozen universal multi-modal teacher that emits soft image-to-text, text-to-image, and image-to-image distributions; SASS, a bidirectional KL loss on the cross-modal soft labels, and ISD, a KL loss on intra-modal image similarity, are added to the standard contrastive objective to counter false negatives in a compressed semantic space.","core_discovery":"The paper's central claim is that the single-match assumption underlying most TBIR benchmarks misleads both evaluation and training in specific domains. Its SecMM-TBIR benchmark contains multiple verified matches per query across pedestrian and vehicle surveillance scenarios, and the proposed SAFT framework turns multi-match structure into training signal. Semantic-Aware Soft-Label Supervision (SASS) replaces rigid one-hot labels with a teacher's cross-modal similarity distributions via bidirectional KL divergence, and Intra-modal Structural Distillation (ISD) transfers the teacher's image-image similarity distribution. The paper reports that SAFT yields an average gain of 7.8 mAP@20 over standard ITC fine-tuning across six CLIP-style models, with consistent improvements on Flickr30K and MS-COCO and additional gains on Fashion200K and compositional reasoning benchmarks.","pith_inferences":["A testable extension is to apply SAFT to a separately built multi-match benchmark in another vertical; the paper's pipeline suggests the margin should transfer, but current evidence is limited to surveillance and fashion.","If the SecMM-TBIR labels were replaced by exhaustive annotation, the measured advantage of SAFT could shrink, because part of what it learns compensates for false negatives in the training set; that would change the interpretation without necessarily invalidating the method.","The teacher-dependent design implies a model-selection rule: when the frozen teacher is much stronger than the student, SASS and ISD should help most, and gains should flatten as the student approaches the teacher.","The data engine could be inverted to build training sets as well as benchmarks, turning unlabeled domain video into contrastive pairs enriched with the same multi-expert candidate union."],"forward_implications":["In specific domains with compressed semantics, models fine-tuned by SAFT should retrieve multiple correct images per query instead of being forced to rank all but one as negatives.","Freezing the text encoder during domain fine-tuning is preferable to joint cross-modal tuning; the paper reports that joint tuning consistently hurts retrieval on SecMM-TBIR.","Standard image self-supervision and threshold-based hard negative mining provide marginal or negative gains under dense false negatives, so they are not the right tools for domain-specific retrieval.","Because SAFT also improves Flickr30K, MS-COCO, Fashion200K, and the ARO compositional reasoning benchmark, its benefits are not limited to surveillance and do not sacrifice general-domain capability.","The DSMM-TBIR pipeline is reusable for other verticals such as medical or retail search, using the same LLM/VLM query generation and multi-expert verification stages."],"supporting_citations":[{"why":"Defines the CLIP contrastive learning paradigm and the family of models this paper fine-tunes.","marker":"Radford et al. 2021"},{"why":"Supplies the Flickr30K benchmark used as a general-domain testbed and as part of the general training set.","marker":"Plummer et al. 2017"},{"why":"Supplies MS-COCO, the other general-domain benchmark and training split used in the experiments.","marker":"Lin et al. 2014"},{"why":"Provides UniME-V2, the frozen universal multi-modal teacher whose cross-modal and intra-modal distributions define SASS and ISD.","marker":"Gu et al. 2025b"},{"why":"CUSA, the architecture-independent unimodal soft-label baseline that SAFT compares against and outperforms.","marker":"Huang et al. 2024"},{"why":"SoftCLIP, the prior soft-label supervision method positioned as unimodal-proxy and distinct from SAFT's cross-modal soft labels.","marker":"Gao et al. 2024"},{"why":"One of the four expert embedders whose top-1000 candidate union defines the multi-match candidate set in Phase 2.","marker":"Li et al. 2026"},{"why":"RZenEmbed, another expert embedder in the multi-expert collaborative filtering step.","marker":"Jian et al. 2025b"},{"why":"Jina-v4, another expert embedder in the multi-expert collaborative filtering step.","marker":"Günther et al. 2025"},{"why":"SigLIP2, the fourth expert embedder that contributes to the candidate union used for manual verification.","marker":"Tschannen et al. 2025"}],"fun_headline_variants":["Multi-match benchmark lifts domain retrieval by 7.8 mAP","Rethinking TBIR: single-match assumption fails in specific domains","Soft labels and distillation solve domain retrieval false negatives","Surveillance retrieval: multi-match supervision yields 7.8-point gain"],"cache_read_input_tokens":19968,"weakest_assumption_plain":"The load-bearing premise is that the union of the four expert embedders' top-1000 candidates plus manual verification finds every relevant query-image pair, with no missed positives that would be counted as negatives during training or evaluation.","fun_headline_variants_meta":{"raw":{"variants":["Multi-match benchmark lifts domain retrieval by 7.8 mAP","Rethinking TBIR: single-match assumption fails in specific domains","Soft labels and distillation solve domain retrieval false negatives","Surveillance retrieval: multi-match supervision yields 7.8-point gain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000946,"raw_usage":{"total_tokens":4048,"prompt_tokens":962,"completion_tokens":3086,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":578,"completion_tokens_details":{"reasoning_tokens":3014}},"tokens_in":578,"tokens_out":3086,"duration_ms":20820,"temperature":1.0,"reasoning_tokens":3014,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:17:34.179191+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of images outside the top-1000 union for each query and have independent annotators judge relevance; if any nontrivial fraction are relevant, the benchmark's multi-match labels are incomplete and reported mAP gains are partly an artifact of unlabeled positives. The benchmark release should include the full verification protocol so this check can be run.","supporting_citations":[],"review_version":1}