{"id":"90280a58-221e-4fc9-83c9-20d80a601ecc","arxiv_id":"2509.04376","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A training-free cloze and re-ranking pipeline lets large multimodal models improve text-based person anomaly retrieval over a standard text-image retriever.","lead":"This paper proposes AnomalyLMM, a pipeline that uses large multimodal models to search for people behaving abnormally in images, given a plain-language description (for example, 'a person falling on ice skates'). The method is notable because it adapts strong generative models to a discriminative retrieval task without fine-tuning them, and it reports a small but positive gain on the PAB benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set hyperparameter selection and absent significance testing leave the +0.96 R@1 gain potentially within noise.","rationale":"The paper presents a coarse-to-fine pipeline that improves a strong baseline by a small margin. The reader identifies the dependency on the coarse retriever's top-3 recall as the weakest assumption. While this is a genuine limitation of any coarse-to-fine approach, the paper's own ablation on the candidate list length (Table 2(d)) shows that N=5, which relaxes the top-3 constraint, yields lower R@1 than N=3. This suggests the re-ranking stage is the performance bottleneck, not the coarse-stage recall. The more critical threat to the central claim is the methodology: hyperparameters are chosen using the test set itself, and no uncertainty quantification is provided. With a small improvement and a single test split, the reported gain may not reflect a real advantage. Therefore, a held-out validation or significance test is the decisive check. I partially agree with the reader because the coarse-stage concern is real but secondary; the reliability of the measured gain is the primary issue.","tokens_in":15138,"tokens_out":8446,"duration_ms":71709,"concrete_test":"Split the 1,978 test queries randomly into two halves. Use the first half to select N, α1, α2, and β (e.g., by grid search as in the paper). Then evaluate the final pipeline on the second half and compute the R@1 difference between X2VLM and X2VLM+Ours, along with a 95% bootstrap confidence interval. If the difference is not positive or the interval includes zero, the reported gain is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on a +0.96 R@1 improvement over the fine-tuned X2VLM baseline (Table 1). However, all key hyperparameters (initial candidate count N, fusion weights α1/α2, decay base β) are selected directly on the PAB test set (Tables 2(b)-(f)), and no error bars or significance tests are reported. Because the test set is a single split of 1,978 pairs, the reported improvement could result from overfitting these hyperparameters to the specific test queries and may not generalize. The coarse-stage dependence (reader's concern) is a necessary condition but is not the weakest link: the paper's own ablation on N (Table 2(d)) shows that increasing N from 3 to 5 decreases R@1, indicating that the re-ranker, not the coarse-retriever recall, is the performance bottleneck. The more load-bearing assumption is that the measured gain is real rather than an artifact of test-set optimization.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AnomalyLMM, a coarse-to-fine pipeline for text-based person anomaly search on the PAB benchmark. A fine-tuned text-to-image retriever (X2VLM or CMP) produces an initial top-N ranking; an LLM then masks verbs and color terms in the query; an LMM fills those masks for each candidate image using visual evidence; an LLM compares the completions with the original query under semantic equivalence rules; and a weighted score fusion combines the initial and re-ranked lists. The central empirical claim is a +0.96 R@1 improvement (84.73 vs. 83.77) and +0.46 mAP over a fine-tuned X2VLM baseline on the PAB test set, with ablations showing both cloze completion and re-ranking contribute. The approach is described as training-free for the LMM components, though the underlying retriever is fine-tuned on PAB.","tokens_in":15395,"tokens_out":3625,"duration_ms":35717,"significance":"If the reported gain is robust, the paper provides a useful and interpretable recipe for leveraging off-the-shelf LMMs in fine-grained retrieval without training them. The idea of reformulating anomaly search as masked cloze completion and semantic re-ranking is novel for this task, and the qualitative analysis supports the interpretability claim. The strengths are the clear decomposition into components, the use of a public benchmark, and the ablation structure. However, the headline improvement is small (0.96 R@1 on a single 1,978-pair test split), all free hyperparameters are selected on that same test set, and no error bars or significance tests are reported; these issues currently leave the central quantitative claim vulnerable to overfitting and noise. The work is therefore significant as a proof-of-concept, but its empirical standing needs strengthening before publication.","major_comments":[{"comment":"The headline result is a +0.96 R@1 gain over X2VLM on a single split of the PAB test set. The hyperparameters that control the method—initial candidate count N, decay base β, and fusion weights α1/α2—are all selected by ablation on this same test set (Table 2(b), 2(d), 2(e), 2(f)). Since the individual component gains in Table 2(a) are only +0.35 and +0.46 R@1, test-set selection could plausibly account for the entire reported improvement. No error bars, multiple runs, or significance tests are reported. To support the central claim, the authors should select hyperparameters on a validation split (or use nested cross-validation) and report variance or paired tests (e.g., bootstrapping over queries). This is load-bearing because the paper's main conclusion is the numerical improvement over a strong baseline.","section":"Sec. 4.2, Table 1; Tables 2(b)-(f); Eq. (1)"},{"comment":"The pipeline can only succeed if the ground-truth image survives the coarse top-N stage, yet the paper does not report the baseline's top-N recall for the N values considered. Table 2(d) shows N=3 is best and N=5 is worse, which suggests the re-ranker is currently the bottleneck rather than coarse recall, but this does not remove the dependence: if top-3 recall of X2VLM is well below the R@1 ceiling, the proposed re-ranking cannot recover those misses and the measured gain is contingent on a shortlist that happens to contain the target. The authors should report oracle top-N recall (for N=2..5) of the fine-tuned X2VLM and analyze failure cases where the ground-truth is outside the shortlist. Without this, the coarse-to-fine dependence is unquantified.","section":"Sec. 3.1 and Table 2(d)"},{"comment":"The ablation over LLM/LMM choices is used to conclude that reasoning capability is important and that smaller models 'exhibit poorer performance.' However, the differences among Qwen2.5-3B, Qwen2.5-7B, Qwen3-8B, and QWQ-Max are within 0.1-0.3 R@1, which is below the likely noise level given the single-run protocol criticized above. This claim should be softened or supported by repeated runs / significance testing, especially since the final model choice (QWQ-Max) is also selected on the test set.","section":"Sec. 4.3, Table 2(c)"}],"minor_comments":[{"comment":"The numbers in the 'X2VLM + Ours' rows appear duplicated ('84.73 (+0.96)84.73 (+0.96)...'), a formatting error that should be fixed.","section":"Table 1 and Table 2"},{"comment":"The abstract and contributions call the method 'training-free,' but the pipeline requires fine-tuning the X2VLM or CMP retriever on PAB. The authors state this in Sec. 4.1, but the wording should be clarified to say that the LMM/LLM components are training-free, not the whole method.","section":"Abstract and Sec. 4.1"},{"comment":"The figure contains typos in the generated descriptions ('waveing', 'UNKNWON'). These should be corrected or marked as LMM outputs if intended.","section":"Figure 4"},{"comment":"The notation S1_n and S2_n is not fully defined: it is unclear whether n indexes the same image in the initial and re-ranked lists, and how ties in the re-ranked list are assigned scores. Please clarify.","section":"Sec. 3.4, Eq. (1)"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable empirical contribution but the central result rests on a small, single-split improvement with test-set hyperparameter selection. The authors should treat the lack of significance testing as a core issue, not a cosmetic one. If the method is intended for a top venue, the community would likely expect either multiple runs, a validation-based hyperparameter selection, or a stronger effect size. I would not reject outright because the pipeline is novel for this task and the qualitative results are suggestive, but the current evidence is not yet convincing enough for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a reasonable engineering paper that combines masked cloze prompting with LLM-based semantic reranking for text-based person anomaly search. The idea is not new in its pieces — LMM captioning, LLM reranking, masked prompts all appear in the cited literature — but the combination, with verb/color placeholders and equivalence rules, targeted at this task, is a legitimate extension. The paper ships ablations on the public PAB benchmark and both components (cloze completion, re-ranking) contribute something. The qualitative examples are honest, including a failure case.\n\nThe soft spots are real but not fatal. All key hyperparameters (initial candidate count N, fusion weights alpha1/alpha2, score decay beta) are tuned on the same test set that produces the headline number, and there are no error bars or significance tests. With a single split of 1,978 pairs, a +0.96 R@1 gain (83.77 to 84.73) could easily be noise. The stress-test note is right that the coarse-stage recall is not the binding constraint — the ablation shows N=5 hurts, so the re-ranker is the bottleneck and it is sensitive to candidate set size. The deeper problem is test-set overfitting. Also, the abstract and contribution list advertise 'behavioral saliency prediction' as a distinct module, but I don't see it defined or implemented anywhere in the method; it seems to be absorbed into cloze completion. That is a mismatch that needs fixing.\n\nOn balance, the central argument holds up as a plausible training-free plug-in: you can attach it to any text-image retriever and it gives a small, consistent boost across two baselines (CMP +3.64, X2VLM +0.96). That consistency is evidence the effect is not pure noise, though the magnitude is small. The paper is honest about limitations and the writing is clear.\n\nWho gets value from this? People working on text-based person retrieval, LMM adaptation for retrieval, or fine-grained reranking. It does not reorganize the field, but it is a useful, well-scoped extension.\n\nMy recommendation: send it to peer review, but with clear requirements — release code and prompts, report variance across multiple runs or bootstraps, and either freeze hyperparameters on a validation split or show sensitivity over a range. It is a borderline accept rather than a desk reject. I would not cite it as a strong result until those are addressed, but as a related method it is citable.","headline":"Plausible training-free LMM reranking recipe for person anomaly search, but the +0.96 R@1 gain is small and likely overfit to the test set.","tokens_in":15828,"tokens_out":2696,"would_cite":false,"duration_ms":23827,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AnomalyLMM claims that blanking out the verbs and colors in a person-anomaly text query, having a large multimodal model fill those blanks from each candidate image's visual evidence, and semantically re-ranking the completions improves ret","keywords":["text-based person anomaly search","large multimodal models","cloze completion","cross-modal retrieval","re-ranking","training-free adaptation","anomaly detection","PAB benchmark"],"falsifier":"Run the full pipeline on the PAB test set again, but feed the masked query to the LMM without showing it the candidate images (or with the image replaced by a blank caption). If Recall@1 stays close to the reported 84.73, the claimed visual grounding is not doing the work; if it drops sharply, the visual cloze is the operative mechanism.","tokens_in":15075,"feed_emoji":"🔍","tokens_out":10116,"duration_ms":90387,"temperature":0.7,"pith_summary":"The paper sets out to show that large multimodal models can be made useful for text-based person anomaly search even though they are trained for generation, not retrieval. The proposed AnomalyLMM keeps a standard text-to-image retriever for a coarse top-N ranking, then blanks out the action verbs and color words in the query and asks an LMM to fill those blanks separately for each candidate image, using only visual evidence and 'UNKNOWN' when uncertain. A second LLM compares the completions semantically and re-ranks the candidates, and the two rankings are fused. If this works, it matters because anomaly searches are sparse and expensive to fine-tune for, so a training-free way to inject world knowledge into retrieval could improve public-safety applications. On the only public benchmark (PAB), the reported improvement over the stronger baseline is +0.96 Recall@1 and +0.46 mAP, with larger gains over a weaker retriever.","feed_headline":"Cloze trick lifts anomaly search recall by 0.96 points","feed_subtitle":"Masking verbs and colors turns generative vision models into discriminative rerankers without retraining them.","key_machinery":"The load-bearing mechanism is masked cloze completion: a prompt template with structured <VERB> and <COLOR> placeholders, filled by an LMM under strict visual-evidence instructions. The cloze does two jobs: it forces the model to localize the specific action and appearance cues named by the query instead of judging whole-image similarity, and it turns each candidate image into a short text that can be compared with the original query by an LLM in a semantically flexible way. The companion mechanism is knowledge-aware re-ranking—an LLM that counts semantically equivalent slot matches under explicit equivalence rules (same physical motion, same hue family)—whose rank is converted to a score by","core_discovery":"The central claim is that only the ground-truth image can supply the visual cues needed to fill every masked slot in an anomaly query, so the ranking should prefer images whose completions match the query semantically. AnomalyLMM operationalizes this by (1) retrieving top-N candidates with a pretrained text-to-image model, (2) replacing action verbs and color terms with <VERB>/<COLOR> placeholders using an LLM, (3) prompting an LMM to complete the placeholders for each candidate from visual evidence, with 'UNKNOWN' allowed to suppress hallucination, and (4) using an LLM to compare each completion with the original query under equivalence rules for actions and hues, then fusing the semantic r","pith_inferences":["The gap between gains on weak and strong retrievers (+3.64 vs +0.96 R@1) suggests the cloze reranker mostly repairs coarse-stage errors; as the coarse retriever improves, the marginal gain from this exact recipe may shrink unless the masked slots are made harder or richer.","The UNKNOWN output offers an untapped signal: a slot-by-slot UNKNOWN rate on the true match could flag query descriptions that are too subtle for current LMMs, and might be used as a data-collection or prompt-engineering cue.","The same text-side compare-and-rerank design should transfer to other retrieval tasks with sparse positive samples and strong text-to-image retrievers—unusual actions, rare objects, or fine-grained attribute queries—where fine-tuning is too costly."],"forward_implications":["Both tested coarse retrievers improve when wrapped in the pipeline (CMP +3.64 R@1, X2VLM +0.96 R@1), so the benefit is not specific to one text-to-image model.","Because no LMM or LLM parameters are updated, the method is deployable in sparse-data or resource-limited settings where fine-tuning a vision-language model is impractical.","The per-candidate cloze completions double as an explanation of the ranking: a user can see which verbs and colors matched for each image.","The optimal initial candidate list is short (N=3 in the experiments), which keeps the added reasoning cost bounded in practice.","The scheme can be layered on top of whatever fine-tuned retriever is available, so future improvements to the coarse stage can compound with it."],"supporting_citations":[{"why":"Supplies the PAB benchmark, the only public evaluation set for text-based person anomaly search, and is the baseline method compared throughout.","marker":"Yang et al. (2024b)"},{"why":"Provides X2VLM, the pretrained text-to-image model used for the coarse retrieval stage and the main baseline whose fine-tuned weights initialize the retriever.","marker":"Zeng et al. (2024)"},{"why":"The latent diffusion approach used to synthesize the 1,013,605 training pairs in PAB, on which the coarse retriever is fine-tuned.","marker":"Rombach et al. (2022)"},{"why":"Defines the contrastive CLIP baseline and the general image-text alignment paradigm that the paper contrasts with generative LMMs.","marker":"Radford et al. (2021)"},{"why":"Representative of the LMMs whose generative pretraining motivates the domain-gap problem the proposed cloze adaptation addresses.","marker":"Li et al. (2023)"},{"why":"Supplies LLaVA, a typical instruction-tuned generative LMM used to illustrate why generative models need a retrieval-oriented adaptation strategy.","marker":"Liu et al. (2023)"}],"fun_headline_variants":["Masked cloze test reranks anomaly search, +0.96 Recall@1","Zero-shot anomaly search gains via masked visual completion","Fill-in-the-blank prompts improve anomaly search by 0.96","AnomalyLMM: generative model becomes fine-grained retriever"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The true matching image must already be among the handful of candidates the ordinary text-to-image retriever returns; if it is not in that short list, the cloze and re-ranking stages cannot bring it back.","fun_headline_variants_meta":{"raw":{"variants":["Masked cloze test reranks anomaly search, +0.96 Recall@1","Zero-shot anomaly search gains via masked visual completion","Fill-in-the-blank prompts improve anomaly search by 0.96","AnomalyLMM: generative model becomes fine-grained retriever"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000231,"raw_usage":{"total_tokens":1376,"prompt_tokens":853,"completion_tokens":523,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":446}},"tokens_in":597,"tokens_out":523,"duration_ms":5445,"temperature":1.0,"reasoning_tokens":446,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T10:12:34.635647+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the full pipeline on the PAB test set again, but feed the masked query to the LMM without showing it the candidate images (or with the image replaced by a blank caption). If Recall@1 stays close to the reported 84.73, the claimed visual grounding is not doing the work; if it drops sharply, the visual cloze is the operative mechanism.","supporting_citations":[{"cited_title":"X \\^ \\ 2\\ 2-vlm: All-in-one pre-trained model for vision-language tasks","cited_arxiv_id":null,"evidence_quote":"Provides X2VLM, the pretrained text-to-image model used for the coarse retrieval stage and the main baseline whose fine-tuned weights initialize the retriever."},{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Defines the contrastive CLIP baseline and the general image-text alignment paradigm that the paper contrasts with generative LMMs."},{"cited_title":"Visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"Supplies LLaVA, a typical instruction-tuned generative LMM used to illustrate why generative models need a retrieval-oriented adaptation strategy."}],"review_version":1}