{"id":"665b85de-7eca-45d9-910d-629d673669c4","arxiv_id":"2411.17776","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Introduces the Pedestrian Anomaly Behavior (PAB) benchmark and a Cross-Modal Pose-aware model for retrieving pedestrians from text descriptions of normal or anomalous actions, reporting 84.93% R@1.","lead":"This paper proposes a new task, text-based person anomaly search, and builds a 1.01 million pair synthetic benchmark plus a 1,978 image real-world test set from OOPS! videos. A pose-aware retrieval model reaches 84.93% recall@1, but the benchmark's train and test both draw from OOPS! without a stated split, which risks inflating the numbers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training captions and test frames both come from OOPS!, with no stated split; if test videos contributed to the synthetic caption pool, the 84.93% R@1 reflects event-memorization rather than unseen generalization, undermining the central benchmark claim.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the paper uses OOPS! captions for synthetic training (Section 3.2) and OOPS! videos for the real test set (Section 3.1), with no explicit guarantee that the test videos were excluded from the training caption pool. This is the single most consequential issue because it directly determines whether the headline empirical claim, 'synthetic training data facilitates fine-grained behavior retrieval on the real-world test set,' is valid. I considered other possible objections, such as the synthetic-to-real domain gap, MLLM caption noise, and noisy attribute annotations, but these are either acknowledged by the authors or do not invalidate the central comparison. The caption-overlap concern is different: if test videos are among the caption sources, the model has seen event-level text descriptions and synthetic renderings of those events during training, so the 84.93% R@1 could be achieved by short-cutting behavior recognition rather than by learning to match unseen individuals. Since the dataset and code are released, this is checkable from metadata, and the proposed test would settle it definitively. The OOD UCC experiment is encouraging but does not repair the PAB evaluation. For these reasons I agree with the reader's verdict: the paper should be rejected in its current form unless the disjointness of the training caption pool and test videos is verified and the evaluation is rerun with a clean split.","tokens_in":21078,"tokens_out":5298,"duration_ms":52402,"concrete_test":"With the released dataset metadata, compute the intersection of source video IDs between the 989 OOPS! test pairs and the 6,739 Cn / 6,979 Ca captions used to synthesize training images. If the intersection is empty, the concern is resolved. If non-empty, retrain CMP under the same settings on the training captions whose source video IDs are excluded from the test split, and report R@1 on the unchanged test set. A material drop (e.g., >2 points) or any non-empty intersection would indicate that the published 84.93% overstates unseen generalization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing premise of the PAB evaluation is that the synthetic training captions (Section 3.2, derived from OOPS! Cn/Ca, yielding 6,739 + 6,979 captions) are disjoint from the videos used to build the real test set (Section 3.1, 989 normal/anomaly pairs from OOPS!). The paper never states this disjointness. Because both subsets come from OOPS!, if any of the 989 test videos' captions appear in the training caption pool, the training set contains synthetic images generated from prompts describing the same events and people as test queries. The model could then solve retrieval by recognizing event/caption combinations (e.g., 'woman in white dress falling on staircase') rather than performing person-level anomaly search on unseen appearances. The reported 84.93% R@1 and the Figure 5 scaling curve would be inflated. This is an unstated assumption rather than an internal inconsistency, but it is directly load-bearing: without a disjoint split, the central claim that synthetic training data facilitates fine-grained behavior retrieval on the real-world test set is unsupported. The OOD UCC result offers partial evidence of generalization, but it does not validate the headline PAB number.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a new task, text-based person anomaly search, in which a model retrieves a pedestrian image from a text query that describes appearance, action, and scene, including anomalous behaviors. To support this task, the authors construct the PAB benchmark: a training set of 1,013,605 synthetic image-text pairs generated from OOPS! captions using a diffusion model, and a real-world test set of 1,978 image-text pairs extracted from OOPS! videos. They also propose a Cross-Modal Pose-aware (CMP) framework that adds a pose-aware image encoder and identity-based hard negative mining to an X-VLM style baseline. Experiments report 84.93% R@1 on PAB and 55.23% R@1 on a newly assembled out-of-distribution UCC test set.","tokens_in":21313,"tokens_out":4742,"duration_ms":42232,"significance":"If the evaluation is sound, the paper addresses a practical gap: most text-based person search benchmarks focus on appearance and common actions, while safety and emergency applications require fine-grained retrieval of anomalous behaviors. The PAB benchmark is large in scale, and the authors provide a clear task formulation, a method, and a public release of dataset, model, and code. The inclusion of an OOD test set is a good practice and offers partial evidence of generalization. However, the central quantitative claim depends critically on a train/test disjointness condition that the paper does not establish, and the OOD result is considerably weaker than the headline PAB number.","major_comments":[{"comment":"The training-data scaling experiment is interpreted as showing that 10% of the data already achieves a competitive recall rate, but the plotted curve is non-monotonic: the 0% point (X-VLM zero-shot) is 71.94% R@1 on PAB, while the 10% point drops to 50.79% R@1. This large drop is not discussed, and it weakens the claim that synthetic training data “facilitates” retrieval in a smooth, predictable way. Please explain the drop and clarify the comparison, or the scaling claim should be tempered.","section":"Figure 5 and §5"}],"minor_comments":[{"comment":"The abstract says “1,013,605 synthesized image-text pairs” while Table 1 reports 1,015,583 total image-text pairs; the difference between training and total should be stated consistently in one place.","section":"Abstract and §5"},{"comment":"The sentence “This process yields 6,739 Cn and 6,979 Ca captions” should specify whether these counts are before or after the deduplication and filtering steps described in Section 3.1; currently the pipeline order is ambiguous.","section":"§3.2"},{"comment":"The identity-based hard negative mining assumes that images generated from Cn and Ca of the same OOPS! video depict the same identity with similar appearance. Since the diffusion model may not preserve identity across independently generated images, please report a measure of appearance consistency or acknowledge this limitation.","section":"§4.2"},{"comment":"The compared methods are trained on 0.1M PAB pairs, but the paper does not state whether the same data sampling and augmentation procedures were used for all methods; this should be clarified for a fair comparison.","section":"Table 3 and §5"}],"recommendation":"major_revision","confidential_remarks":"The decisive issue is the unstated overlap between the OOPS! captions used for synthetic training and the OOPS! videos used for the real test set. The paper currently reads as if the authors used all filtered OOPS! captions for training while also drawing the test set from OOPS!, which makes the 84.93% R@1 number essentially unsupported. This is fixable in principle by demonstrating a disjoint split and, if necessary, retraining; however, if the authors cannot provide such a split, the benchmark as described cannot support the headline claim and the manuscript would be close to reject. The OOD UCC result is a useful existence proof, but it does not salvage the main PAB evaluation without the disjointness guarantee."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the task and dataset are genuinely new, and the CMP baseline is reasonable, but the headline number (84.93% R@1) is not trustworthy until the authors confirm that the OOPS! videos used for the real test set were excluded from the caption pool that generated the synthetic training set. Section 3.1 builds the test from OOPS! videos; Section 3.2 says training prompts come from OOPS! Cn and Ca. No disjoint split is stated. Since the test queries are MLLM captions of the same events, a model that memorized event-caption pairs during training could retrieve the right test frame without doing person-level anomaly search. The issue is directly load-bearing. The OOD UCC test (55.23% R@1) is a useful sanity check but does not validate the PAB number; if anything, the gap suggests the PAB result is inflated.\n\nWhat the paper does well: it is the first to define text-based person anomaly search, and the benchmark construction is thoughtful. The identity-based hard negative mining (using the known normal/anomaly pairing of OOPS! captions) is a legitimate and clever way to create hard negatives. The pose-aware encoder is simple and the ablations show consistent but modest gains. The authors ship dataset and code and include an OOD evaluation. For a dataset paper, this is the right shape.\n\nMinor issues: the multi-weather evaluation is described only by referencing MuSe-Net, and the attribute annotations (action, anomaly, scene) are MLLM-generated without human validation. Those are minor beside the overlap question.\n\nBottom line: if the split is confirmed disjoint, this is a solid benchmark paper. If not, the central empirical claim is unsupported. The fix is straightforward: either state the exclusion clearly (with statistics) or rerun with a clean split. Either way, the resource has value for the community. This deserves a serious referee — desk rejection would be wrong — but the review should be conditional on resolving the data leakage question.","headline":"A genuinely new benchmark and task, but the headline R@1 is compromised by an unstated OOPS! split that must be resolved before the evaluation is credible.","tokens_in":21844,"tokens_out":2516,"would_cite":true,"duration_ms":24941,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that text-based person anomaly search can be learned from one million synthetic image-text pairs and evaluated on real-world frames, with a pose-aware cross-modal model reaching 84.93% recall@1 on the new PAB test set.","keywords":["text-based person anomaly search","image-text retrieval","pedestrian anomaly detection","pose-aware representation","hard negative mining","synthetic training data","cross-modal retrieval","benchmark"],"falsifier":"One concrete check is to compare the set of video identities behind the 989 real test image pairs with the caption pool used as prompts for synthetic training generation; any overlap in identity or event description would undermine the reported 84.93% recall@1. A second check is to retrain CMP on synthetic images generated only from captions whose source videos are explicitly withheld from the test set and observe whether the in-distribution recall drops materially.","tokens_in":20866,"feed_emoji":"🔎","tokens_out":5772,"duration_ms":55093,"temperature":0.7,"pith_summary":"The paper introduces a new image-text retrieval task: given a sentence describing a pedestrian's appearance and action—especially an abnormal one like falling, lying, or being hit—find that specific person in a gallery. To support this, the authors build PAB, a 1.01 million-pair synthetic training benchmark paired with a 1,978-pair real-world test set, and claim that training on the synthetic data transfers to real-world anomaly retrieval. They then propose a Cross-Modal Pose-aware framework that injects human pose maps and uses identity-based hard negatives to push the model to distinguish actions, not just appearances. On PAB, the full model reaches 84.93% recall@1 and 91.66% mean average precision, and on an out-of-distribution test built from surveillance videos it reaches 55.23% recall@1. The contribution matters because existing text-based person search benchmarks are dominated by walking and standing, while existing anomaly datasets mostly offer binary labels rather than retrievable text descriptions.","feed_headline":"Text search finds falling pedestrians at 84.93% recall","feed_subtitle":"A 1M-pair synthetic benchmark teaches models to spot abnormal actions in real crowd images.","key_machinery":"The central mechanism is the identity-based hard negative pair. Because each synthetic image is generated from a caption describing either the normal or the anomalous moment of the same person, the opposite caption yields an image with similar appearance and background but a different action; pairing a text with the counterpart image creates a hard negative that differs only in behavior, forcing the model to learn action-discriminative features. The supporting mechanism is the pose-aware image encoder, which extracts keypoint maps with a human pose detector, passes them through a small convolutional module, and injects the resulting pose features into the image encoder via multi-head cross-attention so posture differences between actions become salient.","core_discovery":"The authors claim that text-based person anomaly search is a distinct task requiring behavior-level discrimination beyond appearance, and that their PAB benchmark and Cross-Modal Pose-aware method make that task learnable and measurable. The training set is synthesized by using normal and anomaly captions from a public unintentional-action video dataset as prompts for a photorealistic diffusion generator, then re-captioning each generated image with a multimodal large language model; the real test set is extracted from the same video source with manual caption correction. The CMP method adds a pose-aware image encoder, which fuses human keypoint maps through cross-attention, and an identity-based hard negative mining scheme that forms image-text pairs differing only in action. With the full one million training pairs, CMP attains 84.93% recall@1 and 91.66% mAP on PAB, surpassing CLIP, X-VLM, and six text-based person search baselines, and 55.23% recall@1 on an out-of-distribution surveillance test, supporting the claim that synthetic data plus pose and hard-negative cues enable fine-grained anomaly retrieval.","pith_inferences":["Not stated in the paper, but if the video identities behind the real test pairs were not excluded from the caption pool used to prompt the synthetic training images, the reported generalization would be inflated; an explicit holdout audit would settle this.","The pose-guided hard-negative idea could extend naturally to video anomaly search, where sequences of pose maps rather than single frames would distinguish falling from kneeling or running from stumbling.","The synthetic training pipeline could be adapted to other anomalous human actions such as accidents or medical distress by changing the caption source, provided the test source is kept disjoint from training prompts.","The benchmark’s frame-level text annotations for appearances, actions, and scenes may support future tasks like action grounding and scene-conditioned retrieval, not just end-to-end ranking."],"forward_implications":["If the benchmark is sound, a model trained on 1M synthetic pairs can retrieve real-world anomaly frames at 84.93% recall@1, meaning a text query such as “falling into a pool” can localize one specific person among 1,978 candidates.","The paired normal/anomaly synthetic generation scheme could be reused to grow behavior-centric retrieval datasets without manual video annotation, since the data pipeline is automatic apart from quality filtering.","The CMP results imply that adding pose maps and action-differing hard negatives is a reliable way to sharpen behavior discrimination beyond appearance-driven text-based person search.","The out-of-distribution result of 55.23% recall@1 on a surveillance-derived test set suggests the model transfers to unseen anomaly types, though with a sizeable drop from the in-distribution score.","Behavior search is harder than identity search on the same data—CMP gets 84.93% versus 94.34% recall@1—so anomaly retrieval should be measured separately from ordinary person search rather than treated as an incidental extension."],"supporting_citations":[{"why":"Provides the real-world videos and paired normal/anomaly captions used to build the test set and to prompt the synthetic training images.","marker":"[13]"},{"why":"Supplies the MLLM captioner used to re-caption synthetic images and to annotate action, anomaly, and scene attributes.","marker":"[4]"},{"why":"Supplies the diffusion model that generates the photorealistic pedestrian training images from captions.","marker":"[42]"},{"why":"Supplies the human keypoint detector used to filter invalid images and to produce the pose maps fed into the pose-aware image encoder.","marker":"[7]"},{"why":"Supplies the pretrained cross-modal weights that initialize the model and the strongest baseline it must beat.","marker":"[65]"},{"why":"Supplies the large-scale synthetic image-text pretraining and contrastive-learning template that CMP adapts to anomaly search.","marker":"[59]"}],"fun_headline_variants":["Text-based anomaly search: pose-aware method hits 84.93% recall","PAB: 1M synthetic pairs train models to find abnormal actions in crowds","1M-pair synthetic benchmark enables fine-grained person anomaly search","Pose-aware cross-modal search beats baselines in person anomaly retrieval"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole generalization claim stands on the unstated assumption that the real test videos were never used to generate any of the one million synthetic training images; if that disjointness fails, the reported recall could reflect memorization of event descriptions rather than learned anomaly recognition.","fun_headline_variants_meta":{"raw":{"variants":["Text-based anomaly search: pose-aware method hits 84.93% recall","PAB: 1M synthetic pairs train models to find abnormal actions in crowds","1M-pair synthetic benchmark enables fine-grained person anomaly search","Pose-aware cross-modal search beats baselines in person anomaly retrieval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000764,"raw_usage":{"total_tokens":3424,"prompt_tokens":1017,"completion_tokens":2407,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":633,"completion_tokens_details":{"reasoning_tokens":2328}},"tokens_in":633,"tokens_out":2407,"duration_ms":16211,"temperature":1.0,"reasoning_tokens":2328,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:21:21.922544+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One concrete check is to compare the set of video identities behind the 989 real test image pairs with the caption pool used as prompts for synthetic training generation; any overlap in identity or event description would undermine the reported 84.93% recall@1. A second check is to retrain CMP on synthetic images generated only from captions whose source videos are explicitly withheld from the test set and observe whether the in-distribution recall drops materially.","supporting_citations":[{"cited_title":"Oops! pre- dicting unintentional action in video","cited_arxiv_id":null,"evidence_quote":"Provides the real-world videos and paired normal/anomaly captions used to build the test set and to prompt the synthetic training images."},{"cited_title":"Realvisxl v4.0, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the diffusion model that generates the photorealistic pedestrian training images from captions."},{"cited_title":"Realtime multi-person 2d pose estimation using part affinity fields","cited_arxiv_id":null,"evidence_quote":"Supplies the human keypoint detector used to filter invalid images and to produce the pose maps fed into the pose-aware image encoder."},{"cited_title":"Multi-grained vi- sion language pre-training: Aligning texts with visual con- cepts","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained cross-modal weights that initialize the model and the strongest baseline it must beat."},{"cited_title":"Towards unified text-based person retrieval: A large-scale multi-attribute and language search benchmark","cited_arxiv_id":null,"evidence_quote":"Supplies the large-scale synthetic image-text pretraining and contrastive-learning template that CMP adapts to anomaly search."}],"review_version":1}