{"id":"40e13682-c6bb-4c22-ab06-001c1baf486a","arxiv_id":"2506.01466","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"SVTA is a large synthetic video-text benchmark with 41,315 videos covering 68 anomaly types, created by LLM-generated captions and text-to-video generation, and evaluated with three retrieval baselines.","lead":"This paper introduces SVTA, a synthetic benchmark of over 41,000 AI-generated videos with text captions for testing video anomaly retrieval, where systems find clips matching natural-language descriptions of unusual events. It shows three standard video-text retrieval models achieve moderate accuracy on SVTA and transfer reasonably to real-world anomaly datasets, suggesting synthetic data may support privacy-preserving anomaly search.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Zero-shot transfer numbers are inflated by text overlap: SVTA training captions are sourced from UCFCrime-AR and OOPS!, the exact evaluation datasets.","rationale":"Good-faith read: The paper proposes a synthetic benchmark for video anomaly retrieval, which is timely and useful. The dataset scale (41K pairs) and broad category coverage are real contributions, and the human-inspection qualitative check, while weak, is not nothing. However, the strongest claim in the abstract is about zero-shot transfer. That claim is directly undermined by the train/test text overlap described in §3.1 and §4.2. This is not a matter of consensus or taste; it is a methodological inconsistency within the paper: training captions are drawn from the same annotation pools as the evaluation captions. The reader's weakest_assumption focused on video-text alignment; I agree that alignment is unverified, but the text leakage is more load-bearing because it attacks the quantitative result directly and is verifiable from the paper's own statements. I therefore recommend keeping the CONDITIONAL verdict, with the added condition that the authors demonstrate the zero-shot result holds after removing target-dataset captions from training, and ideally provide a video-text alignment audit (e.g., CLIPScore or human evaluation on a sample) to support the dataset's visual fidelity.","tokens_in":13253,"tokens_out":5296,"duration_ms":57839,"concrete_test":"Compute exact and near-duplicate overlap between SVTA training captions and the query/gallery text sets of UCFCrime-AR and the filtered OOPS! subset (§4.2). Then retrain the three baselines on SVTA with all captions derived from those target datasets removed (or, at minimum, with test captions held out) and re-run the same zero-shot protocol. If R@1 drops substantially (e.g., more than ~5 absolute points) for CLIP4Clip/X-CLIP/GRAM, contamination is confirmed. A cleaner design is to generate SVTA videos from a disjoint prompt set and only then evaluate on the original real captions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central evidence for synthetic-to-real transfer is the zero-shot evaluation in §4.2. However, §3.1 states SVTA text prompts are acquired by 'retaining all video descriptions from UCA and UCFCrime-AR' and filtering OOPS! descriptions. The same datasets are then used as test targets in Tables 3 and 4. Consequently, the exact test captions (or near equivalences) appear in the SVTA training set. During training, the text encoder is optimized on those sentences paired with synthetic videos. In video-to-text retrieval, the ground-truth caption is therefore a familiar training text; its embedding may be ranked highly for any query video, independent of visual content. Reported R@1 values (e.g., 34.5% on UCFCrime-AR, 18.9% on OOPS!) thus cannot be attributed to faithful video understanding or to distribution transfer from synthetic to real footage. The headline claim that 'models trained on SVTA achieve competitive zero-shot retrieval performance' is not established by these experiments; the evaluation protocol violates the zero-shot assumption on the text side. This is a correctness risk on the paper's main quantitative contribution, distinct from the video-text alignment issue.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces SVTA, a synthetic video-text benchmark for cross-modal video anomaly retrieval. The dataset contains 41,315 video-text pairs (33 frames, 15 fps) generated by using LLM-written captions from UCA, UCFCrime-AR, OOPS!, and LLM-expanded anomaly categories to prompt the Wan2.1 text-to-video model. The authors benchmark CLIP4Clip, X-CLIP, and GRAM on SVTA and report zero-shot transfer to UCFCrime-AR and OOPS!. The main contributions claimed are the dataset's scale, category coverage (68 anomaly and 30 normal types), diversity, and privacy-safe construction.","tokens_in":13436,"tokens_out":4854,"duration_ms":46218,"significance":"The dataset is potentially useful as a large-scale training resource for text-based anomaly retrieval, and the reproducible generation pipeline (fixed random seed, open-source models) is a strength. However, the paper's main quantitative claim—that synthetic training transfers to real-world benchmarks—is not established as reported, because the zero-shot evaluation captions are drawn from the same text sources used to construct the training set, and because the paper provides no quantitative evidence that the synthetic videos depict the events in their captions. The baseline benchmark on SVTA itself is informative, but the validity of SVTA as a measure of visual grounding depends on controls that are currently missing.","major_comments":[{"comment":"The zero-shot evaluation protocol has a text-side leakage problem. Section 3.1 states that SVTA text descriptions are obtained by retaining all video descriptions from UCA and UCFCrime-AR and filtering OOPS! descriptions. Tables 3 and 4 then evaluate zero-shot retrieval on UCFCrime-AR and OOPS! using those same datasets as test targets. Since the exact or near-identical test captions are present in the SVTA training set, a model can rank ground-truth videos highly by text-to-text similarity without learning any visual-semantic understanding from synthetic videos. The reported gains (e.g., R@1 = 34.5 in Table 3) cannot therefore be attributed to synthetic-to-real visual transfer. A valid protocol would remove overlapping captions from training, or use novel query paraphrases for the test queries, and report both settings.","section":"3.1, 4.2, Tables 3-4"},{"comment":"The core assumption that Wan2.1-generated videos faithfully depict the paired captions is validated only by the statement 'human inspection reveals that the content of generated videos is realistic and reasonable to some extent' (Section 3.2). No quantitative alignment measure, sample size, or inter-annotator agreement is provided. If the videos frequently fail to depict the described anomaly, SVTA becomes a text-to-text benchmark, and the cross-domain results in Tables 3 and 4 are not evidence of video understanding. The paper should include a quantitative video-text alignment evaluation (e.g., human verification on a random sample, CLIP-style similarity, or generation-based metrics) and report per-category agreement.","section":"3.2"},{"comment":"The claim that LLM-based attribute annotation 'exceeds 97%' accuracy is not verifiable as reported. The text says 'We manually evaluate 10% of the annotated results through random sampling' but does not specify the sample size, the number of annotators, the annotation instructions, or how disagreements were resolved. Please provide a precise evaluation protocol with confidence intervals.","section":"3.3"},{"comment":"The OOPS! zero-shot evaluation uses a modified test set: 'our test set differs from the original OOPS! benchmark. We exclude videos lacking complete anomaly descriptions and retain only one matched text per video, resulting in 3,468 curated video-text pairs.' The paper does not report the overlap between these 3,468 captions and the text sources used to build SVTA (Section 3.1), nor does it compare results on the official OOPS! split. Without this, it is unclear whether the reported R@1 values reflect transfer or text memorization, and the results are not directly comparable to prior work.","section":"4.1, Table 4"}],"minor_comments":[{"comment":"The percentages in Figure 3(a) appear to be swapped; the text states 25,170 anomaly and 16,145 normal videos, which corresponds to approximately 60.9% anomaly and 39.1% normal, the opposite of the labels shown in the figure.","section":"Figure 3(a)"},{"comment":"The sentence 'X-CLIP [22] initialized with CLIP (ViT-B/32)' cites CLIP4Clip's reference number; it should cite X-CLIP [24].","section":"Section 4.2"},{"comment":"Reference [13] contains a typo: 'PCVPR' should be 'CVPR'.","section":"References"},{"comment":"The paper does not report the number of descriptions contributed by each source (UCA, UCFCrime-AR, OOPS!, and LLM-expanded); please add a breakdown to clarify how the total of 41,315 is composed.","section":"Section 3.1"},{"comment":"Table 1 lists #Texts as 23,542 for UCA and 1,900 for UCFCrime-AR; the relationship between these counts and the SVTA caption pool should be stated explicitly.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The text-leakage issue is the main threat to the paper's headline claim and is fixable with additional experiments (e.g., removing overlapping captions or using novel queries). The dataset itself may still be a useful community resource if the construction details and validation protocols are tightened. I therefore recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this paper has a real artifact—a 41k-pair synthetic video-text benchmark with 68 anomaly types—and the construction is transparent. But the paper's flagship generalization evidence doesn't hold up as presented. The same text descriptions used to build SVTA (Section 3.1, retained from UCA, UCFCrime-AR, and OOPS!) are the test queries and gallery captions in the UCFCrime-AR and OOPS! evaluations (Tables 3-4). For V2T retrieval, the correct caption is literally a training sentence, so high recall can come from text-side familiarity rather than visual understanding. That makes the 'competitive zero-shot' claim uninterpretable as evidence of synthetic-to-real transfer. The authors don't mention this overlap—a genuine omission.\n\nWhat's good: the scale and diversity are a step change over UCFCrime-AR (1,900 videos, 13 types). The taxonomy is sensible, the split is category-balanced, and they evaluate three standard baselines. The paper is honest about modifying OOPS!, and the fixed-seed generation pipeline is reproducible in principle. But the dataset itself is not released, which is a serious gap for a benchmark paper.\n\nTwo other soft spots, in proportion. First, no quantitative check that the generated videos actually depict the captions; 'human inspection reveals... to some extent' is not a benchmark. If Wan2.1 often fails at fine-grained anomalies, SVTA's video side could be noisy in ways that dwarf the text leakage. Second, the within-SVTA numbers (Table 2) give no floor: without a caption-to-image alignment measure, we can't tell how much signal is visual vs. text-only. This is fixable with a simple CLIP-based alignment statistic or a text-only retrieval baseline.\n\nWho benefits: researchers building video-text retrieval models or anomaly detection systems that need a large, customizable training set, provided the data is eventually released. The paper deserves a serious referee—the resource is worth the community's attention—but revision needs to address the leakage and the alignment check. I'd ask for the data (or a large sample) and a re-analysis of the zero-shot results with a clean protocol.","headline":"SVTA is a genuinely new large-scale synthetic anomaly-retrieval dataset, but the zero-shot transfer numbers are compromised because the test captions overlap the training captions.","tokens_in":14015,"tokens_out":2972,"would_cite":true,"duration_ms":32887,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SVTA, a synthetic text-video benchmark of 41,315 pairs over 68 anomaly and 30 normal classes, claims generative data can replace scarce real-world footage for training cross-modal anomaly retrieval models that transfer zero-shot to real…","keywords":["video anomaly retrieval","synthetic video-text benchmark","text-to-video generation","zero-shot cross-modal retrieval","long-tail anomaly categories","privacy-preserving dataset","video-text retrieval evaluation"],"falsifier":"Take a random sample of, say, 200 SVTA clips, show each to independent annotators with its paired caption, and ask whether the video visibly depicts the described event; if more than about 10% are judged not to depict it, the zero-shot transfer results cannot be taken as evidence of visual anomaly understanding. A corroborating check is to compare against a caption-only retriever that matches query text to stored captions and ignores video content, since similar scores would indicate language priors rather than video grounding.","tokens_in":13028,"feed_emoji":"🎥","tokens_out":9194,"duration_ms":89984,"temperature":0.7,"pith_summary":"The paper introduces SVTA, which it calls the first large-scale dataset for cross-modal video anomaly retrieval: 41,315 video-text pairs and 1.36 million frames spanning 68 anomalous and 30 normal event categories. Because real anomalous footage is scarce, long-tailed, and privacy-sensitive, the authors generate both the captions and the videos: off-the-shelf LLMs produce diverse event descriptions, and the text-to-video model Wan2.1 renders matching clips. The central claim is that this synthetic corpus can support cross-modal anomaly retrieval at scale and that models trained on it transfer to real-world anomaly benchmarks. Evaluations with CLIP4Clip, X-CLIP, and GRAM show the dataset is challenging, and zero-shot evaluation on UCFCrime-AR and OOPS! gives competitive retrieval results, supporting synthetic data as a viable substitute for real anomaly footage.","feed_headline":"Synthetic clips can teach models to retrieve real anomalies","feed_subtitle":"A 41,315-video benchmark over 68 anomaly types shows zero-shot transfer to real-world tests.","key_machinery":"The load-bearing object is the generative pipeline: LLMs turn 23 seed anomaly categories into 50 diverse captions each, and Wan2.1, a 14B text-to-video diffusion model, renders each caption into a 33-frame, 15 fps, 480x832 clip that becomes the video half of a paired sample. Category labels for samples without source annotations are assigned by an LLM, unified by K-means clustering, and manually refined, yielding the final 30-normal and 68-anomaly taxonomy organized into four anomaly groups. The retrieval baselines CLIP4Clip, X-CLIP, and GRAM then test whether video-text alignment learned from these generated pairs generalizes to real footage.","core_discovery":"The central claim is that a large-scale cross-modal anomaly retrieval benchmark can be built entirely from generated content. SVTA comprises 41,315 video-text pairs covering 68 anomaly types and 30 normal activities, produced by expanding seed anomaly categories with LLM-written descriptions and rendering them with the text-to-video model Wan2.1. The authors report that models trained on SVTA achieve competitive zero-shot retrieval on two real-world benchmarks, with GRAM reaching R@1 of 34.5% on UCFCrime-AR and X-CLIP (ViT-B/16) reaching R@1 of 18.9% on the filtered OOPS! set, alongside a best SVTA R@1 of 57.3%. The intended upshot is that synthetic footage can supply the scale, category diversity, and privacy compliance that real anomaly collection cannot.","pith_inferences":["Whether SVTA teaches visual grounding is an open question: the paper validates generator fidelity only by informal human inspection, so a systematic alignment audit would tell whether the zero-shot transfer comes from video understanding or from caption-text priors.","Because SVTA's OOPS! evaluation uses a filtered subset with one caption per video, a natural next check is to run the same zero-shot models on the unfiltered OOPS! benchmark to see how much the filtering affects the reported numbers.","A diagnostic the paper does not report is per-category retrieval accuracy across the four anomaly taxonomies, which would show whether models fail on environmental anomalies such as explosions and natural disasters or on social violations such as fighting and theft.","The same generative pipeline could become an augmentation strategy for real anomaly datasets, mixing synthetic clips with real footage during training rather than using synthetic data only as a standalone benchmark."],"forward_implications":["If the zero-shot transfer holds up, synthetic footage can stand in for real CCTV footage when training anomaly retrieval, removing the privacy barrier that has capped real anomaly datasets at roughly 1,900 clips.","A model trained on SVTA can be evaluated over 68 anomaly types rather than the 13 in UCFCrime-AR, enabling finer-grained and more realistic retrieval evaluation.","Because SVTA is generated, its category space can be extended on demand by prompting the LLM and generator again, making anomaly retrieval benchmarks scalable to novel events.","The 2:3 normal-to-anomaly ratio plus explicit category labels supports training and evaluating models for both retrieval and downstream anomaly analysis beyond the binary normal and abnormal framing."],"supporting_citations":[{"why":"Wan2.1, the text-to-video model that generates every SVTA clip from the paired captions.","marker":"[34]"},{"why":"Qwen3, the LLM used to expand anomaly categories into 50 descriptions each and to annotate category labels for samples lacking them.","marker":"[42]"},{"why":"CLIP4Clip, one of the three retrieval baselines trained on SVTA and evaluated zero-shot on UCFCrime-AR and OOPS!.","marker":"[22]"},{"why":"X-CLIP, the second retrieval baseline, whose ViT-B/16 variant gives the best OOPS! zero-shot R@1 among the CLIP-based methods.","marker":"[24]"},{"why":"GRAM, the state-of-the-art multimodal baseline that achieves the best SVTA retrieval and the best UCFCrime-AR zero-shot R@1.","marker":"[5]"},{"why":"UCFCrime-AR, both a source of real anomaly captions and the real-world zero-shot target for cross-domain evaluation.","marker":"[41]"},{"why":"OOPS!, a source of textual descriptions and the second real-world zero-shot target, evaluated on a filtered subset.","marker":"[8]"},{"why":"UCA, a surveillance video-text dataset whose captions are retained as SVTA descriptions and whose limitations motivate the synthetic approach.","marker":"[44]"},{"why":"UBnormal, a synthetic anomaly benchmark that supplies 23 seed anomaly categories expanded by LLM prompting.","marker":"[1]"}],"fun_headline_variants":["Synthetic clips train models to spot real anomalies","41K synthetic videos power anomaly retrieval benchmark","SVTA: synthetic benchmark for cross-modal anomaly search","Synthetic data scales anomaly retrieval to 68 event types","Generated videos bridge gap for anomaly retrieval models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark rests on the assumption that Wan2.1 generates videos that faithfully show the events described in their captions; the only validation reported is informal human inspection, so if the generated clips do not actually depict the described anomalies, SVTA may measure text-to-text similarity rather than video understanding of the event.","fun_headline_variants_meta":{"raw":{"variants":["Synthetic clips train models to spot real anomalies","41K synthetic videos power anomaly retrieval benchmark","SVTA: synthetic benchmark for cross-modal anomaly search","Synthetic data scales anomaly retrieval to 68 event types","Generated videos bridge gap for anomaly retrieval models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000261,"raw_usage":{"total_tokens":1613,"prompt_tokens":987,"completion_tokens":626,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":554}},"tokens_in":603,"tokens_out":626,"duration_ms":5793,"temperature":1.0,"reasoning_tokens":554,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:39:35.994597+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of, say, 200 SVTA clips, show each to independent annotators with its paired caption, and ask whether the video visibly depicts the described event; if more than about 10% are judged not to depict it, the zero-shot transfer results cannot be taken as evidence of visual anomaly understanding. A corroborating check is to compare against a caption-only retriever that matches query text to stored captions and ignores video content, since similar scores would indicate language priors rather than video grounding.","supporting_citations":[{"cited_title":"Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning","cited_arxiv_id":null,"evidence_quote":"CLIP4Clip, one of the three retrieval baselines trained on SVTA and evaluated zero-shot on UCFCrime-AR and OOPS!."},{"cited_title":"X-clip: End-to-end multi-grained con- trastive learning for video-text retrieval","cited_arxiv_id":null,"evidence_quote":"X-CLIP, the second retrieval baseline, whose ViT-B/16 variant gives the best OOPS! zero-shot R@1 among the CLIP-based methods."},{"cited_title":"Gramian multimodal representation learning and alignment","cited_arxiv_id":null,"evidence_quote":"GRAM, the state-of-the-art multimodal baseline that achieves the best SVTA retrieval and the best UCFCrime-AR zero-shot R@1."},{"cited_title":"Toward video anomaly retrieval from video anomaly detection: New benchmarks and model.IEEE Transactions on Image Processing, 33:2213–2225, 2024","cited_arxiv_id":null,"evidence_quote":"UCFCrime-AR, both a source of real anomaly captions and the real-world zero-shot target for cross-domain evaluation."},{"cited_title":"Oops! pre- dicting unintentional action in video","cited_arxiv_id":null,"evidence_quote":"OOPS!, a source of textual descriptions and the second real-world zero-shot target, evaluated on a filtered subset."},{"cited_title":"Towards surveillance video-and-language understanding: New dataset baselines and challenges","cited_arxiv_id":null,"evidence_quote":"UCA, a surveillance video-text dataset whose captions are retained as SVTA descriptions and whose limitations motivate the synthetic approach."},{"cited_title":"Ub- normal: New benchmark for supervised open-set video anomaly detection","cited_arxiv_id":null,"evidence_quote":"UBnormal, a synthetic anomaly benchmark that supplies 23 seed anomaly categories expanded by LLM prompting."}],"review_version":1}