{"id":"20ff6fff-07c8-4add-b0a4-222a1634a7a5","arxiv_id":"2412.06171","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A new multi-granularity benchmark and an anomaly-score-guided frame sampler are presented to improve long-term video anomaly understanding with multimodal LLMs.","lead":"This paper introduces HIVAU-70k, a large benchmark of over 70,000 hierarchical annotations for video anomaly understanding across clips, events, and full videos. It also proposes an anomaly-focused temporal sampler that selects frames for a multimodal language model to describe anomalies in long videos.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Detection comparison is unfair: Holmes-VAU trains its anomaly scorer with frame-level labels while baselines use weak or unsupervised labels, so Table 1 does not establish that ATS+VLM outperform traditional methods.","rationale":"The reader's verdict is CONDITIONAL, and the rationale already notes the unfair detection comparison; however, the reader's stated weakest assumption is annotation quality, not the supervision mismatch. I agree that annotation quality matters, but the most direct threat to the paper's central claim is the comparison in Table 1: the proposed method is given frame-level supervision that no baseline receives. If the concrete test shows UR-DMU alone achieves similar detection performance, then the claim that the integrated ATS+VLM outperforms traditional methods collapses, although the benchmark and the ATS ablation (Table 4) would still have value. The reasoning evaluation is also self-constructed and lacks external validation, but the detection comparison is the more immediate and falsifiable weakness. I therefore keep the verdict CONDITIONAL: the paper should be revised to include properly supervised baselines or a weakly-supervised variant of the method, and the reasoning claims should be validated with human evaluation or an external benchmark.","tokens_in":22661,"tokens_out":5124,"duration_ms":53666,"concrete_test":"Train the UR-DMU anomaly scorer (the same architecture used inside ATS) with the frame-level labels from HIVAU-70k on the XD-Violence and UCF-Crime training splits, then evaluate AP/AUC on the same test sets without the VLM or ATS. If UR-DMU alone matches or exceeds 87.68 AP and 88.96 AUC, the detection advantage of Holmes-VAU does not come from ATS or the VLM but from the extra supervision.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the integrated ATS and VLM outperform traditional methods rests on Table 1, but the comparison is confounded by supervision level. The anomaly scorer is trained with a binary cross-entropy loss (Eq. 5) on frame-level labels derived from manual event boundaries, whereas the baselines (RTFM, S3R, UR-DMU, VadCLIP, etc.) use video-level weak labels or are unsupervised. The reported gains—87.68 AP vs. 84.51 on XD-Violence and 88.96 AUC vs. 88.02 on UCF-Crime—could therefore be entirely attributable to the additional supervision, not to ATS or the VLM. The paper implicitly acknowledges this in Sec. 4.3, stating that the frame-level labels provide 'more accurate anomaly supervision compared to previous unsupervised and weakly-supervised methods.' The ATS is only compared within the proposed pipeline (Table 4), so its marginal benefit is internally consistent but does not validate the headline claim against traditional methods. Without a fully-supervised baseline or a weakly-supervised variant of the proposed method, the claim that the integrated system outperforms traditional VAD is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces HIVAU-70k, a hierarchical video anomaly understanding benchmark with clip-, event-, and video-level annotations, built from UCF-Crime and XD-Violence using a semi-automated pipeline that combines manual event-boundary labeling with LLM-generated summaries. The paper also proposes Holmes-VAU, a multimodal LLM pipeline with an Anomaly-focused Temporal Sampler (ATS) that scores frames with a trained anomaly scorer and non-uniformly samples frames for the LLM. Experiments report state-of-the-art detection results on UCF-Crime and XD-Violence, large gains over general MLLMs on the new reasoning test set, and ablations supporting the ATS and hierarchical instruction data.","tokens_in":22914,"tokens_out":4337,"duration_ms":45304,"significance":"If properly validated, HIVAU-70k would be a valuable community resource: it is large, publicly released, and addresses a real gap by providing multi-granular anomaly annotations. The ATS idea, using anomaly scores to guide temporal sampling for a VLM, is also interesting and could transfer to other long-video understanding tasks. However, the current evidence for the two headline claims ('outperforms traditional methods' and 'hierarchical instruction data markedly improves anomaly comprehension') is weakened by a supervision confound in the detection comparison and by a self-consistent evaluation protocol for reasoning. The paper would be significantly strengthened by controlled experiments that separate the contribution of the extra frame-level supervision from the contribution of the ATS/VLM.","major_comments":[{"comment":"The detection comparison is not apples-to-apples. Holmes-VAU's anomaly scorer is initialized from UR-DMU and trained with frame-level labels derived from manual event boundaries using binary cross-entropy (Eq. 5), whereas all non-explainable baselines in Table 1 are unsupervised or weakly supervised with only video-level labels. The reported gains (87.68 AP on XD-Violence, 88.96 AUC on UCF-Crime) could therefore be due entirely to the added frame-level supervision. I recommend adding a fully supervised baseline, most naturally UR-DMU trained with the same frame-level labels and the same visual features, and/or a weakly supervised variant of the proposed scorer. Without such a control, the abstract's claim that the integrated ATS and visual-language model outperform traditional methods is not established.","section":"Table 1 / Sec 4.3 / Eq. (5)"},{"comment":"The reasoning evaluation is strongly self-referential. The test ground-truth text is generated with the same LLM pipeline (LLaMA3-70B summaries built from clip captions with the same prompt templates) that produced the instruction-tuning data, and the model is fine-tuned to imitate that style. Under this protocol, high BLEU/CIDEr scores may reflect style mimicry rather than anomaly comprehension. The manual review described in Sec 5.1 does not remove this bias, since the underlying text and its style are inherited from the annotation engine. Please add an evaluation on independently written human annotations, a human preference study, or an existing external anomaly reasoning benchmark; also consider reporting judgment accuracy as a binary classification metric rather than only lexical overlap.","section":"Sec 5.1 / Sec 3.2 / Table 2"},{"comment":"The efficiency and accuracy claims for ATS are only partially supported. In Table 4, latency values are reported only for Uniform sampling; the Top-K and ATS rows have empty latency cells, so the paper does not demonstrate that ATS is more efficient than the alternatives. In addition, the accuracy comparison is limited to video-level BLEU and CIDEr on the authors' own test set; no detection metrics (AUC/AP) are reported for different samplers, even though detection is a central claim of the paper. Please report latency for all samplers and, if detection is claimed to improve, the corresponding detection results.","section":"Table 4 / Sec 5.3"}],"minor_comments":[{"comment":"Typo: 'off-the-shell' should be 'off-the-shelf'.","section":"Sec 3.2"},{"comment":"The checkmark alignment in Table 3 is hard to read; the table would be clearer if each row explicitly listed the included granularities (e.g., 'C only', 'E only', 'C+E') rather than leaving the reader to infer them from the checkmark columns.","section":"Table 3"},{"comment":"Figure 6(a) shows loss curves but the axes labels are minimal ('iter' and 'loss'), and Figure 6(b) does not report the numerical values for Video-MME; please provide the exact scores so readers can judge the trade-off quantitatively.","section":"Fig. 6"},{"comment":"The annotation quality control is described qualitatively ('manual checking', 'carefully reviewed'), but no inter-annotator agreement or consistency statistics are reported. Even a small-scale agreement study on event boundaries and summary correctness would increase confidence in the benchmark.","section":"Sec 3.3 / Supp. A.2"}],"recommendation":"major_revision","confidential_remarks":"The dataset release is potentially a significant contribution, and the ATS idea has merit. I would be willing to look at a revised version that adds the controlled detection baseline and a less self-referential reasoning evaluation. The current framing of the main results is too strong given the supervision confound."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The dataset is the real contribution here. HIVAU-70k provides clip-, event-, and video-level annotations for over 70k segments, and that hierarchical structure is genuinely new relative to UCA, CUVA, and Hawk. The semi-automated pipeline—manual event boundaries plus LLM summarization with light manual checking—is sensible, and the public release gives the community a resource that could standardize explainable video anomaly evaluation. Credit where due: the authors also include a limitations section and acknowledge the supervision gap in their own detector training.\n\nThat said, the experimental section does not support the strong claims in the abstract. Table 1 compares a detector trained with frame-level labels against weakly-supervised and unsupervised baselines, so the reported AP/AUC gains could easily be pure supervision effect. The paper's own Sec. 4.3 admits this. The ATS ablation in Table 4 is internally consistent, but it only shows marginal gains over uniform sampling (e.g., 0.576 vs 0.558 BLEU at N=16), and it does not validate the headline 'outperform traditional methods' claim.\n\nThe reasoning evaluation has a separate problem: the test set is built with the same LLM prompt-and-summarize pipeline as the training data, and no error bars or human evaluation are reported. The n-gram metrics are also weak for judging anomaly reasoning quality. These issues are fixable but require actual work: add a fully-supervised baseline (e.g., UR-DMU with the same frame labels), add a weakly-supervised variant of the proposed method, and validate the reasoning set with human judgments or an independent annotation source.\n\nThe math and citation pattern look fine. ATS is a simple combination of existing components—no invented machinery—and the related work is covered honestly. The benchmark itself deserves citation and could be a solid evaluation tool once the experiments are cleaned up. The right audience is researchers working on video anomaly detection and multimodal video understanding; they will want this dataset even if they skip the method.\n\nA serious editor should send this to peer review. The dataset is important enough to warrant referee time, and the evaluation flaws are correctable in revision rather than fatal. My recommendation: invite a major revision, push the authors to fix the detection comparison and the reasoning evaluation, and then it could be a solid publishable resource.","headline":"HIVAU-70k is a genuinely useful hierarchical anomaly-understanding benchmark; the paper's detection and reasoning evaluations do not support the headline claims as currently written.","tokens_in":23448,"tokens_out":1659,"would_cite":true,"duration_ms":18219,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that hierarchical clip, event, and video annotations let a single model both localize and explain anomalies across time scales.","keywords":["video anomaly detection","video anomaly understanding","hierarchical annotations","multimodal large language models","Anomaly-focused Temporal Sampler","long-video reasoning","benchmark construction","visual-language models"],"falsifier":"Re-annotate a random sample of the videos with an independent team that marks event boundaries and checks whether the existing summaries correctly describe the anomalies; if inter-annotator agreement on boundaries is low, or if the LLM summaries frequently conflict with independent event descriptions, then both the training signal and the reasoning-test ground truth are unreliable. Separately, a controlled test with long videos containing two well-separated anomaly events can show whether ATS misses the second event after concentrating on the first at an equal frame budget.","tokens_in":22477,"feed_emoji":"🎥","tokens_out":9417,"duration_ms":83325,"temperature":0.7,"pith_summary":"This paper tries to establish that video anomaly understanding should be treated as a multi-scale reasoning problem, not just frame-level scoring. It introduces HIVAU-70k, a benchmark with more than 70,000 annotations organized at clip, event, and video levels, built from the UCF-Crime and XD-Violence collections using a semi-automated engine that pairs manual event segmentation with LLM-generated summaries. The paper also proposes the Anomaly-focused Temporal Sampler (ATS), which uses a lightweight anomaly scorer to concentrate a visual-language model's sampled frames in anomaly-dense stretches of long videos. On its test set, the resulting Holmes-VAU model reports higher anomaly-reasoning metrics than general video MLLMs, and higher detection scores than previously reported video anomaly detectors, with a much smaller model size. The point of the work is that hierarchical annotations plus adaptive sampling let one model both find and explain anomalies across temporal scales.","feed_headline":"Over 70,000 anomaly annotations span clips, events, and full videos","feed_subtitle":"A multi-granular benchmark and a sampler let video-language models reason about brief incidents and long-running threats alike.","key_machinery":"The load-bearing object is the Anomaly-focused Temporal Sampler (ATS): a two-part module that combines an anomaly scorer with a density-aware sampler. The scorer, following the UR-DMU architecture, maps each frame's class token to an anomaly score; the sampler treats those scores as a probability mass, accumulates them along time as $S_{\\text{cumsum}}(t)=\\sum_{i=1}^{t}(s_i+\\tau)$, and uniformly samples $N$ points on the cumulative axis to choose which frames enter the language model. The $\\tau$ term keeps a floor on uniform coverage, so sampling clusters around anomaly-rich moments without discarding context. The other load-bearing piece is the annotation engine: manual event segmentation followed by recursive LLM summarization from clip captions to event summaries to video summaries, which yields the Judgment, Description, and Analysis instruction data. Together they let a fixed 2B visual-language model spend its visual tokens where anomalies actually occur and learn to verbalize judgments at any timescale.","core_discovery":"The central claim is that anomaly comprehension improves when a model is trained and evaluated at three temporal granularities at once: short clips, anomaly events, and whole videos. The paper constructs this hierarchy automatically at scale by having humans mark event boundaries, using a video captioning model for clips, and then asking an LLM to recursively summarize clip captions into event summaries and event summaries into video summaries, with manual checking. The second claim is that the frame budget of a video-language model should be allocated by anomaly content: ATS treats anomaly scores as a temporal distribution, forms a cumulative sum, and samples frames by an inverse-transform-like mapping, so more frames land where anomaly scores are high. In experiments, training on the full hierarchy outperforms training on any single granularity, and ATS beats uniform and top-K sampling at equal frame counts. The paper reports an AP of 87.68% on XD-Violence, an AUC of 88.96% on UCF-Crime, and substantially higher BLEU, CIDEr, METEOR, and ROUGE scores than general video-language baselines on its 3,300-item reasoning test set.","pith_inferences":["The cumulative-sum sampler is inverse-transform sampling over the frame salience distribution; making the scorer differentiable and training the sampler end-to-end with the language model is a natural next step.","ATS could transfer to other long-video tasks that have a relevance score, such as action localization, highlight detection, or question-guided retrieval.","Because the reasoning ground truth is LLM-generated and only lightly checked, the reported BLEU and CIDEr gaps may partly reward lexical similarity to those summaries; a human-preference or factual-consistency probe would test whether the reasoning gains are real.","Replacing manual event segmentation with a segmentation model and human verification of boundaries could scale the same engine to more anomaly classes and domains at lower cost."],"forward_implications":["The full hierarchy becomes the training recipe: clip data supplies perception, event data supplies anomaly judgment, and video data supplies long-range analysis; dropping any level lowers measured reasoning quality.","With sixteen sampled frames, the ATS-equipped model beats uniform and top-K sampling at the same visual-token budget, so long-video anomaly reasoning can be run with far fewer frames than dense sampling requires.","Because the anomaly scorer is trained on HIVAU-70k's event boundaries, it can serve as a standalone frame-level detector; the paper reports it outperforms weakly-supervised baselines on UCF-Crime and XD-Violence.","The released benchmark lets other groups fine-tune video-language models on hierarchical anomaly instructions without repeating the annotation pipeline."],"supporting_citations":[{"why":"provides the UCF-Crime surveillance videos and video-level anomaly labels that HIVAU-70k's annotation engine segments into events and clips.","marker":"[46]"},{"why":"provides the XD-Violence videos and anomaly classes used as the second source collection for the benchmark and as a detection evaluation set.","marker":"[56]"},{"why":"is the off-the-shelf video perception model used to generate clip-level captions that feed the hierarchical summarization pipeline.","marker":"[28]"},{"why":"supplies the manually annotated clip captions for UCF-Crime that are incorporated directly into the instruction data.","marker":"[64]"},{"why":"is the LLM (Llama 3) prompted to turn clip captions into event summaries and video summaries with judgment, description, and analysis sections.","marker":"[2]"},{"why":"provides the UR-DMU architecture that the Anomaly-focused Temporal Sampler uses as its anomaly scorer.","marker":"[74]"},{"why":"supplies the frozen visual encoder, projector, and large language model that Holmes-VAU initializes and fine-tunes with LoRA.","marker":"[6]"},{"why":"is the low-rank adaptation method that keeps instruction tuning parameter-efficient while adapting the base model to anomaly understanding.","marker":"[15]"}],"fun_headline_variants":["Hierarchical video anomaly benchmark with 70k multi-granular annotations","Anomaly understanding across clips, events, and videos via 70k hierarchy","Density-aware sampler focuses video-language models on anomaly-rich frames","Multi-granular video anomaly: 70k annotations and a density-aware sampler"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's value depends on the manual event-boundary labels and the LLM-written summaries being accurate and consistent enough to serve both as frame-level training supervision for the anomaly scorer and as ground truth for evaluating anomaly reasoning.","fun_headline_variants_meta":{"raw":{"variants":["Hierarchical video anomaly benchmark with 70k multi-granular annotations","Anomaly understanding across clips, events, and videos via 70k hierarchy","Density-aware sampler focuses video-language models on anomaly-rich frames","Multi-granular video anomaly: 70k annotations and a density-aware sampler"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000343,"raw_usage":{"total_tokens":1927,"prompt_tokens":1026,"completion_tokens":901,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":642,"completion_tokens_details":{"reasoning_tokens":820}},"tokens_in":642,"tokens_out":901,"duration_ms":8499,"temperature":1.0,"reasoning_tokens":820,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:55:47.624133+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of the videos with an independent team that marks event boundaries and checks whether the existing summaries correctly describe the anomalies; if inter-annotator agreement on boundaries is low, or if the LLM summaries frequently conflict with independent event descriptions, then both the training signal and the reasoning-test ground truth are unreliable. Separately, a controlled test with long videos containing two well-separated anomaly events can show whether ATS misses the second event after concentrating on the first at an equal frame budget.","supporting_citations":[{"cited_title":"Towards surveillance video-and-language understanding: New dataset, baselines, and challenges, 2023","cited_arxiv_id":null,"evidence_quote":"supplies the manually annotated clip captions for UCF-Crime that are incorporated directly into the instruction data."},{"cited_title":"Dual memory units with uncertainty regulation for weakly supervised video 11 anomaly detection","cited_arxiv_id":null,"evidence_quote":"provides the UR-DMU architecture that the Anomaly-focused Temporal Sampler uses as its anomaly scorer."}],"review_version":1}