{"id":"688f721e-87df-491d-801a-5e52667bd80c","arxiv_id":"2411.10867","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper builds and human-labels a 3,782-video dataset spanning five hallucination categories in text-to-video outputs, and shows that standard classifiers reach only about 35% accuracy on the resulting classification task.","lead":"ViBe is a new benchmark of 3,782 AI-generated videos, each labeled with one of five visual hallucination types, created by prompting ten text-to-video models with MS COCO captions. It gives researchers a much larger test bed than the prior 50-video set for measuring how often video generators ignore or distort the prompt.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dataset selection protocol is undocumented: no total generation count, no exclusion rate, and no non-hallucinated videos are reported, while Table 2's exact repeated 70-counts suggest an unstated quota; this makes 'hallucination detection' and per-model distributions ungrounded.","rationale":"The reader's weakest assumption was the missing generation/exclusion statistics and the absence of non-hallucinated videos; my read agrees. I would not change the reader's CONDITIONAL verdict: the dataset, taxonomy, and kappa scores are plausible contributions, but the paper's framing as a hallucination-detection benchmark is not supported by the reported construction. The near-uniform 70-count cells in Table 2 strengthen the concern: they suggest a quota-based sampling rule that is not documented, and if quotas were used, the table's 'distribution' is an artifact of the annotation budget rather than a property of T2V models. This is not an accusation of fabrication; it is an internal-consistency check that the released data can settle. The kappa values (0.747-0.874) are useful but cover only 100 videos and do not address representativeness. The absence of a no-hallucination class is decisive for the detection claim: no classifier trained only on positive examples can establish a detection threshold. I therefore recommend keeping the CONDITIONAL verdict and requiring, at minimum, disclosure of the sampling protocol, full generation counts, and a detection-oriented evaluation with negative examples.","tokens_in":11212,"tokens_out":12497,"duration_ms":129656,"concrete_test":"Audit the released ViBe annotation records (or request them from the authors if not available): for every model and prompt, list the number of videos generated, the number excluded before annotation, the number assigned to each of the five categories, and any cap or stopping rule used per category. If the per-category counts in the raw records show the same 70-per-category pattern as Table 2, the dataset is quota-selected and the paper must report that; if no non-hallucinated videos are present, add a random sample of non-hallucinated outputs as a sixth class and rerun the TimeSFormer+CNN baseline to test whether the model can actually detect hallucinations rather than only label their type.","verdict_should_be":"UNCHANGED","load_bearing_attack":"ViBe's central contribution is a human-annotated benchmark, so the sampling rule is load-bearing. Section 3.4 says annotators categorized 3,782 videos, but the paper never states how many prompts generated videos per model, how many videos were discarded as non-hallucinated, or whether a per-category quota was used. The evidence in Table 2 is hard to explain under natural annotation: AnimateLCM has counts 2/70/70/70/70, MagicTime and HotShotXL have 70/70/70/69/70, and MS1.7B has 51/50/70/70/70. A cap of roughly 70 per category per model would produce exactly such a table, but this is never disclosed. If a quota or hallucination-only filter was applied, Table 2 is not a natural distribution of T2V hallucinations and no model-level hallucination rate can be computed. Moreover, all 3,782 videos are positive examples; there is no 'no hallucination' class in the classification experiments (Table 6). The reported TimeSFormer+CNN accuracy (0.345) therefore measures five-way categorization conditional on a video being hallucinated, not hallucination detection. The abstract and Section 4 claim 'hallucination detection,' but the benchmark cannot support that claim as constructed. This is independently verifiable from the dataset files, and it directly affects the paper's main claim of a reliable benchmark for T2V hallucination evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ViBe, a benchmark dataset of 3,782 videos generated by ten open-source text-to-video (T2V) models from 837 MS COCO captions. All videos were manually annotated into one of five hallucination categories: Vanishing Subject, Omission Error, Numeric Variability, Subject Dysmorphia, and Visual Incongruity. The authors report inter-annotator Cohen's kappa scores per category, extract video embeddings with TimeSFormer and VideoMAE, and evaluate seven classifiers on the resulting five-way classification task. The best baseline, TimeSFormer+CNN, reaches 0.345 accuracy and 0.342 F1. The paper claims to provide the first large-scale resource for systematically categorizing T2V hallucinations and a benchmark for hallucination detection.","tokens_in":11459,"tokens_out":4601,"duration_ms":44340,"significance":"If the dataset construction is sound, ViBe is a potentially valuable resource: it is substantially larger than the only comparable prior benchmark, T2VHaluBench (50 videos), and the five-category taxonomy is intuitive and illustrated with concrete examples. The reported kappa scores (0.7474--0.8737) indicate reasonable annotation consistency, and the classification baselines provide a useful reference point for future work. However, the current manuscript does not document the sampling/filtering protocol, does not include any non-hallucinated videos, and contains internal inconsistencies in reported counts; these issues must be resolved before the benchmark's representativeness and its 'hallucination detection' framing can be accepted.","major_comments":[{"comment":"The dataset selection protocol is not disclosed. The paper never reports how many videos were generated per model, how many were excluded because they were judged non-hallucinated, or whether a per-category quota or cap was applied. The exact repeated counts of 70 in Table 2 (e.g., AnimateLCM 2/70/70/70/70, MagicTime 70/70/70/69/70, HotShotXL 70/70/70/69/70) strongly suggest an undocumented sampling cap. This is load-bearing because without the denominator and exclusion rule, Table 2 cannot be interpreted as a natural distribution of hallucinations across models, and no model-level hallucination rate can be computed. The authors should report the total number of generated videos per model, the inclusion/exclusion criteria, and any quota used.","section":"Section 3.4, Table 2"},{"comment":"The benchmark contains only hallucinated videos: all 3,782 instances are positive examples, and the classification task has exactly five hallucination-type labels with no 'no hallucination' class. Consequently, the reported TimeSFormer+CNN result of 0.345 accuracy is five-way categorization accuracy conditional on a video already being known to be hallucinated; it is not hallucination detection. The abstract and Section 4 claim 'hallucination detection,' but the experimental setup cannot support such a claim. Either non-hallucinated videos should be included and a detection task defined, or the claims and title framing should be revised to 'hallucination-type classification.'","section":"Abstract, Section 4, Table 6"},{"comment":"The numerical values in Figure 6 are inconsistent with Table 2 and with the prose in the same paragraph. For example, Table 2 reports 282 AnimateLCM videos and 694 MORA videos, all 1 second long, so the cumulative durations should be 282 seconds and 694 seconds, respectively, but Figure 6 lists 349 seconds and 349 seconds. Similarly, zeroscope_v2_XL (363 videos at 2 seconds = 726 seconds) is labeled 311 seconds in the figure, while the text correctly says 726 seconds. This suggests the bar labels are misassigned. The figure must be corrected, and the corresponding values in Table 2 should be cross-checked, because the internal inconsistency weakens confidence in the reported dataset counts.","section":"Section 3.5, Figure 6"}],"minor_comments":[{"comment":"The sentence beginning 'The total duration per model refers to the cumulative time spent annotating all videos...' is garbled (\"as shown in 6. 1 provides a detailed report\"); it should be rewritten for clarity.","section":"Section 3.5"},{"comment":"The appendix states that the average annotation time was 2.25 seconds per video, which is implausible and contradicts Figure 7, where annotation times are hundreds or thousands of minutes per model. This is likely 2.25 minutes; please correct the units.","section":"Appendix B"},{"comment":"The paragraph on image-generation hallucination benchmarks cites HADES (Liu et al., 2022) as an image-generation hallucination dataset, but the reference is actually a token-level text hallucination detection benchmark; the citation or the surrounding description should be corrected.","section":"Section 2"},{"comment":"References to Chu et al. appear as both 2024a and 2024b for what appears to be the same Sora Detector paper; please deduplicate or clarify the distinction.","section":"References"},{"comment":"The kappa values are reported per category, but the number of annotators who labeled the 100 common videos and the procedure for adjudicating disagreements are not described; a brief note on the annotation adjudication protocol would help.","section":"Section 3.6"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the undocumented dataset sampling/filtering process. The exact 70-count repetitions in Table 2 strongly imply a quota or cap, and if so, the paper's model-level distributions and detection framing are not supported. The authors should be asked to provide the full generation counts, filtering criteria, and a negative class or a clear reframing of the task. The Figure 6 numeric mismatch is also concerning for data integrity and should be fixed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ViBe is a real attempt to fill a gap, and the resource itself is the contribution. The five-category taxonomy and the 3,782 human-annotated videos are a step up from T2VHaluBench's 50 clips. The kappa scores (0.75-0.87) are decent for this kind of perceptual annotation, and the appendix gives concrete instructions. The dataset, if properly documented, would be useful to anyone working on T2V failure modes.\n\nThat said, the paper doesn't support its main claim. The selection protocol is unspecified: we don't know how many videos were generated, how many were rejected as non-hallucinated, or why Table 2 contains so many cells at exactly 70. The natural reading is that the authors capped counts per category per model. If that's true, the table is not a natural distribution, and you can't use it to compare hallucination tendencies across models. The paper needs to disclose the quota or the filtering rule. Second, all 3,782 videos are positive examples, so the classifiers are doing five-way labeling, not hallucination detection. The abstract and Section 4 say detection, which is an overclaim. Not fatal, but it changes what the benchmark can claim.\n\nThere are also smaller internal inconsistencies: the annotation-time figures contradict the text and the average annotation time of 2.25 seconds doesn't square with the reported total minutes. The paper mentions a website but doesn't state whether videos and labels will actually be released; that matters for a benchmark.\n\nI'd send this to peer review with a request for major revision. The resource is worth referee time, but the sampling story and the detection/classification distinction need to be addressed before I'd trust the numbers.","headline":"Useful dataset, but the sampling story is undocumented and the 'detection' claim doesn't match the data; worth reviewing with major revision.","tokens_in":12050,"tokens_out":3705,"would_cite":false,"duration_ms":33922,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ViBe introduces a benchmark categorizing hallucinations in text-to-video generation.","keywords":["text-to-video","hallucination","benchmark","video classification","TimeSFormer","VideoMAE","MS COCO","human annotation"],"falsifier":"Sampling the same ten T2V models with the same prompts and counting how many generated videos contain none of the five hallucination types would directly test the claim that these categories are the dominant failure modes; if a large share of videos fall outside the taxonomy, the benchmark's coverage claim collapses.","tokens_in":11025,"feed_emoji":"🎬","tokens_out":3462,"duration_ms":34286,"temperature":0.7,"pith_summary":"The paper introduces ViBe, a human-annotated dataset of 3,782 videos generated by ten open-source text-to-video models from 837 MS COCO captions, with each video labeled as one of five hallucination types: Vanishing Subject, Omission Error, Numeric Variability, Subject Dysmorphia, and Visual Incongruity. The authors argue this is the first large-scale benchmark for systematically categorizing hallucinations in generated video, offering a more substantial resource than the prior 50-video benchmark. They also run classification baselines using VideoMAE and TimeSFormer embeddings with seven classifiers; the best configuration, TimeSFormer with a CNN, reaches 0.345 accuracy and 0.342 F1 score. The modest performance is presented as evidence that automated hallucination detection is difficult and that the dataset should drive the development of better methods.","feed_headline":"A benchmark sorts text-to-video hallucinations into five types","feed_subtitle":"ViBe: 3,782 human-labeled videos from ten generators, plus baseline classifiers to beat.","key_machinery":"The central organizing device is the five-category taxonomy of text-to-video hallucinations: Vanishing Subject, Omission Error, Numeric Variability, Subject Dysmorphia, and Visual Incongruity. This taxonomy drives the manual annotation protocol that labels all 3,782 videos and defines the classification task. The benchmark pipeline then extracts video embeddings with pre-trained models, VideoMAE and TimeSFormer, and feeds them into classifiers; the best-performing configuration is TimeSFormer embeddings combined with a convolutional neural network.","core_discovery":"On its own terms, the paper claims to provide the first dataset specifically designed to systematically categorize hallucinations in text-to-video outputs. ViBe is constructed by prompting ten open-source T2V models with diverse MS COCO captions, manually annotating every generated video, and organizing the results into a five-category taxonomy that captures recurring failure patterns: subjects that vanish mid-video, objects omitted from the prompt, altered counts of objects, distorted forms, and physically impossible scenes. The paper further establishes a classification benchmark using video embeddings and reports baseline results, positioning ViBe as a foundation for evaluating prompt adherence and developing hallucination detectors for T2V models.","pith_inferences":["Since ViBe contains only videos that human annotators judged to be hallucinated and the paper does not report how many generated videos were excluded, the dataset likely overstates how frequently hallucinations occur; a control set of non-hallucinated videos would be needed to build a true detector.","The single-label annotation protocol may compress cases where multiple hallucination types co-occur, so a multi-label variant could change baseline performance and better reflect real T2V errors.","With most videos lasting just one second, the distribution of hallucination types may not transfer to longer generations, where temporal consistency failures could appear differently.","A natural extension would be to use the five categories as fine-grained feedback signals for training T2V models with human preference alignment, rather than only for detection."],"forward_implications":["Any text-to-video model can be evaluated against ViBe's five categories to identify its recurring failure patterns.","The five categories provide a shared vocabulary for comparing hallucination types across different T2V models.","The TimeSFormer + CNN baseline of 0.345 accuracy and 0.342 F1 gives a concrete reference point that future hallucination classifiers must exceed.","If the taxonomy holds, training detectors to flag specific hallucination types becomes a well-defined task, potentially enabling automatic evaluation and mitigation.","The dataset's scale, over 3,700 videos, supports training and testing that the previous 50-video benchmark could not."],"supporting_citations":[{"why":"Provides the 837 MS COCO captions used as prompts for generating the videos in ViBe.","marker":"(Lin et al., 2015)"},{"why":"Introduces the prior 50-video T2V hallucination benchmark that ViBe compares against and aims to supersede.","marker":"(Chu et al., 2024a)"},{"why":"Supplies the TimeSFormer video embedding model that, combined with CNN, yields the best classification results.","marker":"(Bertasius et al., 2021)"},{"why":"Supplies the VideoMAE video embedding model used as an alternative feature extractor in the classification baselines.","marker":"(Tong et al., 2022)"},{"why":"One of the ten open-source T2V models used to generate videos; representative of the model diversity in the dataset.","marker":"(Mullan et al., 2023)"},{"why":"Represents prior work on video hallucination evaluation that motivates the need for a large-scale human-annotated dataset.","marker":"(Wang et al., 2024b)"}],"fun_headline_variants":["Five hallucination types plague text-to-video AI","ViBe: first benchmark to categorize T2V hallucinations","3,782 labeled videos tag AI's five hallucination types","Text-to-video hallucination sorted into five by ViBe","New benchmark catalogs AI video hallucinations in five ways"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dataset is built only from videos that human annotators judged to contain hallucinations, and the paper does not report how many generated videos were excluded, so the benchmark's representativeness of text-to-video hallucination phenomena is unverified.","fun_headline_variants_meta":{"raw":{"variants":["Five hallucination types plague text-to-video AI","ViBe: first benchmark to categorize T2V hallucinations","3,782 labeled videos tag AI's five hallucination types","Text-to-video hallucination sorted into five by ViBe","New benchmark catalogs AI video hallucinations in five ways"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001201,"raw_usage":{"total_tokens":4935,"prompt_tokens":915,"completion_tokens":4020,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":531,"completion_tokens_details":{"reasoning_tokens":3939}},"tokens_in":531,"tokens_out":4020,"duration_ms":35582,"temperature":1.0,"reasoning_tokens":3939,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:12:09.648122+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sampling the same ten T2V models with the same prompts and counting how many generated videos contain none of the five hallucination types would directly test the claim that these categories are the dominant failure modes; if a large share of videos fall outside the taxonomy, the benchmark's coverage claim collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"One of the ten open-source T2V models used to generate videos; representative of the model diversity in the dataset."}],"review_version":1}