{"id":"b7f34eed-f56f-4b1d-9464-8343b8c94166","arxiv_id":"2502.04076","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CRAVE is a new evaluator for AI-generated videos with long prompts, combining text alignment, motion analysis, and visual quality, plus a new content-rich benchmark called CRAVE-DB.","lead":"This paper introduces CRAVE, a model that scores AI-generated videos by checking text-video alignment, motion quality, and visual appeal. It also presents CRAVE-DB, a new benchmark of 1,228 videos from Sora-era generators with long text prompts and human opinion scores.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"CRAVE-DB evaluation protocol is underspecified; without the train/test split and prompt-disjointness guarantees, the reported SOTA margins on the proposed benchmark cannot be confirmed as a valid measurement.","rationale":"The reader's weakest_assumption is the validity of the CRAVE-DB evaluation, and my stress-test converges on the same point: the paper never specifies how the 1,228 videos (from 410 prompts and 4 models) were split into training and test sets. This is the single most load-bearing vulnerability because CRAVE-DB is both the proposed benchmark and the primary evidence for the method's advantage on Sora-era videos. The MTT text-alignment module makes prompt-level leakage a concrete mechanism for inflation: if two videos generated from the same prompt straddle the split, the model can memorize prompt-video score relationships that would not generalize to unseen prompts, which is the actual deployment scenario. I do not believe the paper is fraudulent; the most plausible reading is an omitted implementation detail. However, the omission is consequential because the SOTA margin on CRAVE-DB is modest (SRCC 0.7587 vs 0.7266 for T2VQA) and no error bars are provided. The ablation in Table 4 compounds this because the authors state that underlined settings are used in the final model, implying they selected components using test-set results. The zero-shot VideoGenEval experiment (Figure 7) is also presented as validation of generalization but lacks any human ground truth, so at most it demonstrates that the model produces a ranking, not that the ranking is correct. These issues are all addressable with released code/data and a clearer protocol, which is why CONDITIONAL (matching the reader) is appropriate rather than REJECT. If the author-provided artifacts confirm a prompt-disjoint split and stable improvements, the central claim would hold.","tokens_in":15528,"tokens_out":1842,"duration_ms":16360,"concrete_test":"Request the exact CRAVE-DB split protocol (seed, ratio, video-level vs prompt-level) and rerun Table 2 with a prompt-disjoint split where no prompt appears in both train and test. If the CRAVE SRCC drops by more than ~0.03 or falls below T2VQA/TriVQA, the SOTA claim on CRAVE-DB does not hold as stated. Additionally, recompute Table 4 with 5 random seeds and report mean/std for the final configuration; if the best-configuration advantage over the 'None' baseline is within one standard deviation, the ablation selection is not conclusive.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim that CRAVE achieves leading human-aligned VQA results, especially on the proposed CRAVE-DB, rests on Table 2 and the CRAVE-DB rows of Table 4. Section 5.1 states only 'we train 40 epochs on the training split and evaluate on the test set' without specifying the split ratio, whether the split is at video level or prompt level, or whether any videos share a source prompt across train and test. CRAVE-DB has 1,228 videos from only 410 prompts and a handful of generation models, so a random video-level split is very likely to place different videos from the same prompt into both train and test. Because CRAVE's MTT module explicitly learns text-video alignment per prompt, such leakage can inflate SRCC/PLCC/KRCC relative to a prompt-disjoint split. The 0.7587 SRCC on CRAVE-DB is roughly 0.03 above T2VQA and 0.04 above TriVQA, a margin well within what prompt leakage could plausibly produce. Table 4 also reports design choices (e.g., 'Combined,' 'Hybrid,' 16 flow frames) selected with access to CRAVE-DB test performance, and no error bars are given, so the reported gaps may reflect selection rather than robust method advantage. A second, independent concern is the zero-shot ranking experiment in Section 5.4: Figure 7 is presented without quantitative alignment to human preference, and VideoGenEval has no MOS annotations, so the claim of '0-shot generalization capability' is not actually validated by the experiment as reported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CRAVE, a no-reference video quality assessment model for text-to-video AIGC content, targeting 'Sora-era' videos with long, detailed prompts and complex motion. CRAVE combines a visual harmony branch built on DOVER, a multi-granularity text-temporal fusion module that aligns paragraph-, phrase-, and word-level text embeddings with temporally adapted video features, and a hybrid motion-fidelity module combining optical flow and action-recognition features. The authors also introduce CRAVE-DB, a new benchmark of 1,228 videos generated by Kling, Vidu, Qingying, and Sora from 410 elaborate prompts, with MOS from 29 annotators. Experiments report state-of-the-art SRCC/PLCC/KRCC on CRAVE-DB and T2VQA-DB, plus a zero-shot ranking experiment on VideoGenEval.","tokens_in":15839,"tokens_out":4598,"duration_ms":44324,"significance":"The problem addressed is timely: existing AIGC VQA datasets and methods are built around short prompts and simple motion, while current video generators produce longer, content-rich videos. The proposed three-branch design is reasonable, and the new benchmark addresses a real gap by using dense prompts and 29 annotators per video, which exceeds ITU recommendations. The T2VQA-DB evaluation with 10-fold cross-validation provides some independent grounding beyond the authors' own dataset. If the evaluation-protocol concerns below are resolved, CRAVE and CRAVE-DB would be a useful contribution to the community. The planned public release of data and code is a further strength.","major_comments":[{"comment":"The CRAVE-DB evaluation protocol is not specified at the level needed to validate the reported state-of-the-art results. The text states only that the authors 'train 40 epochs on the training split and evaluate on the test set.' It does not state the split ratio, whether the split is at the video level or the prompt level, or whether any of the 410 prompts appear in both training and test partitions. Because CRAVE's MTT module explicitly learns text-video alignment and CRAVE-DB contains multiple videos per prompt, a random video-level split is likely to place videos sharing a prompt in both partitions. Such prompt leakage can inflate SRCC/PLCC/KRCC, and the margin over T2VQA (0.7587 vs. 0.7266 SRCC) is small enough that leakage could account for part of it. Please specify the exact split and report results under a prompt-disjoint split.","section":"Section 5.1, Table 2"},{"comment":"The ablation study selects the final design ('Combined' text injection, 'Hybrid' motion modeling, 16 flow frames) using test-set performance on CRAVE-DB and T2VQA-DB, and the same test sets are then used for the headline comparisons in Tables 2 and 3. This selection-on-test procedure means the reported margins include model-selection bias. The design choices should be fixed on a validation split (or via nested cross-validation) before evaluating on the test set, or the reported numbers should be adjusted accordingly.","section":"Section 5.5, Table 4"},{"comment":"No error bars or significance tests are reported. T2VQA-DB is evaluated with 10-fold cross-validation, so mean and standard deviation over folds should be given; for CRAVE-DB, multiple random splits or bootstrapping are needed to establish whether the 0.03-0.04 SRCC advantages over prior methods are statistically meaningful.","section":"Tables 2 and 3"},{"comment":"The zero-shot ranking experiment does not validate '0-shot generalization capability' because VideoGenEval has no human MOS and Figure 7 is presented without any quantitative comparison to human preference. As reported, the figure only shows that CRAVE produces rankings. Either obtain human ratings for a subset of VideoGenEval videos and compute correlation, or reframe the figure as a qualitative demonstration and remove the generalization claim.","section":"Section 5.4, Figure 7"}],"minor_comments":[{"comment":"For CRAVE-DB, the training schedule is described only as '40 epochs'; it is unclear whether the linear-probing-then-fine-tuning protocol used for T2VQA-DB also applies. Please state the exact protocol.","section":"Section 5.1"},{"comment":"The sentence 'we curated 14 content-rich prompts and their corresponding outputs from Sora's publicly showcased videos, resulting in a total of 1,228 videos' is ambiguous: does Sora contribute 14 videos or multiple videos per prompt? Please report the number of videos per generation model.","section":"Section 3.2"},{"comment":"The summation over granularity levels uses unweighted cosine distances; the ablated variants in Table 4 suggest sensitivity to how levels are combined, so a brief discussion or sensitivity analysis of the combination weights would help.","section":"Section 4.3, Eq. (11)"},{"comment":"Several '0-shot' baselines (HPSv2, PickScore, ImageReward) are image preference models rather than video quality models; noting their input modality in the table caption would avoid confusion.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The novelty and scope are appropriate for the venue; the main risk is that the CRAVE-DB numbers are inflated by prompt leakage and selection-on-test. I would ask the editor to require a precise split specification, a prompt-disjoint evaluation, and error bars before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real asset here is CRAVE-DB: 1,228 Sora-era videos, 410 long prompts averaging 68 words, 29 annotators, and clean MOS collection with z-scoring and BT.500 screening. That is a genuinely useful resource for the AIGC VQA subfield, and the prompt-construction pipeline (ShareGPT-4o summaries plus manual curation) is thoughtful. The method, CRAVE, is a fair combination of existing pieces—DOVER, BLIP with temporal adapters, StreamFlow optical flow, Uniformer—plus multi-granularity text fusion. That fusion is a reasonable extension of TriVQA and T2VQA, not a breakthrough, but the ablations suggest it does contribute, especially on the new benchmark.\n\nThe soft spots are real but concentrated in the evaluation protocol. The paper never states the CRAVE-DB train/test split ratio, whether the split is at video or prompt level, or whether the same prompt can appear in both train and test. With only 410 prompts for 1,228 videos, a video-level split will almost certainly leak prompts across sets, and since the MTT module explicitly learns text-video alignment, that leakage could plausibly explain most of the 0.03 SRCC gap over T2VQA on CRAVE-DB. That is a load-bearing measurement issue for the headline claim. Relatedly, Table 4's design choices (combined text injection, hybrid motion, 16 flow frames) were selected with access to test performance, and no error bars are reported anywhere. The zero-shot VideoGenEval ranking in Figure 7 has no ground truth, so it is a demo, not a validation. On the positive side, the T2VQA-DB results use 10-fold cross-validation and still show a small but consistent gain (0.8122 vs. 0.7965 SRCC), which is independent evidence that the architecture is not just overfitting to the new dataset.\n\nMy overall take: this is a solid, worthwhile submission for the AIGC VQA audience. The flaws are addressable, not fatal. A careful editor should send it to review, but the referees should insist on the CRAVE-DB split protocol, prompt-disjointness, and error bars before the benchmark claims are accepted. I would cite the benchmark if the data and code actually ship.","headline":"A useful new benchmark and a sensible method for AIGC VQA, but the CRAVE-DB evaluation protocol is underspecified enough that the headline SOTA margins on that benchmark are not yet trustworthy.","tokens_in":16392,"tokens_out":934,"would_cite":true,"duration_ms":11103,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CRAVE is a new quality evaluator that claims to align AI-generated video assessment with human opinion by fusing long text prompts with motion-aware video understanding, and it reports state-of-the-art correlations on two AIGC benchmarks.","keywords":["AIGC video quality assessment","text-to-video evaluation","text-video alignment","motion fidelity","video quality benchmark","multi-granularity text fusion","optical flow","human perception alignment"],"falsifier":"Train CRAVE on CRAVE-DB with several random 80/20, 70/30, and 60/40 train/test splits of the 1,228 videos, keeping the same 40-epoch schedule, and report SRCC/PLCC with standard deviations; if the mean SRCC falls below the T2VQA baseline of 0.7266 under any plausible split, the claimed CRAVE-DB lead is an artifact of the undisclosed split protocol.","tokens_in":15310,"feed_emoji":"🎬","tokens_out":7844,"duration_ms":64358,"temperature":0.7,"pith_summary":"This paper proposes CRAVE, an automatic quality-assessor for AI-generated videos from next-generation text-to-video models such as Sora, which produce long, motion-rich clips and accept elaborate prompts. CRAVE scores a video from three perspectives: traditional visual aesthetics and distortion, semantic alignment between the prompt and the video at paragraph, phrase, and word granularity, and motion fidelity judged from both optical flow and abstract action features. The paper also introduces CRAVE-DB, a benchmark of 1,228 videos generated by Kling, Qingying, Vidu, and Sora from 410 intricate prompts, each scored by 29 human annotators. The paper reports that CRAVE achieves the highest Spearman rank correlation with human scores on both CRAVE-DB and the existing T2VQA-DB, outperforming prior AIGC video quality methods. If those results hold, CRAVE and CRAVE-DB give the field a way to track and compare the perceptual quality of Sora-era video generators automatically.","feed_headline":"CRAVE beats prior AI-video quality metrics on Sora-era videos","feed_subtitle":"It aligns long prompts with clip motion, scoring closer to human raters on two benchmarks.","key_machinery":"The load-bearing components are two named modules. MTT (multi-granularity text-temporal fusion) takes a frozen BLIP visual encoder with a temporal adapter to get video features, splits the prompt into words and phrases with SpaCy, adds a word-to-phrase mapping, and sums cosine distances between video features and text embeddings at paragraph, phrase, and word levels, so the alignment score uses several semantic granularities at once. HMM (hybrid motion-fidelity modeling) concatenates low-level dense motion features from the StreamFlow optical-flow model with high-level action features from Uniformer, capturing both frame-level irregularities and holistic motion plausibility. A third branch reuses DOVER's aesthetic and technical encoders for visual harmony, and the three outputs are regressed to a quality score under a rank-plus-PLCC loss. MTT carries the text-video alignment, HMM carries the temporal artifact scoring, and the DOVER branch carries aesthetics and distortion.","core_discovery":"On its own terms, the paper's central claim is that a three-branch evaluator — visual harmony, multi-granularity text-temporal alignment, and hybrid motion-fidelity modeling — predicts human Mean Opinion Scores for content-rich AIGC videos better than existing VQA methods. The reported evidence is a set of correlation numbers: on CRAVE-DB, CRAVE reaches SRCC 0.7587 and PLCC 0.7581, versus 0.7266 and 0.7098 for the strongest prior method T2VQA; on T2VQA-DB, CRAVE reaches SRCC 0.8122 and PLCC 0.8214, versus 0.7965 and 0.8066 for T2VQA. The authors interpret the gains as showing that long-prompt semantic alignment and motion-aware consistency are the missing ingredients in AIGC video quality assessment, and that the proposed CRAVE-DB closes the gap between existing datasets and the content richness of next-generation generators.","pith_inferences":["Editorial extension: the unspecified train/test split in Section 5.1 means the CRAVE-DB leaderboard numbers should be re-verified under a documented split before being used as a benchmark reference.","Editorial extension: the same multi-granularity text-temporal alignment could in principle serve as a reward model for fine-tuning or reranking video generators, since it produces a differentiable alignment score; the paper does not test this.","Editorial extension: because using 16 optical-flow frames outperformed 4 and 8, motion-quality assessment likely benefits from still longer temporal contexts; testing 32 or 64 frames would show whether the trend continues.","Editorial extension: CRAVE-DB's prompt-collection pipeline (dense captions summarized by GPT-4, then structured into shot/subject/action/scene/detail) could be reused to build benchmarks for other text-conditioned generative media, but the paper only applies it to video."],"forward_implications":["On CRAVE-DB, CRAVE reports SRCC 0.7587, PLCC 0.7581, and KRCC 0.5660, above the best fine-tuned baseline T2VQA (0.7266, 0.7098, 0.5369), so the architecture transfers to next-generation AIGC videos.","On T2VQA-DB, CRAVE reports SRCC 0.8122, PLCC 0.8214, KRCC 0.6338, above T2VQA (0.7965, 0.8066, 0.6058), so the method also holds up on older, shorter-prompt videos.","The ablations show each designed component contributes: temporal-text fusion lifts SRCC on CRAVE-DB from 0.6512 to 0.7587, and combining granularity levels and hybrid motion modeling each add further gains.","CRAVE trained on either dataset can rank next-generation generators (Hunyuan, Sora, Seaweed Pro, Mochi 1 and others) zero-shot on VideoGenEval prompts, giving a tool for model comparison without new human ratings.","CRAVE-DB itself, with 410 elaborate prompts and 29 annotators per video, is proposed as a public benchmark for assessing future AIGC videos."],"supporting_citations":[{"why":"Supplies the DOVER visual-harmony backbone and the aesthetic/technical scores used as the CRAVE visual branch.","marker":"(Wu et al., 2023a)"},{"why":"Provides T2VQA-DB, the principal training and evaluation dataset, and the T2VQA baseline that CRAVE must beat.","marker":"(Kou et al., 2024b)"},{"why":"Provides the frozen BLIP visual and text encoders that carry the multi-granularity text-temporal fusion.","marker":"(Li et al., 2022b)"},{"why":"Supplies StreamFlow, the optical-flow backbone for the low-level dense motion branch of HMM.","marker":"(Sun et al., 2024c)"},{"why":"Supplies Uniformer, the action-recognition backbone for the high-level abstract motion branch of HMM.","marker":"(Li et al., 2023)"},{"why":"Supplies VideoGenEval prompts and generated videos used for the zero-shot model-ranking experiment.","marker":"(Zeng et al., 2024)"},{"why":"Supplies the ShareGPT-4o dense captions that seed the CRAVE-DB prompt collection pipeline.","marker":"(Chen et al., 2023b)"},{"why":"Supplies the word-to-phrase module used to convert word embeddings into phrase-level text embeddings for MTT.","marker":"(Zhu et al., 2023)"},{"why":"Supplies the rank loss used alongside PLCC loss in the training objective.","marker":"(Gao et al., 2019)"}],"fun_headline_variants":["CRAVE aligns text and motion to judge AI videos better","New metric CRAVE outperforms on content-rich AI video quality","CRAVE: Better video quality scores via long-prompt alignment","AI video quality: CRAVE matches human judgment on Sora-era clips","CRAVE benchmark and metric top existing AI video evaluators"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the CRAVE-DB evaluation protocol is sound, yet the paper never states the train/test split ratio or how prompts and videos were assigned, so the reported state-of-the-art numbers on that benchmark rest on an unspecified design choice.","fun_headline_variants_meta":{"raw":{"variants":["CRAVE aligns text and motion to judge AI videos better","New metric CRAVE outperforms on content-rich AI video quality","CRAVE: Better video quality scores via long-prompt alignment","AI video quality: CRAVE matches human judgment on Sora-era clips","CRAVE benchmark and metric top existing AI video evaluators"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000573,"raw_usage":{"total_tokens":2736,"prompt_tokens":1004,"completion_tokens":1732,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":1641}},"tokens_in":620,"tokens_out":1732,"duration_ms":12478,"temperature":1.0,"reasoning_tokens":1641,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T23:36:55.532489+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train CRAVE on CRAVE-DB with several random 80/20, 70/30, and 60/40 train/test splits of the 1,228 videos, keeping the same 40-epoch schedule, and report SRCC/PLCC with standard deviations; if the mean SRCC falls below the T2VQA baseline of 0.7266 under any plausible split, the claimed CRAVE-DB lead is an artifact of the undisclosed split protocol.","supporting_citations":[],"review_version":1}