{"id":"e2bb1d88-0b2e-4539-81f5-a5bad4bd1335","arxiv_id":"2505.19535","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"TDVE-DB and TDVE-Assessor deliver a large MOS-annotated benchmark and a Qwen2.5-VL-based assessor for text-driven video editing quality.","lead":"This paper introduces TDVE-DB, a dataset of 3,857 text-edited videos with 173,565 human ratings on quality, prompt alignment, and structural consistency, plus TDVE-Assessor, an LMM-based model trained to predict those ratings. The work gives video editing researchers a standardized benchmark and an automated scoring tool.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA claim on TDVE-DB is unverified: the two most relevant editing-VQA baselines are absent from Table 3.","rationale":"The reader's CONDITIONAL verdict is appropriate. The strongest claim is SOTA performance on TDVE-DB, and the most load-bearing weakness is that the comparison table omits VE-Bench QA and AIGV-Assessor, the two prior systems built specifically for video editing quality assessment. Their close scores on other benchmarks (Table 4) mean their inclusion could plausibly overturn the SOTA claim. This is a concrete, testable gap rather than a speculative concern. The alignment-input mismatch raised by the reader is real but secondary: the paper explicitly argues in Section 3.2 that training on (edit prompt, edited video, delta-change MOS) tuples lets the model learn the change-centric judgment implicitly, and although cases where the edit is already present in the source could fool the model, the missing-baseline problem directly undermines the paper's headline claim regardless. I also note that the reader's statement that Section 5.3 is contradicted by Table 4 on T2VQA-DB is not accurate: among the models listed there, TDVE-Assessor has the highest SRCC. The concern is the omission, not a contradiction in the reported numbers. A single rerun including the missing baselines would settle whether the SOTA claim holds.","tokens_in":37180,"tokens_out":3532,"duration_ms":34416,"concrete_test":"Run VE-Bench QA and AIGV-Assessor on TDVE-DB under the identical protocol used for Table 3: same 4:1 train-validation split, same per-dimension training pipeline (fine-tune on training folds if that is how they were evaluated in their source papers, otherwise zero-shot), and report SRCC, PLCC, and KRCC for video quality, editing alignment, and structural consistency. If either method exceeds TDVE-Assessor's reported SRCC of 0.8688, 0.8254, or 0.8354 on any dimension, the SOTA claim fails; if both are lower across all dimensions, the claim is provisionally supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that TDVE-Assessor sets a new state-of-the-art on TDVE-DB rests on the comparison in Table 3, which omits the two models most directly designed for text-driven video editing quality assessment: VE-Bench QA [19] and the authors' own AIGV-Assessor [21]. VE-Bench QA is referenced in Table 4 and achieves SRCC 0.8179 on T2VQA-DB, only 0.0043 below TDVE-Assessor's 0.8222, and 0.7415 on VE-Bench DB, only 0.0112 below TDVE-Assessor's 0.7527. AIGV-Assessor achieves 0.9162 SRCC on AIGVQA-DB, where TDVE-Assessor reports 0.9397. These margins are small enough that their performance on TDVE-DB could match or exceed TDVE-Assessor's reported 0.8688, 0.8254, and 0.8354 on the three dimensions. Without these baselines in the main comparison, the SOTA statement is not established. The omission is especially consequential because both are LMM-based or fine-tuned VQA models for exactly this task, not generic image-text scorers. The reader's stated contradiction of Section 5.3 with Table 4 on T2VQA-DB appears factually incorrect (TDVE-Assessor is highest there among listed baselines), but the missing-baseline concern stands independently and is the load-bearing weakness.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TDVE-DB, a large-scale benchmark for text-driven video editing quality assessment, containing 3,857 edited videos from 12 editing models, 340 prompts across 8 categories, and 173,565 human ratings along three dimensions: edited video quality, editing alignment, and structural consistency. The authors report high inter-rater reliability (ICC2k > 0.92) and a thorough subjective evaluation protocol. They also propose TDVE-Assessor, an LMM-based model that ingests spatial and temporal video features and uses a two-stage training procedure (text-level classification pretraining followed by LoRA fine-tuning with a regression head) to predict MOS for the three dimensions. Experiments on TDVE-DB and on external benchmarks VE-Bench DB, T2VQA-DB, and AIGVQA-DB are reported, with the paper claiming state-of-the-art performance on TDVE-DB and strong generalization.","tokens_in":37410,"tokens_out":4070,"duration_ms":33593,"significance":"If the central claims hold, this paper would make a substantial contribution: TDVE-DB is the largest multi-dimensional benchmark for text-driven video editing quality, with a documented subjective rating protocol and excellent aggregate reliability; the proposed TDVE-Assessor demonstrates that LMM-based, prompt-aware assessment can outperform generic VQA and image-text scoring models on all three dimensions. The cross-dataset results on VE-Bench DB, T2VQA-DB, and AIGVQA-DB provide useful independent evidence, and the fact that the dataset and model are planned for release adds practical value. The main qualification is that the SOTA claim on TDVE-DB is currently unverified because the most directly relevant editing-VQA baselines are missing from the main comparison, and the evaluation protocol for baseline adaptation is described ambiguously. The reader's concern about circularity does not land: training and evaluating on the same benchmark is standard supervised evaluation, and the external benchmarks provide grounding.","major_comments":[{"comment":"The SOTA claim on TDVE-DB rests on Table 3, but this table omits VE-Bench QA [19] and AIGV-Assessor [21], the two published models most directly designed for text-driven video editing or AIGV quality assessment. Since these baselines are already cited and used in Table 4, their omission from the main TDVE-DB comparison is unexplained and undermines the claimed state of the art. Please add them to Table 3 (or justify their exclusion with concrete technical reasons) and report their performance on all three TDVE-DB dimensions.","section":"§5.2 / Table 3"},{"comment":"There is a mismatch between the human annotation protocol and the model input for editing alignment. The subjective experiment (Section 3.2 and Figure 7) asked raters to judge alignment relative to the source video and its description, emphasizing 'delta changes,' whereas Appendix D.3 states that TDVE-Assessor receives only the edit prompt and the edited video for this dimension. The paper argues this is learned implicitly from training data, but no evidence is provided that prompt-to-video correspondence alone recovers source-relative judgments. Please add an ablation that includes the source video or source description as input, or present targeted analysis showing that the model's alignment scores correlate with human delta-change ratings without source access.","section":"§3.2 / Appendix D.3"},{"comment":"The description that fine-tunable VQA baselines are 'domain-adapted on the respective test set distributions' is ambiguous and potentially problematic. If baselines are fine-tuned on the test set (or on a split that includes test data), the comparison is not a fair evaluation of generalization, and it could artificially lower their performance relative to TDVE-Assessor. Please clarify the exact protocol: do all methods use the same 4:1 train/validation split, and is any model trained on test labels? In addition, several reported differences are small (e.g., SRCC 0.8222 vs. 0.8179 on T2VQA-DB); the paper should report confidence intervals or significance tests for the main SRCC/PLCC comparisons.","section":"§5.1"},{"comment":"The claim of 'state-of-the-art performance in both instances' is overstated. On T2VQA-DB the margin over VE-Bench QA is only 0.0043 SRCC, and on AIGVQA-DB the margin over AIGV-Assessor is 0.0235 SRCC; no statistical test is provided to establish that these differences are meaningful. Note also that the reader's statement that AIGV-Assessor outperforms TDVE-Assessor on T2VQA-DB is not supported by Table 4: in the T2VQA-DB column, TDVE-Assessor has the highest listed SRCC. The load-bearing issue remains the missing baselines in Table 3, not the ranking within Table 4.","section":"§5.3 / Table 4"}],"minor_comments":[{"comment":"The abstract and Figure 1 caption refer to '170K' subjective scores, while the text and Appendix B.6 report 173,565 ratings; please make these numbers consistent.","section":"Abstract / Figure 1"},{"comment":"The text level list is given as '{bad|pool|fair|good|excellent}' in two places; the second level should be 'poor', not 'pool'.","section":"Eq. (2) and Section 4.2"},{"comment":"The structural-consistency prompt in Figure 5 says 'The first half and the second half of the video are two different videos,' but Appendix D.3 describes concatenating source and edited video segments. Please align the description with the actual input format.","section":"Figure 5 / Appendix D.3"},{"comment":"Reference [2] is cited as 'Tokenflow: Unified image tokenizer for multimodal understanding and generation,' which appears to be a different paper from the TokenFlow video editing method described in the text; the citation should be corrected to the actual TokenFlow work.","section":"References"},{"comment":"Reference [43] is described as Qwen2.5-VL-7B-Instruct, but the cited paper is the original Qwen-VL paper from 2023; please cite the specific Qwen2.5-VL technical report or model card.","section":"References"},{"comment":"The column labeled 'Edited MOS' lists '≈28,000' for VE-Bench DB; this appears to be the number of ratings rather than a MOS value, so the column header should be clarified (e.g., 'Number of subjective ratings').","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope and the dataset contribution is potentially strong. The main risk is that the SOTA claim is built on an incomplete baseline comparison and an ambiguous adaptation protocol; these are fixable with additional experiments and clarification. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this one. First, TDVE-DB is a genuine new resource: 3,857 edited videos from 12 models across 8 editing categories, 173,565 human ratings on three dimensions, ICC2k above 0.92 on all three. That is the largest MOS-annotated text-driven video editing benchmark to date, and the subjective protocol looks careful — ITU-T P.910, per-rater calibration, hidden repeats. Second, the SOTA claim for TDVE-Assessor is not established by the evidence, because Table 3, the head-to-head on TDVE-DB, omits the two baselines that matter most: VE-Bench QA and the authors' own AIGV-Assessor. The latter omission is especially conspicuous since TDVE-Assessor is, by the paper's own description, a close cousin of AIGV-Assessor. Where those models do appear (Table 4), the margins are thin: 0.7527 vs 0.7415 on VE-Bench DB, and 0.9397 vs 0.9162 on AIGVQA-DB. Until VE-Bench QA and AIGV-Assessor are run on TDVE-DB, \"state-of-the-art\" is a claim about an incomplete baseline table.\n\nOne correction to the reader's report: the alleged contradiction between Section 5.3 and Table 4 on T2VQA-DB does not hold up — TDVE-Assessor is highest there too (0.8222 vs 0.8179 for VE-Bench QA). The missing-baseline concern is the real one.\n\nThe model itself is a competent adaptation of the AIGV-Assessor recipe — frozen Qwen2.5-VL encoder, LoRA, MLP regression head — and the cross-dataset numbers do suggest it generalizes. The alignment dimension has a construct-validity wrinkle: raters scored edits against the source video, while the model only sees the edit prompt and edited video. The authors acknowledge this in Section 3.2 and argue the model picks up the \"delta change\" implicitly. Empirically it works, but the assumption is untested; a source-video ablation would settle it. Minor issues: no significance tests or confidence intervals anywhere, the baseline adaptation description (\"domain-adapted on the respective test set distributions\") is ambiguous enough to invite leakage questions, and no code or data is out yet.\n\nCircularity is not the problem here — training and testing on your own benchmark is standard, and the two external benchmarks give independent grounding. The limitations section is honest about the model being a baseline.\n\nRecommendation: send it to review. The dataset deserves to be published; the paper needs the missing baselines in Table 3, significance tests, and ideally the source-input ablation before the assessor claims are taken at face value. I'd bring this to a reading group for the benchmark-design discussion alone, and I'd cite the dataset once it is out.","headline":"TDVE-DB is a solid new benchmark; the assessor's SOTA claim rests on a Table 3 that omits the two most relevant baselines.","tokens_in":38050,"tokens_out":6983,"would_cite":true,"duration_ms":53693,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a large multimodal model fine-tuned on a human-rated benchmark of 3,857 edited videos predicts edited-video quality, editing alignment, and structural consistency better than existing video quality models.","keywords":["text-driven video editing","video quality assessment","large multimodal model","benchmark dataset","mean opinion score","editing alignment","structural consistency","video editing evaluation"],"falsifier":"A direct test is to gather edited videos where the requested change is already present in the source, so humans rate alignment low because nothing changed, while the edited video itself matches the prompt. Since TDVE-Assessor sees only the prompt and the edited video for this dimension, if it does not assign low alignment scores to such no-op edits, the central claim about recovering human 'delta change' judgments fails.","tokens_in":36917,"feed_emoji":"🎬","tokens_out":5215,"duration_ms":43585,"temperature":0.7,"pith_summary":"The paper sets out to fill a gap: text-driven video editing has no dedicated quality assessment model. It builds TDVE-DB, a dataset of 3,857 edited videos from 12 models across 8 editing categories, annotated with 173,565 human ratings on edited video quality, editing alignment, and structural consistency. On this benchmark it trains TDVE-Assessor, a large multimodal model fine-tuned to output both textual comments and a numeric score for each dimension. The paper's central claim is that this model predicts human mean opinion scores better than existing VQA and text-video correspondence methods, with SRCC 0.8688 for video quality, 0.8254 for editing alignment, and 0.8354 for structural consistency on TDVE-DB.","feed_headline":"TDVE-Assessor beats existing metrics on video-edit quality","feed_subtitle":"A fine-tuned vision-language model predicts human ratings of edited-video quality, prompt alignment, and structural consistency.","key_machinery":"The load-bearing construction is a two-stage fine-tuned LMM pipeline. Stage 1 trains the language model to map videos to discrete ITU quality labels; Stage 2 trains a lightweight MLP regression head on the LLM's pooled hidden states, while LoRA low-rank adaptation keeps the LLM update parameter-efficient, to predict continuous MOS scores. The frozen Qwen2.5-VL spatiotemporal encoder supplies variable-resolution, variable-frame-rate video features directly, avoiding lossy resizing or compression preprocessing.","core_discovery":"The central discovery claimed is that an LMM (large multimodal model) fine-tuned on a purpose-built benchmark can serve as a multi-dimensional video-editing quality assessor. TDVE-Assessor takes the edited video (and, for editing alignment, the edit prompt; for structural consistency, side-by-side or concatenated source and edited frames), extracts spatiotemporal features with a frozen vision transformer, fuses them into Qwen2.5-VL-7B via a trainable MLP projector, and adds a regression head on the LLM's hidden states to produce continuous scores. The paper reports that this design outperforms all compared baselines on TDVE-DB across all three dimensions and generalizes to VE-Bench, T2VQA-DB, and AIGVQA-DB.","pith_inferences":["Editorial inference: If editing alignment can be recovered from the prompt plus edited video alone, the same assessor could score text-to-video generation and text-driven image editing without needing a source reference.","Editorial inference: The strong positive correlation the paper reports between video quality and structural consistency hints that one latent 'fidelity to source' factor may drive both scores, which a factor analysis of the MOS could confirm.","Editorial inference: The benchmark could be extended with adversarial no-op edits, where the prompt requests a change already present in the source, to test whether alignment scores truly capture change rather than static prompt matching."],"forward_implications":["If the reported results hold, researchers can replace expensive human MOS collection with one model that ranks edited videos on quality, alignment, and structural consistency.","The benchmark provides a fixed, multi-category testbed for comparing future text-driven video editing models across 12 diffusion-based systems and 8 editing categories.","The cross-dataset gains on T2V and AIGV quality datasets suggest the same architecture can serve as a general AI-generated video quality metric.","The qualitative description output gives users an interpretable reason accompanying every numeric score, which could help debugging of editing models."],"supporting_citations":[{"why":"Supplies the VE-Bench DB dataset used for cross-dataset evaluation of text-driven video editing quality.","marker":"[19]"},{"why":"Provides T2VQA-DB, the text-to-video quality dataset on which TDVE-Assessor is tested for generalization.","marker":"[20]"},{"why":"Provides AIGVQA-DB, the AI-generated video quality dataset whose AIGV-Assessor result the cross-dataset comparison must beat.","marker":"[21]"},{"why":"Qwen2.5-VL-7B-Instruct is the frozen vision encoder and language backbone that carries the model's multimodal reasoning.","marker":"[43]"},{"why":"LoRA low-rank adaptation is the parameter-efficient fine-tuning method used to train the LLM for score regression.","marker":"[45]"},{"why":"DOVER is the strongest deep VQA baseline for video quality and structural consistency on TDVE-DB, the main comparison to beat.","marker":"[14]"},{"why":"FAST-VQA supplies a fragment-sampling VQA baseline used in the TDVE-DB and cross-dataset benchmarks.","marker":"[15]"}],"fun_headline_variants":["TDVE-Assessor: LLM scores video edit quality across three axes","New benchmark and LLM assessor for text-driven video edits","TDVE-Assessor outperforms existing VQA on video edit quality","Multimodal LLM sets SOTA on video editing quality assessment","Benchmarking video editing: TDVE-Assessor beats all baselines"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The SOTA claim for editing alignment rests on the assumption that prompt-plus-edited-video alone can recover human judgments that were defined by comparing the edited video with the source video and its description.","fun_headline_variants_meta":{"raw":{"variants":["TDVE-Assessor: LLM scores video edit quality across three axes","New benchmark and LLM assessor for text-driven video edits","TDVE-Assessor outperforms existing VQA on video edit quality","Multimodal LLM sets SOTA on video editing quality assessment","Benchmarking video editing: TDVE-Assessor beats all baselines"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000672,"raw_usage":{"total_tokens":3065,"prompt_tokens":953,"completion_tokens":2112,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":2034}},"tokens_in":569,"tokens_out":2112,"duration_ms":11075,"temperature":1.0,"reasoning_tokens":2034,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:12:02.665897+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test is to gather edited videos where the requested change is already present in the source, so humans rate alignment low because nothing changed, while the edited video itself matches the prompt. Since TDVE-Assessor sees only the prompt and the edited video for this dimension, if it does not assign low alignment scores to such no-op edits, the central claim about recovering human 'delta change' judgments fails.","supporting_citations":[{"cited_title":"Ve-bench: Subjective-aligned benchmark suite for text-driven video editing quality assessment,","cited_arxiv_id":null,"evidence_quote":"Supplies the VE-Bench DB dataset used for cross-dataset evaluation of text-driven video editing quality."},{"cited_title":"Subjective-aligned dataset and metric for text-to-video quality assessment,","cited_arxiv_id":null,"evidence_quote":"Provides T2VQA-DB, the text-to-video quality dataset on which TDVE-Assessor is tested for generalization."},{"cited_title":"Aigv-assessor: Benchmarking and evaluating the perceptual quality of text-to-video generation with lmm,","cited_arxiv_id":null,"evidence_quote":"Provides AIGVQA-DB, the AI-generated video quality dataset whose AIGV-Assessor result the cross-dataset comparison must beat."},{"cited_title":"Lora: Low-rank adaptation of large language models,","cited_arxiv_id":null,"evidence_quote":"LoRA low-rank adaptation is the parameter-efficient fine-tuning method used to train the LLM for score regression."},{"cited_title":"Exploring video quality assessment on user generated contents from aesthetic and technical perspectives,","cited_arxiv_id":null,"evidence_quote":"DOVER is the strongest deep VQA baseline for video quality and structural consistency on TDVE-DB, the main comparison to beat."},{"cited_title":"Fast-vqa: Efficient end-to-end video quality assessment with fragment sampling,","cited_arxiv_id":null,"evidence_quote":"FAST-VQA supplies a fragment-sampling VQA baseline used in the TDVE-DB and cross-dataset benchmarks."}],"review_version":1}