Pith. sign in

REVIEW 4 major objections 6 minor 71 references

TDVE-Assessor: Benchmarking and Evaluating the Quality of Text-Driven Video Editing with LMMs

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a large multimodal model fine-tuned on a human-rated benchmark of 3,857 edited videos predicts edited-video quality, editing alignment, and structural consistency better than existing video quality models.

desk verdict TDVE-DB is a solid new benchmark; the assessor's SOTA claim rests on a Table 3 that omits the two most relevant baselines. read the letter →

arxiv 2505.19535 v1 pith:ZBR5PFPR submitted 2025-05-26 cs.CV

Juntong Wang , Jiarui Wang , Huiyu Duan , Guangtao Zhai , Xiongkuo Min This is my paper
classification cs.CV
keywords text-drivenvideoeditingqualityassessmentlargemultimodalmodelbenchmarkdatasetmeanopinionscorealignmentstructuralconsistencyevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to fill a gap: text-driven video editing has no dedicated quality assessment model. It builds TDVE-DB, a dataset of 3,857 edited videos from 12 models across 8 editing categories, annotated with 173,565 human ratings on edited video quality, editing alignment, and structural consistency. On this benchmark it trains TDVE-Assessor, a large multimodal model fine-tuned to output both textual comments and a numeric score for each dimension. The paper's central claim is that this model predicts human mean opinion scores better than existing VQA and text-video correspondence methods, with SRCC 0.8688 for video quality, 0.8254 for editing alignment, and 0.8354 for structural consistency on TDVE-DB.

What carries the argument

The load-bearing construction is a two-stage fine-tuned LMM pipeline. Stage 1 trains the language model to map videos to discrete ITU quality labels; Stage 2 trains a lightweight MLP regression head on the LLM's pooled hidden states, while LoRA low-rank adaptation keeps the LLM update parameter-efficient, to predict continuous MOS scores. The frozen Qwen2.5-VL spatiotemporal encoder supplies variable-resolution, variable-frame-rate video features directly, avoiding lossy resizing or compression preprocessing.

What would settle it

A direct test is to gather edited videos where the requested change is already present in the source, so humans rate alignment low because nothing changed, while the edited video itself matches the prompt. Since TDVE-Assessor sees only the prompt and the edited video for this dimension, if it does not assign low alignment scores to such no-op edits, the central claim about recovering human 'delta change' judgments fails.

Watch

Extended reading notes

Core claim

The central discovery claimed is that an LMM (large multimodal model) fine-tuned on a purpose-built benchmark can serve as a multi-dimensional video-editing quality assessor. TDVE-Assessor takes the edited video (and, for editing alignment, the edit prompt; for structural consistency, side-by-side or concatenated source and edited frames), extracts spatiotemporal features with a frozen vision transformer, fuses them into Qwen2.5-VL-7B via a trainable MLP projector, and adds a regression head on the LLM's hidden states to produce continuous scores. The paper reports that this design outperforms all compared baselines on TDVE-DB across all three dimensions and generalizes to VE-Bench, T2VQA-DB, and AIGVQA-DB.

Load-bearing premise

The SOTA claim for editing alignment rests on the assumption that prompt-plus-edited-video alone can recover human judgments that were defined by comparing the edited video with the source video and its description.

Editorial extensions

If this is right

  • If the reported results hold, researchers can replace expensive human MOS collection with one model that ranks edited videos on quality, alignment, and structural consistency.
  • The benchmark provides a fixed, multi-category testbed for comparing future text-driven video editing models across 12 diffusion-based systems and 8 editing categories.
  • The cross-dataset gains on T2V and AIGV quality datasets suggest the same architecture can serve as a general AI-generated video quality metric.
  • The qualitative description output gives users an interpretable reason accompanying every numeric score, which could help debugging of editing models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: If editing alignment can be recovered from the prompt plus edited video alone, the same assessor could score text-to-video generation and text-driven image editing without needing a source reference.
  • Editorial inference: The strong positive correlation the paper reports between video quality and structural consistency hints that one latent 'fidelity to source' factor may drive both scores, which a factor analysis of the MOS could confirm.
  • Editorial inference: The benchmark could be extended with adversarial no-op edits, where the prompt requests a change already present in the source, to test whether alignment scores truly capture change rather than static prompt matching.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces TDVE-DB, a large-scale benchmark for text-driven video editing quality assessment, containing 3,857 edited videos from 12 editing models, 340 prompts across 8 categories, and 173,565 human ratings along three dimensions: edited video quality, editing alignment, and structural consistency. The authors report high inter-rater reliability (ICC2k > 0.92) and a thorough subjective evaluation protocol. They also propose TDVE-Assessor, an LMM-based model that ingests spatial and temporal video features and uses a two-stage training procedure (text-level classification pretraining followed by LoRA fine-tuning with a regression head) to predict MOS for the three dimensions. Experiments on TDVE-DB and on external benchmarks VE-Bench DB, T2VQA-DB, and AIGVQA-DB are reported, with the paper claiming state-of-the-art performance on TDVE-DB and strong generalization.

Significance. If the central claims hold, this paper would make a substantial contribution: TDVE-DB is the largest multi-dimensional benchmark for text-driven video editing quality, with a documented subjective rating protocol and excellent aggregate reliability; the proposed TDVE-Assessor demonstrates that LMM-based, prompt-aware assessment can outperform generic VQA and image-text scoring models on all three dimensions. The cross-dataset results on VE-Bench DB, T2VQA-DB, and AIGVQA-DB provide useful independent evidence, and the fact that the dataset and model are planned for release adds practical value. The main qualification is that the SOTA claim on TDVE-DB is currently unverified because the most directly relevant editing-VQA baselines are missing from the main comparison, and the evaluation protocol for baseline adaptation is described ambiguously. The reader's concern about circularity does not land: training and evaluating on the same benchmark is standard supervised evaluation, and the external benchmarks provide grounding.

major comments (4)
  1. [§5.2 / Table 3] The SOTA claim on TDVE-DB rests on Table 3, but this table omits VE-Bench QA [19] and AIGV-Assessor [21], the two published models most directly designed for text-driven video editing or AIGV quality assessment. Since these baselines are already cited and used in Table 4, their omission from the main TDVE-DB comparison is unexplained and undermines the claimed state of the art. Please add them to Table 3 (or justify their exclusion with concrete technical reasons) and report their performance on all three TDVE-DB dimensions.
  2. [§3.2 / Appendix D.3] There is a mismatch between the human annotation protocol and the model input for editing alignment. The subjective experiment (Section 3.2 and Figure 7) asked raters to judge alignment relative to the source video and its description, emphasizing 'delta changes,' whereas Appendix D.3 states that TDVE-Assessor receives only the edit prompt and the edited video for this dimension. The paper argues this is learned implicitly from training data, but no evidence is provided that prompt-to-video correspondence alone recovers source-relative judgments. Please add an ablation that includes the source video or source description as input, or present targeted analysis showing that the model's alignment scores correlate with human delta-change ratings without source access.
  3. [§5.1] The description that fine-tunable VQA baselines are 'domain-adapted on the respective test set distributions' is ambiguous and potentially problematic. If baselines are fine-tuned on the test set (or on a split that includes test data), the comparison is not a fair evaluation of generalization, and it could artificially lower their performance relative to TDVE-Assessor. Please clarify the exact protocol: do all methods use the same 4:1 train/validation split, and is any model trained on test labels? In addition, several reported differences are small (e.g., SRCC 0.8222 vs. 0.8179 on T2VQA-DB); the paper should report confidence intervals or significance tests for the main SRCC/PLCC comparisons.
  4. [§5.3 / Table 4] The claim of 'state-of-the-art performance in both instances' is overstated. On T2VQA-DB the margin over VE-Bench QA is only 0.0043 SRCC, and on AIGVQA-DB the margin over AIGV-Assessor is 0.0235 SRCC; no statistical test is provided to establish that these differences are meaningful. Note also that the reader's statement that AIGV-Assessor outperforms TDVE-Assessor on T2VQA-DB is not supported by Table 4: in the T2VQA-DB column, TDVE-Assessor has the highest listed SRCC. The load-bearing issue remains the missing baselines in Table 3, not the ranking within Table 4.
minor comments (6)
  1. [Abstract / Figure 1] The abstract and Figure 1 caption refer to '170K' subjective scores, while the text and Appendix B.6 report 173,565 ratings; please make these numbers consistent.
  2. [Eq. (2) and Section 4.2] The text level list is given as '{bad|pool|fair|good|excellent}' in two places; the second level should be 'poor', not 'pool'.
  3. [Figure 5 / Appendix D.3] The structural-consistency prompt in Figure 5 says 'The first half and the second half of the video are two different videos,' but Appendix D.3 describes concatenating source and edited video segments. Please align the description with the actual input format.
  4. [References] Reference [2] is cited as 'Tokenflow: Unified image tokenizer for multimodal understanding and generation,' which appears to be a different paper from the TokenFlow video editing method described in the text; the citation should be corrected to the actual TokenFlow work.
  5. [References] Reference [43] is described as Qwen2.5-VL-7B-Instruct, but the cited paper is the original Qwen-VL paper from 2023; please cite the specific Qwen2.5-VL technical report or model card.
  6. [Table 2] The column labeled 'Edited MOS' lists '≈28,000' for VE-Bench DB; this appears to be the number of ratings rather than a MOS value, so the column header should be clarified (e.g., 'Number of subjective ratings').

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TDVE-Assessor is trained and evaluated on held-out splits of its own TDVE-DB, and cross-dataset results on VE-Bench DB, T2VQA-DB, and AIGVQA-DB provide independent grounding; missing baselines in Table 3 are a comparison-completeness issue, not a circularity.

full rationale

The derivation chain is: construct TDVE-DB with human MOS; benchmark existing VQA models; propose TDVE-Assessor trained with a two-stage loss on the TDVE-DB training split; evaluate on a held-out test split and on three external benchmarks. Each step is separate and none reduces to its inputs by construction. The TDVE-DB SOTA claim is a standard supervised evaluation: all deep baselines are domain-adapted with the same 4:1 train-validation split and averaged over ten randomized trials, so the comparison is not a fit-to-prediction tautology. Cross-dataset numbers on VE-Bench DB, T2VQA-DB, and AIGVQA-DB use benchmarks the authors did not create for those datasets, and including their own prior AIGV-Assessor as a baseline is a self-citation but not a load-bearing one: the cited numbers are external results, not the justification for the architecture or the claim by definition. The editing-alignment protocol difference (human raters saw the source video and its description, while TDVE-Assessor receives only the edit prompt and the edited video) is an input-completeness assumption, not a circular reduction: the human MOS are not defined as the model's output, and the model must learn the mapping from a strictly weaker input. No equation in the paper defines the predicted score as a fitted parameter or as the benchmark MOS by construction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the reliability of the subjective MOS, the separability of the three rating dimensions, the representativeness of the 12 Stable Diffusion based editing models, and the assumption that prompt-to-edited-video correspondence alone can predict human alignment scores collected with source video access. No new physical entities are introduced; the main hand-chosen settings are training hyperparameters and MOS processing constants.

free parameters (5)
  • LoRA rank = 8
    Chosen for parameter-efficient fine-tuning; not fitted to data, but hand-selected.
  • LoRA alpha = 32
    Hand-selected scaling for LoRA updates.
  • Learning rate = 1e-4
    Standard fine-tuning choice; not optimized per dimension.
  • MOS discretization levels = 5
    Eq. 2 divides MOS into five ITU-defined textual levels; the number of levels is chosen by hand.
  • Z-score scaling constants = 3 and 100/6 in Eq. 1a
    Assumes z-scores in [-3,3] map to [0,100]; ad hoc but standard.
assumptions (5)
  • domain assumption Human MOS aggregated from 15 trained raters is a reliable ground truth for text-driven video editing quality.
    Section 3.3 and Appendix B.6 report ICC2k > 0.92 for all dimensions, supporting but not proving population-level validity.
  • domain assumption The three evaluation dimensions (visual quality, editing alignment, structural consistency) are separable and can be rated independently.
    Section 3.2 defines the dimensions; pairwise correlations in Figure 4 show quality and structural consistency are positively correlated, so independence is partial.
  • domain assumption The 12 open-source editing models, all based on Stable Diffusion, represent the distribution of text-driven video editing outputs.
    Table 1 lists the models; no non-SD or commercial editing models are included, so benchmark coverage is limited.
  • domain assumption For editing alignment, prompt-to-edited-video correspondence without source video input suffices to predict human 'delta change' judgments.
    Appendix D.3 specifies the alignment input as edit prompt plus edited video; humans had source video and description (Section 3.2), so this is an unverified transfer.
  • standard math MOS normalization via per-rater Z-scores (Eq. 1) preserves the ordinal structure of raw ratings.
    Z-score standardization is standard practice; the mapping to [0,100] assumes ratings within three standard deviations of each rater mean.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TDVE-Assessor: Benchmarking and Evaluating the Quality of Text-Driven Video Editing with LMMs." pith.science (2026). https://pith.science/paper/ZBR5PFPR

@misc{pith2026250519535,
  author       = {Pith},
  title        = {Pith review of: TDVE-Assessor: Benchmarking and Evaluating the Quality of Text-Driven Video Editing with LMMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZBR5PFPR}},
  note         = {Machine review of arXiv:2505.19535}
}
read the original abstract

Text-driven video editing is rapidly advancing, yet its rigorous evaluation remains challenging due to the absence of dedicated video quality assessment (VQA) models capable of discerning the nuances of editing quality. To address this critical gap, we introduce TDVE-DB, a large-scale benchmark dataset for text-driven video editing. TDVE-DB consists of 3,857 edited videos generated from 12 diverse models across 8 editing categories, and is annotated with 173,565 human subjective ratings along three crucial dimensions, i.e., edited video quality, editing alignment, and structural consistency. Based on TDVE-DB, we first conduct a comprehensive evaluation for the 12 state-of-the-art editing models revealing the strengths and weaknesses of current video techniques, and then benchmark existing VQA methods in the context of text-driven video editing evaluation. Building on these insights, we propose TDVE-Assessor, a novel VQA model specifically designed for text-driven video editing assessment. TDVE-Assessor integrates both spatial and temporal video features into a large language model (LLM) for rich contextual understanding to provide comprehensive quality assessment. Extensive experiments demonstrate that TDVE-Assessor substantially outperforms existing VQA models on TDVE-DB across all three evaluation dimensions, setting a new state-of-the-art. Both TDVE-DB and TDVE-Assessor will be released upon the publication.

Figures

Figures reproduced from arXiv: 2505.19535 by the authors.

Figure 1
Figure 1. (a) Acquisition of the source video (b) Generation of prompt words (c) Obtaining 170K [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Bar charts of the score distribution of each model and editing category across three [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Radar charts of the score distribution of each model and editing category across three [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (11 more)
Figure 4
Figure 4. Figure 4: Scatter plots of score distribution between pairwise dimensions. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: (a) Video features are extracted by a frozen spatiotemporal encoder and aligned to a Large [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: The example of the comparison between the actual score and the score predicted by our [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Subjective Experiment Rating UI.Participants evaluated the overall editing effect by [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Video Quality • Objectivity and Constraints: As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: Editing Alignment action oil painting color multi_color background multi_object Ink painting object Category Slicedit FateZero FRESCO Tokenflow RAVE FLATTEN Pix2Video CCEdit ControlVideo Tune-a-video Vid2Vid-Zero Text2Video-Zero Model 93.4 91.5 90.1 93.1 88.8 85.5 90.0…
Figure 10
Figure 10. Figure 10: Structural Consistency 18 [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: All Average Edit Prompt:A litter of chocolate brown retriever puppies playing in the snow. Their heads pop out of the snow, covered in. Video Quality:65.33(fair) Text Consistency:86.67(excellent) Perceptual Consistency:81.33(good) 0 Edit Prompt:A litter of chocolate b…
Figure 12
Figure 12. Figure 12: Sample 1 B.5 Qualitative Samples Displayed by Dimension and Quality Level For the convenience of the readers, we present the quality scores and quality levels of some video editing samples for reference, as shown in Figures 12, 13, and 14. B.6 Subjective Rating Reliab…
Figure 13
Figure 13. Figure 13: Sample 2 Edit Prompt:A white man in a blue shirt rides a red bicycle in the yard. Video Quality:68(good) Text Consistency:40(pool) Perceptual Consistency:85.33(excellent) 0 Edit Prompt:A lemur plays with a computer at a computer desk. Source Video Edited Video Source …
Figure 14
Figure 14. Figure 14: Sample 3 evaluation dimensions. The comprehensive results of this IRR analysis are presented and discussed below, with detailed statistics available in [PITH_FULL_IMAGE:figures/full_fig_p020_14.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

71 extracted references · 55 canonical work pages

  1. [19]

    Ve-bench: Subjective-aligned benchmark suite for text-driven video editing quality assessment,

    S. Sun, X. Liang, S. Fan, W. Gao, and W. Gao, “Ve-bench: Subjective-aligned benchmark suite for text-driven video editing quality assessment,” 2024

    work page 2024

  2. [21]

    Aigv-assessor: Benchmarking and evaluating the perceptual quality of text-to-video generation with lmm,

    J. Wang, H. Duan, G. Zhai, J. Wang, and X. Min, “Aigv-assessor: Benchmarking and evaluating the perceptual quality of text-to-video generation with lmm,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    work page 2024

  3. [1]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation,

    J. Z. Wu, Y . Ge, X. Wang, W. Lei, Y . Gu, Y . Shi, W. Hsu, Y . Shan, X. Qie, and M. Z. Shou, “Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2023. 9

    work page 2023

  4. [2]

    Tokenflow: Unified image tokenizer for multimodal understanding and generation,

    T. U. I. T. for Multimodal Understanding and Generation, “Tokenflow: Unified image tokenizer for multimodal understanding and generation,” in Proceedings of the International Conference on Learning Representations (ICLR), 2023

    work page 2023

  5. [3]

    Text2video-zero: Text-to-image diffusion models are zero-shot video generators,

    L. Khachatryan, A. Movsisyan, V . Tadevosyan, R. Henschel, Z. Wang, S. Navasardyan, and H. Shi, “Text2video-zero: Text-to-image diffusion models are zero-shot video generators,” inProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2023

    work page 2023

  6. [4]

    Ccedit: Creative and controllable video editing via diffusion models,

    R. Feng, W. Weng, Y . Wang, Y . Yuan, J. Bao, C. Luo, Z. Chen, and B. Guo, “Ccedit: Creative and controllable video editing via diffusion models,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6712–6722, 2024

    work page 2024

  7. [5]

    Controlvideo: Training-free controllable text-to-video generation,

    Y . Zhang, Y . Wei, D. Jiang, X. Zhang, W. Zuo, and Q. Tian, “Controlvideo: Training-free controllable text-to-video generation,” in Proceedings of the International Conference on Learning Representations (ICLR), 2024

    work page 2024

  8. [6]

    Fatezero: Fusing attentions for zero-shot text-based video editing,

    C. Qi, X. Cun, Y . Zhang, C. Lei, X. Wang, Y . Shan, and Q. Chen, “Fatezero: Fusing attentions for zero-shot text-based video editing,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023

    work page 2023

Show all 71 references
  1. [7]

    Flatten: optical flow-guided attention for consistent text-to-video editing,

    Y . Cong, M. Xu, C. Simon, S. Chen, J. Ren, Y . Xie, J.-M. Perez-Rua, B. Rosenhahn, T. Xiang, and S. He, “Flatten: optical flow-guided attention for consistent text-to-video editing,” in Proceedings of the International Conference on Learning Representations (ICLR) , 2024

    2024

  2. [8]

    Fresco: Spatial-temporal correspondence for zero-shot video translation,

    S. Yang, Y . Zhou, Z. Liu, and C. C. Loy, “Fresco: Spatial-temporal correspondence for zero-shot video translation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  3. [9]

    Pix2video: Video editing using image diffusion,

    D. Ceylan, C.-H. P. Huang, and N. J. Mitra, “Pix2video: Video editing using image diffusion,” in Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2023

    2023

  4. [10]

    Rave: Randomized noise shuffling for fast and consistent video editing with diffusion models,

    O. Kara, B. Kurtkaya, H. Yesiltepe, J. M. Rehg, and P. Yanardag, “Rave: Randomized noise shuffling for fast and consistent video editing with diffusion models,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6507–6516, 2024

    2024

  5. [11]

    Slicedit: Zero-shot video editing with text-to-image diffusion models using spatio-temporal slices,

    N. Cohen, V . Kulikov, M. Kleiner, I. Huberman-Spiegelglas, and T. Michaeli, “Slicedit: Zero-shot video editing with text-to-image diffusion models using spatio-temporal slices,” in Proceedings of the International Conference on Machine Learning (ICML) , pp. 9109–9137, 2024

    2024

  6. [12]

    Zero-shot video editing using off-the-shelf image diffusion models,

    W. Wang, Y . Jiang, K. Xie, Z. Liu, H. Chen, Y . Cao, X. Wang, and C. Shen, “Zero-shot video editing using off-the-shelf image diffusion models,” 2024

    2024

  7. [13]

    Video quality assessment: A comprehensive survey,

    Q. Zheng, Y . Fan, L. Huang, T. Zhu, J. Liu, Z. Hao, S. Xing, C.-J. Chen, X. Min, A. C. Bovik, and Z. Tu, “Video quality assessment: A comprehensive survey,” 2024

    2024

  8. [14]

    Exploring video quality assessment on user generated contents from aesthetic and technical perspectives,

    H. Wu, E. Zhang, L. Liao, C. Chen, J. H. Hou, A. Wang, W. S. Sun, Q. Yan, and W. Lin, “Exploring video quality assessment on user generated contents from aesthetic and technical perspectives,” in Proceedings of the International Conference on Computer Vision (ICCV) , 2023

    2023

  9. [15]

    Fast-vqa: Efficient end-to-end video quality assessment with fragment sampling,

    H. Wu, C. Chen, J. Hou, L. Liao, A. Wang, W. Sun, Q. Yan, and W. Lin, “Fast-vqa: Efficient end-to-end video quality assessment with fragment sampling,” in Proceedings of European Conference of Computer Vision (ECCV), 2022

    2022

  10. [16]

    A deep learning based no-reference quality assessment model for ugc videos,

    W. Sun, X. Min, W. Lu, and G. Zhai, “A deep learning based no-reference quality assessment model for ugc videos,” in Proceedings of the ACM International Conference on Multimedia (ACMMM) , p. 856–865, 2022

    2022

  11. [17]

    Quality assessment of in-the-wild videos,

    D. Li, T. Jiang, and M. Jiang, “Quality assessment of in-the-wild videos,” in Proceedings of the ACM International Conference on Multimedia (ACMMM) , p. 2351–2359, ACM, Oct. 2019

    2019

  12. [18]

    Ugc-vqa: Benchmarking blind video quality assessment for user generated content,

    Z. Tu, Y . Wang, N. Birkbeck, B. Adsumilli, and A. C. Bovik, “Ugc-vqa: Benchmarking blind video quality assessment for user generated content,” IEEE Transactions on Image Processing (TIP) , vol. 30, pp. 4449–4461, 2020

    2020

  13. [20]

    Subjective-aligned dataset and metric for text-to-video quality assessment,

    T. Kou, X. Liu, Z. Zhang, C. Li, H. Wu, X. Min, G. Zhai, and N. Liu, “Subjective-aligned dataset and metric for text-to-video quality assessment,” in Proceedings of the ACM International Conference on Multimedia (ACMMM), MM ’24, (New York, NY , USA), p. 7793–7802, Association ...

    2024

  14. [22]

    Cvpr 2023 text guided video editing competition,

    J. Z. Wu, X. Li, D. Gao, Z. Dong, J. Bai, A. Singh, X. Xiang, Y . Li, Z. Huang, Y . Sun, R. He, F. Hu, J. Hu, H. Huang, H. Zhu, X. Cheng, J. Tang, M. Z. Shou, K. Keutzer, and F. Iandola, “Cvpr 2023 text guided video editing competition,” 2023

    2023

  15. [23]

    Harnessing the power of llms in practice: A survey on chatgpt and beyond,

    J. Yang, H. Jin, R. Tang, X. Han, Q. Feng, H. Jiang, B. Yin, and X. Hu, “Harnessing the power of llms in practice: A survey on chatgpt and beyond,” 2023

    2023

  16. [24]

    Señorita- 2m: A high-quality instruction-based dataset for general video editing by video specialists,

    B. Zi, P. Ruan, M. Chen, X. Qi, S. Hao, S. Zhao, Y . Huang, B. Liang, R. Xiao, and K.-F. Wong, “Señorita- 2m: A high-quality instruction-based dataset for general video editing by video specialists,” 2025

    2025

  17. [25]

    The 2017 davis challenge on video object segmentation,

    J. Pont-Tuset, F. Perazzi, S. Caelles, P. Arbeláez, A. Sorkine-Hornung, and L. V . Gool, “The 2017 davis challenge on video object segmentation,” 2018

    2017

  18. [26]

    The kinetics human action video dataset,

    W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, M. Suleyman, and A. Zisserman, “The kinetics human action video dataset,” 2017

    2017

  19. [27]

    JimengAI

    B.Team, “JimengAI.” http://jimeng.jianying.com/, 2025. [Accessed 22-04-2025]

    2025

  20. [28]

    Methodology for the subjective assessment of the quality of television pictures,

    B. Series, “Methodology for the subjective assessment of the quality of television pictures,” Recommenda- tion ITU-R BT, pp. 500–13, 2012

    2012

  21. [29]

    Blind image quality assessment based on high order statistics aggregation,

    J. Xu, P. Ye, Q. Li, H. Du, Y . Liu, and D. Doermann, “Blind image quality assessment based on high order statistics aggregation,” IEEE Transactions on Image Processing (TIP) , vol. 25, no. 9, pp. 4444–4457, 2016

    2016

  22. [30]

    Learning without human scores for blind image quality assessment,

    W. Xue, L. Zhang, and X. Mou, “Learning without human scores for blind image quality assessment,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 995–1002, 2013

    2013

  23. [31]

    Making a “completely blind

    A. Mittal, R. Soundararajan, and A. C. Bovik, “Making a “completely blind” image quality analyzer,” IEEE Signal Processing Letters (SPL) , vol. 20, no. 3, pp. 209–212, 2013

    2013

  24. [32]

    Clipscore: A reference-free evaluation metric for image captioning,

    J. Hessel, A. Holtzman, M. Forbes, R. L. Bras, and Y . Choi, “Clipscore: A reference-free evaluation metric for image captioning,” 2022

    2022

  25. [33]

    Evaluating text-to-visual generation with image-to-text generation,

    Z. Lin, D. Pathak, B. Li, J. Li, X. Xia, G. Neubig, P. Zhang, and D. Ramanan, “Evaluating text-to-visual generation with image-to-text generation,” arXiv preprint arXiv:2404.01291, 2024

    2024 arXiv

  26. [34]

    Lmm4lmm: Benchmarking and evaluating large-multimodal image generation with lmms,

    J. Wang, H. Duan, Y . Zhao, J. Wang, G. Zhai, and X. Min, “Lmm4lmm: Benchmarking and evaluating large-multimodal image generation with lmms,” arXiv preprint arXiv:2504.08358, 2025

    2025 arXiv

  27. [35]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,

    DeepSeek-AI, “Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,” 2025

    2025

  28. [36]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Kolesnikov, A. Dosovitskiy, D. Weissenborn, G. Heigold, J. Uszkoreit, L. Beyer, M. Minderer, M. De- hghani, N. Houlsby, S. Gelly, T. Unterthiner, and X. Zhai, “An image is worth 16x16 words: Transformers for image recognition at scale,” in Proceedings of the International C...

    2021

  29. [37]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby, “An image is worth 16x16 words: Transformers for image recognition at scale,” ICLR, 2021

    2021

  30. [38]

    Mlp-mixer: An all-mlp architecture for vision,

    I. Tolstikhin, N. Houlsby, A. Kolesnikov, L. Beyer, X. Zhai, T. Unterthiner, J. Yung, A. Steiner, D. Keysers, J. Uszkoreit, M. Lucic, and A. Dosovitskiy, “Mlp-mixer: An all-mlp architecture for vision,”arXiv preprint arXiv:2105.01601, 2021

    2021 arXiv

  31. [39]

    How to train your vit? data, augmentation, and regularization in vision transformers,

    A. Steiner, A. Kolesnikov, , X. Zhai, R. Wightman, J. Uszkoreit, and L. Beyer, “How to train your vit? data, augmentation, and regularization in vision transformers,” arXiv preprint arXiv:2106.10270, 2021

    2021 arXiv

  32. [40]

    When vision transformers outperform resnets without pretraining or strong data augmentations,

    X. Chen, C.-J. Hsieh, and B. Gong, “When vision transformers outperform resnets without pretraining or strong data augmentations,” arXiv preprint arXiv:2106.01548, 2021

    2021 arXiv

  33. [41]

    Surrogate gap minimization improves sharpness-aware training,

    J. Zhuang, B. Gong, L. Yuan, Y . Cui, H. Adam, N. Dvornek, S. Tatikonda, J. Duncan, and T. Liu, “Surrogate gap minimization improves sharpness-aware training,” ICLR, 2022

    2022

  34. [42]

    Lit: Zero-shot transfer with locked-image text tuning,

    X. Zhai, X. Wang, B. Mustafa, A. Steiner, D. Keysers, A. Kolesnikov, and L. Beyer, “Lit: Zero-shot transfer with locked-image text tuning,” CVPR, 2022. 11

    2022

  35. [43]

    Qwen-vl: A versa- tile vision-language model for understanding, localization, text reading, and beyond,

    J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang, J. Lin, C. Zhou, and J. Zhou, “Qwen-vl: A versa- tile vision-language model for understanding, localization, text reading, and beyond,” arXiv preprint arXiv:2308.12966, 2023

    2023 arXiv

  36. [44]

    Mlp-net: Multilayer perceptron fusion network for infrared small target detection,

    Z. Wang, C. Wang, X. Li, C. Xia, and J. Xu, “Mlp-net: Multilayer perceptron fusion network for infrared small target detection,” IEEE Transactions on Geoscience and Remote Sensing , vol. 63, pp. 1–13, 2025

    2025

  37. [45]

    Lora: Low-rank adaptation of large language models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” 2021

    2021

  38. [46]

    Imagereward: learning and evaluating human preferences for text-to-image generation,

    J. Xu, X. Liu, Y . Wu, Y . Tong, Q. Li, M. Ding, J. Tang, and Y . Dong, “Imagereward: learning and evaluating human preferences for text-to-image generation,” inProceedings of the International Conference on Neural Information Processing Systems (NeurIPS), pp. 15903–15935, 2023

    2023

  39. [47]

    Blip: Bootstrapping language-image pre-training for unified vision- language understanding and generation,

    J. Li, D. Li, C. Xiong, and S. Hoi, “Blip: Bootstrapping language-image pre-training for unified vision- language understanding and generation,” in Proceedings of the International Conference on Learning Representations (ICML), 2022

    2022

  40. [48]

    Pick-a-pic: An open dataset of user preferences for text-to-image generation,

    Y . Kirstain, A. Polyak, U. Singer, S. Matiana, J. Penna, and O. Levy, “Pick-a-pic: An open dataset of user preferences for text-to-image generation,” in Proceedings of the 37th International Conference on Neural Information Processing Systems(NeurIPS), 2023

    2023

  41. [49]

    Building cnn-based models for image aesthetic score prediction using an ensemble,

    Y . Dai, “Building cnn-based models for image aesthetic score prediction using an ensemble,”Journal of Imaging, vol. 9, p. 30, Jan. 2023

    2023

  42. [50]

    Llava-next: A strong zero-shot video understanding model,

    Y . Zhang, B. Li, h. Liu, Y . j. Lee, L. Gui, D. Fu, J. Feng, Z. Liu, and C. Li, “Llava-next: A strong zero-shot video understanding model,” April 2024

    2024

  43. [51]

    Internvideo: General video foundation models via generative and discriminative learning,

    Y . Wang, K. Li, Y . Li, Y . He, B. Huang, Z. Zhao, H. Zhang, J. Xu, Y . Liu, Z. Wang, S. Xing, G. Chen, J. Pan, J. Yu, Y . Wang, L. Wang, and Y . Qiao, “Internvideo: General video foundation models via generative and discriminative learning,” 2022

    2022

  44. [52]

    Videollama 3: Frontier multimodal foundation models for image and video understanding,

    B. Zhang, K. Li, Z. Cheng, Z. Hu, Y . Yuan, G. Chen, S. Leng, Y . Jiang, H. Zhang, X. Li, P. Jin, W. Zhang, F. Wang, L. Bing, and D. Zhao, “Videollama 3: Frontier multimodal foundation models for image and video understanding,” arXiv preprint arXiv:2501.13106, 2025

    2025 arXiv

  45. [53]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks,

    Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu,et al., “Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (C...

    2024

  46. [54]

    mplug-owl3: Towards long image-sequence understanding in multi-modal large language models,

    J. Ye, H. Xu, H. Liu, A. Hu, M. Yan, Q. Qian, J. Zhang, F. Huang, and J. Zhou, “mplug-owl3: Towards long image-sequence understanding in multi-modal large language models,” 2024

    2024

  47. [55]

    No-reference image quality assessment in the spatial domain,

    A. Mittal, A. K. Moorthy, and A. C. Bovik, “No-reference image quality assessment in the spatial domain,” IEEE Transactions on Image Processing (TIP), vol. 21, no. 12, pp. 4695–4708, 2012

    2012

  48. [56]

    Blind image quality estimation via distortion aggravation,

    X. Min, G. Zhai, K. Gu, Y . Liu, and X. Yang, “Blind image quality estimation via distortion aggravation,” IEEE Transactions on Broadcasting (T-BC), vol. 64, no. 2, pp. 508–517, 2018

    2018

  49. [57]

    Blind quality assessment based on pseudo- reference image,

    X. Min, K. Gu, G. Zhai, J. Liu, X. Yang, and C. W. Chen, “Blind quality assessment based on pseudo- reference image,” IEEE Transactions on Multimedia (TMM) , vol. 20, no. 8, pp. 2049–2062, 2018

    2018

  50. [58]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” 2023

    2023

  51. [59]

    Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models,

    F. Li, R. Zhang, H. Zhang, Y . Zhang, B. Li, W. Li, Z. Ma, and C. Li, “Llava-next-interleave: Tackling multi-image, video, and 3d in large multimodal models,” arXiv preprint arXiv:2407.07895, 2024

    2024 arXiv

  52. [60]

    Llava-next: Improved reasoning, ocr, and world knowledge,

    H. Liu, C. Li, Y . Li, B. Li, Y . Zhang, S. Shen, and Y . J. Lee, “Llava-next: Improved reasoning, ocr, and world knowledge,” January 2024

    2024

  53. [61]

    Improved baselines with visual instruction tuning,

    H. Liu, C. Li, Y . Li, and Y . J. Lee, “Improved baselines with visual instruction tuning,” 2023

    2023

  54. [62]

    Llava-next: Stronger llms supercharge multimodal capabilities in the wild,

    B. Li, K. Zhang, H. Zhang, D. Guo, R. Zhang, F. Li, Y . Zhang, Z. Liu, and C. Li, “Llava-next: Stronger llms supercharge multimodal capabilities in the wild,” May 2024

    2024

  55. [63]

    Llava-next: What else influences visual instruction tuning beyond data?,

    B. Li, H. Zhang, K. Zhang, D. Guo, Y . Zhang, R. Zhang, F. Li, Z. Liu, and C. Li, “Llava-next: What else influences visual instruction tuning beyond data?,” May 2024

    2024

  56. [64]

    Video-llama: An instruction-tuned audio-visual language model for video understanding,

    H. Zhang, X. Li, and L. Bing, “Video-llama: An instruction-tuned audio-visual language model for video understanding,” arXiv preprint arXiv:2306.02858, 2023. 12

    2023 arXiv

  57. [65]

    Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms,

    Z. Cheng, S. Leng, H. Zhang, Y . Xin, X. Li, G. Chen, Y . Zhu, W. Zhang, Z. Luo, D. Zhao, and L. Bing, “Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms,” arXiv preprint arXiv:2406.07476, 2024

    2024 arXiv

  58. [66]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervision,” in Proceedings of the International Conference on Machine Learni...

    2021

  59. [67]

    Unmasked teacher: Towards training- efficient video foundation models,

    K. Li, Y . Wang, Y . Li, Y . Wang, Y . He, L. Wang, and Y . Qiao, “Unmasked teacher: Towards training- efficient video foundation models,” inProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 19948–19960, October 2023

    2023

  60. [68]

    Stablevqa: A deep no-reference quality assessment model for video stability,

    T. Kou, X. Liu, W. Sun, J. Jia, X. Min, G. Zhai, and N. Liu, “Stablevqa: A deep no-reference quality assessment model for video stability,” in Proceedings of the ACM International Conference on Multimedia (ACMMM), MM ’23, (New York, NY , USA), p. 1066–1076, Association for Com...

    2023

  61. [69]

    Q-align: Teaching lmms for visual scoring via discrete text-defined levels,

    H. Wu, Z. Zhang, W. Zhang, C. Chen, C. Li, L. Liao, A. Wang, E. Zhang, W. Sun, Q. Yan, X. Min, G. Zhai, and W. Lin, “Q-align: Teaching lmms for visual scoring via discrete text-defined levels,”arXiv preprint arXiv:2312.17090, 2023. Equal Contribution by Wu, Haoning and Zhang, ...

    2023 arXiv

  62. [70]

    Excellent

    Excellent Reliability of Mean Opinion Scores (MOS): Across all three evaluation dimen- sions, the ICC2k values, which assess the reliability of the average scores from the 15 raters, are exceptionally high. Specifically, Video Quality (ICC2k = 0.955), Editing Alignment (ICC2k ...

  63. [71]

    Excellent

    Consistency Among Individual Raters (ICC2): 20 • For Editing Alignment, the ICC2 value reached 0.753, signifying “Excellent” agreement among individual raters. This indicates that our participants achieved a strong consensus when evaluating the core semantic adherence of edite...

    2016

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.