Pith. sign in

REVIEW 10 cited by

EvalCrafter: Benchmarking and Evaluating Large Video Generation Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.11440 v3 pith:TNDWLB4H submitted 2023-10-17 cs.CV

Yaofang Liu , Xiaodong Cun , Xuebo Liu , Xintao Wang , Yong Zhang , Haoxin Chen , Yang Liu , Tieyong Zeng
show 2 more authors
Raymond Chan Ying Shan
This is my paper · ORCID
classification cs.CV
keywords modelsmetricslargegenerationgenerativequalitiesvideoalignment
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

The vision and language generative models have been overgrown in recent years. For video generation, various open-sourced models and public-available services have been developed to generate high-quality videos. However, these methods often use a few metrics, e.g., FVD or IS, to evaluate the performance. We argue that it is hard to judge the large conditional generative models from the simple metrics since these models are often trained on very large datasets with multi-aspect abilities. Thus, we propose a novel framework and pipeline for exhaustively evaluating the performance of the generated videos. Our approach involves generating a diverse and comprehensive list of 700 prompts for text-to-video generation, which is based on an analysis of real-world user data and generated with the assistance of a large language model. Then, we evaluate the state-of-the-art video generative models on our carefully designed benchmark, in terms of visual qualities, content qualities, motion qualities, and text-video alignment with 17 well-selected objective metrics. To obtain the final leaderboard of the models, we further fit a series of coefficients to align the objective metrics to the users' opinions. Based on the proposed human alignment method, our final score shows a higher correlation than simply averaging the metrics, showing the effectiveness of the proposed evaluation method.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 10 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. KeyFrame-Compass: Towards Comprehensive Evaluation of Keyframe-Conditioned Video Generation

    cs.CV 2026-07 conditional novelty 7.0 of 10

    KeyFrame-Compass tests nine video generators on 386 keyframe-sequence tasks and finds a consistent trade-off between keyframe fidelity and natural video quality, with control degrading under dense keyframes and open-s...

  2. VideoDPO: Omni-Preference Alignment for Video Diffusion Generation

    cs.CV 2024-12 reject novelty 6.0 of 10

    VideoDPO shows that DPO-style training on automatically selected best and worst video pairs improves overall VBench scores on three open text-to-video models, with some sub-metrics degrading and weak gains on external...

  3. AniSora: Exploring the Frontiers of Animation Video Generation in the Sora Era

    cs.GR 2024-12 conditional novelty 6.0 of 10

    AniSora presents a data pipeline, a spatiotemporal-mask diffusion model, and a 948-clip benchmark to push animation video generation past general-purpose models.

  4. InstanceCap: Improving Text-to-Video Generation via Instance-aware Structured Caption

    cs.CV 2024-12 conditional novelty 6.0 of 10

    InstanceCap generates instance-level structured video captions by isolating objects with auxiliary models and prompting MLLMs with class-specific hints, improving text-to-video generation fidelity when used for finetuning.

  5. Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Models

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Evaluation Agent is an LLM-agent framework that evaluates visual generative models with a handful of samples per round, claiming a 10x time reduction while keeping conclusions within one tier of full-benchmark results...

  6. Identity-Preserving Text-to-Video Generation by Frequency Decomposition

    cs.CV 2024-11 conditional novelty 6.0 of 10

    ConsisID generates identity-preserving videos by injecting low-frequency facial features into shallow layers and high-frequency identity features into attention blocks of a DiT video model.

  7. AIGV-Assessor: Benchmarking and Evaluating the Perceptual Quality of Text-to-Video Generation with LMM

    cs.CV 2024-11 conditional novelty 6.0 of 10

    A new large benchmark of AI-generated videos with human ratings across four dimensions, plus an LMM-based video quality assessor that outperforms prior metrics.

  8. AIGVE-MACS: Unified Multi-Aspect Commenting and Scoring Model for AI-Generated Video Evaluation

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A finetuned vision-language model jointly predicts nine aspect scores and written comments for AI-generated videos, with a new benchmark and claims of state-of-the-art alignment with human judgment.

  9. PAROAttention: Pattern-Aware ReOrdering for Efficient Sparse and Quantized Attention in Visual Generation Models

    cs.CV 2025-06 conditional novelty 4.0 of 10

    PAROAttention permutes tokens along frame, height, and width axes to make visual attention block-wise, enabling sparse and INT8/INT4 quantized attention with near-baseline generation quality.

  10. NTIRE 2025 XGC Quality Assessment Challenge: Methods and Results

    cs.CV 2025-06 conditional novelty 4.0 of 10

    All 19 valid entries in the NTIRE 2025 XGC quality assessment challenge outperformed their track baselines at predicting human quality scores for user-generated video, AI-generated video, and talking heads.

Pith tools