Face Consistency Benchmark for GenAI Video

Haohong Wang; Malgorzata Kudelska; Michal Podstawski

REVIEW 4 major objections 5 minor 15 references

Face Consistency Benchmark for GenAI Video

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A benchmark using six face-recognition embeddings finds that every AI video generator it tests keeps faces less consistent than real video does.

desk verdict A simple, plausible face-consistency benchmark whose metric overclaims—reviewable but needs revision. read the letter →

arxiv 2505.11425 v1 pith:6EUNVCKC submitted 2025-05-16 cs.CV cs.MM

Michal Podstawski , Malgorzata Kudelska , Haohong Wang This is my paper · ORCID

classification cs.CVcs.MM

keywords AIvideogenerationcharacterconsistencyfacecosinedistancerecognitionembeddingstext-to-videotemporalbenchmarkingtools

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The Face Consistency Benchmark (FCB) is a proposed standard way to measure how well AI video generators keep a character's face looking like the same person across frames. The method crops faces, encodes each one with six pre-trained face-recognition models, and scores consistency as the cosine distance between those encodings—a similarity measure where lower means more alike—either from a chosen reference frame or across random frame pairs. Applied to four current text-to-video generators and to real footage, the benchmark reports that every generator's average distance is larger than the real-video baseline in both comparison modes, with the best models still far behind real scenes. The paper's point is that facial consistency is a distinct, quantifiable weakness of current models that general video-quality benchmarks do not isolate.

What carries the argument

The machinery is the FCB metric itself: cosine distance between face-recognition embeddings, computed on face crops drawn from each video. Six recognizers—VGG-Face, FaceNet, FaceNet512, ArcFace, SFace, and GhostFaceNet—provide the embeddings, and a face-analysis library bundles them into one pipeline; frames where no face is detected are skipped. The metric is applied in two modes, reference-frame comparison and random-pair comparison, and real video is measured under the same protocol to serve as the baseline that defines the target consistency level. Because the entire verdict rests on this one number, the machinery also includes the design choice of averaging results over 30 videos per model and across six recognizers.

What would settle it

A direct test would be to take the same generated videos and have people rate whether pairs of frames show the same person, then compare their ratings with FCB's cosine distances; if human agreement does not reproduce the real-video-versus-generated gap or the model ranking, the metric is not measuring perceived face consistency.

Watch

Extended reading notes

Core claim

The paper's claim is that facial consistency in AI-generated video can be objectively quantified by embedding face crops and measuring the cosine distance between those embeddings, with lower distance meaning more consistent identity and expression. In Mode 1, every frame is compared with a selected representative frame; in Mode 2, 200 random frame pairs are compared within the same video. On both tables, real video holds the lowest mean distance for every one of the six recognizers, while all four generated models—Runway Gen-3, HunyuanVideo, Vchitect-2.0, and CogVideoX1.5-5B—show substantially higher distances. The paper singles out HunyuanVideo and Runway Gen-3 as relatively better than the other two generators, but still concludes that current generative models 'fall significantly short of real video consistency.' This finding is presented as evidence that standardized face-consistency metrics can expose gaps that generic video-quality benchmarks miss.

Load-bearing premise

The whole benchmark assumes that a computer's face-similarity score tracks what a human viewer would call the same face across frames, yet the paper never compares its scores with human ratings.

Editorial extensions

If this is right

With FCB as a shared metric, model developers can compare face consistency across papers numerically instead of relying on anecdotal examples.
The two modes isolate two failure types: Mode 1 catches drift away from the intended face, and Mode 2 catches incoherent face changes between arbitrary frames.
Because real video is measured under the same protocol, FCB defines a concrete target: closing the gap between a generator's mean distance and the real-video baseline for the same recognizer.
The six recognizer columns let users check whether a result is robust across face encoders; if a model wins under one recognizer but loses under another, the comparison is inconclusive.
The protocol's face-detection skip rule makes the benchmark applicable to unconstrained footage where characters turn away, occlude their face, or leave the frame, but it also means scores describe only frames where a face is visible.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural next experiment the paper does not run is correlating FCB scores with human perceptual ratings; if that correlation is weak, the benchmark's rankings would need recalibration rather than replacement.
The same embedding-distance protocol could be extended to multi-character scenes by tracking each detected identity separately, which would also connect naturally to the full-body coherence the paper lists as future work.
Comparing FCB scores across seeds, prompts, and resolutions would separate model-level consistency from sampling luck; the paper currently averages 30 videos per model but does not report per-prompt variance.
If the community adopts FCB as a standard, it may become a target for optimization, and the six recognizers could be aggregated into a single ensemble score to reduce single-encoder variance.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

FCB is a quick, sensible benchmark that compares face embeddings across frames of AI-generated video, and it finds a consistent identity-drift gap relative to real video. That gap is probably real; the bigger claim—that the metric captures expressions and fine details—is not supported by the evidence.

The paper does a few things right. It uses six off-the-shelf face-recognition models, two comparison modes (representative frame and random pairs), and a real-video baseline. The finding that every generated model scores worse than real video across all six embeddings is consistent and credibly shows current models struggle to keep a face stable. The benchmark is simple enough to adopt.

The soft spots are real but fixable. First, the metric: VGG-Face, ArcFace, FaceNet, etc. are trained for verification/identification and deliberately discard within-identity variation—expression, pose, lighting, fine detail. So cosine distance mostly measures identity substitution, not subtle facial drift. The paper says the models extract 'features representing identity and expressions,' which is misleading. If a character changes expression but keeps identity, FCB would score it as consistent. The paper should either validate the metric against human ratings or reframe the claim as identity consistency only. Second, the related work admits VBench has a 'subject identity inconsistency' dimension, so the novelty is thin—the specific packaging and the real-video baseline are the new parts. Third, the empirical basis is 30 videos per model, with no error bars or significance tests, no code/data/prompts released, and no description of how the representative frame is chosen or how frame skipping affects results. Those are addressable.

The reader's conditional report is fair. The stress-test overstates the damage a bit: a measure of identity consistency is itself valuable, and the paper doesn't need to measure everything to be useful. But the overclaim should be fixed.

This is a workshop/short-conference paper. I'd send it to review rather than desk-reject: the method is coherent and the gap result is useful enough to warrant referee time. A serious referee should ask for the code, the representative-frame selection, and a human-perception sanity check. I would not cite it yet, but I'd track it if the authors release the benchmark artifacts.

Referee Report

4 major / 5 minor

Summary. The paper introduces the Face Consistency Benchmark (FCB), a framework for evaluating facial consistency in AI-generated videos. FCB extracts faces from videos, computes facial embeddings using six publicly available face recognition models via the DeepFace library, and reports the cosine distance between embeddings in two modes: (Mode 1) comparing all frames to a selected representative frame, and (Mode 2) comparing 200 random frame pairs within each video. The authors evaluate four text-to-video generators (HunyuanVideo, Vchitect-2.0, CogVideoX1.5-5B, Runway Gen-3) with 30 videos each, using prompts derived from real videos, and compare the results against a real-video baseline. All generated models show higher mean cosine distances than real videos in both modes, leading the authors to conclude that current models 'fall significantly short of real video consistency.'

Significance. If the proposed metric were validated, the benchmark would address a genuine gap: existing benchmarks such as VBench and AIGCBench focus on temporal consistency and realism but not specifically on facial identity consistency. The paper has several strengths: it uses standard, publicly available face recognition models and library (DeepFace), chooses two complementary comparison schemes, and includes a real-video baseline, making the experimental protocol straightforward to replicate. The central limitation is that the metric is never validated against human perceptual judgments, and the paper makes unsubstantiated claims about measuring expressions and fine details. The reported gap between AI-generated and real videos is plausible but currently lacks statistical support. With added validation and robustness analysis, the framework could be a useful benchmark; as presented, its conclusions rest on unverified assumptions about what cosine distance in face-recognition embedding space measures.

major comments (4)

[Section 3, Tables 1-2] The benchmark's central claim that cosine distance between face-recognition embeddings measures 'identity, expressions, and fine details' is not supported. Face recognition models such as ArcFace are trained to be invariant to expression, pose, and illumination, so the embedding distance primarily captures identity mismatch, not expression or fine-detail variation. A model that changes a character's expression or facial detail while preserving identity would be scored as consistent, contradicting the paper's framing. The authors should either validate the metric against human perceptual ratings (as VBench does) or explicitly restrict the benchmark's claims to identity consistency.
[Tables 1 and 2] The reported mean cosine distances have no error bars, confidence intervals, or significance tests, and each model is represented by only 30 videos. The ranking differences between HunyuanVideo and Runway Gen-3, for example, are within a few hundredths and may not be statistically meaningful. The authors should report per-model distributions and perform paired statistical tests across the same prompts, and clarify how many real videos were used for the baseline.
[Section 3 (frame processing)] The paper does not specify how the 'representative frame' for Mode 1 is selected, and it skips frames in which no face is detected. Both choices can bias the results: selecting a canonical frontal frame inflates Mode 1 similarity, and skipping undetectable faces systematically removes exactly the frames where consistency fails (e.g., profile views). The authors should describe the selection criterion and report the fraction of skipped frames per model to allow assessment of bias.
[Section 3 (experimental data)] The use of ChatGPT to derive prompts from real-video frames, and the absence of a released prompt set, makes the comparison difficult to reproduce. Without releasing the exact prompts, generated videos, and evaluation code, the benchmark cannot serve as a standardized evaluation tool as claimed. The authors should include the prompt set or a public repository to support reproducibility.

minor comments (5)

[Figure 1] The caption of Figure 1 does not indicate the model colors or line styles, so the reader cannot distinguish the curves without the text; consider adding a legend.
[References] In the references, [10] and [12] are only URLs with no author or year; [8] appears without full bibliographic details in the text.
[Section 3] The phrase 'if appropriate, it can be easily switched to Euclidean or L2-normalized Euclidean distance' is vague; specify which condition would make the change appropriate.
[Tables 1 and 2] The captions of Tables 1 and 2 should state the number of videos and the number of pairs (200) for reproducibility.
[Text] The paper uses 'Facenet' and 'Facenet512' inconsistently with 'FaceNet' in reference [4].

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FCB scores are direct measurements using external face-recognition embeddings and a real-video baseline, with no fitted parameter or self-citation chain forcing the reported conclusions.

full rationale

The paper makes no circular derivation. FCB defines its metric as the cosine distance between face embeddings extracted by external face-recognition models (VGG-Face, FaceNet, FaceNet512, ArcFace, SFace, GhostFaceNet) via the DeepFace library. The benchmark values in Tables 1 and 2 are direct measurements computed over generated and real videos; no parameter is fitted to the benchmark outcomes, and no quantity that appears as an output (such as the real-video baseline or the generated-video gap) is used as an input to the construction. The paper does not define 'face consistency' as the embedding distance and then report that same definition as an empirical discovery; rather, it selects a metric and reports measured values. The choice of off-the-shelf recognition models is an independent, non-parametric design choice, and the real-video baseline provides an external reference point. There is no self-citation chain that carries the argument: the cited face-recognition models, VBench, and the video-generation models are all external works. The weakest assumption is that cosine distance between face-recognition embeddings tracks human-perceived face consistency, including expression and fine-detail coherence. That is a validity or correctness concern, not a circularity, because the benchmark's numbers are not forced by the construction to match any target conclusion. Since no step reduces to its own input by definition, and no fitted parameter is renamed as a prediction, the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted. The framework rests on three domain assumptions about face embeddings, frame skipping, and reference-frame selection, none of which are independently validated in the paper.

assumptions (3)

domain assumption Face-recognition embeddings measure facial identity consistency.
The benchmark equates cosine distance between embeddings with character consistency, but this is not validated against human perception anywhere in the paper. This premise enters in Section 3 where the metrics are chosen.
domain assumption Frames without a detected face can be skipped without biasing the comparison.
Skipping frames when a face is not detectable could remove difficult poses or occlusions, potentially inflating consistency scores. This assumption is stated without justification in Section 3.
domain assumption The selected representative frame is a valid reference for mode one.
The paper does not describe how the representative frame is chosen, yet the entire mode-one comparison depends on this choice being a fair representation of the character's face. This is a load-bearing procedural assumption.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Face Consistency Benchmark for GenAI Video." pith.science (2026). https://pith.science/paper/6EUNVCKC

@misc{pith2026250511425,
  author       = {Pith},
  title        = {Pith review of: Face Consistency Benchmark for GenAI Video},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6EUNVCKC}},
  note         = {Machine review of arXiv:2505.11425}
}

read the original abstract

Video generation driven by artificial intelligence has advanced significantly, enabling the creation of dynamic and realistic content. However, maintaining character consistency across video sequences remains a major challenge, with current models struggling to ensure coherence in appearance and attributes. This paper introduces the Face Consistency Benchmark (FCB), a framework for evaluating and comparing the consistency of characters in AI-generated videos. By providing standardized metrics, the benchmark highlights gaps in existing solutions and promotes the development of more reliable approaches. This work represents a crucial step toward improving character consistency in AI video generation technologies.

Figures

Figures reproduced from arXiv: 2505.11425 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 7 canonical work pages

[1]

Aigcbench: Comprehensive eval- uation of image-to-video content generated by ai, 2024

Fan, F., Luo, C., Gao, W., and Zhan, J. Aigcbench: Comprehensive eval- uation of image-to-video content generated by ai, 2024. URL https: //arxiv.org/abs/2401.01651

arXiv 2024
[2]

Podstawski, M

Huang, Z., He, Y ., Yu, J., Zhang, F., Si, C., Jiang, Y ., Zhang, Y ., Wu, T., Jin, Q., Chanpaisit, N., Wang, Y ., Chen, X., Wang, L., Lin, D., Qiao, Y ., and Liu, M. Podstawski, M. Kudelska, H. Wang 7 Z. Vbench: Comprehensive benchmark suite for video generative models,

work page
[3]

M., Vedaldi, A., and Zisserman, A

Parkhi, O. M., Vedaldi, A., and Zisserman, A. Deep face recognition. In BMVC. 2015

work page 2015
[4]

Facenet: A unified embedding for face recognition and clustering

Schro ff, F., Kalenichenko, D., and Philbin, J. Facenet: A unified embedding for face recognition and clustering. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , page 815–823. IEEE, 2015. doi: 10.1109/cvpr.2015.7298682. URL http://dx.doi.org/10.1109/CVPR. 2015.7298682

arXiv 2015
[5]

Arc- face: Additive angular margin loss for deep face recognition

Deng, J., Guo, J., Yang, J., Xue, N., Kotsia, I., and Zafeiriou, S. Arc- face: Additive angular margin loss for deep face recognition. IEEE Trans- actions on Pattern Analysis and Machine Intelligence , 44(10):5962–5979,

work page
[6]

SFace: Privacy-friendly and Accurate Face Recognition using Synthetic Data

Boutros, F., Huber, M., Siebke, P., Rieber, T., and Damer, N. Sface: Privacy- friendly and accurate face recognition using synthetic data, 2022. URL https://arxiv.org/abs/2206.10520

work page Pith review arXiv 2022
[7]

A., Javed, S., Shoufan, A., Zweiri, Y ., and Werghi, N

Alansari, M., Hay, O. A., Javed, S., Shoufan, A., Zweiri, Y ., and Werghi, N. Ghostfacenets: Lightweight face recognition model from cheap operations. IEEE Access, 11:35429–35446, 2023. doi:10.1109/ACCESS.2023.3266068

arXiv 2023
[8]

and Ozpinar, A

Serengil, S. and Ozpinar, A. A benchmark of facial recognition pipelines and co-usability performances of modules. Journal of Information Tech- nologies, 17(2):95–107, 2024. doi:10.17671/gazibtd.1399077. URL https: //dergipark.org.tr/en/pub/gazibtd/issue/84331/1399077

doi:10.17671/gazibtd.1399077 2024

Show all 15 references

[9]

Hunyuanvideo: A systematic framework for large video generative models, 2024

Kong, W., Tian, Q., Zhang, Z., Min, R., Dai, Z., Zhou, J., Xiong, J., Li, X., Wu, B., et al. Hunyuanvideo: A systematic framework for large video generative models, 2024. URL https://arxiv.org/abs/2412.03603

2024 arXiv
[10]

Vchitect

Vchitect. Vchitect. URL https://vchitect.intern-ai.org.cn/. Ac- cessed: 2025-01-02

2025
[11]

Cogvideox: Text-to-video di ffusion models with an expert transformer

Yang, Z., Teng, J., Zheng, W., Ding, M., Huang, S., Xu, J., Yang, Y ., Hong, W., Zhang, X., Feng, G., et al. Cogvideox: Text-to-video di ffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 8 Face Consistency Benchmark for GenAI Video

2024 arXiv
[12]

Runway. Runway. URL https://runwayml.com/. Accessed: 2025-01-02

2025
[13]

OpenAI. Chatgpt. URL https://chatgpt.com/. Accessed: 2025-01-02

2025
[2022]

doi:10.1109 /tpami.2021.3087709

ISSN 1939-3539. doi:10.1109 /tpami.2021.3087709. URL http: //dx.doi.org/10.1109/TPAMI.2021.3087709

1939
[2023]

URL https://arxiv.org/abs/2311.17982

arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Aigcbench: Comprehensive eval- uation of image-to-video content generated by ai, 2024

Fan, F., Luo, C., Gao, W., and Zhan, J. Aigcbench: Comprehensive eval- uation of image-to-video content generated by ai, 2024. URL https: //arxiv.org/abs/2401.01651

arXiv 2024

[2] [2]

Podstawski, M

Huang, Z., He, Y ., Yu, J., Zhang, F., Si, C., Jiang, Y ., Zhang, Y ., Wu, T., Jin, Q., Chanpaisit, N., Wang, Y ., Chen, X., Wang, L., Lin, D., Qiao, Y ., and Liu, M. Podstawski, M. Kudelska, H. Wang 7 Z. Vbench: Comprehensive benchmark suite for video generative models,

work page

[3] [3]

M., Vedaldi, A., and Zisserman, A

Parkhi, O. M., Vedaldi, A., and Zisserman, A. Deep face recognition. In BMVC. 2015

work page 2015

[4] [4]

Facenet: A unified embedding for face recognition and clustering

Schro ff, F., Kalenichenko, D., and Philbin, J. Facenet: A unified embedding for face recognition and clustering. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , page 815–823. IEEE, 2015. doi: 10.1109/cvpr.2015.7298682. URL http://dx.doi.org/10.1109/CVPR. 2015.7298682

arXiv 2015

[5] [5]

Arc- face: Additive angular margin loss for deep face recognition

Deng, J., Guo, J., Yang, J., Xue, N., Kotsia, I., and Zafeiriou, S. Arc- face: Additive angular margin loss for deep face recognition. IEEE Trans- actions on Pattern Analysis and Machine Intelligence , 44(10):5962–5979,

work page

[6] [6]

SFace: Privacy-friendly and Accurate Face Recognition using Synthetic Data

Boutros, F., Huber, M., Siebke, P., Rieber, T., and Damer, N. Sface: Privacy- friendly and accurate face recognition using synthetic data, 2022. URL https://arxiv.org/abs/2206.10520

work page Pith review arXiv 2022

[7] [7]

A., Javed, S., Shoufan, A., Zweiri, Y ., and Werghi, N

Alansari, M., Hay, O. A., Javed, S., Shoufan, A., Zweiri, Y ., and Werghi, N. Ghostfacenets: Lightweight face recognition model from cheap operations. IEEE Access, 11:35429–35446, 2023. doi:10.1109/ACCESS.2023.3266068

arXiv 2023

[8] [8]

and Ozpinar, A

Serengil, S. and Ozpinar, A. A benchmark of facial recognition pipelines and co-usability performances of modules. Journal of Information Tech- nologies, 17(2):95–107, 2024. doi:10.17671/gazibtd.1399077. URL https: //dergipark.org.tr/en/pub/gazibtd/issue/84331/1399077

doi:10.17671/gazibtd.1399077 2024