{"id":"6c0e7787-5f45-4fdc-bdb7-0ed31271b40d","arxiv_id":"2505.11425","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A benchmark based on face-embedding cosine distance shows that AI-generated videos are markedly less face-consistent than real videos, with limited detail on methodology and no released artifacts.","lead":"This paper introduces the Face Consistency Benchmark, which scores how consistently AI-generated videos render a character's face across frames using face-recognition embeddings. It reports that four current text-to-video models, especially Vchitect-2.0 and CogVideoX1.5-5B, fall far short of real-video face consistency.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FCB's metric is an unvalidated proxy for face consistency: face-recognition embeddings intentionally suppress expression and fine-detail variation, so the paper's claim to measure identity, expressions, and fine details is not supported.","rationale":"The reader correctly identifies the missing human validation as the weakest assumption; I partially agree and sharpen it to a specific technical mismatch: face-recognition embeddings are trained to be invariant to the very dimensions (expression, fine detail) the paper claims to evaluate. The paper's quantitative findings are nonetheless internally consistent and the gap is replicated across six embeddings and two comparison modes, which is meaningful evidence for identity-consistency differences. The issue is not that the tables are wrong but that they do not measure what the title and Section 3 claim. Because the authors could remedy this by narrowing claims to identity consistency and adding a perceptual validation study, the appropriate outcome remains conditional rather than acceptance or rejection; hence I do not change the reader's verdict. A secondary issue is that frames with no detectable face are skipped, which excludes the most severe consistency failures from the metric, but the construct-validity problem is the load-bearing one.","tokens_in":4266,"tokens_out":6697,"duration_ms":73272,"concrete_test":"Assemble a set of real and generated clips in which the character's identity is constant but expression and fine details vary (e.g., talking, laughing, hairstyle changes). Compute FCB cosine distances in both modes for these clips and collect at least three human ratings per clip of perceived face consistency. Compute the Spearman correlation between FCB scores and mean human ratings; if rho is below 0.5 overall, or near zero on the expression-change subset, the metric does not measure what the paper claims. As a minimal analytical check, take one real video with strong expression changes: if the FCB distance remains near zero across frames while expression visibly changes, the metric is blind to expression consistency.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that FCB 'objectively' measures face consistency rests on the assumption that cosine distance between face-recognition embeddings tracks human-perceived face consistency. Section 3 selects VGG-Face, FaceNet, FaceNet512, ArcFace, SFace, and GhostFaceNet as 'designed to extract robust features representing identity and expressions.' This conflates identity recognition with face consistency. These models are trained for face verification and identification (e.g., ArcFace's additive angular margin) and are deliberately made invariant to within-identity variation such as expression, pose, illumination, and fine-grained appearance. The cosine distance therefore primarily measures identity mismatch, not expression or fine-detail consistency. The paper presents no human annotation or perceptual validation linking this distance to perceived face consistency. Tables 1 and 2 show a consistent identity-drift gap between generated and real video, but a model that changes a character's expression or facial detail while preserving identity would be scored as consistent, which contradicts the paper's framing. This is the most load-bearing weakness because it determines what the benchmark's numbers mean.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Face Consistency Benchmark (FCB), a framework for evaluating facial consistency in AI-generated videos. FCB extracts faces from videos, computes facial embeddings using six publicly available face recognition models via the DeepFace library, and reports the cosine distance between embeddings in two modes: (Mode 1) comparing all frames to a selected representative frame, and (Mode 2) comparing 200 random frame pairs within each video. The authors evaluate four text-to-video generators (HunyuanVideo, Vchitect-2.0, CogVideoX1.5-5B, Runway Gen-3) with 30 videos each, using prompts derived from real videos, and compare the results against a real-video baseline. All generated models show higher mean cosine distances than real videos in both modes, leading the authors to conclude that current models 'fall significantly short of real video consistency.'","tokens_in":4425,"tokens_out":3873,"duration_ms":36245,"significance":"If the proposed metric were validated, the benchmark would address a genuine gap: existing benchmarks such as VBench and AIGCBench focus on temporal consistency and realism but not specifically on facial identity consistency. The paper has several strengths: it uses standard, publicly available face recognition models and library (DeepFace), chooses two complementary comparison schemes, and includes a real-video baseline, making the experimental protocol straightforward to replicate. The central limitation is that the metric is never validated against human perceptual judgments, and the paper makes unsubstantiated claims about measuring expressions and fine details. The reported gap between AI-generated and real videos is plausible but currently lacks statistical support. With added validation and robustness analysis, the framework could be a useful benchmark; as presented, its conclusions rest on unverified assumptions about what cosine distance in face-recognition embedding space measures.","major_comments":[{"comment":"The benchmark's central claim that cosine distance between face-recognition embeddings measures 'identity, expressions, and fine details' is not supported. Face recognition models such as ArcFace are trained to be invariant to expression, pose, and illumination, so the embedding distance primarily captures identity mismatch, not expression or fine-detail variation. A model that changes a character's expression or facial detail while preserving identity would be scored as consistent, contradicting the paper's framing. The authors should either validate the metric against human perceptual ratings (as VBench does) or explicitly restrict the benchmark's claims to identity consistency.","section":"Section 3, Tables 1-2"},{"comment":"The reported mean cosine distances have no error bars, confidence intervals, or significance tests, and each model is represented by only 30 videos. The ranking differences between HunyuanVideo and Runway Gen-3, for example, are within a few hundredths and may not be statistically meaningful. The authors should report per-model distributions and perform paired statistical tests across the same prompts, and clarify how many real videos were used for the baseline.","section":"Tables 1 and 2"},{"comment":"The paper does not specify how the 'representative frame' for Mode 1 is selected, and it skips frames in which no face is detected. Both choices can bias the results: selecting a canonical frontal frame inflates Mode 1 similarity, and skipping undetectable faces systematically removes exactly the frames where consistency fails (e.g., profile views). The authors should describe the selection criterion and report the fraction of skipped frames per model to allow assessment of bias.","section":"Section 3 (frame processing)"},{"comment":"The use of ChatGPT to derive prompts from real-video frames, and the absence of a released prompt set, makes the comparison difficult to reproduce. Without releasing the exact prompts, generated videos, and evaluation code, the benchmark cannot serve as a standardized evaluation tool as claimed. The authors should include the prompt set or a public repository to support reproducibility.","section":"Section 3 (experimental data)"}],"minor_comments":[{"comment":"The caption of Figure 1 does not indicate the model colors or line styles, so the reader cannot distinguish the curves without the text; consider adding a legend.","section":"Figure 1"},{"comment":"In the references, [10] and [12] are only URLs with no author or year; [8] appears without full bibliographic details in the text.","section":"References"},{"comment":"The phrase 'if appropriate, it can be easily switched to Euclidean or L2-normalized Euclidean distance' is vague; specify which condition would make the change appropriate.","section":"Section 3"},{"comment":"The captions of Tables 1 and 2 should state the number of videos and the number of pairs (200) for reproducibility.","section":"Tables 1 and 2"},{"comment":"The paper uses 'Facenet' and 'Facenet512' inconsistently with 'FaceNet' in reference [4].","section":"Text"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short workshop-style contribution and may be better suited to a workshop than a full journal in its current form. The central quantitative claim is plausible, but the metric-validation and statistical issues are load-bearing and would require substantive additional experiments to resolve. The authors might also consider following VBench's example by collecting human preference annotations for face consistency, which would strengthen the benchmark considerably."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"FCB is a quick, sensible benchmark that compares face embeddings across frames of AI-generated video, and it finds a consistent identity-drift gap relative to real video. That gap is probably real; the bigger claim—that the metric captures expressions and fine details—is not supported by the evidence.\n\nThe paper does a few things right. It uses six off-the-shelf face-recognition models, two comparison modes (representative frame and random pairs), and a real-video baseline. The finding that every generated model scores worse than real video across all six embeddings is consistent and credibly shows current models struggle to keep a face stable. The benchmark is simple enough to adopt.\n\nThe soft spots are real but fixable. First, the metric: VGG-Face, ArcFace, FaceNet, etc. are trained for verification/identification and deliberately discard within-identity variation—expression, pose, lighting, fine detail. So cosine distance mostly measures identity substitution, not subtle facial drift. The paper says the models extract 'features representing identity and expressions,' which is misleading. If a character changes expression but keeps identity, FCB would score it as consistent. The paper should either validate the metric against human ratings or reframe the claim as identity consistency only. Second, the related work admits VBench has a 'subject identity inconsistency' dimension, so the novelty is thin—the specific packaging and the real-video baseline are the new parts. Third, the empirical basis is 30 videos per model, with no error bars or significance tests, no code/data/prompts released, and no description of how the representative frame is chosen or how frame skipping affects results. Those are addressable.\n\nThe reader's conditional report is fair. The stress-test overstates the damage a bit: a measure of identity consistency is itself valuable, and the paper doesn't need to measure everything to be useful. But the overclaim should be fixed.\n\nThis is a workshop/short-conference paper. I'd send it to review rather than desk-reject: the method is coherent and the gap result is useful enough to warrant referee time. A serious referee should ask for the code, the representative-frame selection, and a human-perception sanity check. I would not cite it yet, but I'd track it if the authors release the benchmark artifacts.","headline":"A simple, plausible face-consistency benchmark whose metric overclaims—reviewable but needs revision.","tokens_in":4945,"tokens_out":2385,"would_cite":false,"duration_ms":23164,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A benchmark using six face-recognition embeddings finds that every AI video generator it tests keeps faces less consistent than real video does.","keywords":["AI video generation","character consistency","face consistency","cosine distance","face recognition embeddings","text-to-video generation","temporal consistency","AI benchmarking tools"],"falsifier":"A direct test would be to take the same generated videos and have people rate whether pairs of frames show the same person, then compare their ratings with FCB's cosine distances; if human agreement does not reproduce the real-video-versus-generated gap or the model ranking, the metric is not measuring perceived face consistency.","tokens_in":4063,"feed_emoji":"🎬","tokens_out":10431,"duration_ms":97163,"temperature":0.7,"pith_summary":"The Face Consistency Benchmark (FCB) is a proposed standard way to measure how well AI video generators keep a character's face looking like the same person across frames. The method crops faces, encodes each one with six pre-trained face-recognition models, and scores consistency as the cosine distance between those encodings—a similarity measure where lower means more alike—either from a chosen reference frame or across random frame pairs. Applied to four current text-to-video generators and to real footage, the benchmark reports that every generator's average distance is larger than the real-video baseline in both comparison modes, with the best models still far behind real scenes. The paper's point is that facial consistency is a distinct, quantifiable weakness of current models that general video-quality benchmarks do not isolate.","feed_headline":"Every tested AI video model drifts from real-face consistency","feed_subtitle":"Cosine-distance face embeddings score all four generators worse than real footage in both modes.","key_machinery":"The machinery is the FCB metric itself: cosine distance between face-recognition embeddings, computed on face crops drawn from each video. Six recognizers—VGG-Face, FaceNet, FaceNet512, ArcFace, SFace, and GhostFaceNet—provide the embeddings, and a face-analysis library bundles them into one pipeline; frames where no face is detected are skipped. The metric is applied in two modes, reference-frame comparison and random-pair comparison, and real video is measured under the same protocol to serve as the baseline that defines the target consistency level. Because the entire verdict rests on this one number, the machinery also includes the design choice of averaging results over 30 videos per model and across six recognizers.","core_discovery":"The paper's claim is that facial consistency in AI-generated video can be objectively quantified by embedding face crops and measuring the cosine distance between those embeddings, with lower distance meaning more consistent identity and expression. In Mode 1, every frame is compared with a selected representative frame; in Mode 2, 200 random frame pairs are compared within the same video. On both tables, real video holds the lowest mean distance for every one of the six recognizers, while all four generated models—Runway Gen-3, HunyuanVideo, Vchitect-2.0, and CogVideoX1.5-5B—show substantially higher distances. The paper singles out HunyuanVideo and Runway Gen-3 as relatively better than the other two generators, but still concludes that current generative models 'fall significantly short of real video consistency.' This finding is presented as evidence that standardized face-consistency metrics can expose gaps that generic video-quality benchmarks miss.","pith_inferences":["A natural next experiment the paper does not run is correlating FCB scores with human perceptual ratings; if that correlation is weak, the benchmark's rankings would need recalibration rather than replacement.","The same embedding-distance protocol could be extended to multi-character scenes by tracking each detected identity separately, which would also connect naturally to the full-body coherence the paper lists as future work.","Comparing FCB scores across seeds, prompts, and resolutions would separate model-level consistency from sampling luck; the paper currently averages 30 videos per model but does not report per-prompt variance.","If the community adopts FCB as a standard, it may become a target for optimization, and the six recognizers could be aggregated into a single ensemble score to reduce single-encoder variance."],"forward_implications":["With FCB as a shared metric, model developers can compare face consistency across papers numerically instead of relying on anecdotal examples.","The two modes isolate two failure types: Mode 1 catches drift away from the intended face, and Mode 2 catches incoherent face changes between arbitrary frames.","Because real video is measured under the same protocol, FCB defines a concrete target: closing the gap between a generator's mean distance and the real-video baseline for the same recognizer.","The six recognizer columns let users check whether a result is robust across face encoders; if a model wins under one recognizer but loses under another, the comparison is inconclusive.","The protocol's face-detection skip rule makes the benchmark applicable to unconstrained footage where characters turn away, occlude their face, or leave the frame, but it also means scores describe only frames where a face is visible."],"supporting_citations":[{"why":"Supplies the VGG-Face embedding, one of the six face-recognition backends used to produce the consistency scores.","marker":"[3]"},{"why":"Supplies the FaceNet and FaceNet512 embeddings, two more backends whose cosine distances appear in both result tables.","marker":"[4]"},{"why":"Supplies the ArcFace embedding, a recognizer with additive angular margin loss used in the benchmark.","marker":"[5]"},{"why":"Supplies the SFace embedding, one of the six synthetic-data-trained backends measured in the tables.","marker":"[6]"},{"why":"Supplies the GhostFaceNet embedding, a lightweight recognizer used as another consistency measure.","marker":"[7]"},{"why":"Provides the integration library that runs the face-detection and embedding pipeline across all six recognizers.","marker":"[8]"},{"why":"Defines VBench, the 16-dimension benchmark whose top performers the authors selected as the tested generators, and the gap FCB targets because VBench does not isolate face consistency.","marker":"[2]"},{"why":"Identifies HunyuanVideo, one of the text-to-video generators scored by FCB.","marker":"[9]"},{"why":"Identifies CogVideoX1.5-5B, another scored generator.","marker":"[11]"},{"why":"Identifies Runway Gen-3, the API-accessed scored generator.","marker":"[12]"}],"fun_headline_variants":["Real video beats all AI generators on face consistency","Benchmark: all AI video models drift from real face consistency","No AI video model matches real-face consistency yet","HunyuanVideo and Runway lead, but all AI faces drift in video"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole benchmark assumes that a computer's face-similarity score tracks what a human viewer would call the same face across frames, yet the paper never compares its scores with human ratings.","fun_headline_variants_meta":{"raw":{"variants":["Real video beats all AI generators on face consistency","Benchmark: all AI video models drift from real face consistency","No AI video model matches real-face consistency yet","HunyuanVideo and Runway lead, but all AI faces drift in video"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00074,"raw_usage":{"total_tokens":3229,"prompt_tokens":797,"completion_tokens":2432,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":413,"completion_tokens_details":{"reasoning_tokens":2363}},"tokens_in":413,"tokens_out":2432,"duration_ms":17033,"temperature":1.0,"reasoning_tokens":2363,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:52:09.072801+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would be to take the same generated videos and have people rate whether pairs of frames show the same person, then compare their ratings with FCB's cosine distances; if human agreement does not reproduce the real-video-versus-generated gap or the model ranking, the metric is not measuring perceived face consistency.","supporting_citations":[{"cited_title":"M., Vedaldi, A., and Zisserman, A","cited_arxiv_id":null,"evidence_quote":"Supplies the VGG-Face embedding, one of the six face-recognition backends used to produce the consistency scores."},{"cited_title":"Arc- face: Additive angular margin loss for deep face recognition","cited_arxiv_id":null,"evidence_quote":"Supplies the ArcFace embedding, a recognizer with additive angular margin loss used in the benchmark."},{"cited_title":"SFace: Privacy-friendly and Accurate Face Recognition using Synthetic Data","cited_arxiv_id":"2206.10520","evidence_quote":"Supplies the SFace embedding, one of the six synthetic-data-trained backends measured in the tables."},{"cited_title":"Podstawski, M","cited_arxiv_id":null,"evidence_quote":"Defines VBench, the 16-dimension benchmark whose top performers the authors selected as the tested generators, and the gap FCB targets because VBench does not isolate face consistency."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Identifies Runway Gen-3, the API-accessed scored generator."}],"review_version":1}