{"id":"b610c860-b73c-4361-b3ae-ddb7a9de02e0","arxiv_id":"2505.11109","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"MAVOS-DD, a multilingual audio-video deepfake dataset with controlled open-set splits for unseen languages and generators, shows that state-of-the-art detectors degrade substantially outside their training distribution.","lead":"MAVOS-DD is a new benchmark of over 60,000 real and fake videos in eight languages, generated with seven deepfake methods and split into in-domain and open-set tests. It shows that current audio-video deepfake detectors, while accurate in-domain, lose significant accuracy on unseen languages and generation models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Open-set splits may leak content-level overlap because generated fakes use the same real YouTube clips that are then split into train/test; the reported generalization drops are not clean unless this is ruled out.","rationale":"The reader's weakest_assumption points at identity overlap from the real video pool; I agree that is a related concern, but the load-bearing version is clip-level train/test leakage. The paper's split description and Table 2 give no evidence that source/target videos were excluded from the opposite split. For face-swap and LivePortrait fakes, the generated video shares most pixels and audio with its source real clip; if that real clip is in training, the model has been exposed to near-identical content during fine-tuning. This would not necessarily erase the performance drop, but it would make the 'open-set' label inaccurate and could bias the magnitude of the reported gap. The proposed audit is feasible because the paper promises to release data and code. If the audit finds no overlap, the central claim stands. If overlap is found, the benchmark's open-set validity is compromised and the verdict would need to be REJECT pending reconstruction of clean splits. Since the overlap is not yet confirmed, the current CONDITIONAL verdict remains appropriate, and my read does not move the verdict.","tokens_in":14125,"tokens_out":8432,"duration_ms":91879,"concrete_test":"Using the released dataset metadata, build a bipartite map from each generated fake video to its source real video (audio source for talking heads, source video for LivePortrait, target video and source identity for face swaps). Check whether any source real video or its clip appears in the training split while the corresponding fake appears in an open-set test split, and whether any test real video has a generated fake in the training split. Then recompute Table 3 accuracy and AUC on the subset of open-set fake samples whose source content is disjoint from the training split; if the in-domain-to-open-set drop changes materially, the reported gap is not a clean open-set measurement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires the open-set test sets to measure generalization to unseen generators and languages. Section 3 describes fake generation as operating on the collected real videos: talking-head methods use audio that 'originates from the real video set'; LivePortrait uses 'source videos are represented by the real videos collected from YouTube'; face swaps use 'target videos selected from the collected set of real YouTube videos'; and source identities include 'identities found in our real videos.' The paper never states that the train/validation/test splits are disjoint at the level of these source/target clips. If a real clip used as a target for a fake in an open-set test split also appears, or its near-duplicate fake appears, in the training set, then the detector has seen the original content during fine-tuning. Because the splits are defined only by language and generator membership (Table 2), this is a live possibility. This is sharper than 'identity overlap': it is exact-video-content leakage, and it can affect the open-set model, language, and full scenarios. The EchoMimic per-language fine-tuning is an additional model-variant confound, but the clip-level overlap is the more load-bearing threat to the benchmark's validity.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MAVOS-DD, a large-scale multilingual audio-video deepfake detection benchmark comprising 60,364 real and synthetic videos in eight languages, generated with seven modern deepfake methods (talking-head, portrait animation, and face swapping). The authors define training, validation, and test splits to create four evaluation scenarios: in-domain, open-set model, open-set language, and open-set full. They evaluate three recent detectors (AVFF, MRDF, TALL) in both pre-trained and fine-tuned form, reporting accuracy, AUC, and mAP. The central empirical claim is that state-of-the-art detectors degrade substantially when tested on unseen generators or languages, with the best fine-tuned detector (AVFF) dropping from 86.93% in-domain accuracy to 75.34% on the open-set model split and 77.68% on the open-set full split. The data and code are publicly released.","tokens_in":14349,"tokens_out":2233,"duration_ms":22044,"significance":"If the reported results are valid, MAVOS-DD is a valuable community resource: it is the largest multilingual audio-video deepfake benchmark with explicit open-set train/test splits, and it provides a concrete stress test that separates in-domain proficiency from cross-generator and cross-language generalization. The benchmark design, with seven generators spanning three generative tasks and a relatively balanced language distribution, addresses a real gap in existing datasets. The paper also ships a reproducible evaluation protocol (public data, code, and fine-tuning hyperparameters) and applies McNemar's test for the fine-tuning comparison, which strengthens the empirical section. The central qualitative finding—that detectors fine-tuned in-domain lose performance in open-set settings—is supported by the reported numbers, provided the splits are cleanly disjoint at the content level, which is not yet explicitly established.","major_comments":[{"comment":"The open-set generalization claims require that the train/validation/test splits are disjoint at the level of the underlying real video clips and their derived fake videos. The paper does not state this disjointness. Sections 3 states that talking-head generation uses audio that 'originates from the real video set', LivePortrait uses 'source videos are represented by the real videos collected from YouTube', and face swaps use 'target videos selected from the collected set of real YouTube videos', with source identities including 'identities found in our real videos'. Because Table 2 defines splits only by language and generator membership, a real clip (or a fake video derived from it) could appear in both the training set and the open-set test sets. If a detector sees the original real clip or its near-duplicate fake during fine-tuning, the reported drops, especially in the open-set model and open-set full scenarios, could partly reflect content leakage rather than generalization failure. The authors must either confirm that all splits are disjoint at the source/target video level, or revise the benchmark and the headline claims accordingly.","section":"Section 3, 'Deepfake videos' and Table 2"},{"comment":"The per-language fine-tuning of EchoMimic changes the identity of the generator across languages. The text says that EchoMimic is 'individually fine-tune[d] on additional languages, such as Romanian and Arabic' before generation, with 1,000 real videos per language and 10 epochs. If EchoMimic is also fine-tuned on German and Hindi (the held-out languages) before generating their fakes, then the open-set language test does not present a single EchoMimic generator unseen at training time; it presents language-specific variants of EchoMimic. This confounds the 'open-set language' condition with a generator-variant condition. The authors should specify exactly which EchoMimic variant was used for each language and, if held-out languages used a differently fine-tuned model, report whether results change when excluding EchoMimic from the open-set language analysis.","section":"Section 3, 'Deepfake videos'"},{"comment":"The paper claims that '60% of data being generated', but Table 2 gives 35,169 fake videos out of 60,364 total, which is 58.3%. The 60% figure appears to be a rounding that could mislead readers; the exact fraction should be reported in the abstract and Section 3, or the dataset construction should be adjusted so the stated number matches.","section":"Abstract and Section 3"},{"comment":"The benchmark's authenticity assumption for real videos is reasonable but should be stated more carefully. The real videos are scraped from YouTube and filtered by TalkNet and a face detector; there is no guarantee that some 'real' clips are not themselves deepfakes or contain manipulation. Given that the central claim is about generalization, the authors should acknowledge this as a residual risk and provide at least a basic manual or automated sanity check (e.g., a small annotation study) to estimate the contamination rate.","section":"Section 3, 'Real videos'"}],"minor_comments":[{"comment":"There is a typo in 'MA VOD-DD' in the sentence 'To this end, MA VOD-DD contains samples ...' — should be 'MAVOS-DD'.","section":"Section 2, paragraph after Table 1"},{"comment":"The phrase 'three state-of-the-art deepfake detectors' is written as 'thee state-of-the-art'; please fix the typo.","section":"Section 4, 'Baselines and hyperparameters'"},{"comment":"The figure would benefit from explicitly indicating which real videos are used as sources/targets for fake generation, to make the generation pipeline transparent and to support the split-disjointness claim.","section":"Figure 1"},{"comment":"The broader impact section focuses on hardware limitations and demographic bias, but it would be helpful to also mention the potential for benchmark-specific overfitting: detectors fine-tuned on MAVOS-DD may not generalize to other data, which is a general limitation of any benchmark.","section":"Section 5, 'Broader Impact and Limitations'"},{"comment":"The ethical statement mentions a license and user opt-out, but does not address consent for using individuals' faces in the dataset; consider discussing how identity rights are handled beyond the right-to-be-forgotten request.","section":"Ethical Statement"}],"recommendation":"major_revision","confidential_remarks":"The paper is a useful resource and the experiments are reproducible in principle, but the clip-level disjointness of the splits is the load-bearing assumption for the headline open-set generalization claim. The authors need to provide the source/target CLIP identifiers for each fake video or otherwise demonstrate that no training-test content overlap exists. If the overlap is confirmed to be absent, the remaining issues (EchoMimic variant, 58.3% vs 60%) are fixable in revision. I do not see evidence of circularity; the benchmark is independent of the evaluated detectors."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe useful thing here is the resource: a 250-hour, eight-language audio-video deepfake set with seven generation methods and three open-set test scenarios. That is genuinely new scale and language coverage. The empirical headline—fine-tuned AVFF drops from 86.93% in-domain to 75.34% on open-set model and 77.68% on open-set full—repeats across all three detectors, so the qualitative conclusion that current detectors degrade in open-set settings is probably right.\n\nThe soft spot is more than structural. The stress-test note is correct: fakes are generated from the same real YouTube pool that is later partitioned into train/val/test. Talking-head audio comes from the real video set, face-swap targets are selected from the collected real videos, and LivePortrait source videos are the real videos. The paper never confirms that a real clip used to produce a test fake is absent from the training real set, nor that the fake itself is not in train. Since splits are defined only by language and generator membership, exact-video-content leakage is a live possibility. That would not just bend the numbers; it would undermine the benchmark's claim to measure open-set generalization. The authors should check overlap at the clip level and re-partition if necessary. This is the main thing I would ask for in review.\n\nOther caveats are minor by comparison. EchoMimic is fine-tuned per language, so the generator changes identity across languages; that is a confound for the language split but not fatal. The 60% generated claim is actually 58.3% by the paper's own counts. The results are single-run with no error bars. Code release is not evidenced beyond the dataset link. None of these are disqualifying.\n\nWhat I do want to credit: the language balance is much better than Deepfake-Eval-2024 (which is 80% English), the generation methods are modern and cover three tasks, and the open-set protocol is a sensible way to stress-test detectors. The baseline methods are public, so the evaluation is reproducible in principle. The paper is also honest about limitations like demographic bias.\n\nWho is this for? Anyone working on deepfake detection or audio-visual forensics. It deserves a serious referee, but the referee should treat the open-set numbers as provisional until the overlap question is settled. I would still cite the dataset as a resource, with a caveat.\n\nRecommendation: send to peer review, and make the overlap analysis a condition of acceptance.","headline":"Solid dataset, but the open-set splits may leak clip-level overlap; the generalization numbers need a disjointness check before they can be trusted.","tokens_in":14888,"tokens_out":4205,"would_cite":true,"duration_ms":41059,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces MAVOS-DD, a 250-hour, eight-language deepfake benchmark whose open-set splits show that state-of-the-art audio-video detectors lose substantial accuracy on unseen generators and languages.","keywords":["deepfake detection","open-set benchmark","multimodal audio-video","multilingual dataset","generalization","talking-head generation","face swapping","video forensics"],"falsifier":"Rebuild the open-set language test so the same talking-head generator checkpoint is used for all languages and face-swap source identities are excluded from the training real pool; if fine-tuned detector accuracy no longer drops, the claimed open-set failure is an artifact of per-language generator variants and identity overlap rather than genuine generalization.","tokens_in":13949,"feed_emoji":"🎭","tokens_out":8033,"duration_ms":75475,"temperature":0.7,"pith_summary":"MAVOS-DD is a new benchmark built to test whether audio-video deepfake detectors can recognize fakes produced by generators and in languages they never saw during training. It contains more than 60,000 videos and 250 hours of content across eight languages, with fakes made by seven different generation models spanning talking-head synthesis, portrait animation, and face swapping. The paper's central finding is a generality gap: fine-tuned detectors score high in-domain, but lose a meaningful slice of accuracy on open-set splits, most sharply when generators are held out. The reason a reader should care is that real-world deepfake encounters are open-set by definition, so benchmarks that only test matched generators overstate a detector's practical value.","feed_headline":"Deepfake detectors fall hard on new languages and models","feed_subtitle":"A 250-hour, eight-language open-set benchmark shows fine-tuned detectors lose up to 11 accuracy points outside training data.","key_machinery":"The machine that carries the argument is the benchmark itself, MAVOS-DD, together with its four-way test split. Training and validation cover six languages and four generators; the in-domain test samples the same distribution, while open-set model adds three held-out generators, open-set language adds two held-out languages, and open-set full combines both. This split is what turns 'does the detector work?' into 'does the detector generalize?' by holding the training recipe fixed and varying only what is unseen at test time. The seven generation methods span talking-head synthesis, portrait animation, and face swapping, giving the open-set splits a realistic mix of forgery types.","core_discovery":"The paper's central claim is that MAVOS-DD is the first large-scale open-set benchmark for multilingual audio-video deepfake detection, and that state-of-the-art detectors cannot hold their performance under its open-set conditions. The evidence is a measured degradation: the strongest fine-tuned detector reaches 86.93% accuracy on the in-domain test set but 75.34% when fakes from unseen generators are added and 77.68% on the combined open-set full set; the other detectors show similar or larger relative declines. The paper argues this gap demonstrates that current models learn generator- and language-specific shortcuts rather than a robust notion of authenticity.","pith_inferences":["The paper does not isolate whether the open-set language drop comes from language itself or from per-language generator checkpoints; a single generator checkpoint across all languages would separate the two.","Because face-swap source identities are drawn partly from the same real-video pool used in training, part of in-domain accuracy may reflect identity familiarity; a disjoint-identity split would test this.","The benchmark's open-set protocol could be extended to timestamp-level localization, which would reveal whether detectors rely on genuine artifacts rather than clip-level biases.","The near-chance performance of pre-trained detectors suggests that many existing public checkpoints are tuned to generator-specific traces; retraining with method augmentation or feature-level noise may close some of the gap."],"forward_implications":["Fine-tuned detectors that exceed 90% accuracy on in-domain data fall to 75 to 79% accuracy on open-set model and full splits, so accuracy reported on matched generator and language benchmarks overstates deployment readiness.","The largest drops occur when unseen generators are introduced, indicating detectors latch onto generator-specific cues instead of a general forgery signal.","Unimodal video analysis lags behind audio-visual fusion on this benchmark, supporting the design of detectors that combine both modalities.","Pre-trained, off-the-shelf detectors perform near chance on MAVOS-DD, meaning the generated fakes are realistic enough to challenge existing public models.","MAVOS-DD provides a reusable stress test for future detectors and is designed to be extended with new generators as they are released."],"supporting_citations":[{"why":"It is the large-scale prior video deepfake dataset used in the comparison table to anchor MAVOS-DD's scale and open-set novelty.","marker":"[18]"},{"why":"It is the concurrent in-the-wild multimodal deepfake benchmark used as the main comparison for multilingual coverage and fake-content volume.","marker":"[7]"},{"why":"It is the earlier audio-video deepfake dataset used to contrast the new benchmark's open-set and multilingual design.","marker":"[37]"},{"why":"It supplies the strongest fine-tuned detector, whose in-domain accuracy of 86.93 percent falls to 75.34 percent on the open-set model test.","marker":"[52]"},{"why":"It is the second multimodal detector; its additional false alarms on held-out languages support the paper's language-as-artifact observation.","marker":"[80]"},{"why":"It is the unimodal video detector used to demonstrate the accuracy advantage of combining audio with video.","marker":"[71]"},{"why":"It is the talking-head generator whose per-language fine-tuning affects how the open-set language split should be interpreted.","marker":"[9]"},{"why":"It is the portrait-animation generator whose generated samples the error analysis finds are mislabeled as real 80 percent of the time.","marker":"[25]"},{"why":"It is a face-swap generator held out of training, forming part of the open-set model and open-set full test obstacles.","marker":"[68]"},{"why":"It is a talking-head generator included in the training set and in the in-domain and open-set language content.","marker":"[75]"}],"fun_headline_variants":["New deepfake benchmark exposes detectors' open-set blind spots","Open-set deepfake detection: detectors struggle on unseen languages","First multilingual open-set deepfake test: detectors drop on new generators","Open-set deepfake: state-of-the-art detectors drop 11 points on unseen fakes","Open-set multilingual deepfake benchmark: detectors can't keep up with unseen fakes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result depends on the real YouTube videos being genuinely authentic and the generated fakes, made with specific models and source identities, being representative enough that the drop on held-out languages and generators is due to generalization rather than to dataset artifacts.","fun_headline_variants_meta":{"raw":{"variants":["New deepfake benchmark exposes detectors' open-set blind spots","Open-set deepfake detection: detectors struggle on unseen languages","First multilingual open-set deepfake test: detectors drop on new generators","Open-set deepfake: state-of-the-art detectors drop 11 points on unseen fakes","Open-set multilingual deepfake benchmark: detectors can't keep up with unseen fakes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001509,"raw_usage":{"total_tokens":5993,"prompt_tokens":833,"completion_tokens":5160,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":449,"completion_tokens_details":{"reasoning_tokens":5065}},"tokens_in":449,"tokens_out":5160,"duration_ms":36075,"temperature":1.0,"reasoning_tokens":5065,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:56:55.042987+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rebuild the open-set language test so the same talking-head generator checkpoint is used for all languages and face-swap source identities are excluded from the training real pool; if fine-tuned detector accuracy no longer drops, the claimed open-set failure is an artifact of per-language generator variants and identity overlap rather than genuine generalization.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It is the earlier audio-video deepfake dataset used to contrast the new benchmark's open-set and multilingual design."},{"cited_title":"A VFF: Audio-Visual Feature Fusion for Video Deepfake Detection","cited_arxiv_id":null,"evidence_quote":"It supplies the strongest fine-tuned detector, whose in-domain accuracy of 86.93 percent falls to 75.34 percent on the open-set model test."},{"cited_title":"Cross- modality and within-modality regularization for audio-visual deepfake detection","cited_arxiv_id":null,"evidence_quote":"It is the second multimodal detector; its additional false alarms on held-out languages support the paper's language-as-artifact observation."},{"cited_title":"TALL: Thumbnail Layout for Deepfake Video Detection","cited_arxiv_id":null,"evidence_quote":"It is the unimodal video detector used to demonstrate the accuracy advantage of combining audio with video."},{"cited_title":"EchoMimic: Life- like Audio-Driven Portrait Animations through Editable Landmark Conditions","cited_arxiv_id":null,"evidence_quote":"It is the talking-head generator whose per-language fine-tuning affects how the open-set language split should be interpreted."},{"cited_title":"HifiFace: 3D Shape and Semantic Prior Guided High Fidelity Face Swapping","cited_arxiv_id":null,"evidence_quote":"It is a face-swap generator held out of training, forming part of the open-set model and open-set full test obstacles."}],"review_version":1}