{"id":"826125d4-fcfb-4675-98f5-ad470784d87c","arxiv_id":"2608.09593","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"MADBench introduces a component-aware audio-visual deepfake benchmark with independently manipulated speech and environmental audio, and shows environmental manipulation is easier to detect than synthetic speech.","lead":"MADBench is a new benchmark that treats speech and background audio as separate components that can be faked independently in otherwise real videos. It shows that current deepfake detectors fail on both components, while general audio-visual encoders detect environmental audio manipulation better than fake speech.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Real and fake components are not on equal footing: real components are MossFormer2 outputs while fake components are clean generations, so separation artifacts can serve as a shortcut label and confound the reported component-level findings.","rationale":"The reader's weakest assumption correctly identifies the load-bearing risk: the real components are MossFormer2 outputs, whereas fake components are clean synthetic generations that never pass through the separation model. For MADBench's central claim to hold, the component labels must reflect manipulation rather than differences in how real and fake audio are produced. The current design controls the mixing procedure but not the source-side asymmetry, so the reported findings—especially the environmental-audio advantage and the G_S interference gap—could be driven by separation artifacts. The paper's QC validates absence of speech leakage and presence of environmental energy, but it does not validate that the residual is artifact-free; separation models optimized for speech reconstruction can leave structured artifacts in the residual by construction. This is a concrete, testable confound, and the proposed separation-symmetric control would settle it. Other concerns, such as missing error bars and the dataset not yet being released, are real but secondary; they affect reliability and reproducibility rather than the construct validity of the benchmark. The conditional verdict remains appropriate until the symmetric-pipeline check is performed.","tokens_in":17404,"tokens_out":5876,"duration_ms":62688,"concrete_test":"Construct a separation-symmetric control: for each test fake sample, mix the chosen fake speech and fake environmental components, run the same MossFormer2 model used in Section 3.2 on that mixture, and use the separated output (or its remix) as the fake components, so that real and fake components both undergo identical separation. Recompute the component-level AUCs and interference gaps (Δ_E−S and G_S) for the three frozen A-V encoders in Table 3. If Δ_E−S or G_S drops by more than about 0.05 AUC relative to the published values, the reported environmental advantage and asymmetrical interference are at least partly separation-artifact confounds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's central claim—that component labels isolate manipulation rather than construction artifacts—depends on real components being faithful. Section 3.2 defines the environmental component as the MossFormer2 residual e = x − s_hat, and the speech component as the model output s_hat. Fake components (Sections 3.3–3.4) are generated cleanly and are never passed through the same separation model; fake speech even leaves non-speech regions silent. Section 3.5 controls mixing via source-relative RMS normalization, but it does not control this generation asymmetry. MossFormer2 is a monaural speech estimator, so its estimation error is not optimized for the residual: e_hat contains separation artifacts such as musical noise, spectral holes, and suppressed non-speech events that are absent from clean generated environmental audio. The QC checks (ASR/VAD/energy) rule out gross speech leakage but do not rule out these artifacts. Detectors can therefore separate real from fake by detecting 'separation artifact present' instead of 'manipulation present.' This would inflate the reported environmental advantage (Δ_E−S in Table 3) and the interference gap G_S, making the headline findings an artifact of the pipeline rather than a property of environmental sound deepfakes. Because the same asymmetry underlies every real-versus-fake comparison, both the validity of MADBench as a benchmark and the specific empirical findings rest on this assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MADBench, a benchmark for audio deepfake detection in authentic-video settings where speech and environmental audio are treated as separate forgery components. The dataset is built from AVSpeech clips, with speech separated using MossFormer2 and the residual treated as environmental audio; fake speech is generated via TTS and voice conversion, and fake environmental audio via TTA, VTA, and ATA generation, producing real, speech-fake, environment-fake, and joint-fake variants under scene-matched and scene-mismatched protocols. The authors evaluate pretrained A-V detectors, frozen A-V encoders, audio-only baselines, and zero-shot omni models, reporting that pretrained detectors and omni models perform near chance, frozen A-V encoders transfer well, environmental manipulation is easier to detect than speech, and fake environmental audio asymmetrically degrades speech detection.","tokens_in":17597,"tokens_out":10578,"duration_ms":96070,"significance":"If the construction asymmetry were removed, MADBench would fill a genuine gap: no existing benchmark independently controls speech and environmental audio over authentic video, and the component-level protocol plus leakage controls (speaker-level splits, fixed video, balanced generation) are thoughtful. The finding that frozen A-V encoders provide transferable component-level cues while task-specific detectors do not is potentially useful, and the paper's detailed QC and leakage controls are strengths. However, the central empirical claims are currently undermined because real components are MossFormer2 separation outputs while fake components are clean generations, so reported differences may reflect separation artifacts rather than manipulation. The paper is therefore not yet a rigorous foundation for component-aware audio deepfake detection, but the issue appears addressable with a substantial revision.","major_comments":[{"comment":"The construction treats real and fake components asymmetrically. In Section 3.2, the real speech component is MossFormer2's output s_hat and the real environmental component is the residual e_hat = x - s_hat. In Sections 3.3 and 3.4, fake speech and fake environmental components are generated cleanly and are never passed through the same separation model. Consequently, every real component is a separation output while every fake component is not, so a detector can separate real from fake by detecting the presence or absence of separation artifacts (e.g., musical noise, spectral holes, leakage) rather than manipulation. The QC checks described in Section 3.2 (ASR coverage, VAD/ASR absence of speech, minimum energy) rule out gross speech leakage but do not rule out artifact-level shortcuts. Because this asymmetry affects every real-versus-fake comparison in Tables 2 through 5, the validity of MADBench as a benchmark and the specific findings (e.g., the environmental advantage ΔE−S and the interference gaps G_S in Table 3) rest on an untested assumption.","section":"Section 3.2 and 3.5"},{"comment":"The paper states that “all sample types share an identical construction process, so that labels reflect component differences rather than assembly differences,” but this is inaccurate. The mixing procedure is identical, but the component-generation process is not: real components come from MossFormer2 separation, while fake components are clean generations. The claim should be revised, and the authors should either (a) pass fake components through the same separation pipeline so that both classes receive identical processing, or (b) provide a control experiment showing that detectors cannot distinguish real from fake on the basis of separation artifacts alone (e.g., train a detector to separate the residual e_hat from clean generated environmental audio, or s_hat from clean TTS, and show that this detector does not transfer to MADBench). Without such a control, the paper's central claims are not supported.","section":"Section 3.5"},{"comment":"The reported environmental-audio advantage and interference gap G_S are confounded by the construction asymmetry. For example, G_S compares AUC(S vs. R) with AUC(Q vs. E). In S vs. R, both groups contain real (separated) environmental audio; in Q vs. E, both groups contain fake (clean) environmental audio. The detector's ability to exploit the presence or absence of separation artifacts differs between these two set pairs, so the gap does not isolate interference from manipulation. A positive G_S could simply reflect the model using the environmental component's processing history as a shortcut. The authors should re-analyze these quantities on a sympathetically processed subset before making the interference claim.","section":"Table 3 and Section 5.2"}],"minor_comments":[{"comment":"The table header does not clearly separate the columns for ΔE−S and G_S/G_E; the S and E rows contain different numbers of numeric entries, making it hard to determine which value belongs to which metric. Please reformat the header and clarify the caption.","section":"Table 3 caption"},{"comment":"The description that fake speech is “placed into the original speech regions, with non-speech regions left silent” creates another asymmetry: the real speech stem from MossFormer2 may contain residual noise outside speech regions, while the fake speech component is silent there. Although mixing with an environmental component may partially mask this, it should be acknowledged and controlled.","section":"Section 3.3"},{"comment":"For scene-mismatched VTA and ATA samples, the paper states that the generated track is replaced with one from an incompatible scene rather than being generated for that mismatch. Please clarify whether the generation-branch balance reported in Table 1 reflects the final samples or the initially generated components, since the replacement changes the effective branch distribution.","section":"Section 3.4"},{"comment":"The paper does not state where the dataset and benchmark code will be released. Please include a repository or availability statement for reproducibility.","section":"General"},{"comment":"The text notes that the evaluated AudioMosaic checkpoint is fine-tuned on the EnvSDD TTA split, which is important context. Please add this information to the table caption so that readers do not mistake it for a frozen pretrained model.","section":"Table 5"},{"comment":"The conclusion states that MADBench establishes a rigorous foundation, but given the construction asymmetry discussed above, this claim is too strong for the current version. A limitation paragraph should be added, or the claim should be softened.","section":"Section 6"}],"recommendation":"major_revision","confidential_remarks":"This is a carefully engineered dataset paper with strong leakage controls and a useful evaluation protocol. The main issue is the asymmetric processing of real and fake components, which is a load-bearing confound for all empirical findings. The issue is fixable in principle by regenerating fake components through the same separation pipeline or by providing a control experiment that rules out separation-artifact shortcuts, but the current manuscript cannot support its central claims without such work. I would not reject the paper outright, but the revision needs to be substantive."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know: MADBench is the first benchmark to treat speech and environmental audio as separately manipulable components over authentic video. That is a real gap, and the construction is thoughtful: fixed video across all variants, balanced generator coverage, speaker-cluster splits, and sensible QC. The evaluation is broad—pretrained A-V detectors, frozen encoders, and omni models—and the finding that frozen A-V encoders transfer while dedicated detectors fail is worth checking.\n\nThe novelty is legitimate relative to EnvSDD, Compspoof, and VCapAV. The scene-consistency axis and component-level annotation are useful, and the paper gives enough detail to reproduce the pipeline. Credit where it is due: this is a well-structured benchmark.\n\nThe load-bearing flaw is the asymmetry between real and fake components. Real speech and real environmental audio are MossFormer2 outputs; fake components are clean TTS/VC or generated audio, never passed through the separation model. So \"real\" in MADBench is not an original recording but a separation artifact. A detector can label real vs. fake by detecting musical noise, spectral holes, or residual speech leakage in the separation output rather than by detecting manipulation. The paper says all sample types share an identical construction process, but that refers to reassembly and RMS mixing, not the generation path. This confound affects every real-vs-fake comparison, including the reported environmental advantage (Δ_E−S) and the interference gap G_S. The QC checks rule out gross speech leakage but not separation artifacts.\n\nSecondary issues: no error bars or significance tests on the AUC differences, and the dataset is not released. Both are fixable and not fatal.\n\nThe benchmark deserves a serious referee. I would ask the authors for a control: can a model distinguish original mixtures from reassembled real samples? Or pass fake components through the same separation pipeline? Without that, the empirical claims are provisional; with it, MADBench is a solid, citable contribution. Accept with major revisions, conditional on the control and on releasing data or a clear timeline.","headline":"A genuinely new component-aware audio-visual benchmark with a thoughtful design, but the real/fake component asymmetry lets detectors shortcut on separation artifacts, so the headline findings are provisional until a control is added.","tokens_in":18183,"tokens_out":3499,"would_cite":false,"duration_ms":33949,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MADBench is the first audio deepfake benchmark that treats speech and background audio as independent manipulable components, and its results show that fake background audio is easier to detect than synthetic speech while also degrading…","keywords":["audio deepfake detection","environmental audio","speech synthesis","component-level evaluation","audio-visual benchmark","scene consistency","multimodal large language models","deepfake detection"],"falsifier":"Train or run any detector on the task of distinguishing original unseparated real clips from the benchmark's reassembled real clips (real speech stem plus real environmental residual re-mixed through the same pipeline). If it exceeds chance, separation artifacts are present and the construction is asymmetric; applying the same check with fake components passed through an identical separation step would test whether the reported environmental-audio advantage survives when both sides see the same processing.","tokens_in":17155,"feed_emoji":"🎧","tokens_out":11806,"duration_ms":85714,"temperature":0.7,"pith_summary":"This paper argues that audio deepfakes in real video should be evaluated as two independent acoustic components rather than one undifferentiated stream: the speech track and the environmental soundscape, each with its own generative pipeline and artifact profile. It introduces MADBench, a benchmark built from real videos with a fixed visual stream, in which the speech component, the background component, or both are replaced by generated audio, and every sample is labeled accordingly. Using this testbed, the paper shows that general-purpose audio-visual encoders detect manipulated background audio more reliably than synthetic speech, that existing pretrained deepfake detectors operate near chance on both components, and that a fake background track degrades detection of fake speech. The purpose of the benchmark is to make these cross-component effects visible, rather than collapsing all audio manipulation into a single real/fake label.","feed_headline":"New benchmark splits audio deepfakes into speech and background","feed_subtitle":"MADBench tests each acoustic component separately; background fakes are easier to spot and mask speech forgery cues.","key_machinery":"The load-bearing construction is the modality-component hierarchy: each video's audio stream is decomposed into a speech stem $\\hat{s}$ and an environmental residual $\\hat{e} = x - \\hat{s}$ using speech separation, with mixture consistency requiring the components to sum back to the original waveform. Fake speech is produced by same-identity and cross-identity TTS and by voice conversion; fake environmental audio is produced by text-to-audio, video-to-audio, and audio-to-audio generators, under scene-matched and scene-mismatched conditions. Assembly re-mixes real speech with real environmental audio through the same RMS-normalization pipeline used for fake samples, so the four labels R/S/E/Q differ only in which acoustic component was manipulated. This controlled construction is what lets the benchmark attribute detection differences to the speech or environmental component rather than to visual or assembly differences.","core_discovery":"On its own terms, the paper claims that component-level annotation changes what audio deepfake detection can measure. From each source video it constructs four sample types—real, speech-fake, environment-fake, and joint-fake—with the visual stream held fixed, and reports that pretrained audio-visual deepfake detectors transfer near chance (mean binary AUC 0.518–0.534 even after fitting new heads), while frozen general-purpose audio-visual encoders reach above 0.91 AUC for any-fake detection and support four-way R/S/E/Q attribution. It further reports that environmental audio manipulation is consistently easier to detect than speech manipulation across encoders and protocols, including scene-matched ones, and that fake environmental audio interferes with speech-specific detection (interference gap up to 0.138) while the converse does not hold. The paper interprets these results as evidence that environmental audio is a distinct forensic domain whose manipulation is obscured when benchmarks use a single audio label.","pith_inferences":["Beyond the paper's claims: if the asymmetry holds in the wild, background-audio manipulation is the more dangerous attack surface—it is easier to spot, yet it can hide a simultaneous voice forgery, so practical detectors should flag suspicious ambience even when speech sounds clean.","Because the real samples are re-assembled from separated stems while fake components are not re-separated, a detector that exploits separation artifacts could inflate the reported gaps; this could be tested by separating the fake components too or by training on the original unseparated mixes.","The near-chance zero-shot omni-model results suggest that component-level attribution requires explicit forensic supervision or component-conditioned prompting, both of which MADBench's labels now make trainable.","The scene-matched/mismatched taxonomy is a tunable difficulty axis: future generators could be prompted with the fine-grained scene labels to produce forgeries that close the gap between synthetic and real ambience, stress-testing the benchmark's hardest settings."],"forward_implications":["Pretrained audio-visual deepfake detectors should not be assumed transferable to authentic-video, component-level forgeries; with native scores or fitted heads they remain near chance (mean AUC below 0.54).","Frozen general-purpose audio-visual encoders are strong baselines for this setting, reaching above 0.91 AUC for any-fake detection and macro-F1 up to 0.745 for four-way R/S/E/Q classification.","Manipulated environmental audio is a distinct and more detectable forgery source than synthetic speech, so background-audio synthesis deserves its own evaluation axis and countermeasure research.","Fake background audio can obscure speech-specific forgery cues while the converse does not, so single-label benchmarks overestimate detector robustness in realistic jointly manipulated clips.","Audio alone, not the visual stream, carries the strongest direct forensic signal for manipulation labels; video's main role is scene-consistency verification."],"supporting_citations":[{"why":"Supplies the source video corpus (AVSpeech) of real-world clips with visible speakers and paired audio from which all benchmark samples are built.","marker":"Ephrat et al. 2018"},{"why":"Provides MossFormer2, the separation model that yields the real speech stem and defines the environmental residual as $\\hat{e} = x - \\hat{s}$, the core decomposition underlying the benchmark.","marker":"Zhao et al. 2024"},{"why":"Contributes F5-TTS, one of the same-identity text-to-speech models used to generate fake speech components.","marker":"Chen et al. 2025"},{"why":"Contributes AudioLDM2, used in both text-to-audio and audio-to-audio branches to generate fake environmental components.","marker":"Liu et al. 2024a"},{"why":"Contributes AudioGen, a text-to-audio generator for environmental fakes in the TTA branch.","marker":"Kreuk et al. 2022"},{"why":"One of the three pretrained audio-visual deepfake detectors benchmarked; its near-chance transfer establishes the paper's claim that existing detectors lack component-level cues.","marker":"Cai et al. 2023"},{"why":"ImageBind serves as a frozen audio-visual encoder baseline whose strong transfer supports the claim that general-purpose pretraining captures component-level forgery signals.","marker":"Girdhar et al. 2023"},{"why":"CAV-MAE Sync is the frozen encoder with the strongest environmental detection and the only one that consistently benefits from the correct video in scene-consistency tests.","marker":"Araujo et al. 2025"}],"fun_headline_variants":["Background fakes are easier to spot than fake speech","New benchmark separates speech and audio to expose deepfake blind spots","Fake background audio hides speech forgery cues in deepfake detectors","MADBench: component-aware testing reveals background fakes are more obvious","Audio deepfakes: background manipulation is easier to detect than speech"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's real audio components come from a separation model, and only the fake components are generated cleanly; if that separation leaves detectable artifacts in the real stems, the measured differences between real and fake components could reflect the separation pipeline rather than the forgery itself.","fun_headline_variants_meta":{"raw":{"variants":["Background fakes are easier to spot than fake speech","New benchmark separates speech and audio to expose deepfake blind spots","Fake background audio hides speech forgery cues in deepfake detectors","MADBench: component-aware testing reveals background fakes are more obvious","Audio deepfakes: background manipulation is easier to detect than speech"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00023,"raw_usage":{"total_tokens":1485,"prompt_tokens":951,"completion_tokens":534,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":446}},"tokens_in":567,"tokens_out":534,"duration_ms":5529,"temperature":1.0,"reasoning_tokens":446,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:25:45.442534+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train or run any detector on the task of distinguishing original unseparated real clips from the benchmark's reassembled real clips (real speech stem plus real environmental residual re-mixed through the same pipeline). If it exceeds chance, separation artifacts are present and the construction is asymmetric; applying the same check with fake components passed through an identical separation step would test whether the reported environmental-audio advantage survives when both sides see the same processing.","supporting_citations":[{"cited_title":"ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages=","cited_arxiv_id":null,"evidence_quote":"Provides MossFormer2, the separation model that yields the real speech stem and defines the environmental residual as $\\hat{e} = x - \\hat{s}$, the core decomposition underlying the benchmark."}],"review_version":1}