REVIEW 4 major objections 6 minor 1 cited by
Localizing Audio-Visual Deepfakes via Hierarchical Boundary Modeling
T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read HBMNet, a hierarchical boundary modeling network, localizes audio-visual deepfakes at 96.8 AP@0.5, far ahead of prior baselines.
desk verdict A well-executed engineering paper with a large empirical claim that deserves serious review but needs artifact release and a clarified backward-label description before the result is fully verifiable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The core object is the hierarchical boundary model: a coarse boundary-matching map (built with the Boundary-Matching layer from BMN) over all candidate proposals, combined with a fine-grained bidirectional probability stream for start, end, and content frames (produced by a Nested U-Net, supervised by focal loss, with the input sequence also processed in reverse). The central identity is the fused confidence score $s' = M'_{i,j} \times P_s'(j) \times P_e'(j+i) \times \mathrm{mean}(P_c'(j:j+i))$, which multiplies proposal-level confidence with frame-level boundary-content probabilities and then applies Soft-NMS. What carries the argument is the claim that these two scales and two directions are complementary: proposal cues give global context and recall, frame-level cues give boundary precision, and bidirectional processing captures both real-to-fake and fake-to-real transitions.
What would settle it
Re-running HBMNet against UMMAFormer on the full AV-Deepfake-1M test set, or on a new independently sampled 8,000-video subset with the same composition as the benchmark, and finding that the AP@0.5 gap collapses from roughly 21 points to near zero or reverses would falsify the claim of consistent superiority.
Extended reading notes
Core claim
The paper's central claim is that temporal deepfake localization is best treated as two complementary problems: finding plausible manipulated segments at the proposal level and pinning down exact boundaries at the frame level, with both directions of each boundary transition taken into account. Concretely, the Audio-Visual Feature Encoder uses an SENet audio encoder and a 3D-CNN plus ResNet-18 visual encoder, followed by audio-to-visual and visual-to-audio cross-attention and a self-attention fusion block, supervised by a frame-level contrastive loss that pulls real-frame cross-modal features together and pushes fake ones apart. The Coarse Proposal Generator produces a boundary-matching confidence map over all start times and durations, and the Fine-grained Probabilities Generator runs a Nested U-Net over both forward and reversed sequences to output start, end, and content probabilities, fused at inference by geometric mean. The final score multiplies coarse proposal confidence with fine-grained boundary-content probabilities, and Soft-NMS removes redundant proposals. On the benchmark subset, the model reports 96.82 AP@0.5 versus 75.93 for UMMAFormer and 38.89 for BA-TFD, with recall of 88.83 AR@50 versus 78.31 and 42.56; ablations show audio-visual fusion mainly raises precision, frame-level supervision raises recall, and bidirectionality lifts strict-threshold precision.
Load-bearing premise
The results rest on the assumption that the 8,000-video sampled subset of AV-Deepfake-1M, drawn with a fixed random seed but without a reported breakdown of speakers, manipulation types, or segment lengths, fairly represents the benchmark and that the BA-TFD and UMMAFormer baselines were reproduced faithfully on that same subset.
Editorial extensions
If this is right
- If HBMNet's results hold, audio-visual deepfake localization can be done with much higher precision and recall than the prior state of the art on partial manipulations.
- The architectural decomposition suggests that adding hierarchical boundary modeling (two scales plus bidirectionality) is a transferable recipe beyond this specific network.
- The scaling experiment indicates that the model's gains grow with more training data, suggesting it is not saturated on the sampled subset.
- The ablations isolate distinct contributions: encoding and fusion for precision, frame-level supervision for recall, and bidirectionality for strict-threshold precision, so future systems can choose components based on which metric matters.
Reading between the lines
- If the performance gap persists on the full benchmark with proper stratification, then hierarchical boundary modeling may become a default component in audio-visual deepfake localization, similar to its established role in temporal action localization.
- The bidirectional boundary-content design could transfer to uni-modal partial deepfake audio detection, where only one modality is available and boundaries may be abrupt.
- A testable extension is to evaluate HBMNet on longer continuous videos with rare, very short manipulations, which would stress whether the coarse proposal map remains tractable and whether the recall gains persist.
- The absolute numbers depend on the fixed 8,000-video subset, so releasing its composition or running on the full AV-Deepfake-1M dataset would tell whether the reported margins are stable.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes HBMNet, a three-module network for audio-visual temporal deepfake localization on AV-Deepfake-1M. The audio-visual feature encoder uses dedicated backbones, cross-attention/self-attention fusion, and a frame-level contrastive loss; the coarse proposal generator outputs a boundary-matching confidence map; the fine-grained probabilities generator predicts frame-level start/end/content probabilities in both forward and backward directions. On a self-sampled 8,000-video training subset, HBMNet is reported to achieve 96.82 AP@0.5 and 88.83 AR@50, substantially above BA-TFD (38.89 AP@0.5) and UMMAFormer (75.93 AP@0.5). Ablations show that encoding and fusion improve precision, frame-level supervision improves recall, and CPG/FPG combination and bidirectional training are complementary. The paper also reports consistent improvements when training data scale increases from 0.33x to 3x of the subset.
Significance. If the empirical results hold, HBMNet would represent a substantial advance in a difficult task, and the decomposition of contributions (precision vs. recall, coarse vs. fine temporal scales, uni- vs. bidirectional) is informative and internally consistent. The use of three resamples is a positive reproducibility practice. However, the central empirical claim is currently unverifiable because the evaluation subset is not characterized, baseline reproductions are not documented, the bidirectional loss/fusion mechanism is underspecified, and no code or split files are released. These issues are fixable within the scope of a revision.
major comments (4)
- [Experimental Setup] The paper reports results on a self-selected 8,000-video training subset (with 1,000 validation and 2,000 test videos) but does not describe its composition—speaker overlap, manipulation-type balance, segment-length distribution, or real/fake ratio—and the appendix referenced for these details is not included in the manuscript. This is load-bearing because localization metrics are strongly sensitive to segment-length distribution: the CPG caps proposal duration at L=60 and T=512, so a subset enriched in short manipulations would advantage HBMNet over baselines that do not use the same cap. Without the subset metadata or released split files, the reported large gains over BA-TFD and UMMAFormer cannot be verified as transferable to the full AV-Deepfake-1M benchmark.
- [Experimental Setup] The manuscript states that BA-TFD and UMMAFormer were reproduced from official repositories but gives no protocol details (input length, sampling rate, epoch count, loss weights, or whether their hyperparameters were re-tuned on the subset). Since these baselines have their own temporal modeling assumptions, and the reported numbers may depend on the subset sampling, the absence of a detailed reproduction setup leaves open the possibility that the comparison is not apples-to-apples. Please report the exact adaptation steps and, ideally, release the evaluation scripts.
- [Deepfake Localization Loss on FPG / Eqs. (7)-(8)] The bidirectional loss and fusion are underspecified. Eq. (7) applies the same label sequence P_b to the backward predictions ←P_b' as to the forward predictions, but the text never states whether the backward labels are time-reversed and start/end swapped to match the reversed input. Similarly, Eq. (8) fuses forward and backward predictions by element-wise geometric mean without specifying whether the backward sequence is reversed back to original temporal order before fusion. Without these definitions, the 'Flip' improvement in Table 5 cannot be attributed to genuine fake-to-real boundary learning, and the method cannot be reproduced. Please specify the backward label construction and the temporal alignment in Eqs. (7)-(8).
- [Related Work] Related Work names Vigo and Maya as top-performing frame-level methods from the AV-Deepfake-1M Challenge 2024, but these baselines are not included in Table 1. Given that these systems reportedly outperform UMMAFormer on the benchmark, omitting them makes it difficult to judge whether HBMNet advances the state of the art; please add them to the comparison or justify their exclusion.
minor comments (6)
- [Problem Formulation] The text contains blank cross-references ('Section .') in the Problem Formulation and Motivation sections; these should be resolved to the actual section numbers.
- [Deepfake Localization Loss on FPG] Eq. (6) is not self-consistent: β is defined as a balanced weight β0·P + (1-β0)(1-P) and also described as the focusing parameter, the exponent β1 is undefined, and the sigmoid σ(·) is mentioned but not used in the formula. Please correct the notation.
- [Results and Discussions] The header of Table 2 appears misaligned: the columns 'Con Our' are ambiguous, and the legend does not define the symbols in the last column. Please reformat the table.
- [Can HBMNet benefit from more data?] The scaling experiment in Table 6 is conducted on 0.33x–3x of the hand-sampled subset, not on the full AV-Deepfake-1M dataset; the claim of 'potential scalability' should be qualified accordingly.
- [Model Training] There is a grammatical error in the sentence 'The loss trains HBMNet on learns bidirectional temporal dependencies for boundary localization'; please rephrase.
- [Experimental Setup] The phrase 'See Appendix for dataset and HBMNet model configuration details' points to an appendix that is not present in the manuscript; please include the appendix or remove the pointer.
Circularity Check
No circularity: HBMNet's claims are empirical, evaluated against an external benchmark with held-out test data, and no prediction reduces to a fitted input or self-citation by construction.
full rationale
The paper's central claims are empirical: HBMNet outperforms BA-TFD and UMMAFormer on the AV-Deepfake-1M benchmark (Table 1). The evaluation uses a held-out test subset of an external benchmark, baselines are reproduced from official repositories, and the reported numbers are evaluated with standard AP/AR metrics against ground-truth segment annotations. The training losses in Eqs. (1)-(7) supervise network outputs with labels derived directly from ground-truth segments; they do not encode the evaluation metric as a training target in a way that would force the reported result. Hyperparameters such as alpha=0.1 and L=60 are tuned on the validation subset, which is standard practice and not circular. The self-citations to the authors' prior work (Chen et al. 2023, 2024) are used only to justify adopting a cross-attention architecture, not to establish the paper's empirical superiority, so they are not load-bearing for the main claim. Concerns about the undocumented composition of the 8,000-video training subset, baseline reproduction fidelity, and the underspecified backward-label handling in Eq. (7) are legitimate verifiability and correctness risks, but they are not instances of circular derivation: no fitted parameter is renamed as a prediction, no target result is assumed inside its own derivation, and no uniqueness theorem is imported from the authors' prior work. The derivation chain is therefore self-contained in the sense relevant to circularity analysis, and the appropriate score is 0.
Assumptions & free parameters
free parameters (5)
- alpha (contrastive loss weight) =
0.1
- Max proposal duration L =
60 frames
- Max input frames T =
512 frames
- Boundary ground-truth interval d_f =
not specified
- Focal loss balancing parameters =
not specified
assumptions (3)
- domain assumption AV-Deepfake-1M frame-level annotations are correct and temporally aligned.
- domain assumption The 8k/1k/2k sampled subset is representative of the full AV-Deepfake-1M benchmark.
- ad hoc to paper Reversed audio-visual inputs carry equivalent frame-level supervision and useful complementary cues.
Cite this review
Pith. "Pith review of Localizing Audio-Visual Deepfakes via Hierarchical Boundary Modeling." pith.science (2026). https://pith.science/paper/PB6T43WH
@misc{pith2026250802000,
author = {Pith},
title = {Pith review of: Localizing Audio-Visual Deepfakes via Hierarchical Boundary Modeling},
year = {2026},
howpublished = {\url{https://pith.science/paper/PB6T43WH}},
note = {Machine review of arXiv:2508.02000}
}
read the original abstract
Audio-visual temporal deepfake localization under the content-driven partial manipulation remains a highly challenging task. In this scenario, the deepfake regions are usually only spanning a few frames, with the majority of the rest remaining identical to the original. To tackle this, we propose a Hierarchical Boundary Modeling Network (HBMNet), which includes three modules: an Audio-Visual Feature Encoder that extracts discriminative frame-level representations, a Coarse Proposal Generator that predicts candidate boundary regions, and a Fine-grained Probabilities Generator that refines these proposals using bidirectional boundary-content probabilities. From the modality perspective, we enhance audio-visual learning through dedicated encoding and fusion, reinforced by frame-level supervision to boost discriminability. From the temporal perspective, HBMNet integrates multi-scale cues and bidirectional boundary-content relationships. Experiments show that encoding and fusion primarily improve precision, while frame-level supervision boosts recall. Each module (audio-visual fusion, temporal scales, bi-directionality) contributes complementary benefits, collectively enhancing localization performance. HBMNet outperforms BA-TFD and UMMAFormer and shows improved potential scalability with more training data.
Figures
Forward citations
Cited by 1 Pith paper
-
Towards Generalized Source Tracing for Codec-Based Deepfake Speech
SASTNet, which fuses Whisper semantic features with Wav2Vec2 and AudioMAE acoustic features, improves source tracing for codec-based deepfake speech on CodecFake+, while exposing that prior models overfit to silence.
Reference graph
Works this paper leans on
-
[5]
ADD 2023: the Second Audio Deepfake Detection Challenge. In DADA@IJCAI. Zhang, C.-L.; Wu, J.; and Li, Y . 2022. ActionFormer: Local- izing Moments of Actions with Transformers. In Computer Vision – ECCV 2022. Zhang, L.; Wang, X.; Cooper, E.; Evans, N.; and Yamagishi, J. 2023a. The PartialSpoof Database and Countermeasures for the Detection of Short Fake S...
work page 2023
-
[2017]
In Proceedings of the IEEE international conference on computer vision , 2980–2988
Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision , 2980–2988. Liu, M.; Wang, J.; Qian, X.; and Li, H. 2023a. Audio- visual temporal forgery detection using embedding-level fu- sion and multi-dimensional contrastive loss. IEEE Transac- tions on Circuits and Systems for Video Technology, 34. Liu, ...
-
[2022]
In Image Analysis and Process- ing – ICIAP 2022
Combining EfficientNet and Vision Transformers for Video Deepfake Detection. In Image Analysis and Process- ing – ICIAP 2022. Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; Uszkoreit, J.; and Houlsby, N. 2021. An Image is Worth 16x16 Words: Transformers for Image ...
arXiv 2022
-
[2023]
In 2022 IEEE Spoken Language Technology Workshop (SLT)
Push-Pull: Characterizing the Adversarial Robustness for Audio-Visual Active Speaker Detection. In 2022 IEEE Spoken Language Technology Workshop (SLT). Chen, X.; Wu, H.; Wang, C.-C.; Lee, H.-Y .; and Jang, J.- S. R. 2024. Multimodal Transformer Distillation for Audio- Visual Synchronization. In ICASSP 2024 - 2024 IEEE Inter- national Conference on Acousti...
work page 2022
-
[2024]
How Do Neural Spoofing Countermeasures Detect Partially Spoofed Audio? In Proc. INTERSPEECH. Liu, X.; Wang, X.; Sahidullah, M.; Patino, J.; Delgado, H.; Kinnunen, T.; et al. 2023b. ASVspoof 2021: To- wards Spoofed and Deepfake Speech Detection in the Wild. IEEE/ACM Trans. Audio, Speech, Lang. Process., 31. Martinez, B.; Ma, P.; Petridis, S.; and Pantic, M...
arXiv 2021
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.