{"id":"e703d903-e08c-40f6-960e-9653ffd41c84","arxiv_id":"2505.06528","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A MTCNN-EfficientNet-B5 pipeline achieves 0.4278 log loss, 0.938 AUC, and 0.8682 F1 on the Kaggle DFDC deepfake video dataset.","lead":"This paper tests a deepfake video detector that uses MTCNN to crop faces and EfficientNet-B5 to classify each frame, reporting 93.8% AUC and 86.8% F1 on the DFDC dataset. A smart generalist might read it as a compact example of how standard face detection plus a modern CNN performs on a public deepfake benchmark.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significant objection identified beyond the reader's evaluation-protocol concern, which is real, checkable, and already decisive for the conditional verdict.","rationale":"The reader's weakest_assumption is precisely the load-bearing concern I would also flag: an evolving validation protocol without a clearly held-out test split makes the headline numbers potentially optimistically biased. The paper never specifies a final untouched test set, so the reproducibility of the central claim is not established. I agree with the reader's conditional verdict. The evidence is otherwise ordinary for a short empirical paper: standard components (MTCNN + EfficientNet-B5) plus a claimed performance slightly better on log loss than one published ensemble baseline and slightly worse than a hybrid ViT approach. There are no mathematical or architectural correctness issues that I could find with the stated method. The comparison table omits key details (same split, same frame sampling, same aggregation), which adds uncertainty but does not by itself invalidate the proposal's contribution. The Discussion honestly admits the EfficientNet-ViT gap and the face-detection vulnerability, which counts as independent support for the report's credibility. My verdict remains conditional accept: no change to the reader's verdict is needed, but the evaluation-protocol concern should be addressed in revision.","tokens_in":6718,"tokens_out":1353,"duration_ms":11553,"concrete_test":"Request or reconstruct the exact evaluation protocol: ask the authors to provide the test-set folder/file indices and the aggregation thresholds used to compute 0.4278/0.938/0.8682, then rerun inference on that untouched split (or, if the split was used for any model selection or threshold adjustment, recompute the metrics on a held-out split of DFDC not involved in validation). If the recomputed log loss, AUC, and F1 are within 0.01 of the reported values, the central claim holds; otherwise the claim should be reported with a clear confidence interval, and the abstract should state which split produced the numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"No significant objection identified. The central claim is that the MTCNN-EfficientNet-B5 pipeline achieves log loss 0.4278, AUC 0.938, and F1 0.8682 on DFDC. The paper explicitly discloses in Section 4.5 that 'the validation approach evolved throughout the development process,' and it never states that a final, untouched test split exists. If the reported numbers were produced by the same evolving validation loop used to choose EfficientNet-B5 and the confidence-weighted aggregation thresholds, the performance claim is optimistically biased and not reproducible on unseen DFDC videos. This is the one load-bearing concern. The rest of the pipeline is standard: MTCNN face detection and EfficientNet-B5 are well-established components, so the empirical outcome is not surprising. The comparison to the EfficientNet-Vision Transformer model [9] and the ensemble CNN [7] is too thin; whether those models used the same DFDC test split, frame sampling, or aggregation is not established. The Discussion independently acknowledges the gap to the ViT hybrid and the face-detection vulnerabilities, which supports the credibility of the report but does not resolve the split issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes a deepfake video detection pipeline that combines MTCNN face detection, an EfficientNet-B5 frame-level classifier, and a confidence-weighted aggregation of 32 sampled frames per video. The model is trained and evaluated on the Kaggle DFDC dataset, and the paper reports a log loss of 0.4278, an AUC of 0.9380, and an F1 score of 0.8682. The authors compare these numbers with an EfficientNet-Vision Transformer baseline and an ensemble CNN baseline from the literature, and they argue that the proposed pipeline is competitive without extensive preprocessing or ensembling.","tokens_in":6880,"tokens_out":5259,"duration_ms":49364,"significance":"If the performance numbers survive a properly defined untouched test split, this is a modest and useful empirical contribution: the pipeline uses established components (MTCNN, EfficientNet) plus a simple confidence-weighted aggregation rule, and it is described transparently enough to be reproduced from public data and pretrained models. The authors are also candid about the remaining gap to the ViT hybrid and about face-detection vulnerabilities. The significance is, however, conditional: the evaluation protocol in Section 4.5 is ambiguous, and the state-of-the-art comparison in Table 1 is not controlled. The paper's central claim of competitive performance therefore cannot be accepted as stated without a clarified and, ideally, untouched evaluation split.","major_comments":[{"comment":"The evaluation protocol is not sufficiently defined to support the headline numbers. Section 4.5 states that 'the validation approach evolved throughout the development process' and only describes an initial holdout of folders 0-2; no final, untouched test split is defined. Because Section 4.3 reports selecting EfficientNet-B5 through experiments and Section 4.4 introduces confidence thresholds for aggregation, the reader cannot rule out that the log loss 0.4278, AUC 0.938, and F1 0.8682 were produced by the same validation loop used for model selection and threshold tuning. Please specify a final test split (e.g., a set of DFDC folders or video IDs) that was never used for early stopping, variant selection, or threshold tuning, and report all headline metrics on that split; if the reported numbers are validation numbers, say so explicitly and temper the claims accordingly.","section":"Section 4.5, Section 5.2, Table 1"},{"comment":"The comparison with the two baselines is uncontrolled. The values for EfficientNet-Vision Transformer [9] and Ensemble CNN [7] are taken from the literature, and it is not established that they were computed on the same test split, same frame sampling, same face detector, or same aggregation method. The claim in the contributions that the proposed model 'outperformed many state-of-the-art models' is therefore not supported by the evidence presented. Please either run the baselines under the same protocol on the final test split or state clearly that Table 1 is a literature comparison with no claim of a controlled head-to-head.","section":"Section 5.2, Table 1"},{"comment":"The central design choices are not quantitatively justified. The paper states that EfficientNet-B5 provided the best trade-off and that confidence-weighted aggregation was beneficial, but it reports no ablation comparing variants (B0-B7) or simple averaging versus the weighted aggregation. Without these ablations, the contribution of the proposed fusion and heuristic cannot be separated from the choice of the backbone. Please add ablation results on the validation set, and confirm that the final configuration was fixed before computing metrics on the untouched test split.","section":"Sections 4.3 and 4.4"}],"minor_comments":[{"comment":"The abstract and Section 5.2 report '42.78% log loss' and '93.80% AUC'; log loss and AUC are not percentages and should be reported as 0.4278 and 0.9380.","section":"Abstract, Section 5.2"},{"comment":"Section 4.5 refers to 'folders 0 through 2' but the folder partition of DFDC is never defined; clarify how the data were split into folders and how the final test split relates to them.","section":"Section 4.5"},{"comment":"Section 4.1 mentions computing SSIM masks and storing them as PNG files, but the rest of the paper never states whether or how these masks are used by the model; either integrate them into the pipeline description or remove them.","section":"Section 4.1"},{"comment":"Equation (1) is typeset incorrectly in the submitted text; the summation and variables should be formatted properly, and the definitions of y-hat and y should be stated in the text rather than only in a parenthetical.","section":"Equation (1)"},{"comment":"The manuscript contains numerous language and typographical issues (e.g., 'now a days', 'intent to enhance', 'Architechture', 'EfficinetNet', 'maneuver', 'contribute important perceptions'); a careful copyedit is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The central issue is the evaluation split; if the authors cannot identify a final untouched test split, the paper's main performance claim should be substantially weakened. The literature comparison in Table 1 is also too thin to support the stated contribution. If the authors can provide a clean held-out evaluation and a same-protocol comparison, the paper could be acceptable as a short empirical contribution, but in its current form it is not ready for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is a routine deepfake-detection pipeline (MTCNN crops + EfficientNet-B5) with competitive numbers on DFDC, but the paper never defines a final untouched test split, so the headline result is not fully verified.\n\nWhat it does well: the write-up is honest and workmanlike. The pipeline is described clearly enough to reproduce the trained-model behavior, and the Discussion admits the ViT hybrid beats them and that face detection is a vulnerability. The authors also report log loss, AUC, and F1 rather than cherry-picking a single metric. The related-work citations are reasonable and not padded.\n\nThe soft spots are real, but none are fatal. The evaluation protocol is the load-bearing one. Section 4.5 says the validation approach 'evolved' and that folders 0-2 were initially held out, but there is no statement that a final untouched test set was set aside before model selection and threshold tuning. If the 0.4278/0.938/0.8682 numbers come from the same evolving validation loop, they are optimistically biased relative to unseen DFDC videos. A referee should require a clear split definition and a statement that the test set was frozen before any selection. The comparison table is also thin: only two prior systems, with no evidence that evaluation protocols match, and the confidence-weighted aggregation is described at a high level with no ablation. Minor gap: SSIM masks are computed and stored, but the paper never explains whether or how they enter the training or inference pipeline. Error bars are absent, which is common but matters more in a paper whose contribution is a single performance number.\n\nNone of this makes me doubt the authors ran the experiments; the architecture is standard, so the results are plausible. It just means the central claim is weakly verified as written.\n\nWho it's for: practitioners who want another reference point for EfficientNet on DFDC, and referees who care about evaluation hygiene in this literature. I would send it to peer review because the empirical claim is checkable and the problem is addressable with a clear split and an ablation. The revision should center on the test-set question.","headline":"Routine MTCNN+EfficientNet-B5 pipeline with plausible DFDC numbers, but the missing untouched test split means the headline result is unverified.","tokens_in":7505,"tokens_out":3167,"would_cite":false,"duration_ms":30750,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that MTCNN face detection plus an EfficientNet-B5 frame classifier detects deepfake videos on the DFDC dataset with 0.4278 log loss, 0.938 AUC, and 0.8682 F1, beating one published ensemble baseline on log loss.","keywords":["Deepfake detection","MTCNN","EfficientNet-B5","DFDC dataset","Log loss","AUC","F1 score","Video authenticity"],"falsifier":"Re-run the same MTCNN-EfficientNet-B5 training recipe on the DFDC data with a pre-registered held-out test set, one whose folders are untouched by early stopping and threshold selection, and recompute the three metrics; if the log loss rises materially above 0.4278 or the AUC falls materially below 0.938, the paper's central performance claim is not reproducible.","tokens_in":6462,"feed_emoji":"🎭","tokens_out":4903,"duration_ms":44228,"temperature":0.7,"pith_summary":"This paper claims that a deepfake video detector built from two off-the-shelf pieces, MTCNN for finding faces and EfficientNet-B5 for classifying frames, can match or beat far heavier systems on the DFDC benchmark. The authors report a log loss of 0.4278, an AUC of 0.938, and an F1 score of 0.8682, which places the pipeline ahead of one published ensemble-CNN baseline on log loss and only slightly behind a hybrid that adds vision transformers. The practical point is that accurate detection does not necessarily require ensembling many models or extensive preprocessing; a single well-scaled CNN on cropped faces may be enough for many real-world filters. The paper frames the result as evidence that precise face localization plus compound-scaled feature extraction is a strong, efficient recipe for video authenticity checking.","feed_headline":"Face-crop CNN detects deepfakes with 93.8% AUC","feed_subtitle":"MTCNN and EfficientNet-B5 beat an ensemble CNN on log loss in the DFDC test.","key_machinery":"The mechanism is a two-stage pipeline. MTCNN (Multi-task Cascaded Convolutional Networks) detects faces and five landmarks through a P-Net, R-Net, O-Net cascade, producing cropped face regions at original resolution. EfficientNet-B5, a compound-scaled CNN, classifies each cropped frame; training adds a 30 percent margin around the crop, uses balanced batches, SGD with momentum, a polynomial learning-rate scheduler, and label smoothing. Video-level predictions come from confidence-weighted aggregation over 32 frames per video, where high-confidence frames weigh more and low-confidence frames are discarded, and SSIM masks computed during preprocessing are stored as auxiliary PNG files.","core_discovery":"The central claim is that the MTCNN-EfficientNet-B5 pipeline distinguishes fake from real videos on the DFDC dataset with a video-level log loss of 0.4278, an AUC of 0.938, and an F1 score of 0.8682, using only face crops, SSIM-based difference masks, and confidence-weighted averaging of 32 frame predictions per video. The authors present this as competitive with published DFDC results: it beats an ensemble CNN on log loss and trails an EfficientNet-Vision Transformer hybrid by 0.013 in AUC and 0.012 in F1. They attribute the result to MTCNN's robust face localization, EfficientNet-B5's compound scaling, and a confidence-weighted pooling scheme that downweights uncertain frames instead of using simple averaging.","pith_inferences":["If the numbers reproduce on a true holdout, the log-loss advantage over the ensemble baseline suggests single-model pipelines can be better calibrated, and confidence-weighted pooling may transfer to other video-classification tasks.","Because the pipeline depends on face detection, its accuracy on videos with occluded, tiny, off-center, or multiple faces is an open question the paper does not address; those conditions are a likely failure mode in the wild.","The method's components are standard and the DFDC benchmark is public, so the natural next check is a direct, matched-split comparison with the two baselines cited in the paper, plus evaluation on other deepfake datasets, which would show whether the margins generalize."],"forward_implications":["Deployable single-model detection: a pipeline with one CNN and a face detector can reach a log loss of 0.4278 on DFDC, so high-accuracy screening does not require large ensembles.","Calibrated confidence: the low log loss implies the model's probabilities are well-calibrated enough to rank and filter videos, not just classify them.","Rejected uncertain frames: confidence-weighted averaging means frame-level uncertainty is explicitly discounted, which should improve robustness on blurry or occluded clips.","Architecture verdict: EfficientNet-B5 beats its smaller and larger siblings in this task, giving a concrete cost-accuracy sweet spot for the detection pipeline."],"supporting_citations":[{"why":"Introduces the EfficientNet compound-scaling family; supplies the B4/B5 base configuration used as the classifier backbone.","marker":"[19]"},{"why":"Source of the MTCNN architecture diagram; the cascaded P-Net/R-Net/O-Net detector is the face-crop engine.","marker":"[20]"},{"why":"Published EfficientNet-Vision Transformer baseline whose AUC and F1 the proposed model is compared against.","marker":"[9]"},{"why":"Published ensemble-CNN baseline whose log loss the proposed model is compared against.","marker":"[7]"}],"fun_headline_variants":["MTCNN-EfficientNet-B5 beats ensemble CNNs on log loss","Face-crop pipeline hits 93.8% AUC on DFDC","EfficientNet-B5 deepfake detector scores 93.8% AUC","Deepfake detection with MTCNN and EfficientNet-B5","93.8% AUC: MTCNN and EfficientNet-B5 detect fakes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reported log loss, AUC, and F1 come from a test split that was never used to choose the model, tune the confidence thresholds, or stop training; the paper only describes an evolving validation setup and never states that such a final untouched split exists.","fun_headline_variants_meta":{"raw":{"variants":["MTCNN-EfficientNet-B5 beats ensemble CNNs on log loss","Face-crop pipeline hits 93.8% AUC on DFDC","EfficientNet-B5 deepfake detector scores 93.8% AUC","Deepfake detection with MTCNN and EfficientNet-B5","93.8% AUC: MTCNN and EfficientNet-B5 detect fakes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000171,"raw_usage":{"total_tokens":1230,"prompt_tokens":863,"completion_tokens":367,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":268}},"tokens_in":479,"tokens_out":367,"duration_ms":3529,"temperature":1.0,"reasoning_tokens":268,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:39:21.483064+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same MTCNN-EfficientNet-B5 training recipe on the DFDC data with a pre-registered held-out test set, one whose folders are untouched by early stopping and threshold selection, and recompute the three metrics; if the log loss rises materially above 0.4278 or the AUC falls materially below 0.938, the paper's central performance claim is not reproducible.","supporting_citations":[{"cited_title":"Minor privacy protection through real-time video processing at the edge","cited_arxiv_id":null,"evidence_quote":"Source of the MTCNN architecture diagram; the cascaded P-Net/R-Net/O-Net detector is the face-crop engine."},{"cited_title":"Combining efficientnet and vision transformers for video deepfake detec- tion","cited_arxiv_id":null,"evidence_quote":"Published EfficientNet-Vision Transformer baseline whose AUC and F1 the proposed model is compared against."},{"cited_title":"Video face manipulation detection through ensem- ble of cnns","cited_arxiv_id":null,"evidence_quote":"Published ensemble-CNN baseline whose log loss the proposed model is compared against."}],"review_version":1}