{"id":"66f03ee2-1c7e-478d-86b5-4eb4e98c5934","arxiv_id":"2411.09266","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ChatGPT with carefully engineered prompts detects audiovisual deepfakes at about the same accuracy as humans (65%), below specialized detectors (87.5% to 97.5%), on a 40-video benchmark.","lead":"Researchers tested whether OpenAI's ChatGPT can spot fake videos with manipulated faces or voices, using 40 benchmark clips. They found well-designed prompts let ChatGPT match average human accuracy, though specialized AI detectors remain far more accurate.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No evidence ChatGPT actually processes the audio track; the 'audiovisual' claim rests on an unverified and internally inconsistent mechanism.","rationale":"The reader's conditional verdict already identifies the same weakest assumption, and my analysis agrees: the central claim depends on GPT-4 actually analyzing the audio track. The paper's own text is internally inconsistent about how this happens: Section III-B says no preprocessing and direct video input, while Section V-B attributes the analysis to OpenCV, librosa, numpy, wav, and skimage, which would require code execution or tool use that is never specified. Without that detail, the acoustic feature lists and artifact attributions in the outputs cannot be distinguished from LLM hallucination. The modality-ablation test is the minimal decisive check: if removing or replacing the audio track does not change P6/P7 outputs, then the 'audiovisual' interpretation of Table II is unsupported, even though the raw accuracy numbers may be reproducible. This concern is load-bearing because it targets the paper's main contribution, not a peripheral robustness issue. I would keep the reader's CONDITIONAL verdict pending this test; if the test shows no audio sensitivity, the claim would need to be narrowed to visual-only or rejected.","tokens_in":12734,"tokens_out":5048,"duration_ms":105102,"concrete_test":"Run a modality ablation on the same 40 videos with P6/P7 and identical session settings: (a) full video with audio, (b) the same video with the audio track replaced by digital silence, and (c) the same video with audio replaced by unrelated speech or noise. If accuracy and the generated artifact reports for (a) are unchanged from (b), or if condition (b) still produces reports such as 'clipping detected' or 'spectral artifacts' that require audio content, then GPT-4 is not processing the audio channel, and the audiovisual claim collapses.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result (Table II, P6/P7 = 65%) is interpreted as audiovisual deepfake detection. That interpretation requires GPT-4 to genuinely consume the audio stream and compute the acoustic features listed in Section III-C (spectral centroid, MFCC, silence ratio, etc.). Section III-B states that videos are fed in 'without performing any preprocessing,' yet Section V-B says ChatGPT 'mainly relies on hand-crafted features and traditional functions in computer vision and speech processing libraries, including OpenCV, librosa, numpy, wav, and skimage.' No API configuration, tool-use mode, or code-interpreter setting is reported, so the reader cannot tell whether the acoustic descriptions in the outputs are real measurements or plausible-sounding text generated from visual frames and learned priors. If the latter, then even if the 65% accuracy number is real, the paper's central claim that ChatGPT performs audiovisual forgery triage and gives artifact attributions is unsupported: the result would at best be a visual/prior-based detector, and the comparison to audiovisual forensic models and human perception would be misleading. The manuscript does not provide per-manipulation (FVRA/RVFA/FVFA) results or repeated runs, so this mechanism cannot be checked from the reported data.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether ChatGPT (GPT-4) can detect audiovisual deepfakes by analyzing both video and audio streams. Using 40 videos from the FakeAVCeleb benchmark, the authors test seven prompting strategies, compare the best results against human accuracy from their prior study [19] and against several audiovisual forensic models, and report that artifact-focused prompts (P6 and P7) achieve 65.00% accuracy, similar to the human overall accuracy of 65.64%, while the specialized models reach 87.50–97.50%. The paper also claims that ChatGPT provides interpretable artifact attributions and concludes that, with appropriate prompting, ChatGPT performs on par with humans but well below end-to-end forensic detectors.","tokens_in":12935,"tokens_out":5654,"duration_ms":52796,"significance":"If fully supported, the central claim would be a significant and timely result: a general-purpose LLM performing zero-shot audiovisual forgery triage at human level, with interpretable outputs, would be practically useful for initial screening. The paper has clear strengths: it uses an external benchmark dataset, a systematic set of prompts, and an explicit comparison table that imports human and AI baselines from a prior study. The authors also include a limitations section, which is good scientific practice. However, the load-bearing evidence is presently too thin to support the strong conclusions. The mechanism by which ChatGPT processes audio is not documented, the human-comparative claim lacks any uncertainty quantification, and per-manipulation results are not reported. As an exploratory study the paper is valuable, but as a comparative claim it needs substantial additional support.","major_comments":[{"comment":"The claim that ChatGPT performs genuine acoustic analysis is not supported by the reported experimental setup. Section III-B states that videos are fed directly 'without performing any preprocessing' and that 'the model extracts audio from the video and performs visual and acoustic analysis based on input prompts,' while Section III-C lists concrete acoustic features such as spectral centroid, MFCC, and silence ratio. However, the paper does not report the ChatGPT interface or API version, whether any code-execution or tool-use mode was enabled, or how the audio track is actually ingested. Section V-B then asserts that 'ChatGPT mainly relies on hand-crafted features and traditional functions in computer vision and speech processing libraries, including OpenCV, librosa, numpy, wav, and skimage,' which is inconsistent with a plain LLM operating without tool access. The example outputs in Fig. 4 (e.g., 'Clipping detected,' 'Silence detected') are consistent with plausible text generation rather than actual measurements. Unless the authors document the audio-processing mechanism, the 'audiovisual' interpretation of the 65% result is not established; the result may reflect visual-only or prior-based detection.","section":"Section III-B, III-C, V-B"},{"comment":"The headline claim that ChatGPT 'performs on par with humans' is not statistically supported. The comparison uses a single run of seven prompts on 40 videos; P6 and P7 yield 65.00% accuracy, which is only 0.64 percentage points below the human overall average of 65.64% taken from the authors' prior study [19]. No confidence intervals, repeated trials, or significance tests are provided. For n=40, the standard error of an accuracy near 65% is roughly 7.5 percentage points, so the observed difference is negligible. In addition, P6 and P7 were selected after inspecting results across all seven prompts, which inflates the expected best-case accuracy. The human baseline is also imported from a different experimental protocol (a two-phase subjective test) without human variance being reported. To support the 'comparable to human' claim, the authors should provide per-run variability, statistical tests, and a clear statement of the prompt-selection procedure.","section":"Table II / Section IV-C-2"},{"comment":"The abstract and conclusions claim that ChatGPT can 'account for spatial and spatiotemporal artifacts and inconsistencies that may exist within or across modalities,' but the results are reported only as aggregate accuracy on the 40-video set. The FakeAVCeleb subset spans three manipulation types (FVRA, RVFA, FVFA) described in Fig. 1, and the paper never reports accuracy per manipulation type. Without this breakdown, the reader cannot determine whether the 65% result reflects genuine cross-modal detection (e.g., of audio artifacts in RVFA) or mostly visual/prior-based behavior. Reporting per-manipulation accuracy is essential to substantiate the audiovisual forgery triage claim.","section":"Abstract / Section IV-A / Fig. 1"}],"minor_comments":[{"comment":"The table header and the main text contain the typo 'ChatGP' instead of 'ChatGPT'; for example, 'ChatGP performs on par with humans' in Section IV-C-2.","section":"Table II / Section IV-C-2"},{"comment":"Fig. 4 contains two typos: 'Predication' should be 'Prediction' and 'lable' should be 'label.'","section":"Fig. 4"},{"comment":"The ablation prompts are described as 'video-only mention prompts,' yet the last prompt in the list includes '10-lack of synchronization between audio and video,' which is an audiovisual attribute; this is confusing and should be clarified or removed from the video-only set.","section":"Section V-A"},{"comment":"The seventh prompt in the ablation list says 'Check if the video has the artifact attribute in the following list and ONLY return the attribute number in this image,' but the task is video-based; the phrase 'in this image' appears to be copied from an image-forensics study and should be adapted to the video setting.","section":"Section V-A"},{"comment":"The paper states 'we used OpenAI's GPT-4' but does not specify the exact model version, the interface (API vs. web), the date of deployment, or any parameter settings; this information is needed for reproducibility.","section":"Section III"},{"comment":"The heading 'Ablation Study' is somewhat misleading because the section varies prompt design rather than ablating model components; consider renaming it to 'Prompt-Modality Analysis' or 'Prompt Design Experiments.'","section":"Section V"}],"recommendation":"major_revision","confidential_remarks":"The paper relies heavily on the authors' own prior work: the human baseline is taken from [19], and several of the compared AI models are from the same group ([26], [27], [28], [30]). This is not improper, but the comparison would be more credible with independent replication or at least an explicit disclosure of the overlap. The empirical claims also need stronger statistical grounding before publication. The topic fits the journal's scope, but the current evidence supports an exploratory result rather than a definitive comparative claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is my read. The paper tries something genuinely new: it puts ChatGPT (GPT-4) on audiovisual deepfake detection, not just images, and does a systematic seven-prompt comparison. That alone is worth a look. The best artifact-focused prompts (P6/P7) reach 65.00% accuracy on the same 40-video FakeAVCeleb subset used in the authors' prior human study, matching the human baseline of 65.64%. Specialized audiovisual models are far ahead at 87.50-97.50%, so the headline is modest: an off-the-shelf LLM can do first-pass triage at human level in a low-resource setting.\n\nWhat the paper does well: the ablation with video-only prompts shows higher rejection rates and lower accuracy, supporting the point that you need to ask for audio analysis. The failure-case discussion is candid, especially about silence ratio and modality-score averaging. Reusing the exact video subset for human, AI, and ChatGPT comparison is the right way to make the comparison fair.\n\nThe soft spots are real but not disqualifying. Forty videos is small; the 65.00% versus 65.64% difference is within one sample and there are no error bars or repeated runs. So 'comparable to humans' is an observation, not a robust claim. The bigger issue is the mechanism: Section III-C says ChatGPT computes spectral centroid, MFCCs, silence ratio, etc., but Section V-B says it relies on hand-crafted features and traditional libraries like OpenCV, librosa, and skimage. No API configuration or tool-use mode is reported. I cannot tell whether the audio descriptions are actual signal measurements or plausible text generated from visual frames and learned priors. The stress-test note calls this an internal inconsistency, and I think it is a fair concern. If the model never truly processes the audio track, the 'audiovisual' interpretation is unsupported, even if the accuracy numbers are reproducible.\n\nThe citation pattern is mostly fine. Several compared models are from the same group, and the human baseline is the authors' own prior study, but the overlap is transparent and not a flaw by itself. What is missing is independent verification of the mechanism.\n\nWho is this for? Researchers working on LLM-based media forensics or practical deepfake triage. They will get a useful prompt-design template and a sanity-check data point. It is not a definitive benchmark paper. It deserves a serious referee, but that referee should ask for code, data, repeated runs, and a clear statement of how audio is actually handled by the model.\n\nI would bring it to reading group as a maybe—timely, but the evidence is thin. My recommendation: engage it, require revision before acceptance.","headline":"First systematic look at ChatGPT for audiovisual deepfakes, but the human-parity claim rests on 40 videos and an unverified audio mechanism.","tokens_in":813,"tokens_out":1729,"would_cite":true,"duration_ms":36309,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A general-purpose chatbot with no forensic training can detect audiovisual deepfakes at about the same accuracy as human viewers, provided the prompt directs it to look for concrete artifacts.","keywords":["audiovisual deepfake detection","ChatGPT","GPT-4","multimodal forensics","prompt engineering","zero-shot detection","FakeAVCeleb","human perception"],"falsifier":"Re-test prompts P6 and P7 on the same 40 videos after replacing every audio track with a single unrelated soundtrack, and also report ChatGPT's accuracy on the real-video/fake-audio (RVFA) subset separately; if audio-track replacement leaves accuracy unchanged or RVFA accuracy is near chance, then the model is deciding from visual content alone and the acoustic-analysis mechanism is falsified.","tokens_in":12492,"feed_emoji":"🎭","tokens_out":8299,"duration_ms":78816,"temperature":0.7,"pith_summary":"ChatGPT, a general-purpose chatbot with no training on forensic data, can flag audiovisual deepfakes about as reliably as human viewers once the prompt points it at specific artifacts, this paper argues. On a balanced 40-video subset of the FakeAVCeleb benchmark, the two artifact-focused prompts reach 65.00% accuracy, nearly matching the 65.64% human average, while simple 'is this real?' prompts hover near chance. The paper positions this as evidence that prompt engineering can unlock an LLM's implicit multimodal knowledge for video forgery triage, with the bonus of interpretable explanations naming the offending artifact. It also stresses the gap to state-of-the-art audiovisual forensic models (87.50–97.50%), which remain far more accurate.","feed_headline":"ChatGPT detects audiovisual deepfakes about as well as humans","feed_subtitle":"Artifact-focused prompts lift ChatGPT to 65%, near human accuracy but well below specialized detectors.","key_machinery":"The mechanism is prompt-conditioned multimodal analysis: the video (frames and audio) and a text prompt are fed to GPT-4, and the prompt dictates which low-level and cross-modal cues the model must check. The paper's operative tool is the artifact attribute list, ten numbered cues spanning visual forgery signs (unnatural face edges, inconsistent lighting/shading, irregular teeth, lip-movement problems, skin-texture inconsistencies), auditory signs (spectral artifacts, phoneme artifacts, speech-pattern inconsistencies, voice-quality issues), and one synchronization cue (audio-video mismatch). Prompts P6 and P7 force a structured answer—either [region, artifact] pairs or a bare attribute number—which eliminates rejected answers and raises recall compared with binary 'yes/no' prompts; the paper attributes the gain to the model's use of hand-crafted features from OpenCV, librosa, and related libraries for visual and acoustic analysis.","core_discovery":"The paper's central claim is that ChatGPT (GPT-4) performs audiovisual deepfake detection at a human-competitive level when the input prompt enumerates the artifacts to look for, and that it does so without any task-specific fine-tuning. Using the same 40 videos from FakeAVCeleb that a prior human-perception study used, the authors report 65.00% accuracy for prompts P6 and P7—where the model must report synthesis artifacts in the visual frames or audio, or must return attribute numbers from a ten-item artifact list—against a 65.64% human average and state-of-the-art model scores between 87.50% and 97.50%. The authors interpret the result as showing that an LLM can generalize across manipulations, explain its decisions by naming visual, acoustic, or synchronization artifacts, and serve as an interpretable first-pass screening tool, while acknowledging that its reliance on hand-crafted signal-processing features and prompt sensitivity cap its accuracy well below specialized detectors.","pith_inferences":["We infer that the paper's accuracy numbers do not by themselves prove the model is analyzing the audio track; a direct test would swap each video's audio with an unrelated soundtrack and check whether ChatGPT's decisions change, which would separate visual-cue-based inference from genuine cross-modal analysis.","We infer that the 'human-level' comparison on 40 videos should be treated as an existence proof rather than a stable ranking; repeating the prompt protocol on other audiovisual deepfake datasets or on larger subsets could move the 65% figure substantially.","We infer that the probability prompts (P3/P4) could be recalibrated against human confidence judgments from the same benchmark, giving a practical way to combine LLM probability scores with human oversight in a triage system."],"forward_implications":["A zero-shot LLM can serve as a first-pass deepfake screener on audiovisual content, producing a human-readable artifact report before a specialized model or human examiner is called in.","Because the model names the artifact it sees, LLM triage could categorize manipulations by type (face swap vs. voice clone vs. both) without separate per-type classifiers.","The sharp drop from P6/P7 to the naive prompts implies that any user-facing deepfake tool built on an LLM must embed artifact-specific scaffolding; otherwise accuracy falls to near chance.","The accuracy gap between ChatGPT and trained forensic models confirms that LLMs are not a substitute for dedicated detectors, but the interpretability difference suggests hybrid pipelines—LLM rationale plus deep-model score—as the natural next step."],"supporting_citations":[{"why":"supplies the human perception baseline and the 40-video selection protocol that makes the ChatGPT comparison fair","marker":"[19]"},{"why":"provides the FakeAVCeleb dataset and the subset of 40 balanced real/fake videos used in all experiments","marker":"[46]"},{"why":"grounds the prompt-design strategy, as the paper adapts its LLM-based image forensics prompts to audiovisual content","marker":"[36]"},{"why":"LipForensics is one of the audiovisual forensic models in the accuracy comparison table","marker":"[10]"},{"why":"AV-Lip-Sync is one of the audiovisual forensic models in the accuracy comparison table","marker":"[27]"},{"why":"AV-Lip-Sync+ is one of the audiovisual forensic models in the accuracy comparison table","marker":"[28]"},{"why":"CNN-Ensemble is one of the audiovisual forensic models in the accuracy comparison table","marker":"[26]"},{"why":"AVTENet is one of the audiovisual forensic models in the accuracy comparison table","marker":"[30]"}],"fun_headline_variants":["ChatGPT rivals human accuracy in audiovisual deepfake detection","With artifact prompts, ChatGPT matches humans on deepfakes","ChatGPT's artifact-focused prompts hit human-level deepfake detection","ChatGPT matches human deepfake detection given artifact prompts","ChatGPT falls short of specialized AIs but matches humans on deepfakes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that ChatGPT genuinely processes each video's audio track—extracting features like spectral centroid, MFCCs, and silence ratio—rather than producing its audio commentary from visual frames and learned priors; if the model is not actually analyzing sound, the paper's cross-modal and audio-artifact claims are unsupported.","fun_headline_variants_meta":{"raw":{"variants":["ChatGPT rivals human accuracy in audiovisual deepfake detection","With artifact prompts, ChatGPT matches humans on deepfakes","ChatGPT's artifact-focused prompts hit human-level deepfake detection","ChatGPT matches human deepfake detection given artifact prompts","ChatGPT falls short of specialized AIs but matches humans on deepfakes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000717,"raw_usage":{"total_tokens":3222,"prompt_tokens":944,"completion_tokens":2278,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":2193}},"tokens_in":560,"tokens_out":2278,"duration_ms":19008,"temperature":1.0,"reasoning_tokens":2193,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:49:43.588990+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-test prompts P6 and P7 on the same 40 videos after replacing every audio track with a single unrelated soundtrack, and also report ChatGPT's accuracy on the real-video/fake-audio (RVFA) subset separately; if audio-track replacement leaves accuracy unchanged or RVFA accuracy is near chance, then the model is deciding from visual content alone and the acoustic-analysis mechanism is falsified.","supporting_citations":[{"cited_title":"Hashmi, S","cited_arxiv_id":null,"evidence_quote":"CNN-Ensemble is one of the audiovisual forensic models in the accuracy comparison table"},{"cited_title":"Khalid, S","cited_arxiv_id":null,"evidence_quote":"provides the FakeAVCeleb dataset and the subset of 40 balanced real/fake videos used in all experiments"},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"grounds the prompt-design strategy, as the paper adapts its LLM-based image forensics prompts to audiovisual content"},{"cited_title":"Haliassos, K","cited_arxiv_id":null,"evidence_quote":"LipForensics is one of the audiovisual forensic models in the accuracy comparison table"},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"AV-Lip-Sync is one of the audiovisual forensic models in the accuracy comparison table"}],"review_version":1}