REVIEW 5 major objections 5 minor 21 references
Can ChatGPT Perform Image Splicing Detection? A Preliminary Study
T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper claims that GPT-4V, a general-purpose multimodal model, detects image splicing out of the box with over 85% zero-shot accuracy, and that chain-of-thought prompting yields the most balanced performance across authentic and…
desk verdict Honest preliminary measurement, but the 85% out-of-the-box claim is genuinely unverifiable given closed training data and a public benchmark. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is the prompting strategy applied to GPT-4V over a curated JPEG-only subset of CASIA v2.0 containing 3,100 authentic and 600 spliced images from three categories. Zero-shot, few-shot, and chain-of-thought prompts are the three experimental levers; the model's free-form explanations are what make the mechanism visible, letting the authors attribute decisions to visual artifacts or to real-world knowledge. The central object is therefore not a new architecture but a behavior: an MLLM's sensitivity to instruction format, demonstrated through accuracy and category-wise breakdowns.
What would settle it
Construct a splicing test set from images created after GPT-4.1's training cutoff (or from a private, never-published corpus) and run the same zero-shot and chain-of-thought prompts; if accuracy on unseen forgeries drops to near chance while accuracy on CASIA v2.0 stays high, the headline numbers are leakage artifacts, not a general detection capability.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that an out-of-the-box multimodal LLM can perform binary image splicing detection on a curated subset of CASIA v2.0 with 87.65% overall accuracy under zero-shot prompting, while few-shot prompting raises overall accuracy to 94.02% but biases the model toward calling images authentic, dropping spliced recall from 85.50% to 76.50%. Chain-of-thought prompting, which supplies four annotated examples with step-by-step reasoning, restores a more balanced trade-off at 93.76% overall accuracy with 81.33% spliced recall and 96.16% authentic accuracy. The paper further claims that the model's explanations show two kinds of evidence: inherited visual heuristics about lighting, shadows, edges, texture, focus, and color, and encyclopedic reasoning about implausible scale, habitat mismatches, and architectural facts. The authors are explicit that this does not beat specialized splicing detectors on raw accuracy, but they argue the combination of generalization, interpretability, and world knowledge makes the model a promising complementary tool.
Load-bearing premise
The whole reading depends on the assumption that the reported accuracies reflect genuine out-of-the-box detection ability, not the model having already seen these benchmark images during pre-training.
Editorial extensions
If this is right
- If the central claim holds, forensic analysts can use a general-purpose multimodal model as a zero-shot triage tool on JPEG splicing cases before deploying specialized detectors.
- Chain-of-thought prompts provide natural-language rationales that could make forensic decisions auditable and explainable, not just accurate.
- Prompt format is a first-order variable: the same model swings from 85.50% spliced recall under zero-shot to 76.50% under few-shot, so forensic deployments must choose and validate prompts carefully.
- Category dependence means architectural images are the hard case for this approach, with spliced-architecture accuracy dropping to 62.82% under few-shot prompting.
Reading between the lines
- The paper's own data-leakage caveat suggests a direct test: build a splicing test set from images released after the model's training cutoff and rerun the zero-shot and chain-of-thought prompts; if accuracy falls to chance, the reported numbers reflect benchmark memorization rather than a general forensic skill.
- The same world-knowledge reasoning could plausibly extend to copy-move forgeries, inpainting, or AI-generated image detection, since those manipulations also create semantic or geometric implausibilities the model can vocalize.
- Because the model's explanations name specific cues, one could turn those explanations into a checklist that a human analyst verifies, effectively using the MLLM as a hypothesis generator rather than a final judge.
- Category-level results imply future benchmarks should stratify by scene type; an aggregate accuracy number will hide systematic blind spots such as the architecture category.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents a preliminary study of whether a general-purpose multimodal large language model, OpenAI's GPT-4.1 (referred to throughout as GPT-4V), can detect image splicing without task-specific fine-tuning. The authors evaluate three prompting strategies—zero-shot (ZS), few-shot (FS), and few-shot chain-of-thought (FS-CoT)—on a curated subset of the CASIA v2.0 dataset containing 3,100 authentic and 600 spliced JPEG images drawn from the animal, architecture, and character categories. They report overall detection accuracies of 87.65% for ZS, 94.02% for FS, and 93.76% for FS-CoT, with FS-CoT yielding a more balanced trade-off between authentic and spliced classes than FS. Qualitative examples illustrate that the model sometimes justifies its decisions using both low-level visual artifacts (lighting, shadows, edge blending) and higher-level world knowledge (object scale, habitat plausibility, architectural facts). The paper concludes that GPT-4.1 offers promising out-of-the-box capability as a complementary tool for image forensics, while acknowledging limitations including potential training-data leakage and the absence of comparisons with other MLLMs.
Significance. If the reported results are robust, this paper provides an early demonstration that a closed-source MLLM can perform a specialized forensics task with competitive accuracy and interpretable rationales, without any adaptation. The explicit prompt protocols, public code repository, and category-specific analysis are useful contributions for a preliminary study. However, the central claim of 'out-of-the-box' capability is threatened by the well-known data-leakage risk of public benchmark datasets and undisclosed training corpora, a risk the authors explicitly acknowledge but do not experimentally address. Additionally, the paper makes comparative claims about specialized state-of-the-art models without running any baseline on the same protocol. The internal inconsistency found in Table 1's category counts further tempers confidence. The paper is a defensible preliminary capability study, but its quantitative conclusions need verification and its presentation needs correction.
major comments (5)
- [Section 3, Table 1] The category breakdown for spliced images is internally inconsistent: for each prompting strategy, the counts for Animal, Architecture, and Character sum to 618, while Section 2.1 states that the spliced test set contains 600 images. For example, in the ZS row, 222 + 78 + 318 = 618, and the same discrepancy appears in the FS and FS-CoT rows. The overall totals in the table (e.g., 513 correct + 87 incorrect = 600) contradict the category subtotals. This must be corrected or explicitly explained, for instance by stating whether categories are non-exclusive (a spliced image can belong to more than one category) and adjusting the analysis accordingly.
- [Section 5, Limitation (1)] The data-leakage concern is acknowledged but not resolved. Since CASIA v2.0 is a public benchmark and GPT-4.1's training data are undisclosed, the reported accuracies—particularly the headline "more than 85% accuracy" in the Abstract—cannot be unambiguously attributed to out-of-the-box generalization. The authors' counter-argument that observed misclassifications preclude full memorization is insufficient, because partial memorization of a subset of images or labels could still inflate accuracy. The paper needs a leakage-mitigating experiment, such as evaluating on newly created forgeries not present on the web, or a membership-inference-style test, before claiming that the model generalizes to unseen splices.
- [Abstract and Section 3] The claim that GPT-4V "lags behind specialized state-of-the-art splicing detection models" is unsupported by any direct comparison in the paper. No specialized detector is evaluated on the same curated subset or with the same protocol, so this statement is at best an extrapolation from prior literature. The authors should either remove this claim or substantiate it with a same-protocol baseline experiment.
- [Section 2.2] The "Chain-of-Thought" (CoT) strategy includes four manually annotated examples with step-by-step explanations, making it a few-shot variant rather than a pure reasoning prompt. Consequently, the comparison between FS and FS-CoT is confounded: the two conditions differ both in the presence of reasoning chains and in the content of the examples. The concluding statement that "CoT prompting effectively mitigates this bias" should be tempered to acknowledge that the improvement may be due to the specific examples or explanations rather than the chain-of-thought mechanism itself.
- [Section 3] The text states that CoT "results in an overall accuracy gain of approximately 5% compared to FS," but the overall accuracy (both classes combined) actually decreases from 94.02% (FS) to 93.76% (FS-CoT). The approximately 5% gain applies only to the spliced-class accuracy (76.50% to 81.33%). This wording is misleading and should be revised to state explicitly that the gain is in spliced-class accuracy, not overall accuracy.
minor comments (5)
- [Throughout] The model designation is inconsistent: the abstract, introduction, and qualitative analysis refer to "GPT-4V," Section 2.3 specifies "GPT-4.1 (version 2025-04-14)," and the title says "ChatGPT." Please use a single, accurate designation throughout the manuscript.
- [Section 2.2] The procedure for selecting the four few-shot and CoT example images is not described in sufficient detail: the paper does not specify whether the examples are fixed, randomly sampled, or selected to balance categories and manipulation types. This information is necessary for reproducibility, even if the examples are provided in the repository.
- [Section 4] The qualitative analysis shows only examples where the model's reasoning led to correct or plausible conclusions. Including counterexamples where the model gives a confident but incorrect explanation would provide a more balanced view and help readers assess the reliability of the model's rationales.
- [Section 2.3] The paper states that the temperature is fixed to 0, but it does not mention other decoding parameters (e.g., top_p, max_tokens) or whether the evaluation consists of a single deterministic run. Please state the full decoding configuration and, if only one run was performed, acknowledge the absence of variance estimates.
- [Section 2.1] The random sampling of 3,100 authentic and 600 spliced images should be reproducible: please report the random seed or provide the exact image indices/counts in the repository.
Circularity Check
No circularity: the accuracy figures are direct external measurements against a public benchmark, with no fitted parameter, derivation, or load-bearing self-citation.
full rationale
This paper is an empirical evaluation, not a derivation. It queries GPT-4.1 via the OpenAI API on a curated subset of CASIA v2.0 and computes detection accuracy directly from the model's labels versus ground truth. There is no parameter fitted to the test labels, no prediction derived from an assumed law, and no uniqueness theorem or ansatz smuggled in through citation. The only self-references are the author's own project repository [11] and a prior technical paper [12] cited in the introduction as related work; neither is load-bearing for the measured results. The paper's Section 5 explicitly flags the data-leakage risk: 'we cannot definitively ensure that the images or their associated metadata used in our experiments were not seen during pre-training.' That is a legitimate external-validity and contamination concern, not a circularity concern: even if the accuracy were inflated by memorization, the measurement would still be an empirical observation rather than a construct that reduces to its own inputs. The qualitative examples are illustrative outputs from the model and do not constitute a derivation chain. Accordingly, no circular step can be exhibited, and the appropriate score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption CASIA v2.0 ground-truth labels correctly mark each image as authentic or tampered.
- domain assumption The curated random subset of 3,100 authentic and 600 spliced JPEGs is representative enough to support claims about the full splicing detection task.
- domain assumption Temperature 0 plus the fixed model version produces outputs that are stable and interpretable as the model's capability.
Cite this review
Pith. "Pith review of Can ChatGPT Perform Image Splicing Detection? A Preliminary Study." pith.science (2026). https://pith.science/paper/HTSN34VI
@misc{pith2026250605358,
author = {Pith},
title = {Pith review of: Can ChatGPT Perform Image Splicing Detection? A Preliminary Study},
year = {2026},
howpublished = {\url{https://pith.science/paper/HTSN34VI}},
note = {Machine review of arXiv:2506.05358}
}
read the original abstract
Multimodal Large Language Models (MLLMs) like GPT-4V are capable of reasoning across text and image modalities, showing promise in a variety of complex vision-language tasks. In this preliminary study, we investigate the out-of-the-box capabilities of GPT-4V in the domain of image forensics, specifically, in detecting image splicing manipulations. Without any task-specific fine-tuning, we evaluate GPT-4V using three prompting strategies: Zero-Shot (ZS), Few-Shot (FS), and Chain-of-Thought (CoT), applied over a curated subset of the CASIA v2.0 splicing dataset. Our results show that GPT-4V achieves competitive detection performance in zero-shot settings (more than 85% accuracy), with CoT prompting yielding the most balanced trade-off across authentic and spliced images. Qualitative analysis further reveals that the model not only detects low-level visual artifacts but also draws upon real-world contextual knowledge such as object scale, semantic consistency, and architectural facts, to identify implausible composites. While GPT-4V lags behind specialized state-of-the-art splicing detection models, its generalizability, interpretability, and encyclopedic reasoning highlight its potential as a flexible tool in image forensics.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al
Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)
arXiv 2023
-
[2]
Al-Qershi, O. M., and Khoo, B. E. Passive detection of copy-move forgery in digital images: State-of-the-art. Forensic science international 231, 1-3 (2013), 284–295
work page 2013
-
[3]
Bhowal, A., Neogy, S., and Naskar, R. Deep learning-based forgery detection and localization for compressed images using a hybrid optimization model. Multimedia Systems 30, 3 (2024), 128
work page 2024
-
[4]
D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901
2020
-
[5]
Dong, J., W ang, W., and Tan, T.CASIA image tampering detection evaluation database. In 2013 IEEE China Summit and International Conference on Signal and Information Processing (July 2013), IEEE
work page 2013
-
[6]
He, K., Gkioxari, G., Dollár, P., and Girshick, R. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision (2017), pp. 2961–2969
work page 2017
-
[7]
Hosny, K. M., Mortda, A. M., Lashin, N. A., and Fouda, M. M. A new method to detect splicing image forgery using convolutional neural network. Applied Sciences 13, 3 (2023), 1272
work page 2023
-
[8]
Deep learning-based efficient and robust image forgery detection
KASIM, Ö. Deep learning-based efficient and robust image forgery detection. Multimedia Tools and Applications 83, 21 (2024), 59819–59838
work page 2024
Show all 21 references
-
[9]
Copy move and splicing image forgery detection using cnn
Mallick, D., Shaikh, M., Gulhane, A., and Maktum, T. Copy move and splicing image forgery detection using cnn. In ITM Web of Conferences (2022), vol. 44, EDP Sciences, p. 03052
2022
-
[10]
B., and Tyagi, V.A deep learning based method for image splicing detection
Meena, K. B., and Tyagi, V.A deep learning based method for image splicing detection. InJournal of physics: conference series (2021), vol. 1714, IOP Publishing, p. 012038
2021
-
[11]
Llm image splicing detection project repository
Nath, S. Llm image splicing detection project repository. https://github.com/confusedDip/LLM-Image-Splicing- Detection. , Vol. 1, No. 1, Article . Publication date: August 2025. 10 Souradip Nath
2025
-
[12]
Automated image splicing detection using deep cnn-learned features and ann-based classifier
Nath, S., and Naskar, R. Automated image splicing detection using deep cnn-learned features and ann-based classifier. Signal, Image and Video Processing 15 , 7 (2021), 1601–1608
2021
-
[13]
An introduction to convolutional neural networks
O’shea, K., and Nash, R. An introduction to convolutional neural networks. arXiv preprint arXiv:1511.08458 (2015)
2015 arXiv
-
[14]
Language models are unsupervised multitask learners
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog 1, 8 (2019), 9
2019
-
[15]
A deep learning approach to detection of splicing and copy-move forgeries in images
Rao, Y., and Ni, J. A deep learning approach to detection of splicing and copy-move forgeries in images. In 2016 IEEE international workshop on information forensics and security (WIFS) (2016), IEEE, pp. 1–6
2016
-
[16]
K., Saha, S., Jain, V., Mondal, S., and Chadha, A
Sahoo, P., Singh, A. K., Saha, S., Jain, V., Mondal, S., and Chadha, A. A systematic survey of prompt engineering in large language models: Techniques and applications. arXiv preprint arXiv:2402.07927 (2024)
2024 arXiv
-
[17]
S., Reddy, D
Tallapragada, V. S., Reddy, D. V., and Kumar, G. P.Blind forgery detection using enhanced mask-region convolutional neural network. Multimedia Tools and Applications (2024), 1–24
2024
-
[18]
Is chatgpt a good sentiment analyzer? a preliminary study
W ang, Z., Xie, Q., Feng, Y., Ding, Z., Y ang, Z., and Xia, R. Is chatgpt a good sentiment analyzer? a preliminary study. arXiv preprint arXiv:2304.04339 (2023)
2023 arXiv
-
[19]
V., Zhou, D., et al
Wei, J., W ang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35 (2022), 24824–24837
2022
-
[20]
An experimental evaluation of llm on image classification
Wu, J., Tang, X., Y ang, Z., Hao, K., Lai, L., and Liu, Y. An experimental evaluation of llm on image classification. In Australasian Database Conference (2025), Springer, pp. 506–518
2025
-
[21]
Zhou, C., and Paffenroth, R. C. Anomaly detection with robust deep autoencoders. In Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining (2017), pp. 665–674. , Vol. 1, No. 1, Article . Publication date: August 2025
2017
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.