REVIEW 4 major objections 4 minor 1 cited by
Detecting Text Manipulation in Images using Vision Language Models
T0 review · 4 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read General-purpose vision-language models, prompted zero-shot, can detect manipulated text in images, and GPT-4o outperforms both open-source VLMs and specialized forgery detectors on this task.
desk verdict Useful zero-shot benchmark, but the headline GPT-4o vs. Qwen gap is partly a resolution confound; the ranking is conditional until matched-resolution runs are done. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is zero-shot prompting: a pretrained VLM receives a user prompt that defines the manipulation-detection task and an output format, plus a query image; the model's textual answer is thresholded into a binary pristine/manipulated label. The prompt design is systematically varied (L1 generic manipulation, L2 text-specific, L3 detailed with artifact descriptions), operating as a task specification that steers the model's reasoning. Image resolution acts as a second control variable, with GPT-4o-high (up to 768×2000) and Qwen's native resolution outperforming low-resolution variants. The specialized baselines FakeShield and SIDA, by contrast, use fixed prompts and lower-reso
What would settle it
Construct a separate test set of tampered ID documents produced by a different pipeline (e.g., an off-the-shelf generative editor rather than TextDiffuser-2, or real redacted IDs edited by hand) and run the same zero-shot prompts: if GPT-4o's F1 drops to the level of the specialized detectors or if FakeShield/SIDA overtake it, the paper's central ranking and its real-world relevance would be refuted.
Extended reading notes
Core claim
On its own terms, the paper's central claim is that text manipulation detection is not solved by existing image-forgery detectors and is best handled by a large vision-language model in a zero-shot setting: GPT-4o achieves AvgF1 0.86 on OSTF and 0.85 on FantasyID, while FakeShield and SIDA remain near 0.51 and 0.47/0.34, essentially random or worse. The open-source Qwen-2.5-VL-72B reaches 0.79 on OSTF and 0.56 on FantasyID, closer but still behind. The authors attribute success to two factors: the prompt must describe the text-manipulation context precisely (otherwise, as with FantasyID, the model labels everything manipulated), and the vision encoder must receive high-resolution images to s
Load-bearing premise
The evaluation assumes FantasyID's synthetic manipulations, generated by fine-tuning a text-rendering diffusion model on fantasy card templates, faithfully represent how a real fraudster would tamper with an ID document, so that a model that wins on FantasyID will also win on real KYC injections.
Editorial extensions
If this is right
- GPT-4o zero-shot can be used directly as a text manipulation detector for KYC-style ID documents, reaching AvgF1 0.85 on FantasyID.
- Specialized image-forgery VLMs (FakeShield, SIDA) do not transfer to text manipulation and need text-specific training or evaluation data.
- Open-source VLMs are closing the gap but remain behind closed-source on both scene text and ID-card manipulations (7% and 29% AvgF1 gaps for Qwen-VL).
- Prompt detail is decisive: insufficient context causes GPT-4o to label all FantasyID cards as manipulated, so deployment requires careful prompt engineering.
- Higher input resolution consistently improves detection, especially for small manipulated text regions in documents.
Reading between the lines
- GPT-4o's correct explanations could be used to distill a text-manipulation detector into a smaller open-source VLM, likely closing part of the 29% FantasyID gap (the paper suggests distillation as a direction but does not test it).
- The same zero-shot recipe could be turned into a localization tool by asking the model to enumerate manipulated words first, then verify each; the paper notes the model misses some regions and suggests this prompting strategy.
- If transfer holds, organizations could evaluate VLM-based detectors on synthetic 'fantasy' documents before committing to real ID data, avoiding the legal constraints of using real government IDs.
- A natural test is to vary the fraudster's generation pipeline: if a different fine-tuned diffusion model produces artifacts of different appearance, GPT-4o's ranking may shift, so robustness across generation methods needs explicit measurement.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents a benchmark for detecting text manipulation in images using zero-shot vision-language models (VLMs). The authors evaluate GPT-4o (low and high resolution), Qwen2.5-VL-72B, Llama-3.2-90B-Vision, two fine-tuned manipulation-detection VLMs (FakeShield, SIDA), and the non-VLM detector TruFor on the OSTF and FantasyID datasets. They report that GPT-4o-high achieves the best AvgF1 (0.86 on OSTF, 0.85 on FantasyID), that open-source VLMs are competitive but behind (Qwen: 0.79/0.56), and that specialized manipulation-detection VLMs perform near chance. They also present ablations showing that prompt detail and input image resolution affect performance, and they provide qualitative examples of GPT-4o reasoning.
Significance. If the comparisons were properly controlled, the paper would be a useful benchmark for an underexplored task and would support practical guidance: high input resolution matters, prompt context matters, and specialized manipulation detectors do not automatically transfer to text tampering. The prompt and resolution ablations are a strength, as is the inclusion of a document-style dataset. However, the headline closed-vs-open comparison is confounded by image resolution, and the FantasyID proxy may not transfer to real KYC scenarios. These issues need to be addressed before the ranking can be considered established.
major comments (4)
- [Sec. 3.2, Table 1, Fig. 4] The main closed-vs-open comparison is not resolution-controlled. GPT-4o-high processes images at 768x2000, while Qwen2.5-VL-72B uses its default resolution, described only as 'a sequence of 282 non-overlapping patches', and Llama uses up to four 560x560 patches. The paper's own Fig. 4 (right) and Table 1 show GPT-4o-low at 0.62 AvgF1 on FantasyID, only 0.06 above Qwen (0.56), compared with a 0.29 gap for GPT-4o-high. Since the paper demonstrates that resolution alone can cause a 0.23 drop (0.85 to 0.62), the headline claim that GPT-4o 'significantly outperforms' open-source VLMs is not established. Please include matched-resolution experiments, e.g., Qwen at a comparable effective pixel count, and report the effective resolution for each model.
- [Sec. 3.4, Fig. 4 (left)] Prompt sensitivity is only measured for GPT-4o. The ablation shows that L1/L2 prompts degrade FantasyID performance dramatically (GPT-4o labels all images as manipulated when the fantasy context is not given). The main results use a single detailed prompt for all three zero-shot models, but there is no evidence that this prompt is equally well matched to Qwen and Llama. A different prompt might change the ranking. Please report prompt-level results for all zero-shot models, or justify why a model-specific prompt search is unnecessary.
- [Sec. 3.3, Table 1, Sec. 5] The abstract and conclusions use 'significantly higher' without uncertainty quantification. Table 1 contains point estimates only; no confidence intervals, bootstrap resampling, or repeated runs are reported. Since the test sets are finite and API outputs may vary, the significance claims need statistical support, especially for differences that are small (e.g., Qwen 0.79 vs GPT-4o-low 0.84 on OSTF).
- [Sec. 3.1, Sec. 5] FantasyID is described as mimicking real-world KYC misuse, but it consists of synthetic fantasy cards manipulated by fine-tuning TextDiffuser-2 on pristine templates. The paper provides no evidence that these manipulations are representative of real fraudster injection attacks, e.g., no comparison to real tampered IDs or human-judged realism. This limits the practical conclusion about KYC risk; the limitation should be stated explicitly, and the transfer claim should be softened or supported.
minor comments (4)
- [Sec. 4, 'VLM vs TruFor' paragraph] The sentence 'For fair comparison, we do not include OSTF [26] as it is a purely localization baseline' is unclear: OSTF is a dataset, not a baseline. Please rephrase or correct.
- [Sec. 3.2, Qwen-VL] Please specify the patch size and effective resolution for Qwen, not only '282 non-overlapping patches'. This information is necessary to assess the resolution confound.
- [Sec. 5 and Fig. 2 caption] The conclusion refers to 'the proposed new dataset', but FantasyID was introduced in reference [15]; this paper proposes a benchmark, not a new dataset. The Fig. 2 caption says 'our proposed, FantasyID' and has the same issue. Please adjust the wording.
- [Sec. 3.3 and 4] The code availability statement only promises release upon acceptance. For a benchmark paper, consider providing at least the prompts and evaluation scripts as supplementary material during review.
Circularity Check
No circular derivation; empirical benchmark with a minor, non-load-bearing self-citation to the authors' own FantasyID dataset.
full rationale
The paper is an empirical benchmark, not a derivation: it reports zero-shot VLM outputs on fixed test sets (OSTF and FantasyID) with no fitted parameters and no equation-level construction. The central ranking (GPT-4o > Qwen > Llama, and specialized detectors near random) is read off Table 1 from model outputs, so it is not equivalent to the benchmark inputs by construction. OSTF [26] is an external dataset, giving the closed-vs-open-source claim independent grounding. The only self-referential element is FantasyID [15], a dataset from the authors' prior work (also labeled 'our proposed' in Fig. 2), used as an evaluation set. This is a minor self-citation and is not load-bearing: FantasyID's manipulated images were generated by fine-tuning TextDiffuser-2 on pristine cards before any VLM was queried, and the VLM predictions are externally reproducible, so the evaluation does not reduce to the citation. The reported resolution mismatch between GPT-4o-high and Qwen is a correctness/validity confound, not a circularity, and the paper's own resolution ablation (Fig. 4) does not turn the benchmark into a self-fulfilling prediction. No step in the paper reduces to its inputs, so the circularity score stays low.
Assumptions & free parameters
assumptions (3)
- domain assumption OSTF and FantasyID ground-truth labels are correct and representative of real text tampering, including KYC injection attacks.
- domain assumption Temperature 0 makes VLM outputs deterministic enough to compare without repeated sampling.
- domain assumption Macro F1 with a fixed 0.5 threshold for TruFor is an adequate comparison metric.
Cite this review
Pith. "Pith review of Detecting Text Manipulation in Images using Vision Language Models." pith.science (2026). https://pith.science/paper/6P6JWWP4
@misc{pith2026250910278,
author = {Pith},
title = {Pith review of: Detecting Text Manipulation in Images using Vision Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/6P6JWWP4}},
note = {Machine review of arXiv:2509.10278}
}
read the original abstract
Recent works have shown the effectiveness of Large Vision Language Models (VLMs or LVLMs) in image manipulation detection. However, text manipulation detection is largely missing in these studies. We bridge this knowledge gap by analyzing closed- and open-source VLMs on different text manipulation datasets. Our results suggest that open-source models are getting closer, but still behind closed-source ones like GPT- 4o. Additionally, we benchmark image manipulation detection-specific VLMs for text manipulation detection and show that they suffer from the generalization problem. We benchmark VLMs for manipulations done on in-the-wild scene texts and on fantasy ID cards, where the latter mimic a challenging real-world misuse.
Forward citations
Cited by 1 Pith paper
-
From Forgeries to Foundation Models: A Systematic Survey of Identity Document Attack and Detection
A systematic survey unifies presentation, digital injection, and GenAI synthesis attacks on identity documents, audits datasets for a reality gap, identifies SDGI in multimodal models, and reports APCER above 25% for ...
Reference graph
Works this paper leans on
-
[1]
Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report.ar...
arXiv 2025
-
[2]
Textdiffuser- 2: Unleashing the power of language models for text rendering
Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, and Furu Wei. Textdiffuser- 2: Unleashing the power of language models for text rendering. InEuropean Conference on Computer Vision, pages 386–402. Springer, 2024
2024
-
[3]
Image manipulation detection by multi-view multi-scale supervision
Xinru Chen, Chengbo Dong, Jiaqi Ji, Juan Cao, and Xirong Li. Image manipulation detection by multi-view multi-scale supervision. InProceedings of the IEEE/CVF international conference on computer vision, pages 14185–14193, 2021
2021
-
[4]
Chatbot arena: An open platform for evaluating llms by human preference
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. InForty-first International Conference on Machine Learning, 2024
2024
-
[5]
On the detection of digital face manipulation
Hao Dang, Feng Liu, Joel Stehouwer, Xiaoming Liu, and Anil K Jain. On the detection of digital face manipulation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern recognition, pages 5781–5790, 2020
2020
-
[6]
Mvss-net: Multi-view multi- scale supervised networks for image manipulation detection.IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 45(3):3539–3553, 2022
Chengbo Dong, Xinru Chen, Ruohan Hu, Juan Cao, and Xirong Li. Mvss-net: Multi-view multi- scale supervised networks for image manipulation detection.IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 45(3):3539–3553, 2022
2022
-
[7]
Casia image tampering detection evaluation database
Jing Dong, Wei Wang, and Tieniu Tan. Casia image tampering detection evaluation database. In2013 IEEE China summit and international conference on signal and information processing, pages 422–426. IEEE, 2013
2013
-
[8]
Nicholas Dufour, Arkanath Pathak, Pouya Samangouei, Nikki Hariri, Shashi Deshetti, Andrew Dudfield, Christopher Guess, Pablo Hernández Escayola, Bobby Tran, Mevan Babakar, et al. Ammeba: A large-scale survey and dataset of media-based misinformation in-the-wild.arXiv preprint arXiv:2405.11697, 1(8), 2024. VIDIT ET. AL.: DETECTING TEXT MANIPULA TION IN IMA...
arXiv 2024
Show all 17 references
-
[9]
The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ah- mad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024
2024 arXiv
-
[10]
Tru- for: Leveraging all-round clues for trustworthy image forgery detection and localization
Fabrizio Guillaro, Davide Cozzolino, Avneesh Sud, Nicholas Dufour, and Luisa Verdoliva. Tru- for: Leveraging all-round clues for trustworthy image forgery detection and localization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 20...
2023
-
[11]
Hier- archical fine-grained image forgery detection and localization
Xiao Guo, Xiaohong Liu, Zhiyuan Ren, Steven Grosz, Iacopo Masi, and Xiaoming Liu. Hier- archical fine-grained image forgery detection and localization. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3155–3165, 2023
2023
-
[12]
Sida: Social media image deepfake detection, localization and explanation with large multimodal model.arXiv preprint arXiv:2412.04292, 2024
Zhenglin Huang, Jinwei Hu, Xiangtai Li, Yiwei He, Xingyu Zhao, Bei Peng, Baoyuan Wu, Xi- aowei Huang, and Guangliang Cheng. Sida: Social media image deepfake detection, localization and explanation with large multimodal model.arXiv preprint arXiv:2412.04292, 2024
2024 arXiv
-
[13]
The point where reality meets fan- tasy: Mixed adversarial generators for image splice detection.Advances in neural information processing systems, 32, 2019
Vladimir V Kniaz, Vladimir Knyaz, and Fabio Remondino. The point where reality meets fan- tasy: Mixed adversarial generators for image splice detection.Advances in neural information processing systems, 32, 2019
2019
-
[14]
Exploring chatgpt for face presentation attack detection in zero and few-shot in-context learning
Alain Komaty, Hatef Otroshi Shahreza, Anjith George, and Sebastien Marcel. Exploring chatgpt for face presentation attack detection in zero and few-shot in-context learning. InIEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2025
2025
-
[15]
Fantasyid: A dataset for detecting digital manipulations of id-documents, 2025
Pavel Korshunov, Amir Mohammadi, Vidit Vidit, Christophe Ecabert, and Sébastien Marcel. Fantasyid: A dataset for detecting digital manipulations of id-documents, 2025. URLhttps: //arxiv.org/abs/2507.20808
2025 arXiv
-
[16]
Cat-net: Compression artifact tracing network for detection and localization of image splicing
Myung-Joon Kwon, In-Jae Yu, Seung-Hun Nam, and Heung-Kyu Lee. Cat-net: Compression artifact tracing network for detection and localization of image splicing. InProceedings of the IEEE/CVF winter conference on applications of computer vision, pages 375–384, 2021
2021
-
[17]
Lisa: Reasoning segmentation via large language model
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579–9589, 2024
2024
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.