Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Detecting Text Manipulation in Images using Vision Language Models

T0 review · 4 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Pith's one-line read General-purpose vision-language models, prompted zero-shot, can detect manipulated text in images, and GPT-4o outperforms both open-source VLMs and specialized forgery detectors on this task.

desk verdict Useful zero-shot benchmark, but the headline GPT-4o vs. Qwen gap is partly a resolution confound; the ranking is conditional until matched-resolution runs are done. read the letter →

arxiv 2509.10278 v1 pith:6P6JWWP4 submitted 2025-09-12 cs.CV

classification cs.CV
keywords textmanipulationdetectionvisionlanguagemodelszero-shotimageforgeryGPT-4oQwen-VLFantasyIDpromptdesign
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that general-purpose vision-language models, prompted zero-shot with a clear description of the text-manipulation task, can detect altered text in images, and that the closed-source GPT-4o does this substantially better than open-source VLMs or detectors fine-tuned for general image forgery. It benchmarks on in-the-wild scene text (OSTF) and on FantasyID, a dataset that mimics text injection attacks on ID documents for KYC fraud. The authors find that specialized manipulation detectors FakeShield and SIDA perform near chance on text manipulations, while GPT-4o reaches AvgF1 of 0.86 on OSTF and 0.85 on FantasyID. They also show that performance depends strongly on prompt detail and input image resolution. If this is right, zero-shot VLM prompting is a practical baseline for catching document text forgery, and existing specialized detectors need text-specific training.

What carries the argument

The central mechanism is zero-shot prompting: a pretrained VLM receives a user prompt that defines the manipulation-detection task and an output format, plus a query image; the model's textual answer is thresholded into a binary pristine/manipulated label. The prompt design is systematically varied (L1 generic manipulation, L2 text-specific, L3 detailed with artifact descriptions), operating as a task specification that steers the model's reasoning. Image resolution acts as a second control variable, with GPT-4o-high (up to 768×2000) and Qwen's native resolution outperforming low-resolution variants. The specialized baselines FakeShield and SIDA, by contrast, use fixed prompts and lower-reso

What would settle it

Construct a separate test set of tampered ID documents produced by a different pipeline (e.g., an off-the-shelf generative editor rather than TextDiffuser-2, or real redacted IDs edited by hand) and run the same zero-shot prompts: if GPT-4o's F1 drops to the level of the specialized detectors or if FakeShield/SIDA overtake it, the paper's central ranking and its real-world relevance would be refuted.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that text manipulation detection is not solved by existing image-forgery detectors and is best handled by a large vision-language model in a zero-shot setting: GPT-4o achieves AvgF1 0.86 on OSTF and 0.85 on FantasyID, while FakeShield and SIDA remain near 0.51 and 0.47/0.34, essentially random or worse. The open-source Qwen-2.5-VL-72B reaches 0.79 on OSTF and 0.56 on FantasyID, closer but still behind. The authors attribute success to two factors: the prompt must describe the text-manipulation context precisely (otherwise, as with FantasyID, the model labels everything manipulated), and the vision encoder must receive high-resolution images to s

Load-bearing premise

The evaluation assumes FantasyID's synthetic manipulations, generated by fine-tuning a text-rendering diffusion model on fantasy card templates, faithfully represent how a real fraudster would tamper with an ID document, so that a model that wins on FantasyID will also win on real KYC injections.

Editorial extensions

If this is right

  • GPT-4o zero-shot can be used directly as a text manipulation detector for KYC-style ID documents, reaching AvgF1 0.85 on FantasyID.
  • Specialized image-forgery VLMs (FakeShield, SIDA) do not transfer to text manipulation and need text-specific training or evaluation data.
  • Open-source VLMs are closing the gap but remain behind closed-source on both scene text and ID-card manipulations (7% and 29% AvgF1 gaps for Qwen-VL).
  • Prompt detail is decisive: insufficient context causes GPT-4o to label all FantasyID cards as manipulated, so deployment requires careful prompt engineering.
  • Higher input resolution consistently improves detection, especially for small manipulated text regions in documents.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • GPT-4o's correct explanations could be used to distill a text-manipulation detector into a smaller open-source VLM, likely closing part of the 29% FantasyID gap (the paper suggests distillation as a direction but does not test it).
  • The same zero-shot recipe could be turned into a localization tool by asking the model to enumerate manipulated words first, then verify each; the paper notes the model misses some regions and suggests this prompting strategy.
  • If transfer holds, organizations could evaluate VLM-based detectors on synthetic 'fantasy' documents before committing to real ID data, avoiding the legal constraints of using real government IDs.
  • A natural test is to vary the fraudster's generation pipeline: if a different fine-tuned diffusion model produces artifacts of different appearance, GPT-4o's ranking may shift, so robustness across generation methods needs explicit measurement.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper presents a benchmark for detecting text manipulation in images using zero-shot vision-language models (VLMs). The authors evaluate GPT-4o (low and high resolution), Qwen2.5-VL-72B, Llama-3.2-90B-Vision, two fine-tuned manipulation-detection VLMs (FakeShield, SIDA), and the non-VLM detector TruFor on the OSTF and FantasyID datasets. They report that GPT-4o-high achieves the best AvgF1 (0.86 on OSTF, 0.85 on FantasyID), that open-source VLMs are competitive but behind (Qwen: 0.79/0.56), and that specialized manipulation-detection VLMs perform near chance. They also present ablations showing that prompt detail and input image resolution affect performance, and they provide qualitative examples of GPT-4o reasoning.

Significance. If the comparisons were properly controlled, the paper would be a useful benchmark for an underexplored task and would support practical guidance: high input resolution matters, prompt context matters, and specialized manipulation detectors do not automatically transfer to text tampering. The prompt and resolution ablations are a strength, as is the inclusion of a document-style dataset. However, the headline closed-vs-open comparison is confounded by image resolution, and the FantasyID proxy may not transfer to real KYC scenarios. These issues need to be addressed before the ranking can be considered established.

major comments (4)
  1. [Sec. 3.2, Table 1, Fig. 4] The main closed-vs-open comparison is not resolution-controlled. GPT-4o-high processes images at 768x2000, while Qwen2.5-VL-72B uses its default resolution, described only as 'a sequence of 282 non-overlapping patches', and Llama uses up to four 560x560 patches. The paper's own Fig. 4 (right) and Table 1 show GPT-4o-low at 0.62 AvgF1 on FantasyID, only 0.06 above Qwen (0.56), compared with a 0.29 gap for GPT-4o-high. Since the paper demonstrates that resolution alone can cause a 0.23 drop (0.85 to 0.62), the headline claim that GPT-4o 'significantly outperforms' open-source VLMs is not established. Please include matched-resolution experiments, e.g., Qwen at a comparable effective pixel count, and report the effective resolution for each model.
  2. [Sec. 3.4, Fig. 4 (left)] Prompt sensitivity is only measured for GPT-4o. The ablation shows that L1/L2 prompts degrade FantasyID performance dramatically (GPT-4o labels all images as manipulated when the fantasy context is not given). The main results use a single detailed prompt for all three zero-shot models, but there is no evidence that this prompt is equally well matched to Qwen and Llama. A different prompt might change the ranking. Please report prompt-level results for all zero-shot models, or justify why a model-specific prompt search is unnecessary.
  3. [Sec. 3.3, Table 1, Sec. 5] The abstract and conclusions use 'significantly higher' without uncertainty quantification. Table 1 contains point estimates only; no confidence intervals, bootstrap resampling, or repeated runs are reported. Since the test sets are finite and API outputs may vary, the significance claims need statistical support, especially for differences that are small (e.g., Qwen 0.79 vs GPT-4o-low 0.84 on OSTF).
  4. [Sec. 3.1, Sec. 5] FantasyID is described as mimicking real-world KYC misuse, but it consists of synthetic fantasy cards manipulated by fine-tuning TextDiffuser-2 on pristine templates. The paper provides no evidence that these manipulations are representative of real fraudster injection attacks, e.g., no comparison to real tampered IDs or human-judged realism. This limits the practical conclusion about KYC risk; the limitation should be stated explicitly, and the transfer claim should be softened or supported.
minor comments (4)
  1. [Sec. 4, 'VLM vs TruFor' paragraph] The sentence 'For fair comparison, we do not include OSTF [26] as it is a purely localization baseline' is unclear: OSTF is a dataset, not a baseline. Please rephrase or correct.
  2. [Sec. 3.2, Qwen-VL] Please specify the patch size and effective resolution for Qwen, not only '282 non-overlapping patches'. This information is necessary to assess the resolution confound.
  3. [Sec. 5 and Fig. 2 caption] The conclusion refers to 'the proposed new dataset', but FantasyID was introduced in reference [15]; this paper proposes a benchmark, not a new dataset. The Fig. 2 caption says 'our proposed, FantasyID' and has the same issue. Please adjust the wording.
  4. [Sec. 3.3 and 4] The code availability statement only promises release upon acceptance. For a benchmark paper, consider providing at least the prompts and evaluation scripts as supplementary material during review.

Circularity Check

0 steps flagged · score 2.0 of 10

No circular derivation; empirical benchmark with a minor, non-load-bearing self-citation to the authors' own FantasyID dataset.

full rationale

The paper is an empirical benchmark, not a derivation: it reports zero-shot VLM outputs on fixed test sets (OSTF and FantasyID) with no fitted parameters and no equation-level construction. The central ranking (GPT-4o > Qwen > Llama, and specialized detectors near random) is read off Table 1 from model outputs, so it is not equivalent to the benchmark inputs by construction. OSTF [26] is an external dataset, giving the closed-vs-open-source claim independent grounding. The only self-referential element is FantasyID [15], a dataset from the authors' prior work (also labeled 'our proposed' in Fig. 2), used as an evaluation set. This is a minor self-citation and is not load-bearing: FantasyID's manipulated images were generated by fine-tuning TextDiffuser-2 on pristine cards before any VLM was queried, and the VLM predictions are externally reproducible, so the evaluation does not reduce to the citation. The reported resolution mismatch between GPT-4o-high and Qwen is a correctness/validity confound, not a circularity, and the paper's own resolution ablation (Fig. 4) does not turn the benchmark into a self-fulfilling prediction. No step in the paper reduces to its inputs, so the circularity score stays low.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No parameters are fitted to data and no new entities are invented. The benchmark rests on dataset-validity, determinism, and metric assumptions.

assumptions (3)
  • domain assumption OSTF and FantasyID ground-truth labels are correct and representative of real text tampering, including KYC injection attacks.
    Section 3.1 describes FantasyID as synthetic edits on fantasy cards that mimic real ID misuse; the paper's practical conclusions depend on this proxy being valid.
  • domain assumption Temperature 0 makes VLM outputs deterministic enough to compare without repeated sampling.
    Section 3.2 sets temperature to 0 for GPT-4o, Qwen, and Llama, but no confidence intervals or repeated API calls are reported, leaving API non-determinism unquantified.
  • domain assumption Macro F1 with a fixed 0.5 threshold for TruFor is an adequate comparison metric.
    Section 3.3 fixes the threshold at 0.5; changing it would alter TruFor's numbers, and macro F1 treats classes equally, which interacts with the imbalanced OSTF split.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Detecting Text Manipulation in Images using Vision Language Models." pith.science (2026). https://pith.science/paper/6P6JWWP4

@misc{pith2026250910278,
  author       = {Pith},
  title        = {Pith review of: Detecting Text Manipulation in Images using Vision Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6P6JWWP4}},
  note         = {Machine review of arXiv:2509.10278}
}
read the original abstract

Recent works have shown the effectiveness of Large Vision Language Models (VLMs or LVLMs) in image manipulation detection. However, text manipulation detection is largely missing in these studies. We bridge this knowledge gap by analyzing closed- and open-source VLMs on different text manipulation datasets. Our results suggest that open-source models are getting closer, but still behind closed-source ones like GPT- 4o. Additionally, we benchmark image manipulation detection-specific VLMs for text manipulation detection and show that they suffer from the generalization problem. We benchmark VLMs for manipulations done on in-the-wild scene texts and on fantasy ID cards, where the latter mimic a challenging real-world misuse.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. From Forgeries to Foundation Models: A Systematic Survey of Identity Document Attack and Detection

    cs.CR 2026-07 unverdicted novelty 7.0 of 10

    A systematic survey unifies presentation, digital injection, and GenAI synthesis attacks on identity documents, audits datasets for a reality gap, identifies SDGI in multimodal models, and reports APCER above 25% for ...

Reference graph

Works this paper leans on

17 extracted references · 5 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report.ar...

  2. [2]

    Textdiffuser- 2: Unleashing the power of language models for text rendering

    Jingye Chen, Yupan Huang, Tengchao Lv, Lei Cui, Qifeng Chen, and Furu Wei. Textdiffuser- 2: Unleashing the power of language models for text rendering. InEuropean Conference on Computer Vision, pages 386–402. Springer, 2024

  3. [3]

    Image manipulation detection by multi-view multi-scale supervision

    Xinru Chen, Chengbo Dong, Jiaqi Ji, Juan Cao, and Xirong Li. Image manipulation detection by multi-view multi-scale supervision. InProceedings of the IEEE/CVF international conference on computer vision, pages 14185–14193, 2021

  4. [4]

    Chatbot arena: An open platform for evaluating llms by human preference

    Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael Jordan, Joseph E Gonzalez, et al. Chatbot arena: An open platform for evaluating llms by human preference. InForty-first International Conference on Machine Learning, 2024

  5. [5]

    On the detection of digital face manipulation

    Hao Dang, Feng Liu, Joel Stehouwer, Xiaoming Liu, and Anil K Jain. On the detection of digital face manipulation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern recognition, pages 5781–5790, 2020

  6. [6]

    Mvss-net: Multi-view multi- scale supervised networks for image manipulation detection.IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 45(3):3539–3553, 2022

    Chengbo Dong, Xinru Chen, Ruohan Hu, Juan Cao, and Xirong Li. Mvss-net: Multi-view multi- scale supervised networks for image manipulation detection.IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 45(3):3539–3553, 2022

  7. [7]

    Casia image tampering detection evaluation database

    Jing Dong, Wei Wang, and Tieniu Tan. Casia image tampering detection evaluation database. In2013 IEEE China summit and international conference on signal and information processing, pages 422–426. IEEE, 2013

  8. [8]

    Ammeba: A large-scale survey and dataset of media-based misinformation in-the-wild.arXiv preprint arXiv:2405.11697, 1(8), 2024

    Nicholas Dufour, Arkanath Pathak, Pouya Samangouei, Nikki Hariri, Shashi Deshetti, Andrew Dudfield, Christopher Guess, Pablo Hernández Escayola, Bobby Tran, Mevan Babakar, et al. Ammeba: A large-scale survey and dataset of media-based misinformation in-the-wild.arXiv preprint arXiv:2405.11697, 1(8), 2024. VIDIT ET. AL.: DETECTING TEXT MANIPULA TION IN IMA...

Show all 17 references
  1. [9]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ah- mad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

  2. [10]

    Tru- for: Leveraging all-round clues for trustworthy image forgery detection and localization

    Fabrizio Guillaro, Davide Cozzolino, Avneesh Sud, Nicholas Dufour, and Luisa Verdoliva. Tru- for: Leveraging all-round clues for trustworthy image forgery detection and localization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 20...

  3. [11]

    Hier- archical fine-grained image forgery detection and localization

    Xiao Guo, Xiaohong Liu, Zhiyuan Ren, Steven Grosz, Iacopo Masi, and Xiaoming Liu. Hier- archical fine-grained image forgery detection and localization. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3155–3165, 2023

  4. [12]

    Sida: Social media image deepfake detection, localization and explanation with large multimodal model.arXiv preprint arXiv:2412.04292, 2024

    Zhenglin Huang, Jinwei Hu, Xiangtai Li, Yiwei He, Xingyu Zhao, Bei Peng, Baoyuan Wu, Xi- aowei Huang, and Guangliang Cheng. Sida: Social media image deepfake detection, localization and explanation with large multimodal model.arXiv preprint arXiv:2412.04292, 2024

  5. [13]

    The point where reality meets fan- tasy: Mixed adversarial generators for image splice detection.Advances in neural information processing systems, 32, 2019

    Vladimir V Kniaz, Vladimir Knyaz, and Fabio Remondino. The point where reality meets fan- tasy: Mixed adversarial generators for image splice detection.Advances in neural information processing systems, 32, 2019

  6. [14]

    Exploring chatgpt for face presentation attack detection in zero and few-shot in-context learning

    Alain Komaty, Hatef Otroshi Shahreza, Anjith George, and Sebastien Marcel. Exploring chatgpt for face presentation attack detection in zero and few-shot in-context learning. InIEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2025

  7. [15]

    Fantasyid: A dataset for detecting digital manipulations of id-documents, 2025

    Pavel Korshunov, Amir Mohammadi, Vidit Vidit, Christophe Ecabert, and Sébastien Marcel. Fantasyid: A dataset for detecting digital manipulations of id-documents, 2025. URLhttps: //arxiv.org/abs/2507.20808

  8. [16]

    Cat-net: Compression artifact tracing network for detection and localization of image splicing

    Myung-Joon Kwon, In-Jae Yu, Seung-Hun Nam, and Heung-Kyu Lee. Cat-net: Compression artifact tracing network for detection and localization of image splicing. InProceedings of the IEEE/CVF winter conference on applications of computer vision, pages 375–384, 2021

  9. [17]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579–9589, 2024

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.