{"id":"b95ec842-8924-485b-95b6-acc6a295d272","arxiv_id":"2509.10278","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In zero-shot benchmarks, GPT-4o outperforms open-source VLMs and specialized manipulation detectors on text tampering detection in scene images and fantasy ID documents.","lead":"This paper tests how well vision language models, including GPT-4o, Qwen, and Llama, detect edited text in photos and fake ID cards. It finds GPT-4o performs best, open-source models trail behind, and specialized image forgery detectors fail to generalize to text tampering.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GPT-4o's reported advantage may be confounded with image resolution; Qwen and Llama were not given matched high-resolution inputs despite the paper's own ablation showing resolution is a major factor on FantasyID.","rationale":"The reader's verdict (CONDITIONAL) is appropriate and I agree with the overall cautious stance. However, I identify a different load-bearing concern than the one listed as the weakest_assumption. The reader focused on FantasyID's validity as a proxy for real fraudster attacks, which affects the practical relevance of the 'KYC' framing. My concern is more fundamental: even if FantasyID were a perfect proxy, the performance comparison itself is confounded by input resolution. The paper's own Fig. 4 shows that resolution alone can shift GPT-4o's FantasyID AvgF1 from 0.85 to 0.62. The open-source models, especially Qwen, run at effective resolutions far below GPT-4o-high, so the reported 29-point gap on FantasyID may not reflect inherent detection capability. This is not an external criticism but an internal inconsistency: the paper acknowledges resolution as a key factor in the ablation yet does not control for it in the main comparison. The concrete test I propose would settle this: matching Qwen's max_pixels to GPT-4o-high's and re-running the evaluation. If the gap collapses, the paper's central claim would need to be reframed as 'GPT-4o with high-resolution input outperforms' rather than 'GPT-4o outperforms.' If the gap persists, the claim is secure. Given the paper's transparency in showing the resolution ablation, this is a fixable issue, and the verdict remains CONDITIONAL pending this check. I mark agreement as 'partial' because the reader did mention resolution as part of the 'not fully controlled' caveat, but they chose a different weakest_assumption; I see the resolution confound as the more load-bearing issue for the central claim.","tokens_in":9197,"tokens_out":6261,"duration_ms":64248,"concrete_test":"Re-run Qwen2.5-VL-72B and Llama-3.2-90B-Vision on the FantasyID and OSTF test sets with a resolution configuration matched to GPT-4o-high. For Qwen, set max_pixels to 768*2000 (or the closest supported value) so that the input is processed at roughly the same pixel count as GPT-4o-high; for Llama, use a patch layout that yields at least 768x2000 effective resolution (e.g., increase the number of 560x560 patches to cover the full image at that density). Keep the L3 prompts from Fig. 3 and temperature 0. Recompute AvgF1 and report F1(P), F1(M). If Qwen's FantasyID AvgF1 rises from 0.56 to above 0.7, the original gap is largely a resolution artifact; if it remains near 0.56, the central claim is robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that GPT-4o significantly outperforms open-source VLMs rests on Table 1, which shows GPT-4o-high achieving AvgF1 0.85 on FantasyID versus 0.56 for Qwen. However, the comparison does not control for input image resolution. In Section 3.2, GPT-4o-high processes images at 768x2000 pixels, while Qwen2.5-VL-72B uses its default resolution, described as 'a sequence of 282 non-overlapping patches' (Section 3.2), which implies a significantly smaller effective pixel count than GPT-4o-high. Llama-3.2-90B-Vision uses up to 4 patches of 560x560, which for a portrait ID card may be roughly comparable to GPT-4o-high, yet it still scores lower; however, the more critical comparison is GPT-4o-high vs. Qwen, where resolution disparity is large. The paper's own ablation (Fig. 4, right panel) shows GPT-4o performance on FantasyID drops from 0.85 to 0.62 when using low resolution (longest side 512), a 0.23 drop nearly equal to the entire Qwen gap. Since Qwen's effective resolution is likely below GPT-4o-low, the reported gap may largely reflect resolution rather than model capability. This is an internal inconsistency: the paper demonstrates that resolution has a strong effect, yet the main benchmark does not match resolutions across models. Without matching resolution, the headline 'GPT-4o significantly outperforms' is not established. The FantasyID proxy validity (the reader's weakest_assumption) affects the practical relevance, but the resolution confound affects the validity of the performance claim itself, making it the more load-bearing concern.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a benchmark for detecting text manipulation in images using zero-shot vision-language models (VLMs). The authors evaluate GPT-4o (low and high resolution), Qwen2.5-VL-72B, Llama-3.2-90B-Vision, two fine-tuned manipulation-detection VLMs (FakeShield, SIDA), and the non-VLM detector TruFor on the OSTF and FantasyID datasets. They report that GPT-4o-high achieves the best AvgF1 (0.86 on OSTF, 0.85 on FantasyID), that open-source VLMs are competitive but behind (Qwen: 0.79/0.56), and that specialized manipulation-detection VLMs perform near chance. They also present ablations showing that prompt detail and input image resolution affect performance, and they provide qualitative examples of GPT-4o reasoning.","tokens_in":9583,"tokens_out":6551,"duration_ms":65958,"significance":"If the comparisons were properly controlled, the paper would be a useful benchmark for an underexplored task and would support practical guidance: high input resolution matters, prompt context matters, and specialized manipulation detectors do not automatically transfer to text tampering. The prompt and resolution ablations are a strength, as is the inclusion of a document-style dataset. However, the headline closed-vs-open comparison is confounded by image resolution, and the FantasyID proxy may not transfer to real KYC scenarios. These issues need to be addressed before the ranking can be considered established.","major_comments":[{"comment":"The main closed-vs-open comparison is not resolution-controlled. GPT-4o-high processes images at 768x2000, while Qwen2.5-VL-72B uses its default resolution, described only as 'a sequence of 282 non-overlapping patches', and Llama uses up to four 560x560 patches. The paper's own Fig. 4 (right) and Table 1 show GPT-4o-low at 0.62 AvgF1 on FantasyID, only 0.06 above Qwen (0.56), compared with a 0.29 gap for GPT-4o-high. Since the paper demonstrates that resolution alone can cause a 0.23 drop (0.85 to 0.62), the headline claim that GPT-4o 'significantly outperforms' open-source VLMs is not established. Please include matched-resolution experiments, e.g., Qwen at a comparable effective pixel count, and report the effective resolution for each model.","section":"Sec. 3.2, Table 1, Fig. 4"},{"comment":"Prompt sensitivity is only measured for GPT-4o. The ablation shows that L1/L2 prompts degrade FantasyID performance dramatically (GPT-4o labels all images as manipulated when the fantasy context is not given). The main results use a single detailed prompt for all three zero-shot models, but there is no evidence that this prompt is equally well matched to Qwen and Llama. A different prompt might change the ranking. Please report prompt-level results for all zero-shot models, or justify why a model-specific prompt search is unnecessary.","section":"Sec. 3.4, Fig. 4 (left)"},{"comment":"The abstract and conclusions use 'significantly higher' without uncertainty quantification. Table 1 contains point estimates only; no confidence intervals, bootstrap resampling, or repeated runs are reported. Since the test sets are finite and API outputs may vary, the significance claims need statistical support, especially for differences that are small (e.g., Qwen 0.79 vs GPT-4o-low 0.84 on OSTF).","section":"Sec. 3.3, Table 1, Sec. 5"},{"comment":"FantasyID is described as mimicking real-world KYC misuse, but it consists of synthetic fantasy cards manipulated by fine-tuning TextDiffuser-2 on pristine templates. The paper provides no evidence that these manipulations are representative of real fraudster injection attacks, e.g., no comparison to real tampered IDs or human-judged realism. This limits the practical conclusion about KYC risk; the limitation should be stated explicitly, and the transfer claim should be softened or supported.","section":"Sec. 3.1, Sec. 5"}],"minor_comments":[{"comment":"The sentence 'For fair comparison, we do not include OSTF [26] as it is a purely localization baseline' is unclear: OSTF is a dataset, not a baseline. Please rephrase or correct.","section":"Sec. 4, 'VLM vs TruFor' paragraph"},{"comment":"Please specify the patch size and effective resolution for Qwen, not only '282 non-overlapping patches'. This information is necessary to assess the resolution confound.","section":"Sec. 3.2, Qwen-VL"},{"comment":"The conclusion refers to 'the proposed new dataset', but FantasyID was introduced in reference [15]; this paper proposes a benchmark, not a new dataset. The Fig. 2 caption says 'our proposed, FantasyID' and has the same issue. Please adjust the wording.","section":"Sec. 5 and Fig. 2 caption"},{"comment":"The code availability statement only promises release upon acceptance. For a benchmark paper, consider providing at least the prompts and evaluation scripts as supplementary material during review.","section":"Sec. 3.3 and 4"}],"recommendation":"major_revision","confidential_remarks":"The paper relies heavily on the authors' own FantasyID dataset, and the wording 'proposed new dataset' in the conclusion may overstate novelty; this is not a disqualifying issue but should be framed correctly. The main risk is the resolution confound in the closed-vs-open comparison; if the authors can run Qwen (and Llama, where possible) at matched resolution and still observe a large gap, the central claim would be substantially strengthened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful, clearly written zero-shot evaluation of VLMs for text manipulation detection, and the finding that specialist forgery detectors (FakeShield, SIDA) essentially fail on this task is probably robust. But the paper's headline ranking of GPT-4o over Qwen rests on a comparison that is not resolution-matched, and the paper's own ablation shows resolution alone can account for most of the gap. That needs to be fixed before the ranking is usable.\n\nWhat's new: first systematic comparison of GPT-4o, Qwen2.5-VL, and Llama-3.2-Vision on OSTF and FantasyID for text tampering, with a clean binary classification setup. The prompt and resolution ablations on GPT-4o are genuinely informative; the finding that detailed prompts matter a lot, and that low resolution kills performance on FantasyID, is practically useful. Benchmarking TruFor and the two specialized VLMs against the same data is also a useful data point.\n\nSoft spots: (1) Resolution confound. Qwen runs at default resolution (282 patches) while GPT-4o-high gets 768x2000. The paper itself shows GPT-4o drops from 0.85 to 0.62 AvgF1 on FantasyID when going to low resolution. The Qwen gap is 0.29. So the claim that GPT-4o significantly outperforms is not established. They should run matched-resolution experiments, or at least report Qwen with a comparable high-resolution setting. (2) FantasyID is the authors' previous work, but the text calls it \"our proposed, FantasyID\" in a figure caption and \"the proposed new dataset\" in the conclusion; that overstates novelty. (3) No code or data released; \"make public upon acceptance\" is not enough for a benchmark paper. (4) No error bars or repeats, though that is common in API-based evaluation. (5) The synthetic fantasy-ID proxy is a reasonable first step but should be defended more; it is not a real forgery test.\n\nBottom line: worth refereeing, but the resolution confound is a load-bearing weakness. The paper should be revised to either match resolutions or clearly frame the comparison as \"default settings\" rather than \"capability.\" If they cannot rerun, the low-resolution GPT-4o number (0.62) is the honest comparison against Qwen's default; with that, Qwen is much closer. I would send it to peer review expecting major revision. Practitioners should not take the GPT-4o-vs-Qwen gap at face value.","headline":"Useful zero-shot benchmark, but the headline GPT-4o vs. Qwen gap is partly a resolution confound; the ranking is conditional until matched-resolution runs are done.","tokens_in":10093,"tokens_out":2210,"would_cite":false,"duration_ms":24385,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"General-purpose vision-language models, prompted zero-shot, can detect manipulated text in images, and GPT-4o outperforms both open-source VLMs and specialized forgery detectors on this task.","keywords":["text manipulation detection","vision language models","zero-shot detection","image forgery detection","GPT-4o","Qwen-VL","FantasyID","prompt design"],"falsifier":"Construct a separate test set of tampered ID documents produced by a different pipeline (e.g., an off-the-shelf generative editor rather than TextDiffuser-2, or real redacted IDs edited by hand) and run the same zero-shot prompts: if GPT-4o's F1 drops to the level of the specialized detectors or if FakeShield/SIDA overtake it, the paper's central ranking and its real-world relevance would be refuted.","tokens_in":9121,"feed_emoji":"🔍","tokens_out":3714,"duration_ms":37483,"temperature":0.7,"pith_summary":"The paper tries to establish that general-purpose vision-language models, prompted zero-shot with a clear description of the text-manipulation task, can detect altered text in images, and that the closed-source GPT-4o does this substantially better than open-source VLMs or detectors fine-tuned for general image forgery. It benchmarks on in-the-wild scene text (OSTF) and on FantasyID, a dataset that mimics text injection attacks on ID documents for KYC fraud. The authors find that specialized manipulation detectors FakeShield and SIDA perform near chance on text manipulations, while GPT-4o reaches AvgF1 of 0.86 on OSTF and 0.85 on FantasyID. They also show that performance depends strongly on prompt detail and input image resolution. If this is right, zero-shot VLM prompting is a practical baseline for catching document text forgery, and existing specialized detectors need text-specific training.","feed_headline":"GPT-4o beats specialized forgery detectors on tampered text","feed_subtitle":"A zero-shot vision model catches edited text in ID cards and scene images where dedicated manipulation detectors fail.","key_machinery":"The central mechanism is zero-shot prompting: a pretrained VLM receives a user prompt that defines the manipulation-detection task and an output format, plus a query image; the model's textual answer is thresholded into a binary pristine/manipulated label. The prompt design is systematically varied (L1 generic manipulation, L2 text-specific, L3 detailed with artifact descriptions), operating as a task specification that steers the model's reasoning. Image resolution acts as a second control variable, with GPT-4o-high (up to 768×2000) and Qwen's native resolution outperforming low-resolution variants. The specialized baselines FakeShield and SIDA, by contrast, use fixed prompts and lower-reso","core_discovery":"On its own terms, the paper's central claim is that text manipulation detection is not solved by existing image-forgery detectors and is best handled by a large vision-language model in a zero-shot setting: GPT-4o achieves AvgF1 0.86 on OSTF and 0.85 on FantasyID, while FakeShield and SIDA remain near 0.51 and 0.47/0.34, essentially random or worse. The open-source Qwen-2.5-VL-72B reaches 0.79 on OSTF and 0.56 on FantasyID, closer but still behind. The authors attribute success to two factors: the prompt must describe the text-manipulation context precisely (otherwise, as with FantasyID, the model labels everything manipulated), and the vision encoder must receive high-resolution images to s","pith_inferences":["GPT-4o's correct explanations could be used to distill a text-manipulation detector into a smaller open-source VLM, likely closing part of the 29% FantasyID gap (the paper suggests distillation as a direction but does not test it).","The same zero-shot recipe could be turned into a localization tool by asking the model to enumerate manipulated words first, then verify each; the paper notes the model misses some regions and suggests this prompting strategy.","If transfer holds, organizations could evaluate VLM-based detectors on synthetic 'fantasy' documents before committing to real ID data, avoiding the legal constraints of using real government IDs.","A natural test is to vary the fraudster's generation pipeline: if a different fine-tuned diffusion model produces artifacts of different appearance, GPT-4o's ranking may shift, so robustness across generation methods needs explicit measurement."],"forward_implications":["GPT-4o zero-shot can be used directly as a text manipulation detector for KYC-style ID documents, reaching AvgF1 0.85 on FantasyID.","Specialized image-forgery VLMs (FakeShield, SIDA) do not transfer to text manipulation and need text-specific training or evaluation data.","Open-source VLMs are closing the gap but remain behind closed-source on both scene text and ID-card manipulations (7% and 29% AvgF1 gaps for Qwen-VL).","Prompt detail is decisive: insufficient context causes GPT-4o to label all FantasyID cards as manipulated, so deployment requires careful prompt engineering.","Higher input resolution consistently improves detection, especially for small manipulated text regions in documents."],"fun_headline_variants":["GPT-4o beats specialized forgery detectors on tampered text","Zero-shot GPT-4o outshines dedicated forgery detectors on text tampering","Text manipulation in images: GPT-4o tops dedicated forgery detectors","Open-source VLMs catch up, but GPT-4o still wins on tampered text","Vision-language models uncover text forgeries that specialized detectors miss"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The evaluation assumes FantasyID's synthetic manipulations, generated by fine-tuning a text-rendering diffusion model on fantasy card templates, faithfully represent how a real fraudster would tamper with an ID document, so that a model that wins on FantasyID will also win on real KYC injections.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4o beats specialized forgery detectors on tampered text","Zero-shot GPT-4o outshines dedicated forgery detectors on text tampering","Text manipulation in images: GPT-4o tops dedicated forgery detectors","Open-source VLMs catch up, but GPT-4o still wins on tampered text","Vision-language models uncover text forgeries that specialized detectors miss"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00118,"raw_usage":{"total_tokens":4670,"prompt_tokens":662,"completion_tokens":4008,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":406,"completion_tokens_details":{"reasoning_tokens":3909}},"tokens_in":406,"tokens_out":4008,"duration_ms":33172,"temperature":1.0,"reasoning_tokens":3909,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T17:59:12.658725+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a separate test set of tampered ID documents produced by a different pipeline (e.g., an off-the-shelf generative editor rather than TextDiffuser-2, or real redacted IDs edited by hand) and run the same zero-shot prompts: if GPT-4o's F1 drops to the level of the specialized detectors or if FakeShield/SIDA overtake it, the paper's central ranking and its real-world relevance would be refuted.","supporting_citations":[],"review_version":1}