{"id":"4a21c2ff-4617-49fa-9d18-1228510a2e72","arxiv_id":"2607.21617","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"General-purpose VLMs systematically rewrite perturbed words back to the original — up to 4.5 WER points on English — with rewriting tied to representation similarity and word length.","lead":"VLMs often 'fix' imperfect text instead of transcribing it literally, and this study quantifies how often. It introduces FaithC4, a multilingual perturbation benchmark, and finds that general-purpose VLMs rewrite up to 4.5 percentage points more than traditional OCR when words are scrambled or substituted.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Mechanistic gating claim rests on a single-model, correlational probe; if it fails to replicate across architectures or full-document decoding, the paper's explanation—though not the core behavioral finding—is unsupported.","rationale":"The reader's weakest_assumption correctly identifies the mechanistic claim as the load-bearing point most exposed to correctness risk. The behavioral phenomenon—VLMs rewrite imperfect text more than traditional OCR—is convincingly demonstrated by the error-classification data and the three-tier degradation ordering. But the paper goes beyond that phenomenon to assert a specific internal mechanism: rewriting is gated by how much the final-layer FFN representation diverges from the clean encoding. That mechanism is inferred from a single architecture, a single-word probe, a small number of rewritten instances, and an in-sample quartile split. The paper's own limitation section acknowledges architecture dependence. Because the abstract and conclusion present this mechanism as a 'consistent pattern' and use it to explain word-length effects, the mechanism is a substantive part of the paper's contribution. If it fails to replicate, the central behavioral claim survives but the explanatory narrative and the proposed representation-distance account would need to be withdrawn. This does not change the CONDITIONAL verdict: the paper should be accepted only if the mechanistic claims are either replicated across architectures/tasks or explicitly reframed as speculative.","tokens_in":19834,"tokens_out":10124,"duration_ms":113906,"concrete_test":"Run the Section 4.2 probe on at least two additional open-weight general VLMs (e.g., InternVL3.5-4B, Gemma4-E4B) and repeat the quartile analysis on full-document extraction by computing per-perturbed-word FFN similarity during generation. Preregister three checks: (i) Q4-vs-Q1 rewritten-rate difference is positive, (ii) rewritten rate decreases monotonically across quartiles, (iii) rewritten rate is 0% for words of length ≥8. If any check fails, the mechanistic claim should be downgraded to a Qwen3-VL-specific observation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's advertised mechanistic conclusion—rewriting is gated by final-layer FFN representation similarity (Section 4.2)—is supported only by correlational probing of Qwen3-VL-4B on a highlighted single-word task. The effect is small (Table 9: 4.66% vs. 0.0% rewritten across FFN-similarity quartiles, i.e., 19 samples in Q4), the quartile partition is defined on the same similarity variable being tested, and the probe rewrite rate (4.9–6.0%, Table 8) is an order of magnitude below full-document rewrite rates (29–65%, Table 5). The paper itself concedes 'the internal dynamics may differ across architectures' (Section 6). If the FFN-similarity threshold is Qwen3-VL-specific or an artifact of the highlighted-word setup, then the explanation of why some VLMs rewrite and why short words show an 8-character cutoff is not established. The behavioral three-tier ordering might still hold; what fails is the causal story that the abstract and conclusion present as a 'consistent pattern.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether vision-language models (VLMs) faithfully transcribe imperfect text or silently rewrite it into more plausible forms. It introduces FaithC4, a multilingual benchmark of 1,455 synthetic single-page documents in English, Chinese, and Korean, with three perturbation families (scramble, random, visual) at 8% of eligible words. The authors evaluate 15 systems across three categories—general-purpose VLMs, OCR-specialized VLMs, and traditional OCR—reporting WER/CER degradation relative to clean baselines, an error-type taxonomy distinguishing rewrites from ordinary OCR errors, and evidence of non-local error propagation to unperturbed words. They also probe Qwen3-VL-4B layer-by-layer to argue that rewriting is gated by the cosine similarity of final-layer FFN representations between original and perturbed inputs, and that word length affects rewriting with a sharp cutoff at 8 characters. The central behavioral claim is that general-purpose VLMs are substantially less faithful transcribers than OCR-specialized models and traditional OCR when text is imperfect.","tokens_in":20042,"tokens_out":6240,"duration_ms":59391,"significance":"If the behavioral finding holds, the paper addresses an important and underexplored failure mode: clean-text OCR benchmarks overstate VLM transcription reliability for degraded or noisy real-world documents. The controlled perturbation benchmark, the 15-system comparison, and the error taxonomy are valuable contributions. The non-local error propagation result is practically important. The mechanistic probing, while suggestive, is preliminary and not yet at the level of evidence claimed in the abstract and conclusion. The paper also ships falsifiable empirical results: the three-tier ordering of degradation, the rewrite-rate differences, and the word-length cutoff can be tested by other groups. Overall this is a solid empirical contribution whose significance is currently tempered by overgeneralized interpretive claims.","major_comments":[{"comment":"The abstract states that OCR-specialized VLMs degrade by 0.2–2 points, but Table 2 reports olmOCR-2-7B at +2.32 pp under scramble, which is outside this range. Also, PaddleOCR-VL-1.5 (OCR-specialized) degrades more than Qwen3.5-4B (general-purpose) under random substitution (+1.45 vs +0.74), so the three-tier ordering is not exception-free even in English. Please either report per-model ranges with exceptions or soften the category-level claim to 'most OCR-specialized VLMs'.","section":"Abstract / §4.1, Table 2"},{"comment":"The paper claims the three-tier ordering (traditional OCR < OCR-specialized VLM < general-purpose VLM) 'holds in all three languages.' Table 3 contradicts this for Chinese: docTR degrades by +9.31 pp under visual substitution, higher than most general VLMs (e.g., Gemma4-E2B +6.66, InternVL3.5 +7.72), and Tesseract under random degrades +4.63 pp, comparable to Qwen3.5-4B +4.82. Section 6 acknowledges traditional OCR has limited multilingual support, but the conclusion still asserts the cross-lingual ordering. The authors should restrict the claim to English or provide a per-script analysis that accounts for baseline failures/truncation (e.g., docTR's high failure rate on Chinese in Table A.3).","section":"§4.1 / Table 3 / Conclusion"},{"comment":"The mechanistic conclusion that rewriting is gated by layer-35 FFN representation similarity is based on a single model (Qwen3-VL-4B), a single-word highlighted probe, and very few rewritten samples: Table 9 reports 19 rewritten words in Q4, 1 in Q3, and 0 in Q1/Q2. No confidence intervals or significance tests are reported, and the quartile partition is defined on the same similarity variable whose association with rewriting is being tested. Moreover, the probe rewrite rate (4.9–6.0%, Table 8) is an order of magnitude below the full-document rewrite rates (29–65%, Table 5), so the mechanism may not transfer to the setting that motivates the paper. The paper itself concedes that 'the internal dynamics may differ across architectures' (§6). At minimum, the abstract and conclusion should rephrase this as a hypothesis specific to Qwen3-VL, or the authors should add a second model and a prop","section":"§4.2 / Tables 8–9 / §6"}],"minor_comments":[{"comment":"Missing space: 'We introduceFaithC4' should be 'We introduce FaithC4'.","section":"Abstract"},{"comment":"The column arrangement for Orig/Scramble/Random/Visual with paired Pert/Non-pert columns is very difficult to parse. Please reformat, e.g., with separate subheaders or a wide-to-long layout.","section":"Table 6"},{"comment":"The PaddleOCR-VL-1.5 reference describes the model as 0.9B, while Table A.2 and Table A.3 list it as 1.5B. Please reconcile.","section":"References / Table A.2"},{"comment":"Equation (1) defines WER with N as the reference token count, but for Chinese the metric is character-level. Please state explicitly that N is the reference character count in that case.","section":"§3.3, Eq. (1)"},{"comment":"The caption mentions panels (a) and (b), but the panels are not labeled in the figure. Add panel labels.","section":"Figure 2"},{"comment":"The 'Highlight Attn.' values (e.g., 0.00012) are not directly comparable to the attention ratios in Table 7. Consider reporting the same ratio format for consistency.","section":"Table 9"}],"recommendation":"major_revision","confidential_remarks":"This is a well-executed empirical study with a useful benchmark and a clearly presented behavioral finding. The main problem is interpretive overreach: the abstract and conclusion state the cross-lingual ordering and the FFN-similarity gating mechanism more strongly than the evidence supports, and the paper's own limitations section acknowledges key gaps. I would advise the editor that the paper is likely publishable after revision, but the authors must either add supporting analyses (e.g., a second probed model, significance testing, or a per-script break-down for Chinese) or carefully confine the claims to what the current data show."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper's headline behavioral finding is solid and useful—general-purpose VLMs rewrite imperfect text a lot more than OCR-specialized models or traditional OCR—but the advertised mechanism (rewriting gated by final-layer FFN representation similarity) is a single-model, correlational probe and is being oversold.\n\nThe new contribution here is FaithC4, a carefully constructed multilingual perturbation benchmark with 1,455 rendered documents in English, Chinese, Korean, three perturbation families, and evaluation of 15 systems. The three-tier ordering (traditional OCR < OCR-specialized VLM < general-purpose VLM) is broadly supported by the tables, with a few acknowledged exceptions (Gemini-3-Flash behaves like OCR; PaddleOCR-VL does poorly on English random). The non-local error propagation claim is notable: corrupting ~5% of words amplifies errors on the unperturbed remainder by up to ~10x for general VLMs. That's a genuinely new and practically important finding. The word-length cutoff (no rewriting above 7 characters) is also interesting. The ablation in Appendix F shows a simple no-correction prompt cuts degradation by 30–50%, which is actionable.\n\nThe soft spots are mostly around the mechanistic story. The quartile analysis in Table 9 has only 19 rewritten samples in the high-similarity quartile, the probe rewrite rate (5–6%) is an order of magnitude below full-document rates (29–65%), and it's one model (Qwen3-VL-4B) on a single-word highlighted task. The paper's own Limitations section concedes that internal dynamics may differ across architectures. So the behavioral claim stands, but the claim of a 'consistent pattern' for gating is not established. I'd also want error bars or significance tests; the tables report point estimates only, and some between-model differences are within the range where noise matters. The 8-character cutoff rests on only 52 samples for the 10+ bucket. Also, benchmark and code are not released, which limits verification and adoption. And there's a small internal inconsistency: the abstract and body say 5–10x non-local amplification, while the conclusion says 2–5x.\n\nMinor issues aside, this is a serious paper. The behavioral result is reproducible from the tables, the experimental design is careful, and the limitations are honestly stated. For a practitioner deciding between a general VLM and an OCR-specialized model for literal transcription, this is directly relevant. It deserves peer review, but I'd ask the authors to soften the mechanism claims, add significance testing or confidence intervals, release the benchmark, and fix the inconsistency before acceptance.","headline":"Solid behavioral result, oversold mechanism—worth a serious referee.","tokens_in":20573,"tokens_out":3960,"would_cite":true,"duration_ms":39405,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Vision-language models often rewrite imperfect text into more plausible words instead of transcribing what is actually written—a failure mode that clean-text OCR benchmarks cannot detect.","keywords":["vision-language models","OCR faithfulness","transcription","text perturbation","language prior","FaithC4","representation similarity","rewriting behavior"],"falsifier":"A direct test: run the same highlighted-word probe on a second general-purpose VLM family (e.g., a different architecture) and check whether rewritten cases are again confined to the top FFN-similarity quartile. If a model rewrites words whose final-layer representation has diverged far from the original, or fails to rewrite words whose representation stays close, the gating claim is falsified. A simpler behavioral check: find any general-purpose VLM whose WER degradation under scramble perturbation is below 0.6 points on English, matching traditional OCR, which would break the three-tier orde","tokens_in":19679,"feed_emoji":"🔤","tokens_out":5354,"duration_ms":46951,"temperature":0.7,"pith_summary":"The paper claims that vision-language models (VLMs) treat imperfect text as a prediction problem rather than a transcription problem: when a word contains a typo, a scramble, or a visually ambiguous character, the model often outputs the word it expects to see instead of the word that is written. To demonstrate this, the authors build FaithC4, a multilingual benchmark of 1,455 rendered document pages in English, Chinese, and Korean, perturbing a small fraction of words and measuring word error rate degradation across 15 systems. The result is a consistent ordering: traditional OCR degrades by under 0.6 points, OCR-specialized VLMs by 0.2–2 points, and general-purpose VLMs by up to 4.5 points on English. Layer-by-layer probing of one model shows rewriting is gated by how much the perturbation shifts the word's internal representation, and that short words are rewritten far more often than long ones, with a sharp cutoff at 8 characters. The paper matters because clean-text OCR benchmarks cannot see this failure mode, and domains that require literal transcription—legal records, medical notes, historical manuscripts—would silently get 'corrected' output.","feed_headline":"VLMs rewrite imperfect text instead of reading it","feed_subtitle":"A small typo can trigger a silent 'correction,' and errors spill onto surrounding words—clean-text benchmarks miss it.","key_machinery":"The central instruments are two. First, FaithC4—a benchmark of 1,455 single-page documents rendered from perturbed text (scramble, random substitution, visually similar substitution) in English, Chinese, and Korean, using the perturbed text as ground truth so that any output matching the original unperturbed word counts as a 'rewrite.' Second, a layer-wise probing method on Qwen3-VL-4B that records the cosine similarity between feed-forward network (FFN) representations of the original and perturbed inputs at the final token, together with attention to the highlighted word's image patches; the FFN similarity at the final layer is the quantity that gates whether rewriting occurs. The probe is","core_discovery":"The paper establishes that VLMs are not faithful transcribers of imperfect text. Under controlled perturbations of rendered documents, general-purpose VLMs degrade by up to 4.5 WER points on English, OCR-specialized VLMs by 0.2–2 points, and traditional OCR by under 0.6 points, with the same three-tier ordering in Chinese and Korean. Probing Qwen3-VL-4B layer by layer shows rewriting fires only when a perturbed word's final-layer feed-forward representation stays close to the original encoding: top-quartile similarity samples are rewritten at 4.66%, bottom-quartile samples never, with attention playing no decisive role. Word length is a step function: 4–6 character words are rewritten up to","pith_inferences":["If the representation-gating mechanism generalizes, then decoding-time interventions that push the final-layer representation away from the clean encoding—rather than prompt instructions—might be a more direct mitigation than the no-correction prompt tested here.","The 7-to-8-character cutoff suggests a testable prediction for other scripts and models: rewriting should vanish for words whose perturbation exceeds the model's typical 'neighborhood' size in representation space; measuring that neighborhood could predict rewrite rates without running probes.","The same mechanism that produces rewriting may explain context leakage (outputting a salient document word instead of the target), implying that both errors are one phenomenon—representation collapse toward the closest known token—rather than separate failure modes.","For downstream tasks like document QA, faithful transcription may not always be the goal; rewriting could sometimes improve answer quality, so deployments should decide explicitly whether literalness or plausibility is the objective."],"forward_implications":["Clean-text OCR benchmarks systematically overstate VLM transcription faithfulness, because they contain no imperfect text for rewriting to act on.","General-purpose VLMs are a poor default for literal-transcription tasks such as legal, medical, or historical document processing; the paper concludes traditional OCR or carefully selected OCR-specialized VLMs remain the safest choice.","A small corruption rate has outsized cost: corrupting about 5% of words multiplies errors on unperturbed text by 5–10× in general VLMs, so degradation cannot be predicted from perturbation rate alone.","Explicit no-correction prompting roughly halves the degradation on tested models but does not eliminate it, and slightly raises baseline WER.","Rewriting requires the language prior to recognize a plausible target: random substitutions that share no lexical overlap are never rewritten, so the failure mode is specifically about recovering known words."],"fun_headline_variants":["VLMs silently rewrite imperfect text, not read it","Why VLMs 'fix' typos: a word-length cutoff emerges","Vision models rewrite errors only when the face fits","Short words trigger rewriting in VLMs, probing shows"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise for the mechanistic explanation is that the representation-similarity gating observed in one model (Qwen3-VL-4B) with a single-word probe is the actual cause of rewriting across architectures; the paper itself notes that internal dynamics may differ across models, so if that correlation does not hold in other VLMs, the explanation fails even if the behavioral degradation remains.","fun_headline_variants_meta":{"raw":{"variants":["VLMs silently rewrite imperfect text, not read it","Why VLMs 'fix' typos: a word-length cutoff emerges","Vision models rewrite errors only when the face fits","Short words trigger rewriting in VLMs, probing shows"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000121,"raw_usage":{"total_tokens":956,"prompt_tokens":800,"completion_tokens":156,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":90}},"tokens_in":544,"tokens_out":156,"duration_ms":2659,"temperature":1.0,"reasoning_tokens":90,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T12:40:55.926335+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test: run the same highlighted-word probe on a second general-purpose VLM family (e.g., a different architecture) and check whether rewritten cases are again confined to the top FFN-similarity quartile. If a model rewrites words whose final-layer representation has diverged far from the original, or fails to rewrite words whose representation stays close, the gating claim is falsified. A simpler behavioral check: find any general-purpose VLM whose WER degradation under scramble perturbation is below 0.6 points on English, matching traditional OCR, which would break the three-tier orde","supporting_citations":[],"review_version":1}