{"id":"f56dbb9a-47b3-4868-a0de-86f1fb63515a","arxiv_id":"2411.09273","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"GPT-4V answers identical questions much less accurately when they are presented as images than as text, even when it can extract the image content nearly perfectly.","lead":"This paper tests whether a multimodal AI model gives the same answer when the same question is presented as an image or as text, and finds that GPT-4V answers far less accurately from images. The authors build parallel vision-text datasets, report large modality gaps, and show that a simple 'describe the image first' prompt narrows them.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Converter information preservation is the load-bearing assumption; the OCR ablation tests transcription, not task-level usability, so image deficits could be conversion artifacts rather than evidence of modality bias.","rationale":"The reader identified converter information preservation as the weakest assumption; I agree that this is the pivotal condition, but I sharpen it: the paper's own OCR ablation (Sec. 5.4) is insufficient to establish task-level usability of the image representation. The OCR test asks the model to 'extract the exact content' and reports near-perfect accuracy, but this does not prove that the model can recover the spatial structure (row/column positions, equation grouping) needed to solve the task. For Table Understanding, free-form transcription could be perfect while the model still fails to bind numbers to the correct row/column indices; for MES, OCR accuracy is only 0.68/0.76, so a large part of the image deficit could be recognition failure even if conditional consistency shows no correlation. The central finding of cross-modal inconsistency would survive even if the mechanism were visual extraction rather than reasoning, but the paper's specific claim that the gap 'can be attributed solely to the model's internal reasoning processes' and the reader's characterization of a 'language-side bias' would be overstatements. My proposed test directly settles whether the deficit persists when the image is demonstrably and fully usable, which is the minimal condition for the paper's causal interpretation. Since the reader already conditioned acceptance on verifiable data and fixes, my concern does not change the verdict; it reinforces CONDITIONAL by pointing to a specific missing analysis. Agreement is partial because my concern is closely related but distinct: the reader emphasized conversion fidelity, while I emphasize the gap between OCR transcription and task-level information access, which the current ablation does not close.","tokens_in":12556,"tokens_out":8658,"duration_ms":83734,"concrete_test":"Select the 50 Table Understanding (and MES) instances. Independently render each text instance to an image at multiple resolutions and font sizes. Use a deterministic table parser to extract a row/column-indexed representation from each rendered image and compare it with the LaTeX source; keep only instances where the structured extraction exactly matches. Then recompute GPT-4V image accuracy on this 'perfect structured extraction' subset and compare it with text accuracy. If image accuracy remains far below text accuracy on this subset, the deficit is in reasoning; if it rises to text level, the original gap is a conversion/layout artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central inference—that GPT-4V's lower image accuracy reflects a language-side reasoning bias—requires that the text↔image converters (Sec. 4.1) preserve all task-relevant information, including layout. The authors verify OCR for image→text and use screenshots for text→image, but the OCR ablation in Sec. 5.4 only checks that the model can transcribe 'exact content'; it does not verify that the image representation supports the spatial operations the task demands (e.g., row/column lookup in Table Understanding, equation structure in MES). Since the Limitations concede lossless conversion is unavailable, and Table 3's conditional consistency analysis does not report conditional accuracy, the sizable image deficits (TU 0.93→0.03; MMLU 1.00→0.74) could in part be renderer artifacts (small fonts, ambiguous layout) rather than intrinsic cross-modal inconsistency. If a high-fidelity structured rendering eliminated the gap, the 'solely internal reasoning' conclusion would collapse.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper defines cross-modal consistency for multimodal LLMs as output invariance under information-preserving modality conversion (Eq. 1), constructs parallel text-image datasets for seven tasks, evaluates GPT-4V on 50 instances per task, and reports substantial accuracy drops when input is presented as an image rather than text (e.g., Table Understanding 0.93 to 0.03, MMLU 1.00 to 0.74). It also proposes a Vision-Depicting-Prompting (VDP) method that extracts a textual description of the image before answering, which partially recovers the image-mode deficit and improves consistency scores. The central conclusion is that GPT-4V exhibits language-dominant, modality-dependent internal reasoning, rather than a unified multimodal understanding.","tokens_in":12699,"tokens_out":5193,"duration_ms":51518,"significance":"If the result is sound, the paper makes a useful contribution by formalizing a cross-modal consistency metric and providing a concrete evaluation framework. The dataset release (promised but not yet available) and the VDP method are potentially valuable. The paper is honest about the difficulty of information-preserving conversion and explicitly acknowledges limitations. However, the strength of the central claim depends on two premises: that the converters preserve all task-relevant information (including layout), and that the 50-instance subsets are representative. Both premises are currently insufficiently supported, so the evidence for 'solely internal reasoning' remains incomplete.","major_comments":[{"comment":"The ablation study does not rule out perceptual failure as the source of the image deficit. For Table Understanding, an OCR accuracy of 1.00 only shows that the model can transcribe the numbers; the task also requires locating row/column positions in the rendered image, which the OCR check does not verify. The conditional consistency scores in Table 3 are consistency rates, not conditional accuracies, so they cannot isolate whether the gap stems from failed spatial perception or from modality-dependent reasoning. The claim that the gap 'can be attributed solely to the model's internal reasoning processes' (Section 5.4) is therefore not supported by the reported evidence.","section":"Section 5.4, Tables 2-3"},{"comment":"The evaluation uses 50 randomly selected instances per task with no confidence intervals, significance tests, or released data. The MMLU text accuracy of 1.00 suggests the sampled split is unusually easy and may not be representative. With n=50, the 95% confidence interval for a proportion is roughly ±0.14, so the 'sizeable decrease' threshold of 10% is within the noise band for several tasks. The paper should report bootstrap or exact confidence intervals and make the sampled instances public to substantiate the cross-task generalization claim.","section":"Section 5.2, Table 1"},{"comment":"The information-preservation assumption is load-bearing for the entire framework, yet the Limitations section acknowledges that lossless conversion is not generally available. For Math Equation Solving, the OCR accuracy is only 0.68-0.76 (Table 2), so the converter demonstrably does not preserve information for a substantial fraction of instances. The conditional consistency analysis in Table 3 does not report task accuracy on the subset with correct OCR, so it cannot separate conversion artifacts from genuine cross-modal inconsistency. The authors should report image accuracy and text accuracy on the same subset of instances where OCR is verified correct, to establish that the gap persists even when information is actually preserved.","section":"Sections 3 and 4.1, Limitations"}],"minor_comments":[{"comment":"The figure label 'T ext' should read 'Text'; the figure is also not referenced in the body text before it appears.","section":"Figure 1"},{"comment":"The notation for the converter is inconsistent: K^q_{a,b} is used in the formal definition, but K_{a,b} appears in Section 4.1. Clarify whether the converter is query-dependent in all uses.","section":"Section 3"},{"comment":"The statement that 'OCR accuracy approaches nearly 100% for all instances of LogicQA, MMLU, and Table Understanding tasks' should specify that this refers to the 50-image subsets used in the experiments, not the full source datasets.","section":"Section 5.4"},{"comment":"The text versions of math equations contain unrendered LaTeX (e.g., 'x ˆ 2', '$$2x + 4 = |-17 + 3|$$'), which makes the appendix difficult to read; the intended rendered formulas should be shown.","section":"Appendix A and B"},{"comment":"The paper states that 'our results are manually collected for pairwise data instances,' but no inter-annotator agreement or verification protocol is described; at least a brief description would strengthen reproducibility.","section":"Section 5.2"},{"comment":"The ethical consideration paragraph is generic and does not discuss the specific ethical implications of reporting consistency failures in a commercial model or the potential for misuse of the VDP method; this is a presentation issue rather than a technical one.","section":"Ethical Consideration"}],"recommendation":"major_revision","confidential_remarks":"The paper is a timely and readable study of cross-modal consistency in GPT-4V. The core idea is interesting and the datasets, once released, could be a useful community resource. However, the experimental methodology needs strengthening before the central claim can be fully accepted: the OCR ablation does not test task-level spatial reasoning, the 50-instance samples lack statistical support, and the information-preservation assumption is not validated for tasks where OCR accuracy is far from perfect. The paper's own Limitations section concedes the latter point. I would recommend major revision, asking for conditional accuracy analyses, confidence intervals, and public release of the sampled data. The citation of related work (e.g., Chou et al. 2024 on (in-)consistency) is appropriate, but the novelty claim should be softened in light of this prior work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this paper gives the field a clean formalization of cross-modal consistency and a set of parallel vision-language datasets, then reports a robust empirical pattern on GPT-4V: the same task instances get markedly worse accuracy when the input is an image instead of text. The direction holds across all seven tasks, and the Table Understanding drop (0.93 to 0.03) is too large to wave away. I think the core finding is real and worth engaging with.\n\nWhat's actually new: the definition of consistency via information-preserving converters is a useful contribution, and the consistency metric (proportion of paired instances with identical output) cleanly separates agreement from per-modality accuracy. The OCR ablation is a sensible check: for LogicQA, MMLU, and Table Understanding the model transcribes the images nearly perfectly, which supports the claim that low image accuracy isn't simply a perception failure. VDP—first describing the image in text, then answering—recovers a large chunk of the image-mode deficit, which is a practical finding practitioners can use today: transcribe or re-enter text before asking for reasoning.\n\nThe soft spots are real but fixable. The evaluation uses only 50 instances per task, with no error bars or significance tests; the MMLU text accuracy of 1.00 signals the sampled subset is unusually easy, so some gaps may be inflated. The datasets are promised but not yet released, which limits immediate use. VDP is validated on the same tasks it was designed for. And the paper's phrase 'attributed solely to the model's internal reasoning' overreaches: the OCR ablation verifies exact content extraction, not whether the image supports the spatial operations the task demands (row/column lookup in tables, equation structure). Part of the image-mode deficit could be renderer or layout artifacts. That said, the VDP result—where the model's own text description of the image largely closes the gap—suggests the dominant factor is a language-side reasoning bias, not the rendering per se.\n\nBottom line: this is a solid, honest paper with a slightly overclaimed attribution in Section 5.4. It deserves a serious referee, and the fixes are straightforward: more samples, significance testing, release the data, and soften the 'solely' language. I'd cite the framework if I were working on multimodal evaluation, and I'd bring it to a reading group.\n\nRecommendation: send it to peer review with a request for revision.","headline":"Useful formalization of cross-modal consistency with a robust but thinly evidenced GPT-4V finding; deserves peer review with revisions.","tokens_in":13303,"tokens_out":2344,"would_cite":true,"duration_ms":23670,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GPT-4V is cross-modally inconsistent: for task instances carrying the same information, its output depends on whether the input is text or image.","keywords":["cross-modal consistency","multimodal large language models","GPT-4V","vision-language evaluation","information-preserving conversion","Vision-Depicting-Prompting","parallel vision-language dataset"],"falsifier":"Run the seven parallel tasks on a fresh GPT-4V session with the same converters and manually verified conversions; if image-mode accuracy on Table Understanding and MMLU matches text-mode accuracy, or the consistency score $C_t$ exceeds 0.95 across tasks, then the reported cross-modal inconsistency would fail to replicate and the claim would be withdrawn.","tokens_in":12306,"feed_emoji":"🖼️","tokens_out":6969,"duration_ms":62134,"temperature":0.7,"pith_summary":"This paper tries to establish that GPT-4V, a multimodal model that handles both vision and language, does not treat equivalent information equally across modalities. The authors define cross-modal consistency as the requirement that a model give the same answer to a task instance whether it is presented as text or as an image, under an information-preserving conversion. Across seven parallel vision-language datasets they constructed, GPT-4V's accuracy drops substantially when the same problem arrives as an image, in one case from 0.93 in text to 0.03 in image. The paper concludes that GPT-4V has a measurable language-side bias in its internal reasoning, and proposes a two-step prompting method, Vision-Depicting-Prompting, that routes image content through text and recovers much of the gap.","feed_headline":"GPT-4V flunks image versions of problems it solves in text","feed_subtitle":"A new parallel benchmark shows accuracy drops up to 90% when identical tasks arrive as images.","key_machinery":"The framework rests on the notion of an information-preserving converter $K^q_{a,b}$ between modalities, and on the consistency score $C_t$ that measures how often outputs agree across converted pairs. The authors build parallel datasets for seven tasks using OCR with human verification for image-to-text conversion and rendered screenshots for text-to-image conversion, selecting tasks where they argue the conversion preserves all task-relevant information. The Vision-Depicting-Prompting (VDP) method is the corrective mechanism: it first asks the model to transcribe the image into text, then answers with both the transcription and the original image, exploiting the model's stronger text-side reasoning.","core_discovery":"The central discovery is that GPT-4V is cross-modally inconsistent as formalized by the paper's consistency score $C_t = \\frac{1}{n}\\sum_i c_i^M$, the fraction of paired instances where the model's answer is identical under text and image input. Table 1 reports large image-mode accuracy drops on all seven tasks, including Table Understanding (0.93 text vs 0.03 image) and MMLU (1.00 vs 0.74), while a one-step OCR ablation shows near-perfect extraction on LogicQA, MMLU, and Table Understanding. Because the information in the image is available to the model, the authors attribute the gap to the model's internal reasoning processes differing by modality rather than to failed perception. The Vision-Depicting-Prompting results support the same interpretation: having the model first describe the image in text raises image-mode accuracy by up to 57 percentage points on understanding tasks and substantially increases consistency.","pith_inferences":["The same converter-based consistency test could be applied to other multimodal models to determine whether a language-side bias is specific to GPT-4V or a general property of vision-language alignment.","If the bias is architectural, then input modality becomes a reliability surface: the same prompt could yield different factual behavior depending on whether it is delivered as text or as a screenshot.","A natural extension is to use consistency scores as a training objective or as a diagnostic signal for modality-invariant representations.","The framework could be extended to audio or video modalities whenever an information-preserving converter exists, testing whether the language-dominant pattern generalizes."],"forward_implications":["Users of GPT-4V should expect markedly worse answers on image-only inputs than on equivalent text inputs, especially for table and reading-comprehension tasks.","Cross-modal consistency should be reported alongside per-modality accuracy, since a model can score well in each modality taken alone yet be unreliable across them.","For tasks where OCR is nearly perfect, the bottleneck is not visual information extraction but modality-specific reasoning, so improving visual reasoning is the targeted next step.","VDP-style prompting is a practical workaround: on Table Understanding it raises image accuracy from 0.03 to 0.93 and consistency from 0.10 to 0.90."],"supporting_citations":[{"why":"Supplies the LaTeX math questions used to construct the easy Math Equation Solving text side.","marker":"Hendrycks et al., 2021b"},{"why":"Supplies the GSM8K math reasoning instances that are rendered into images for the Math Reasoning task.","marker":"Cobbe et al., 2021"},{"why":"Supplies the LogicQA logical reasoning questions converted into image format.","marker":"Liu et al., 2020"},{"why":"Supplies the MMLU passages and questions used for the Reading Comprehension task.","marker":"Hendrycks et al., 2021a"},{"why":"Demonstrates cross-lingual performance bias in LLMs, the prior that motivates treating modalities like languages under a shared consistency framework.","marker":"Zhang et al., 2023b"},{"why":"Provides the within-modality consistency analysis for multimodal models that this paper extends to cross-modal settings.","marker":"Chou et al., 2024"}],"fun_headline_variants":["GPT-4V flunks image tasks it aces in text","Image inputs slash GPT-4V accuracy by up to 90%","Cross-modal test exposes GPT-4V's weak vision reasoning","GPT-4V's text-to-image gap: accuracy craters on vision","New benchmark: GPT-4V inconsistent across modalities"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The measured gaps rest on the assumption that the converters—OCR with human verification and rendered screenshots—preserve all task-relevant information, including layout, for every one of the seven tasks; the authors concede in the Limitations section that lossless conversion is not generally available, so a conversion artifact could mimic a true inconsistency.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4V flunks image tasks it aces in text","Image inputs slash GPT-4V accuracy by up to 90%","Cross-modal test exposes GPT-4V's weak vision reasoning","GPT-4V's text-to-image gap: accuracy craters on vision","New benchmark: GPT-4V inconsistent across modalities"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000979,"raw_usage":{"total_tokens":4174,"prompt_tokens":977,"completion_tokens":3197,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":3107}},"tokens_in":593,"tokens_out":3197,"duration_ms":25757,"temperature":1.0,"reasoning_tokens":3107,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:49:18.842337+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the seven parallel tasks on a fresh GPT-4V session with the same converters and manually verified conversions; if image-mode accuracy on Table Understanding and MMLU matches text-mode accuracy, or the consistency score $C_t$ exceeds 0.95 across tasks, then the reported cross-modal inconsistency would fail to replicate and the claim would be withdrawn.","supporting_citations":[],"review_version":1}