{"id":"ccbdc9a3-b513-45a2-9894-092bbacf7d19","arxiv_id":"2501.02669","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Training VLMs to explicitly convert images to text before reasoning transfers simple-to-hard generalization from text to image, and this conversion skill can be internalized to keep inference cheap.","lead":"This paper introduces synthetic visual reasoning puzzles that come in both image and text versions, and shows that vision-language models reason much better from text than from images. It proposes training recipes, built around teaching the model to convert images into text before reasoning, that transfer text reasoning to images and shrink this 'modality imbalance'.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The flagship Mix+/Align-Mix+ results are not evidence of simple-to-hard generalization: training includes HARD Text examples, so the image gains reflect transfer of already-trained HARD text reasoning rather than S2H learning.","rationale":"The reader's weakest assumption (CoT-trace metric) is real but local: it affects only Consecutive Table Readout, and even there the paper reports best-of-two extraction. My stress-test focuses on a more general threat to the central claim. The paper is transparent in Section 4 and Algorithm 1 that Mix+ and Align-Mix+ include HARD Text supervision, but the title, abstract, and Figure 5 labels present these as S2H-generalization results. Under the paper's own definition (training on X_SIMPLE only), they are not: the model has seen HARD solutions in text. If the headline numbers are read as \"simple-to-hard generalization on images,\" they overstate what the experiments show. The defensible contribution is the more modest cross-modal transfer finding, which is well supported by the gradient-alignment analysis and the Qwen replication. This is an interpretability and framing issue rather than a falsified mechanism, so it does not justify rejection; it does justify a conditional acceptance with a revised claim and the proposed disjoint-held-out check. The reader's conditional verdict already captures this level of risk, so I do not move the verdict.","tokens_in":53327,"tokens_out":7666,"duration_ms":76932,"concrete_test":"Build a new held-out HARD evaluation set for Table Readout, Grid Navigation, and Visual Analogy whose underlying hard patterns (for Visual Analogy, (attribute, relation) combinations; for Table Readout, path-shape compositions; for Grid Navigation, obstacle/object configurations) are provably disjoint from the HARD Text examples used in Mix+/Align-Mix+ training. Retrain Mix+ and Align-Mix+ with the exact Section 4 protocol and evaluate only on this disjoint set. If accuracy falls to the <=25% Mix baseline, the claimed S2H generalization is largely an artifact of transferring seen HARD-text solutions; if accuracy remains near 64/92/35 and 76/96/56, the concern is refuted. Run the same evaluation on final answers as well, not only the CoT trace, to separate metric effects.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is a mismatch between the central claim and the training protocol. The paper's formal definition (Sections 2.3-2.4) ties S2H generalization to supervision on X_SIMPLE only. For the three non-S2H tasks, however, the headline results in Figure 5 come from Mix+ and Align-Mix+, which by Algorithm 1 (Section 2.4, \"When also training on HARD examples\") mix HARD Text examples into training; Section 4 states \"We then adapt Mix supervision to include HARD Text in training.\" Text/Image/Mix trained only on SIMPLE examples stay below 25%, so the 64/92/35 and 76/96/56 numbers are obtained only after the model has seen solutions to HARD instances in text. Thus these results demonstrate cross-modal transfer of already-trained HARD text reasoning to images, not simple-to-hard generalization. The abstract's \"training on the SIMPLE version\" is therefore misleading for the paper's flagship results; the strongest defensible claim is the narrower one in Section 1.1(ii) about injecting HARD-text capability. Genuine S2H evidence is limited to Consecutive Table Readout and Pattern-Heldout Visual Analogy, and the former relies on the CoT-trace metric flagged by the reader.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces a synthetic benchmark for studying modality imbalance in vision-language models (VLMs) using three algorithmic visual reasoning tasks, each with SIMPLE and HARD versions and matched text/image inputs. It proposes supervision strategies centered on explicit image-to-text conversion and reports that they improve performance on HARD image inputs. The main claims are: (i) when a VLM already shows S2H generalization on text, Mix supervision transfers part of that behavior to images; (ii) on tasks where neither modality S2H-generalizes, including HARD Text examples in training via Mix+ or Align-Mix+ yields large gains on HARD images; and (iii) gradient-alignment scores correlate with these gains. The paper also reports ablations on chain-of-thought, text warm-up, and multi-task training, and a partial replication on Qwen2.5-VL.","tokens_in":53494,"tokens_out":6623,"duration_ms":61730,"significance":"The controlled task suite and the image-to-text conversion intervention are potentially useful for the VLM community, and the paper is commendable for releasing code, comparing two model families, and reporting a large number of ablations. If the central claim were established for genuinely SIMPLE-only training, this would be an important step. As written, however, the headline 'simple-to-hard' generalization on the three main tasks is not what is tested: the successful protocols train on HARD text examples. The strongest defensible contribution is a more modest one about cross-modal transfer of already-learned HARD text reasoning, plus the observation that image-to-text conversion and chain-of-thought are important for such transfer. The paper should be revised to align its framing with its protocol.","major_comments":[{"comment":"The paper's formal framework defines S2H generalization as training on X_SIMPLE and evaluating on X_HARD: Section 2.3 splits X into X_SIMPLE and X_HARD, and Section 2.4 describes supervisions applied to SIMPLE examples. Section 4 then says 'We then adapt Mix supervision to include HARD Text in training,' and Algorithm 1 sets N_HARD = N/2 for Image-via-Text+ and Mix+, adding {x(t), CoT(x), f(x)} for x ~ X_HARD. The headline results in Figure 5, including 64/92/35 for Mix+ and 76/96/56 for Align-Mix+, are therefore not demonstrations of simple-to-hard generalization by the paper's own definition; the SIMPLE-only Text/Image/Mix baselines remain below 25%. The abstract's phrase 'training on the SIMPLE version' is accordingly misleading. Please either rename this phenomenon, for example as cross-modal transfer of HARD-text reasoning, or re-derive the claim so that the definition of S2H is the one actually tested. The genuinely S2H results, such as Consecutive Table Readout and Pattern-Heldout Visual Analogy, should be clearly separated from the HARD-text-injection results.","section":"Sections 2.3–2.4, 4, Algorithm 1"},{"comment":"The motivating modality-gap result for Consecutive Table Readout, Text 80% vs Image 20% on HARD in Figure 3, is measured on the chain-of-thought trace rather than the final answer. Section C.6.1 states: 'we find that the verbose CoT trace shows S2H generalization, and not the final solution that the model reports. Hence, we report our evaluation performance for Consecutive Table Readout on the CoT trace.' Section E.1 says the paper reports the best performance of the two extraction sources. If exact match on the final answer is used instead, the text-side S2H advantage and the modality gap can change substantially. Please report final-answer accuracy separately, state evaluation-set sizes, and justify the CoT-trace choice as the primary metric; otherwise the opening illustration of modality imbalance is not supported by a standard answer-correctness metric.","section":"Section C.6.1 and Section E.1"},{"comment":"All accuracy and loss curves are reported without error bars or multiple random seeds, and the evaluation-set sizes are not stated for the main figures (some appendix tables mention 100 or 500 examples). Without variance information, conclusions such as 'Align-Mix+ further enhances generalization' in Figure 5, or the gradient-alignment comparisons in Figures 7–9, cannot be reliably separated from training noise. Please add at least two or three seeds with standard deviations, or state the evaluation-set sizes and, where feasible, bootstrap confidence intervals.","section":"Figures 3–9 and Tables 5–8"}],"minor_comments":[{"comment":"The statement that including the synthetic datasets in pretraining 'seems useful for improving the performance of VLMs in real-world settings' should be qualified: Table 9 shows MME falling from 1529 to 1242 with the 30k synthetic mixture and TextVQA declining with the 240k mixture, so the utility is task-dependent.","section":"Section 7 / Table 9"},{"comment":"There is a typographical error ('puxxle' for 'puzzle') in Section I.5; the appendix should be proofread for similar typos.","section":"Section I.5"},{"comment":"The conversion prompt P_convert is described verbally but never quoted; please include the exact prompt template used for reproducibility.","section":"Section 2.4 / Appendix D"},{"comment":"For the Qwen2.5-VL replication of Consecutive Table Readout, the paper relaxes HARD to MEDIUM; this should be stated in the main-text discussion of cross-family consistency, not only in Appendix F.3.","section":"Section F.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is an honest empirical study, but the title and abstract oversell simple-to-hard generalization. The discrepancy between the formal definition (SIMPLE-only training) and the Mix+/Align-Mix+ protocols (HARD Text included) is the main obstacle. I would support publication after a major revision that reframes the non-S2H tasks as cross-modal transfer and reports final-answer metrics for Consecutive Table Readout. The CoT-trace metric and lack of error bars are additional concerns that need to be addressed in the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the S2H VLM paper (arXiv:2501.02669). Bottom line: there's a genuine contribution—a controlled synthetic benchmark with matched text/image versions of three algorithmic reasoning tasks, plus training recipes (Mix, Mix+, Align-Mix+) that consistently transfer text-side reasoning gains to image inputs, with ablation support and replication on Qwen2.5-VL. The 'conversion can be internalized' finding is real and cheap to verify.\n\nThat said, the abstract's 'training on the SIMPLE version' claim does not hold for the flagship Section 4 results. Mix+ and Align-Mix+ include HARD Text examples in training by design (Algorithm 1; Section 4). So the 64/92/35 and 76/96/56 numbers show transfer of already-trained HARD-text reasoning to images, not simple-to-hard generalization. The authors admit as much in 1.1(ii) and 4.1, which makes the abstract misleading rather than the work incoherent. Genuine S2H evidence is limited to Consecutive Table Readout and Pattern-Heldout Visual Analogy—and the former is weakened by the post-hoc choice (Section C.6.1) to score the CoT trace rather than the final answer, after observing that only the CoT generalized. Final-answer accuracy would likely shrink the motivating text-vs-image gap.\n\nOther soft spots are secondary: no seeds or error bars, epoch counts for Text and Image tuned to maximize the reported gap (Section C.1.1), and no public code or data URL despite the 'code available' line. The gradient alignment analysis is honest—Theorem H.1 is a standard Taylor statement, not used to fit anything—and the appendix is unusually transparent about task-design failures.\n\nThis deserves real peer review. I would send it out, but ask for corrective changes: fix the abstract, report final-answer numbers for Consecutive Table Readout, and run at least one multi-seed replication. The benchmark is worth citing on its own.","headline":"A genuinely useful controlled benchmark and a plausible training recipe, but the headline S2H claim is overdrawn: the flagship gains are HARD-text transfer, not simple-to-hard generalization, and the motivating gap depends on a post-hoc CoT-trace metric.","tokens_in":54124,"tokens_out":3055,"would_cite":true,"duration_ms":28153,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training a VLM to explicitly convert images into text before reasoning transfers text-side simple-to-hard generalization to images and closes most of the modality gap.","keywords":["simple-to-hard generalization","modality imbalance","vision-language models","image-to-text conversion","chain-of-thought reasoning","gradient alignment","length generalization","synthetic visual reasoning"],"falsifier":"Re-score the HARD split of Consecutive Table Readout using only the final answer block (the reported list of numbers or their sum), ignoring the chain-of-thought trace, for Text- and Image-trained models. If the text/image accuracy gap shrinks to near zero, the paper's motivating modality imbalance is an artifact of its CoT-trace metric. A second test: at inference, replace the internal image-to-text conversion of a Mix+ model with a deliberately corrupted conversion and check whether HARD-image accuracy drops; if it does not, the reasoning transfer is not actually carried by the conversion.","tokens_in":53054,"feed_emoji":"🖼️","tokens_out":10489,"duration_ms":89006,"temperature":0.7,"pith_summary":"Vision-language models solve easy visual versions of a task much worse than they solve the same task as text, and they fail to generalize from easy to hard visual instances. This paper argues that a missing step, the model never turning the image into the symbolic content the reasoning operates on, is what blocks the transfer, and shows that teaching explicit image-to-text conversion transfers text-side reasoning to images. On three synthetic tasks with matched image and text versions, adding image-to-text conversion to the training mixture raises HARD-image accuracy from below 25% to 64–92%, and to 76–96% with an added alignment phase. The conversion skill is internalized: at test time the model solves directly from the image, so the transfer costs no extra inference tokens. A gradient alignment score over training data predicts which mixtures generalize, and chain-of-thought traces are indispensable.","feed_headline":"Image-to-text training lifts VLM reasoning from 25% to 96%","feed_subtitle":"Teaching VLMs to convert images to text transfers text-side reasoning to images without slowing inference.","key_machinery":"The load-bearing object is the Image-via-Text supervision type: on an image input the model is trained to emit the converted text (LaTeX of the table, text version of the grid, or text description of the puzzle), then the chain-of-thought trace, then the final answer. It appears inside three mixtures: Mix (Text + Image + Image-via-Text on SIMPLE examples), Mix+ (Mix plus HARD-text examples, for tasks where text itself does not S2H-generalize), and Align-Mix+ (a first phase with SIMPLE Text and SIMPLE Image-via-Text, then Mix+). The mechanistic account is a gradient alignment score, $\\langle g_{\\text{SIMPLE}}, g_{\\text{HARD}}\\rangle / \\langle g_{\\text{HARD}}, g_{\\text{HARD}}\\rangle$, which measures how much an averaged SIMPLE-image gradient update would reduce the HARD-image loss relative to a HARD-image update; higher scores at large gradient norms distinguish Mix from Text+Image and Align-Mix+ from Mix+. The third essential component is the verbose chain-of-thought: ablations that remove CoT or progressively internalize it collapse HARD-image accuracy to roughly zero.","core_discovery":"The paper's central claim is that explicit image-to-text conversion is the mechanism through which reasoning transfers from the text modality to the image modality. In its cleanest demonstration, supervised training on SIMPLE text examples (tables given as LaTeX) yields 80% accuracy on HARD-text examples, while the same training on SIMPLE images yields only 20% on HARD-image examples; that 60-point gap is the modality imbalance under study. Supervision that trains the model to first convert the image into text (Image-via-Text) and a mixture that combines it with direct image and text training (Mix) close most of the gap, and Mix keeps inference cost low because the conversion is not reproduced at test time. For the three harder tasks where neither modality simple-to-hard generalizes, adding HARD-text examples (Mix+) yields 64%, 92%, and 35% accuracy on HARD images, and a two-phase version with a SIMPLE image-to-text alignment phase (Align-Mix+) reaches 76%, 96%, and 56%. The paper reports that the transfer is carried by training gradients that align SIMPLE-image updates with HARD-image loss, and that chain-of-thought is necessary. It also discloses that for Consecutive Table Readout the reported S2H generalization appears in the chain-of-thought trace and not in the final answer the model gives (Section C.6.1), which is the basis for its headline modality gap.","pith_inferences":["The paper's cleanest illustration, the 80% versus 20% text/image gap on Consecutive Table Readout, is scored on the chain-of-thought trace, and the authors disclose that the final answer does not S2H-generalize; re-scoring on final answers is the direct test of whether the motivating gap is as large as claimed.","The internalization result suggests a production recipe the paper only sketches: fine-tune with explicit conversion (possibly on lossy text descriptions, since the paper's Visual Analogy text is not image-reconstructing), then rely on the model's internalized shortcut at inference; Mix is already this recipe in one stage.","The authors note that minor CoT template changes substantially change transferred accuracy; extrapolating, the method's success is coupled to the trace format, so a robustness-oriented follow-up should randomize trace verbosity during the Mix/Mix+ phase.","Because the gradient alignment score predicts which mixture transfers without waiting for HARD evaluation, a natural extension is to use it online to adjust the Text/Image/Image-via-Text ratio during training, rather than measuring it post hoc as done here."],"forward_implications":["On tasks where the base LLM already shows simple-to-hard generalization in text, mixing Image-via-Text supervision transfers that generalization to images while keeping generation length low, because the conversion skill is internalized and not reproduced at test time.","On tasks where S2H generalization fails in text as well, injecting HARD-text examples (Mix+) is needed; adding a short SIMPLE image-to-text alignment phase first (Align-Mix+) improves the transfer further, so stronger future LLM backbones should amplify the effect.","Chain-of-thought is load-bearing: completely removing it or progressively internalizing it during the Mix+ phase drops HARD-image accuracy to near zero across supervision types.","Including the synthetic datasets in visual instruction tuning improves real-world benchmarks (at least 3%p on MMMU, 5.1%p on chart descriptive questions), indicating the trained conversion-and-reasoning skill is not confined to the synthetic tasks.","The gradient alignment score identifies which training mixture will reduce HARD-image loss before evaluation is needed, which makes it a usable selection signal for supervision design during model development."],"supporting_citations":[{"why":"Provides the Eagle VLM architecture (two visual encoders on a Llama backbone) that all main experiments fine-tune from.","marker":"(Shi et al., 2025)"},{"why":"Provides the Llama3-8B-Instruct LLM backbone whose text-mode reasoning the training strategies aim to transfer.","marker":"(Dubey et al., 2024)"},{"why":"Supplies the CLIP visual encoder used as one of the model's two image encoders.","marker":"(Radford et al., 2021)"},{"why":"Supplies the ConvNeXt visual encoder that gives the second image pathway.","marker":"(Liu et al., 2022)"},{"why":"The Procedurally Generated Matrices benchmark from which the Visual Analogy task is adapted.","marker":"(Barrett et al., 2018)"},{"why":"Contributes the relational-composition design (domains, relations, held-out patterns) on which Visual Analogy's SIMPLE/HARD split rests.","marker":"(Hill et al., 2019)"},{"why":"Frames simple-to-hard generalization as a form of length generalization tied to RASP, the conceptual backdrop for the Consecutive Table Readout demonstration.","marker":"(Zhou et al., 2024a)"},{"why":"Provides the Qwen2.5-VL 3B/7B models used to replicate the main findings across model family and size.","marker":"(Bai et al., 2025)"},{"why":"Supplies the explicit-to-implicit CoT internalization procedure whose failure in the ablations establishes that chain-of-thought is necessary.","marker":"(Deng et al., 2024)"}],"fun_headline_variants":["Teaching VLMs to convert images to text beats modality gap","Image-to-text training transfers reasoning to VLMs: 25% to 96%","SIMPLE-to-HARD reasoning in VLMs fixed by text conversion","Converting images to text unlocks VLM reasoning on hard tasks","Modality imbalance in VLMs mitigated by image-to-text conversion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The motivating 80% versus 20% modality gap for Consecutive Table Readout is computed on the chain-of-thought trace, not the final answer; the paper discloses (Section C.6.1) that only the trace shows simple-to-hard generalization. If final-answer accuracy is the correct metric, the motivating example of modality imbalance weakens.","fun_headline_variants_meta":{"raw":{"variants":["Teaching VLMs to convert images to text beats modality gap","Image-to-text training transfers reasoning to VLMs: 25% to 96%","SIMPLE-to-HARD reasoning in VLMs fixed by text conversion","Converting images to text unlocks VLM reasoning on hard tasks","Modality imbalance in VLMs mitigated by image-to-text conversion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000337,"raw_usage":{"total_tokens":1934,"prompt_tokens":1088,"completion_tokens":846,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":704,"completion_tokens_details":{"reasoning_tokens":755}},"tokens_in":704,"tokens_out":846,"duration_ms":7837,"temperature":1.0,"reasoning_tokens":755,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:07:33.899263+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-score the HARD split of Consecutive Table Readout using only the final answer block (the reported list of numbers or their sum), ignoring the chain-of-thought trace, for Text- and Image-trained models. If the text/image accuracy gap shrinks to near zero, the paper's motivating modality imbalance is an artifact of its CoT-trace metric. A second test: at inference, replace the internal image-to-text conversion of a Mix+ model with a deliberately corrupted conversion and check whether HARD-image accuracy drops; if it does not, the reasoning transfer is not actually carried by the conversion.","supporting_citations":[],"review_version":1}