Pith. sign in

REVIEW 4 major objections 41 references

Interleaved text-image reasoning fails when modalities stop informing each other; supervising the transitions themselves restores long-chain coherence and accuracy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 14:12 UTC pith:OUDIC7A2

load-bearing objection Clean diagnosis of a real interleaved-CoT failure mode, with large gains from transition-level training—but the causal story still rides on unvalidated VLM judges and per-task models. the 4 major comments →

arxiv 2606.12886 v2 pith:OUDIC7A2 submitted 2026-06-11 cs.CV cs.AI

Bridging Modal Isolation in Interleaved Thinking: Supervising Modality Transitions via Stepwise Reinforcement

Tingyu Li , Le Zhou , Siyuan Li , Yujun Wu , Xinglong Xu , Jingxuan Wei , Conghui He , Cheng Tan This is my paper
classification cs.CV cs.AI
keywords interleaved thinkingmodal isolationmodality transition lossunified multimodal modelsReflective SFTFlow-GRPOvisual reasoningcross-modal hallucination
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Unified multimodal models that alternate between writing and drawing can look as if they are thinking visually, yet on long spatial and physical puzzles the two streams often decouple: the image no longer matches the plan, and later text ignores what the image actually shows. The paper names this failure Modal Isolation and traces it to compounding information loss exactly at the text-to-image and image-to-text boundaries. It decomposes each cycle into observe-reason-instruct-draw steps and defines a modality transition loss that measures cross-modal hallucination and visual under-use. The proposed MoTiF pipeline then optimizes those transitions directly—Reflective SFT teaches the text side to notice and recover from bad images, while Flow-GRPO strengthens image fidelity to instructions—without ever rewarding final-answer correctness. On four visual-puzzle suites the same transition-level signals raise both cross-modal coherence and end-task accuracy, arguing that structural supervision at the boundaries is necessary for scalable interleaved reasoning.

Core claim

Effective interleaved reasoning requires explicit structural supervision of modality transitions (via a modality transition loss over cross-modal hallucination and visual utilization deficit) rather than scaling or end-task rewards; optimizing only those transition signals with Reflective SFT and Flow-GRPO substantially improves both coherence and accuracy on long-chain visual puzzles.

What carries the argument

Modality Transition Loss (L_MT): the sum, over each cycle, of world-state divergence between intended and rendered states (text-to-image hallucination) plus rendered and decoded states (image-to-text utilization deficit); MoTiF minimizes it with Reflective SFT on deliberately corrupted images and Flow-GRPO on the image-generation policy.

Load-bearing premise

The binary judgments of a strong vision-language model are treated as faithful stand-ins for true world-state mismatch, so training against those judgments is assumed to reduce real isolation rather than merely matching the judge.

What would settle it

Hold the base model and tasks fixed, replace the VLM-as-Judge rewards with an independent ground-truth world-state extractor (or human annotation of state fidelity), retrain MoTiF, and test whether final accuracy and cross-modal coherence still rise; if they collapse, the claim that transition-level supervision works rests on the judge proxy.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Long-chain interleaved planners can be improved without outcome rewards by supervising only the fidelity of each modality hand-off.
  • Training data that deliberately injects corrupted images and teaches recovery becomes a reusable recipe for reducing visual under-use.
  • Flow-matching image generators can be policy-optimized for instruction fidelity inside a reasoning loop rather than only for aesthetic quality.
  • Diagnostic benchmarks that score single-step text-to-image and image-to-text fidelity become necessary companions to end-task accuracy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Any multimodal chain that mixes discrete symbolic steps with continuous generation is likely to exhibit an analogous isolation failure once chain length grows, so boundary-level losses may apply beyond pure vision-language puzzles.
  • If the judge model itself hallucinates, MoTiF may simply teach the student to echo the judge’s biases; independent state extractors or multi-judge ensembles would be a natural robustness check.
  • The same atomic observe-reason-instruct-draw decomposition could be used as an online monitor at inference time to decide when to re-generate an image rather than continuing a broken chain.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 0 minor

Summary. The paper identifies Modal Isolation in long-chain interleaved thinking with unified multimodal models: text-to-image generation diverges from intended world state while subsequent text fails to use visual evidence, so modalities alternate without mutual information transfer. It decomposes each cycle into observe/reason/instruct/draw operations, defines Modality Transition Loss L_MT as the sum of cross-modal hallucination (Def. 1, und o gen) and visual utilization deficit (Def. 2, gen o und), and proposes MoTiF: Reflective SFT (contrastive recovery from corrupted images) plus Flow-GRPO (RL on the flow-matching image policy), both driven only by transition-level VLM-as-Judge signals rather than end-task accuracy. On four synthetic visual-puzzle suites (Sokoban, Maze, Multi-hop Manipulation, Ball Tracking), Bagel-7B-MoT rises from ~25% to ~71% overall accuracy, with transition rewards rising in tandem (Table 2, Fig. 4); Appendix Table 5 bounds gen o und degradation after Flow-GRPO.

Significance. If the causal story holds, the work supplies a useful diagnostic (modal isolation / L_MT) and a practical process-level training recipe for interleaved UMMs that is more targeted than end-task RL or naive SFT. The atomic decomposition of interleaved cycles, the explicit separation of und o gen and gen o und losses, and the two-stage pipeline that never uses final-answer reward are clear conceptual contributions. Empirical gains on Bagel-7B-MoT are large and consistent across four domains, and the authors report bounded multi-objective interference (Table 5). Promised code, website, and dataset would aid reproducibility. The result would matter for spatial/physical interleaved reasoning and for process supervision in multimodal CoT more broadly.

major comments (4)
  1. §3.2 footnote 1 and §4.2 Reward Design treat binary R_img / R_txt judgments from Qwen3.5-27B as empirical estimators of the abstract divergences d(ŵ_i, w^V_i) and d(w^V_i, w̃_i) in Defs. 1–2, and both training signals and the diagnostic curves in Fig. 4 are these same proxies. No calibration against ground-truth world states (available from the rule solvers and Blender renders used to build the data), no human agreement, and no inter-judge agreement are reported. Without that validation, the claim that gains come from reduced modal isolation (vs. high-quality process imitation under a strong teacher) is under-supported; a calibration study or GT-based d on held-out steps is needed for the central causal story.
  2. The abstract and §1/§6 assert that effective interleaved reasoning requires transition-level supervision “not merely scaling or end-task optimization,” and that all MoTiF signals avoid end-task accuracy. Table 2 and Fig. 4 compare to base Bagel, open-source interleaved models, and frontier models, but not to an end-task RL (or outcome-reward GRPO) baseline on the same base model and data. Without that contrast, the necessity claim over end-task optimization is not established; an outcome-only RL arm (and ideally a joint outcome+transition arm) is load-bearing for the paper’s strongest framing.
  3. §4.2 states “we train a separate model per task” to isolate Modal Isolation effects. Table 2’s overall ~45-point gain is therefore an average of four specialized models, not a single general interleaved reasoner. This weakens the claim that MoTiF yields a transferable structural fix for long-chain interleaved thinking. At minimum, report a multi-task trained model (or zero-shot transfer across the four suites) so the generality of transition-level supervision can be assessed.
  4. Fig. 4 jointly plots task accuracy and the same R_img / R_txt transition rewards used as training signals; partial circularity on the diagnostic metrics is acknowledged in the design but not quantified. End-task accuracy is partly independent, yet no error bars, multiple seeds, or statistical tests appear in Table 2 or Fig. 4. Given binary judges and synthetic domains, seed variance and judge noise could be large; report multi-seed means/std or bootstrap CIs so the magnitude of gains can be trusted.

Circularity Check

2 steps flagged

Mild self-definitional loop: Modal Isolation is defined as compounding of L_MT, so 'optimizing L_MT mitigates isolation' is partly by construction; reported cross-modal coherence gains reuse the same VLM-judge rewards used as training signals.

specific steps
  1. self definitional [§3.2 Defs. 1–2, Eq. 6, and paragraph after Eq. 6; Abstract]
    "This mutual amplification accumulates with chain length K, providing a formal account of Modal Isolation: the progressive decoupling of textual reasoning from visual evidence, where each modality drifts further from the other with every transition. L_MT thus serves as both a diagnostic metric and a fundamental bottleneck on the scalability of interleaving thinking to long chains."

    Modal Isolation is defined/accounted for as the compounding interaction of ℓ_und→gen and ℓ_gen→und. The subsequent claim that MoTiF, by minimizing L_MT, mitigates Modal Isolation therefore holds largely by the paper's own definition of the phenomenon, not as an independent empirical discovery. The interesting non-circular content is the transfer to end-task accuracy, not the isolation diagnosis itself.

  2. fitted input called prediction [§4.2 Reward Design; §5.1 Figure 4 and surrounding text]
    "Figure 4 further demonstrates the synergistic evolution between model performance and cross-modal reward scores throughout the training process. We build two benchmarks adopting the designed VLM-as-Judge reward functions, which quantify the severity of modal isolation by evaluating single-step text and image generation during interleaved thinking. In the figure, ℓ_und→gen and ℓ_gen→und respectively represent the reward scores of the corresponding benchmarks."

    Training Stage 1 filters/supervises with R_txt and Stage 2 optimizes with R_img (binary VLM judges). The paper then reports those same judge-derived transition rewards rising as evidence that modal isolation is reduced. For the diagnostic curves, the 'improvement in cross-modal coherence' is the training objective re-measured, not an independent prediction. End-task accuracy remains a separate outcome and is not forced by this loop.

full rationale

This is an empirical ML methods paper, not a first-principles derivation. The load-bearing scientific claim—that transition-level supervision (Reflective SFT + Flow-GRPO) improves held-out end-task accuracy without using end-task rewards as the primary objective—is not circular: Table 2 accuracy is an independent outcome on rule-solved puzzles. Two weaker loops exist. (1) Modal Isolation is introduced as the failure mode and then formally accounted for as the mutual amplification of the two components of L_MT (Defs. 1–2 and Eq. 6); claiming that minimizing L_MT reduces Modal Isolation is therefore largely definitional rather than an independent prediction. (2) Cross-modal coherence is operationalized and reported via the same binary VLM-as-Judge rewards R_img / R_txt that supply training signals (Fig. 4 dashed lines), so those diagnostic curves rising is expected by construction of the training objective. Neither loop forces the accuracy gains, and there is no self-citation uniqueness theorem or fitted parameter renamed as a novel prediction. Score 3 reflects partial circularity confined to the diagnostic framing, not the central empirical result.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 3 invented entities

The central claim rests on an abstract world-state model, VLM judges as stand-ins for state divergence, architectural separation of text AR and flow-matching image policies, and task-specific synthetic interleaved chains. Free parameters are training and loss weights rather than physics constants. Invented entities are conceptual (losses and the isolation phenomenon), not new physical objects; their independent handle is the reported task-accuracy transfer.

free parameters (4)
  • α, β weights in L_MT
    Equation (6) introduces positive weighting coefficients on the two transition losses; values are not fixed by theory and are effectively design choices for the joint objective.
  • Flow-GRPO β_KL / noise / SDE window
    Appendix Table 4 sets beta=0.04, noise level 1.3, sde window size 3, lr 5e-6, group size 16—hand-chosen RL hyperparameters that affect image-fidelity optimization.
  • Reflective SFT loss weights (mse_weight, ce_weight)
    Table 3 sets mse weight 10 and ce weight 1; the balance between flow-matching and text CE is a free training choice.
  • Corruption / recovery pattern mix in Reflective SFT
    Which intermediate image is replaced and whether Detect-and-Ignore vs Detect-and-Redraw is used is a data-construction choice that shapes L_gen→und supervision.
axioms (5)
  • domain assumption Each visual reasoning task admits a world-state space W and a divergence d that capture task-relevant configuration differences.
    §3.1–3.2 build all of L_MT on W and d without proving uniqueness of W for the four puzzle domains.
  • ad hoc to paper Binary rubric judgments from a frontier VLM approximate d(ŵ, w^V) and d(w^V, w̃) well enough to supervise and evaluate transitions.
    Footnote 1 and §4.2 operationalize the entire theory via R_img/R_txt; this is a paper-specific measurement axiom.
  • domain assumption Text segments functionally decompose into observe / reason / instruct atomic operations in interleaved chains.
    §3.1 imposes this structure on t_i; training formats and judges assume it.
  • domain assumption Flow-matching image generation can be cast as an MDP and optimized with group-relative policy gradients (Flow-GRPO) without destroying understanding.
    §3.2 and Stage 2 rely on Liu et al. Flow-GRPO; Appendix B treats bounded Rg2u drop as acceptable.
  • ad hoc to paper Synthetic interleaved chains from rule solvers plus Gemini rewriting are valid supervision for human-like interleaved reasoning.
    §4.1 data pipeline; quality of MoTiF hinges on this synthetic process.
invented entities (3)
  • Modal Isolation no independent evidence
    purpose: Name the compounding failure where text and image alternate without mutual information transfer.
    Introduced in abstract/§1 as the paper’s diagnostic target; evidence is qualitative examples plus VLM-judge trends, not an external physical measurement.
  • Modality Transition Loss L_MT (cross-modal hallucination + visual utilization deficit) no independent evidence
    purpose: Provide a process-level objective at T↔V boundaries independent of end-task accuracy.
    Definitions 1–2 and Eq. (6); measured only via paper-defined judges.
  • MoTiF (Reflective SFT + Flow-GRPO pipeline) no independent evidence
    purpose: Jointly optimize the two transition directions with separate modality-appropriate trainers.
    Core method contribution; success is internal to the four benchmarks and judge metrics.

pith-pipeline@v1.1.0-grok45 · 20395 in / 3973 out tokens · 43899 ms · 2026-07-12T14:12:53.830826+00:00 · methodology

0 comments
read the original abstract

Interleaved thinking, where a unified multimodal model alternates between textual reasoning and visual generation, has shown promise on spatial and physical tasks. However, in complex long-chain scenarios, we identify a fundamental failure mode: generated images diverge from the textual context while subsequent text ignores the visual evidence, causing the two modalities to alternate without genuinely informing each other. We term this Modal Isolation and attribute it to compounding information loss at modality boundaries. We decompose each reasoning cycle into atomic operations and define modality transition loss, quantifying cross-modal hallucination (text-to-image) and visual utilization deficit (image-to-text) at each boundary. We propose MoTiF (Modality Tiransition Fidelity), a two-stage training framework that directly optimizes these transitions: Reflective SFT trains the model to detect and recover from erroneous visual outputs; Flow-GRPO improves image generation fidelity via reinforcement learning. All training signals in MoTiF derive from transition-level fidelity rather than end-task accuracy. Across four visual puzzle benchmarks, this transition-level supervision substantially improves both cross-modal coherence and final task accuracy. The results demonstrate that effective interleaved reasoning requires explicit structural supervision at modality boundaries, not merely scaling or end-task optimization.

Figures

Figures reproduced from arXiv: 2606.12886 by Cheng Tan, Conghui He, Jingxuan Wei, Le Zhou, Siyuan Li, Tingyu Li, Xinglong Xu, Yujun Wu.

Figure 1
Figure 1. Figure 1: The phenomenon of modal iso￾lation in interleaved thinking. In a maze navigation task, the generated image de￾picts an inconsistent path. Subsequently, the model observes the erroneous image yet fails to detect the discrepancy, incor￾rectly validating the path. through hallucinated intermediate reasoning, leaving modal isolation unaddressed. Similarly, supervised fine￾tuning on correct chains teaches the m… view at source ↗
Figure 2
Figure 2. Figure 2: Modality transition loss and atomic decomposition of interleaving thinking. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Reflective SFT data collection pipeline. t ins i , the model generates an image vi and receives a reward signal derived from the empirical estimate of ℓ (i) und→gen . The policy π V θ is updated via group￾relative advantage estimation with KL divergence regularization against a reference policy, maximiz￾ing the fidelity of image generation to textual in￾structions while preserving generation quality. For L… view at source ↗
Figure 4
Figure 4. Figure 4: Solid lines denote model accuracy scores, with different colors distinguishing the Reflective [PITH_FULL_IMAGE:figures/full_fig_p009_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Changes of Rg2u and Ru2g before and after Flow-GRPO training. Since the two training phases of Reflective SFT and Flow-GRPO optimize the modal transition loss in different dimensions, we systematically analyze whether multi-objective conflict and catastrophic forgetting occur during this process. Given that prior work has indicated that optimizing UMM with Flow-GRPO would impair the model’s rea￾soning abil… view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 7 linked inside Pith

  1. [1]

    Simulation as an engine of physical scene understanding.Proceedings of the national academy of sciences, 110(45):18327–18332, 2013

    Peter W Battaglia, Jessica B Hamrick, and Joshua B Tenenbaum. Simulation as an engine of physical scene understanding.Proceedings of the national academy of sciences, 110(45):18327–18332, 2013

    2013

  2. [2]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling.arXiv preprint arXiv:2501.17811, 2025

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling.arXiv preprint arXiv:2501.17811, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    Mint-cot: Enabling interleaved visual tokens in mathematical chain-of-thought reasoning.Advances in neural information processing systems, 38:69110–69139, 2026

    Xinyan Chen, Renrui Zhang, Dongzhi Jiang, Aojun Zhou, Shilin Yan, Weifeng Lin, and Hongsheng Li. Mint-cot: Enabling interleaved visual tokens in mathematical chain-of-thought reasoning.Advances in neural information processing systems, 38:69110–69139, 2026

    2026

  4. [4]

    Anole: An open, autoregressive, native large multimodal models for interleaved image-text generation.arXiv preprint arXiv:2407.06135, 2024

    Ethan Chern, Jiadi Su, Yan Ma, and Pengfei Liu. Anole: An open, autoregressive, native large multimodal models for interleaved image-text generation.arXiv preprint arXiv:2407.06135, 2024

    Pith/arXiv arXiv 2024

  5. [6]

    Thinking with generated images, 2025

    Ethan Chern, Zhulin Hu, Steffi Chern, Siqi Kou, Jiadi Su, Yan Ma, Zhijie Deng, and Pengfei Liu. Thinking with generated images, 2025. URLhttps://arxiv.org/abs/2505.22525

    Pith/arXiv arXiv 2025

  6. [7]

    Emerging properties in unified multimodal pretraining.arXiv preprint arXiv:2505.14683, 2025

    Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, et al. Emerging properties in unified multimodal pretraining.arXiv preprint arXiv:2505.14683, 2025

    Pith/arXiv arXiv 2025

  7. [8]

    Dreamllm: Synergistic multimodal comprehension and creation

    Runpei Dong, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, et al. Dreamllm: Synergistic multimodal comprehension and creation. In International Conference on Learning Representations, volume 2024, pages 6666–6702, 2024

    2024

  8. [9]

    Grit: Teaching mllms to think with images.Advances in Neural Information Processing Systems, 38: 116522–116543, 2026

    Yue Fan, Xuehai He, Diji Yang, Kaizhi Zheng, Ching-Chen Kuo, Yuting Zheng, Xinze Guan, and Xin Wang. Grit: Teaching mllms to think with images.Advances in Neural Information Processing Systems, 38: 116522–116543, 2026

    2026

  9. [10]

    Interleaved-modal chain-of-thought

    Jun Gao, Yongqi Li, Ziqiang Cao, and Wenjie Li. Interleaved-modal chain-of-thought. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 19520–19529, 2025

    2025

  10. [11]

    Thinkmorph: Emergent properties in multimodal interleaved chain-of-thought reasoning

    Jiawei Gu, Yunzhuo Hao, Huichen Will Wang, Linjie Li, Michael Qizhe Shieh, Yejin Choi, Ranjay Krishna, and Yu Cheng. Thinkmorph: Emergent properties in multimodal interleaved chain-of-thought reasoning. arXiv preprint arXiv:2510.27492, 2025

    arXiv 2025

  11. [12]

    Generating images with multimodal language models

    Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov. Generating images with multimodal language models. Advances in Neural Information Processing Systems, 36:21487–21506, 2023

    2023

  12. [13]

    Imagine while reasoning in space: Multimodal visualization-of-thought

    Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vuli´c, and Furu Wei. Imagine while reasoning in space: Multimodal visualization-of-thought. InInternational Conference on Machine Learning, pages 36340–36364. PMLR, 2025

    2025

  13. [14]

    Flow-grpo: Training flow matching models via online rl.Advances in neural information processing systems, 38:40783–40818, 2026

    Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl.Advances in neural information processing systems, 38:40783–40818, 2026

    2026

  14. [15]

    Towards unified multimodal interleaved generation via group relative policy optimization.Advances in Neural Information Processing Systems, 38: 5332–5353, 2026

    Ming Nie, Chunwei Wang, Jianhua Han, Hang Xu, and Li Zhang. Towards unified multimodal interleaved generation via group relative policy optimization.Advances in Neural Information Processing Systems, 38: 5332–5353, 2026

    2026

  15. [16]

    Uni-cot: Towards unified chain-of-thought reasoning across text and vision.arXiv preprint arXiv:2508.05606, 2025

    Luozheng Qin, Jia Gong, Yuqing Sun, Tianjiao Li, Mengping Yang, Xiaomeng Yang, Chao Qu, Zhiyu Tan, and Hao Li. Uni-cot: Towards unified chain-of-thought reasoning across text and vision.arXiv preprint arXiv:2508.05606, 2025

    arXiv 2025

  16. [17]

    Qwen3.5: Towards native multimodal agents, February 2026

    Qwen Team. Qwen3.5: Towards native multimodal agents, February 2026. URL https://qwen.ai/blog?id= qwen3.5. 11 Supervising Modality Transitions via Stepwise Reinforcement

    2026

  17. [18]

    Generative multimodal models are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14398–14409, 2024

    2024

  18. [19]

    Emu: Generative pretraining in multimodality

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. InInternational conference on learning representations, volume 2024, pages 12352–12380, 2024

    2024

  19. [21]

    Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    Pith/arXiv arXiv 2024

  20. [22]

    Multimodal learning with next-token prediction for large multimodal models.Nature, pages 1–7, 2026

    Xinlong Wang, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Zhen Li, Yuqi Wang, et al. Multimodal learning with next-token prediction for large multimodal models.Nature, pages 1–7, 2026

    2026

  21. [23]

    The trinity of consistency as a defining principle for general world models.arXiv preprint arXiv:2602.23152, 2026

    Jingxuan Wei, Siyuan Li, Yuhang Xu, Zheng Sun, Junjie Jiang, Hexuan Jin, Caijun Jia, Honghao He, Xinglong Xu, Chang Yu, et al. The trinity of consistency as a defining principle for general world models.arXiv preprint arXiv:2602.23152, 2026

    arXiv 2026

  22. [24]

    Janus: Decoupling visual encoding for unified multimodal understanding and generation

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 12966–12977, 2025

    2025

  23. [25]

    Visual generation unlocks human-like reasoning through multimodal world models.arXiv preprint arXiv:2601.19834, 2026

    Jialong Wu, Xiaoying Zhang, Hongyi Yuan, Xiangcheng Zhang, Tianhao Huang, Changjing He, Chaoyi Deng, Renrui Zhang, Youbin Wu, and Mingsheng Long. Visual generation unlocks human-like reasoning through multimodal world models.arXiv preprint arXiv:2601.19834, 2026

    arXiv 2026

  24. [27]

    Show-o: One single transformer to unify multimodal understanding and generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. InInternational Conference on Learning Representations, volume 2025, pages 28240–28264, 2025

    2025

  25. [28]

    The latent space: Foundation, evolution, mechanism, ability, and outlook.arXiv preprint arXiv:2604.02029, 2026

    Xinlei Yu, Zhangquan Chen, Yongbo He, Tianyu Fu, Cheng Yang, Chengming Xu, Yue Ma, Xiaobin Hu, Zhe Cao, Jie Xu, et al. The latent space: Foundation, evolution, mechanism, ability, and outlook.arXiv preprint arXiv:2604.02029, 2026

    Pith/arXiv arXiv 2026

  26. [29]

    Minigpt-5: Interleaved vision-and-language generation via generative vokens.arXiv preprint arXiv:2310.02239, 2023

    Kaizhi Zheng, Xuehai He, and Xin Eric Wang. Minigpt-5: Interleaved vision-and-language generation via generative vokens.arXiv preprint arXiv:2310.02239, 2023

    arXiv 2023

  27. [30]

    Deepeyes: Incentivizing” thinking with images” via reinforcement learning.arXiv preprint arXiv:2505.14362, 2025

    Ziwei Zheng, Michael Yang, Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang, Chao Shen, and Xing Yu. Deepeyes: Incentivizing” thinking with images” via reinforcement learning.arXiv preprint arXiv:2505.14362, 2025

    Pith/arXiv arXiv 2025

  28. [31]

    Pre” denotes the model after Reflective SFT only; “Post

    Yiyang Zhou, Haoqin Tu, Zijun Wang, Zeyu Wang, Niklas Muennighoff, Fan Nie, Yejin Choi, James Zou, Chaorui Deng, Shen Yan, et al. When visualizing is the first step to reasoning: Mira, a benchmark for visual chain-of-thought.arXiv preprint arXiv:2511.02779, 2025. 12 Supervising Modality Transitions via Stepwise Reinforcement Appendix A Training Hyperparam...

    arXiv 2025

  29. [32]

    Describe what you actually see in it and explain specifically why it does NOT match what was expected based on Step{{j}}’s reasoning and drawing plan

    **Observe and describe the error**: Carefully examine the WRONG image. Describe what you actually see in it and explain specifically why it does NOT match what was expected based on Step{{j}}’s reasoning and drawing plan

  30. [33]

    Based on what the CORRECT image SHOULD have shown (as intended by Step{{j}}), continue the logical reasoning for the next step

    **Continue reasoning**: Acknowledge that an image generation error occurred, but decide to move forward rather than redraw. Based on what the CORRECT image SHOULD have shown (as intended by Step{{j}}), continue the logical reasoning for the next step

  31. [34]

    This drawing plan should be consistent with the original next step’s intended target

    **Drawing plan for the next step**: Provide a clear, executable drawing instruction for the NEXT image to generate. This drawing plan should be consistent with the original next step’s intended target. Write in first person. Use natural hesitation markers like ”Hmm...”, ”Wait,”, ”Let me look more closely...”, or ”Actually...” where appropriate. Please thi...

  32. [37]

    The image to be evaluated, which is generated following the explicit visualization instructions of the previous step’s textual reasoning

  33. [38]

    ## Strict Evaluation Principles Focus only on correctness, not standalone aesthetic preference

    The ground-truth image. ## Strict Evaluation Principles Focus only on correctness, not standalone aesthetic preference. The generated image does not need to be pixel-identical to the ground truth, but it must be logically equivalent for the current reasoning step. Score = 1 only if all of the following are true: - The generated image and the ground-truth ...

  34. [39]

    The original multimodal problem image and problem text

  35. [40]

    The full context of completed interleaved multi-modal CoT reasoning steps, where the corresponding images at their respective positions are represented by the ‘¡auxiliary-image¿‘ placeholder

  36. [41]

    The image generated in the immediately preceding step (if any), which the model must observe and reflect upon. If there is no preceding image (i.e., this is the very first reasoning step), you will only see the problem image and must judge whether the model correctly observes the visual elements in the problem image instead

  37. [42]

    The generated text to be evaluated

  38. [43]

    This is an intermediate reasoning step

    The step type: ”non-final” (a <think>...</think> block ending with <create_image>) or ”final” (an<ans>...</ans>block that concludes the entire reasoning chain). This is an intermediate reasoning step. The generated text must follow the format <think>...</think> and end with <create_image>. Score = 1 only if ALL of the following are true: - The generated t...

  39. [44]

    Change the orange cylinder into a gray cylinder

  40. [45]

    After completing all operations, what is the object in front of the red cuboid? Options: A

    Change the blue sphere into a red cylinder. After completing all operations, what is the object in front of the red cuboid? Options: A. red cylinder B. purple sphere C. gray cylinder D. red cuboid Sokoban Images Question: This is a Sokoban puzzle where cartoon person is player, green X is target, brown box with X is box to push, brown tiles are walls, and...

  41. [46]

    Find the path to the finish and count the number of turns it takes to get there

    The coordinates are given in the format (row, col), where row represents the vertical position and col represents the horizontal position. Find the path to the finish and count the number of turns it takes to get there. Model Response: <think> To solve the maze, I will start by identifying the player’s position at (9, 2) and the goal at (1, 7). The goal i...