Pith. sign in

REVIEW 5 major objections 5 minor 51 references

STBridge's three-stage shared-target training lifts a unified multimodal model's caption-image consistency from 0.574 to 0.900 while also improving understanding, generation, and editing benchmarks.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 18:52 UTC pith:UFWQ743Z

load-bearing objection A genuinely useful shared-target post-training recipe, but the consistency claim is overstated and the VLM-judged alignment metric needs external validation. the 5 major comments →

arxiv 2607.17140 v1 pith:UFWQ743Z submitted 2026-07-19 cs.CV

STBridge: Shared-Target Alignment for Bridging Understanding and Generation in UMMs

Ye Wang , Hongjun Wang , Hao Fang , Tongyuan Bai , Zuwei Long , Peixian Chen , Wei Liu , Weibo Gu
show 2 more authors
Xing Sun Rui Ma
This is my paper
classification cs.CV
keywords unified multimodal modelsunderstanding-generation alignmentshared-target post-trainingimage editingtarget captioningdiff rewardreinforcement learningcaption-image consistency
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Unified multimodal models aim to both understand and generate images in one network, but a model can describe an edit correctly and still draw something else. The paper claims this understanding-generation gap exists because the two outputs are trained as separate tasks rather than as two views of the same target result. It proposes STBridge, a post-training recipe built around a shared target state: given a source image and an instruction, the model must first produce a 'target caption' that spells out the desired result, then generate the edited image that matches that caption. Three training stages—supervised alignment, then reinforcement learning on the caption, then reinforcement learning on the image—progressively close the gap, raising measured caption-image consistency from 0.574 to 0.900 and improving the model across understanding, generation, and editing benchmarks.

Core claim

The paper's central claim is that architectural unification alone does not make a unified multimodal model semantically consistent: under the same source image and edit instruction, the model's textual description of the intended result and its generated edited image can diverge in fine-grained entities, attributes, spatial relations, and local details. STBridge treats the target image as a shared semantic anchor, so that target captioning and image editing become two modal expressions of the same target state rather than independent tasks. The method uses a three-stage align-then-optimize procedure: supervised fine-tuning first binds the target caption to the target image, then a Diff Rewar

What carries the argument

The central mechanism is the shared-target state in the image-editing interface: for a source image and instruction, a target image anchors both modalities. The understanding path produces a target caption that expresses that target; the generation path produces the edited image that realizes it. Training proceeds in three stages: (1) Shared-Target Alignment, supervised fine-tuning on tuples of source, instruction, target caption, and target image, using a cross-entropy caption loss plus a flow-matching image loss; (2) Target-Expression Optimization, which uses the Diff Reward—a VLM infers the edit implied by a candidate caption and compares it with the annotated source-to-target difference—

Load-bearing premise

The whole pipeline presumes that VLM-written captions, VLM-computed rewards, and a VLM judge measure true semantic agreement; if those models share systematic blind spots, the reported alignment gain may measure agreement with the judge rather than with human intent.

What would settle it

Ask human raters to score whether the target caption and edited image match on the paper's 400 alignment pairs, and compare human scores with the VLM judge's 0.574-to-0.900 gap. If humans see little or no improvement, the alignment gain is an artifact of the judge.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the central claim holds, architecture alone is not enough to make understanding and generation semantically consistent; post-training around a shared target is a viable lever.
  • Target captions act as a controllable intermediate representation: improving the caption policy first, then freezing it while improving image realization, yields better final edits than joint or independent optimization.
  • Visual-understanding questions can be converted into editing-style transformations, letting one data interface train both capabilities.
  • Reward shaping that focuses on the edit-relevant difference, rather than global image-text similarity, is what makes the expression stage work.
  • Consistent gains on perception, generation, and editing benchmarks imply that shared-target post-training may transfer to other tasks within unified multimodal models.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the shared-target interface could be applied to multi-turn or video editing, where each step's target state becomes the next step's source.
  • Editorial inference: since target captions, RL rewards, and the alignment judge are all VLM outputs, the 0.900 alignment score should be tested against human ratings; otherwise the gain may partly reflect conformity to the judge's preferences.
  • Editorial extension: a strong robustness test would be to swap in a different judge model at evaluation and see whether the 0.574-to-0.900 improvement persists.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper studies the semantic alignment between the textual and visual outputs of unified multimodal models (UMMs) in the image-editing setting. It first diagnoses an 'understanding–generation alignment gap' in a baseline UMM (BAGEL) using a VLM-based consistency protocol, then proposes STBridge, a three-stage post-training procedure: Shared-Target Alignment (SFT on target captions plus target-image flow-matching), Target-Expression Optimization (GRPO with a VLM-based Diff Reward on captions), and Target-Realization Optimization (MixGRPO with a VLM-based editing reward on generated images). The authors report improvements over BAGEL on BLINK, CVBench, OddGridBench, MMVP, VisPuzzle, MM-Vet, WISE, ImgEdit, and RISE, and a large increase in caption–image consistency from 0.5740 to 0.9000 on a custom 400-pair alignment evaluation. They conclude that anchoring understanding and generation to a shared target state narrows the gap between what the model describes and what it generates.

Significance. If the central claim held, STBridge would be a useful and fairly general post-training recipe for UMMs: it converts visual-understanding data into an editing-style interface, uses a shared target caption to condition generation, and applies sequential RL to improve both expression and realization. The paper also ships a diagnostic protocol (Table 7, Fig. 6) that could be reused by the community. The strongest positive evidence is the broad benchmark coverage, the qualitative examples, and the ablation separating source-caption, target-caption, and shared-target SFT. However, the manuscript's own tables contain several sub-score decreases, and all the key rewards and the main alignment metric are produced by VLM judges, with no human agreement study. Those two issues are load-bearing for the abstract's and conclusion's claims of 'consistent improvements' and 'narrowing the gap.' The paper is therefore promising but requires substantial additional analysis before the central claims can be considered established.

major comments (5)
  1. [Tables 1–6 and Sec. 5.3] The abstract, introduction, and conclusion repeatedly claim 'consistent improvements,' but the benchmark tables show multiple decreases at the sub-score level. For example, Table 4: WISE Cultural drops by -9.00 and Time by -0.33; Table 5: ImgEdit Style -0.12, Compose -0.31, Action -0.15; Table 1: BLINK Count. Rel. -0.84, Rel. Depth -0.66, Multi-view Rel. -0.67; Table 2: CVBench-2D Count -1.27. No error bars, confidence intervals, or significance tests are reported anywhere, so it is impossible to tell which of the aggregate gains (e.g., BLINK +5.15, RISE +21.00) are robust versus noise. Please either provide repeated-run variance and significance testing, or temper the 'consistent improvements' claim to 'improvements on the averages of most benchmarks' and explicitly discuss the sub-score regressions.
  2. [Sec. 5.2 vs Sec. 3.1 (WISE evaluation)] The WISE evaluation is performed with Qwen3.5-35B-A3B as the judge (Sec. 5.2), and the same model is used to generate the SFT target captions in Sec. 3.1. This creates a concrete judge-overlap risk: the gain on WISE (+1.20 overall, +21.50 on Biology) may reflect optimization toward the preferences of the captioning/judging model rather than genuine world-knowledge-informed generation. The manuscript does not provide any independent judge or human agreement analysis for WISE. Please add an evaluation with a different VLM family and/or a human-annotated subset, and report per-dimension agreement.
  3. [Sec. 5.5, Table 7] The central alignment claim—that STBridge raises caption–image consistency from 0.5740 to 0.9000—is based solely on Gemini-3.1-Flash-Image-Preview as the judge. The training signals (target captions in Sec. 3.1, Diff Reward in Sec. 4.3, realization reward in Sec. 4.4) come from other VLM systems (Qwen3.5-35B and Qwen3.5-122B). All these VLMs are text-image judges, and no human agreement study or independent-judge replication is reported. If these VLM judges share systematic biases—e.g., favoring verbose scene-level descriptions or template-like phrasing—the reported alignment gains could be inflated by satisfying the judge rather than by genuine convergence of textual and visual semantics. Please provide a human evaluation on a subset of the 400 pairs, an independent VLM judge, and per-item error analysis. Also specify how the 400 pairs were sampled.
  4. [Sec. 4.3, Eq. (6)] The Diff Reward depends on the function Align(·,·) and on an 'offline annotated reference difference' Δ*. Neither is defined anywhere in the paper. It is not specified whether Δ* is human-annotated or generated by a VLM, nor how the alignment between the predicted difference and Δ* is computed (exact string match? semantic similarity? another VLM call?). Since this reward is the sole training signal for the target-caption policy, the lack of specification is load-bearing. Please provide the exact construction of Δ*, the Align function, and the prompt/format used for DiffVLM, or release the code/config.
  5. [Sec. 4.4, Eq. (10) and Sec. 5.3] The realization reward RG = b·g combines five VLM rubric scores (Eq. 7) with a gate g (Eq. 9). The gate parameters α=0.1 and β=0.9 are stated but never ablated, and the VLM judge used to produce the five rubric scores is the same Qwen3.5-122B-A10B family that provides the Diff Reward. No evidence is provided that these rubric scores correlate with human preference or with benchmark scores. Given the large RISE jump (+21.00), the reader cannot distinguish a genuine improvement in editing ability from reward overfitting to the VLM rubric. Please ablate the gate components and validate the reward against human judgments or a held-out independent benchmark.
minor comments (5)
  1. [Sec. 5.5, Table 8] The conclusion that 'the gain comes from coupling target expression with target realization' is based entirely on the BLINK benchmark. This is a single benchmark and the caption-only variants (Source-Caption SFT, Target-Caption SFT) are trained with the same recipe? If they use different numbers of steps or different data, please state it. At minimum, temper the claim or extend the ablation to at least one generation/editing benchmark.
  2. [Sec. 5.1] Please clarify the token limits: the first paragraph says 'global and per-sample token limits are both set to 35,000,' while the RL paragraph says 'maximum token budget of 32,768.' Are these different settings for SFT vs RL, or an inconsistency?
  3. [Tables 1–3] The tables are hard to read because several numbers run together (e.g., '76.9230.77' in Table 1, '71.8386.77' in Table 2). Please add proper spacing or vertical lines, and ensure column headers align with the data.
  4. [Fig. 6 / Table 7] The alignment protocol in Fig. 6 lists seven metrics, but Table 7 reports only six (Global Scene Alignment, Entity Presence, Entity Attribute, Entity Part Attribute, Spatial Relation, Inter-Attribute Relation) and an Overall. Is 'Fine-grained Avg.' the average of the five fine-grained ones? Please define the relationship clearly and state whether the 'Global Scene Alignment' is the same as the 'Global Scene Alignment' in Fig. 6.
  5. [Sec. 3.2] The conversion of visual-understanding data into editing-style instructions via τ(q) and ρ(I,q,a) is described at a high level. The examples in Fig. 11 are helpful, but the exact templates for τ and ρ are not given. Please provide the full prompt templates or point to the released code/data.

Circularity Check

0 steps flagged

No significant circularity: training signals and the Table 7 alignment metric use distinct VLM judges, and the main benchmark claims are anchored by external benchmarks.

full rationale

STBridge's central derivation is not circular. The SFT target captions are produced by Qwen3.5-35B-A3B from the target image, the RL rewards are computed by Qwen3.5-122B-A10B, and the Table 7 alignment score is evaluated by Gemini-3.1-Flash-Image-Preview; these are distinct judges, and the paper does not train against the Gemini metric. No equation in the paper reduces a reported result to its training input: Eq. (4) sums two supervised losses, Eqs. (6) and (10) define rewards from VLM difference/realization scores, and Table 7 is a separate held-out protocol. The main benchmark gains (BLINK, OddGridBench, MMVP, RISE, ImgEdit) are external and do not depend on STBridge's own VLM annotations. The same Qwen3.5-35B-A3B is used to generate SFT captions and as the official WISE judge, which raises an evaluation-validity concern for that one benchmark, but this is a correctness/robustness risk rather than a circular derivation: no parameter is fitted to WISE, and the other benchmarks are unaffected. There are no load-bearing self-citations or imported uniqueness theorems. Therefore the paper is self-contained against external benchmarks and has no significant circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central claim rests on two load-bearing assumptions: VLM judgments are trustworthy for both training and evaluation, and the editing-style data conversion preserves the semantic content of visual-understanding tasks. The only explicit free parameters are reward weights and data filters; no new physical or architectural entities are postulated.

free parameters (3)
  • Reward gate weights alpha, beta = alpha=0.1, beta=0.9
    Sec. 4.4, Eq. 9: chosen by hand to put the realization reward gate in [0.1, 1.0]. No sensitivity analysis is reported.
  • ThinkEdit-RL edit-score filter threshold = first-round edit score >= 3 and < 7
    Sec. 3.2: selects 'appropriately challenging' editing trajectories; threshold is ad hoc and not ablated.
  • Alignment matching function Align(...) = not specified
    Sec. 4.3, Eq. 6: Diff Reward relies on an unspecified scalar alignment function in [0,1]; implementation details are not given.
axioms (4)
  • domain assumption VLM-generated captions and VLM-computed rewards are reliable proxies for target semantics.
    Sec. 3.1 uses Qwen3.5-35B-A3B for target captions; Sec. 5.1 uses Qwen3.5-122B-A10B for rewards. No human validation is provided.
  • domain assumption Visual-understanding questions can be faithfully converted to editing-style transformations without changing task identity.
    Sec. 3.2, Eq. 1: each (I, q, a) is mapped to (Is = I, x = tau(q), It = rho(I,q,a)). If this conversion changes the underlying task, RL on converted data may not transfer to understanding benchmarks.
  • domain assumption The Gemini-3.1-Flash-Image-Preview judge measures understanding-generation consistency independently of the training signal.
    Sec. 5.5: the alignment score is computed entirely by this VLM judge; no human agreement study is reported.
  • standard math Flow-matching loss in image latent space is a valid generative objective.
    Sec. 4.2, Eq. 3: borrowed from prior unified-modal work; treated as given.

pith-pipeline@v1.3.0-alltime-deepseek · 22223 in / 9246 out tokens · 84807 ms · 2026-08-01T18:52:50.151738+00:00 · methodology

0 comments
read the original abstract

Unified multimodal models (UMMs) aim to integrate visual understanding and generation within a single architecture, but architectural unification alone does not ensure semantic consistency. A model may describe the intended target correctly while generating an inconsistent edit. This exposes an understanding-generation alignment gap: linguistic and visual outputs live in different spaces, yet should be governed by the same target semantics. We study this gap in image editing, where an instruction defines a target state that can be both described and visually realized. Given a source image and an edit instruction, we compare a UMM's target caption with its edited image to test whether the two outputs converge on the same result. Our analysis shows that existing UMMs remain weakly aligned, especially for fine-grained entities, attributes, spatial relations, and local details, indicating that semantic unification is not achieved by architecture alone. To bridge this gap, we propose STBridge, a shared-target alignment framework that connects understanding and generation through a common target state. Here the target caption expresses the desired visual result, while the edited image realizes it visually, replacing separate task-specific paths with a shared information flow from target expression to target realization. STBridge follows an align-then-optimize strategy: supervised fine-tuning first establishes the shared-target channel, and sequential reinforcement learning further refines target-centered coordination. Across visual understanding, image generation, and image editing benchmarks, STBridge consistently improves over the initialization model. Alignment analysis confirms that STBridge narrows the gap between what the model describes and what it generates, demonstrating shared-target alignment as an effective post-training strategy for bridging understanding and generation in UMMs.

Figures

Figures reproduced from arXiv: 2607.17140 by Hao Fang, Hongjun Wang, Peixian Chen, Rui Ma, Tongyuan Bai, Weibo Gu, Wei Liu, Xing Sun, Ye Wang, Zuwei Long.

Figure 1
Figure 1. Figure 1: Aggregate results. We compare BAGEL and Ours across visual understanding, image generation, and image editing benchmarks, reporting benchmark-specific metrics and absolute gains. Co-corresponding authors †Project leader ∗Work was done when Ye Wang interned at Tencent YouTu Lab. 1 arXiv:2607.17140v1 [cs.CV] 19 Jul 2026 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Generation and editing examples. Representative qualitative examples from STBridge are arranged as a staggered mosaic. Above: image-generation examples with diverse resolutions and aspect ratios. Below: instruction￾guided image-editing examples. 2 [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Knowledge-rich generation on WISE. We show image generation examples from STBridge on WISE. Each row corresponds to a knowledge dimension, with two sets, each containing a prompt, a thinking trace, and an image. Blue boxes contain the model’s thinking for translating the prompt into a target expression; magenta text highlights key knowledge constraints and visual target cues. 3 [PITH_FULL_IMAGE:figures/fu… view at source ↗
Figure 4
Figure 4. Figure 4: Image-understanding examples. Representative image-understanding examples from STBridge are arranged in a compact two-column qualitative grid. Each panel pairs the visual input on the left with a compact blue question￾answer block on the right. Blue text marks the model answer, and magenta text marks the ground-truth answer. 4 [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: RISE editing examples. Representative STBridge RISE image-editing examples pair the source image with the edited result and show an abbreviated thinking summary with key visual changes or preservation cues highlighted in magenta. 5 [PITH_FULL_IMAGE:figures/full_fig_p005_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Understanding-generation alignment analysis. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Shared-target alignment diagnostics. Representative diagnostics for BAGEL compare the understanding pathway and generation pathway under the same source image and editing instruction. Row (a) shows a generation failure: the caption correctly specifies the requested bicycle, but the generated image omits it. Row (b) shows an understanding failure: the generated image follows the watercolor-style edit, but t… view at source ↗
Figure 8
Figure 8. Figure 8: Training data composition. The SFT pie summarizes the PickNano-250K SFT split. The RL pies summarize the converted visual-understanding data and native image-editing data after grouping related fine-grained subtypes into coarse categories. 3.1 Training Data for the SFT Stage The SFT data is constructed to provide paired supervision for target expression and visual realization in the alignment stage. Starti… view at source ↗
Figure 9
Figure 9. Figure 9: SFT annotation pipeline. We directly use the 250K PickNano SFT split and annotate each example with a target caption. 10 [PITH_FULL_IMAGE:figures/full_fig_p010_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: SFT training example. A concrete SFT example pairs the source-target image pair and instruction with a target caption that describes the target visual result. The presented images are selected from the open-source Pico-Banana-400K dataset [39]. 3.2 Training Data for the RL Stage For RL post-training, we organize all data sources under the same editing-style tuple (Is , x, It). Here, Is is the source image… view at source ↗
Figure 11
Figure 11. Figure 11: Understanding-to-editing interface. We represent visual-understanding examples under the shared editing-style target interface. Each example shows a source-instruction-target transformation derived from a visual￾understanding question, demonstrating highlighting, cropping, correspondence marking, and spatial transformation operations. This shared interface enables both understanding and generation pathway… view at source ↗
Figure 12
Figure 12. Figure 12: STBridge pipeline. Overview of the STBridge three-stage post-training procedure. (a) Shared-target SFT establishes the shared-target alignment by supervising the model to predict a textual target expression and then realize the same shared target as the edited image. (b) Target-expression RL optimizes the textual expression using a Diff Reward without image rollouts. (c) Visual-realization RL freezes the … view at source ↗
Figure 13
Figure 13. Figure 13: Diff Reward. Illustration of the Diff Reward used in Target-Expression Optimization. For each sampled target caption, a VLM infers the visual difference between the source image and the caption-implied target visual result. The inferred difference is compared with the annotated reference difference between the source image and the target image, and the resulting alignment score is used as the reward. This… view at source ↗
Figure 14
Figure 14. Figure 14: Realization-oriented editing reward. Illustration of the reward used in Target-Realization Optimization. The optimized target-expression policy is frozen, and its predicted target caption is used as the condition for image generation. A VLM judge compares each source–generated pair under the instruction and returns five rubric scores: edit success, instruction-detail fidelity, source preservation, visual … view at source ↗
Figure 15
Figure 15. Figure 15: Visual-understanding comparisons. Qualitative BAGEL-vs-Ours comparisons on visual-understanding tasks are arranged in a three-row, two-column grid. Each panel shows the visual input and a compact answer block with the ground-truth answer, the BAGEL baseline prediction, and the Ours prediction. Green smile icons mark correct answers, while red frown icons mark incorrect BAGEL answers. All test images shown… view at source ↗
Figure 16
Figure 16. Figure 16: ImgEdit comparisons. Qualitative ImgEdit benchmark comparisons are arranged in a four-row, two-column grid. Each panel places the editing instruction on top and compares the source image, the BAGEL baseline output, and the Ours output in parallel. All test images shown above are selected from the open-source benchmark ImgEdit [50]. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_16.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 18 linked inside Pith

  1. [1]

    Ofa: Unifying architectures, tasks, and modalities through a simple sequence- to-sequence learning framework

    Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence- to-sequence learning framework. InInternational conference on machine learning, pages 23318–23340. PMLR, 2022

    2022

  2. [2]

    Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action

    Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26439–26455, 2024

    2024

  3. [3]

    Dreamllm: Synergistic multimodal comprehension and creation

    Runpei Dong, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, et al. Dreamllm: Synergistic multimodal comprehension and creation. In International Conference on Learning Representations, volume 2024, pages 6666–6702, 2024

    2024

  4. [4]

    Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    Pith/arXiv arXiv 2024

  5. [5]

    Show-o: One single transformer to unify multimodal understanding and generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. InInternational Conference on Learning Representations, volume 2025, pages 28240–28264, 2025

    2025

  6. [6]

    Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869, 2024

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869, 2024

    Pith/arXiv arXiv 2024

  7. [7]

    Janus: Decoupling visual encoding for unified multimodal understanding and generation

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 12966–12977, 2025

    2025

  8. [8]

    Emerging properties in unified multimodal pretraining.arXiv preprint arXiv:2505.14683, 2025

    Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, et al. Emerging properties in unified multimodal pretraining.arXiv preprint arXiv:2505.14683, 2025

    Pith/arXiv arXiv 2025

  9. [9]

    Seed-x: Multimodal models with unified multi-granularity comprehension and generation.arXiv preprint arXiv:2404.14396, 2024

    Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granularity comprehension and generation.arXiv preprint arXiv:2404.14396, 2024

    Pith/arXiv arXiv 2024

  10. [10]

    Vila-u: a unified foundation model integrating visual understanding and generation

    Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model integrating visual understanding and generation. InInternational Conference on Learning Representations, volume 2025, pages 93620–93638, 2025

    2025

  11. [11]

    Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset.arXiv preprint arXiv:2505.09568, 2025

    Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Sain- ing Xie, Silvio Savarese, et al. Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset.arXiv preprint arXiv:2505.09568, 2025

    Pith/arXiv arXiv 2025

  12. [12]

    Internvl-u: Democratizing unified multimodal models for understanding, reasoning, generation and editing.arXiv preprint arXiv:2603.09877, 2026

    Changyao Tian, Danni Yang, Guanzhou Chen, Erfei Cui, Zhaokai Wang, Yuchen Duan, Penghao Yin, Sitao Chen, Ganlin Yang, Mingxin Liu, et al. Internvl-u: Democratizing unified multimodal models for understanding, reasoning, generation and editing.arXiv preprint arXiv:2603.09877, 2026

    arXiv 2026

  13. [13]

    Tuna: Taming unified visual representations for native unified multimodal models

    Zhiheng Liu, Weiming Ren, Haozhe Liu, Zijian Zhou, Shoufa Chen, Haonan Qiu, Xiaoke Huang, Zhaochong An, Fanny Yang, Aditya Patel, et al. Tuna: Taming unified visual representations for native unified multimodal models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15740–15751, 2026. 25

    2026

  14. [14]

    Tuna-2: Pixel embeddings beat vision encoders for multimodal understanding and generation.arXiv preprint arXiv:2604.24763, 2026

    Zhiheng Liu, Weiming Ren, Xiaoke Huang, Shoufa Chen, Tianhong Li, Mengzhao Chen, Yatai Ji, Sen He, Jonas Schult, Belinda Zeng, et al. Tuna-2: Pixel embeddings beat vision encoders for multimodal understanding and generation.arXiv preprint arXiv:2604.24763, 2026

    Pith/arXiv arXiv 2026

  15. [15]

    Sensenova-u1: Unifying multimodal understanding and generation with neo-unify architecture.arXiv preprint arXiv:2605.12500, 2026

    Haiwen Diao, Penghao Wu, Hanming Deng, Jiahao Wang, Shihao Bai, Silei Wu, Weichen Fan, Wenjie Ye, Wenwen Tong, Xiangyu Fan, et al. Sensenova-u1: Unifying multimodal understanding and generation with neo-unify architecture.arXiv preprint arXiv:2605.12500, 2026

    Pith/arXiv arXiv 2026

  16. [16]

    Unified language-vision pretraining in llm with dynamic discrete visual tokenization

    Yang Jin, Kun Xu, Liwei Chen, Chao Liao, Jianchao Tan, Quzhe Huang, Bin Chen, Chengru Song, Di ZHANG, Wenwu Ou, et al. Unified language-vision pretraining in llm with dynamic discrete visual tokenization. InInternational Conference on Learning Representations, volume 2024, pages 51599–51619, 2024

    2024

  17. [17]

    Vl-gpt: A generative pre-trained transformer for vision and language understanding and generation.arXiv preprint arXiv:2312.09251, 2023

    Jinguo Zhu, Xiaohan Ding, Yixiao Ge, Yuying Ge, Sijie Zhao, Hengshuang Zhao, Xiaohua Wang, and Ying Shan. Vl-gpt: A generative pre-trained transformer for vision and language understanding and generation.arXiv preprint arXiv:2312.09251, 2023

    Pith/arXiv arXiv 2023

  18. [18]

    Generative multimodal models are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14398–14409, 2024

    2024

  19. [19]

    Show-o2: Improved native unified multimodal models.Advances in Neural Information Processing Systems, 38:47490–47518, 2026

    Jinheng Xie, Zhenheng Yang, and Mike Zheng Shou. Show-o2: Improved native unified multimodal models.Advances in Neural Information Processing Systems, 38:47490–47518, 2026

    2026

  20. [20]

    Transfusion: Predict the next token and diffuse images with one multi-modal model

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modal model. InInternational Conference on Learning Representations, volume 2025, pages 6446–6469, 2025

    2025

  21. [21]

    Metamorph: Multimodal understanding and generation via instruction tuning

    Shengbang Tong, David Fan, Jiachen Li, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal understanding and generation via instruction tuning. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 17001–17012, 2025

    2025

  22. [22]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.17811, 2025

    Pith/arXiv arXiv 2025

  23. [23]

    Onecat: Decoder-only auto-regressive model for unified un- derstanding and generation

    Han Li, Xinyu Peng, Yaoming Wang, Zelin Peng, Xin Chen, Rongxiang Weng, Jingang Wang, Xunliang Cai, Wenrui Dai, and Hongkai Xiong. Onecat: Decoder-only auto-regressive model for unified un- derstanding and generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 30235–30245, 2026

    2026

  24. [24]

    Unigen: Enhanced training & test-time strategies for unified multimodal understanding and generation.Advances in Neural Information Processing Systems, 38:152386–152415, 2026

    Rui Tian, Mingfei Gao, Mingze Xu, Jiaming Hu, Jiasen Lu, Zuxuan Wu, Yinfei Yang, and Afshin Dehghan. Unigen: Enhanced training & test-time strategies for unified multimodal understanding and generation.Advances in Neural Information Processing Systems, 38:152386–152415, 2026

    2026

  25. [25]

    Unirl: Self-improving unified multimodal models via supervised and reinforcement learning.arXiv preprint arXiv:2505.23380, 2025

    Weijia Mao, Zhenheng Yang, and Mike Zheng Shou. Unirl: Self-improving unified multimodal models via supervised and reinforcement learning.arXiv preprint arXiv:2505.23380, 2025

    Pith/arXiv arXiv 2025

  26. [26]

    Lavida-r1: Advancing reasoning for unified multimodal diffusion language models

    Shufan Li, Yuchen Zhu, Jiuxiang Gu, Kangning Liu, Zhe Lin, Yongxin Chen, Molei Tao, Aditya Grover, and Jason Kuen. Lavida-r1: Advancing reasoning for unified multimodal diffusion language models. arXiv preprint arXiv:2602.14147, 2026

    Pith/arXiv arXiv 2026

  27. [27]

    Towards unified multimodal interleaved generation via group relative policy optimization.Advances in Neural Information Processing Systems, 38:5332–5353, 2026

    Ming Nie, Chunwei Wang, Jianhua Han, Hang Xu, and Li Zhang. Towards unified multimodal interleaved generation via group relative policy optimization.Advances in Neural Information Processing Systems, 38:5332–5353, 2026. 26

    2026

  28. [28]

    Lvrpo: Language-visual alignment with grpo for multimodal under- standing and generation.arXiv preprint arXiv:2603.27693, 2026

    Shentong Mo and Sukmin Yun. Lvrpo: Language-visual alignment with grpo for multimodal under- standing and generation.arXiv preprint arXiv:2603.27693, 2026

    arXiv 2026

  29. [29]

    Generation enhances understanding in unified multimodal models via multi-representation generation

    Zihan Su, Hongyang Wei, Kangrui Cen, Yong Wang, Guanhua Chen, Chun Yuan, and Xiangxiang Chu. Generation enhances understanding in unified multimodal models via multi-representation generation. arXiv preprint arXiv:2601.21406, 2026

    Pith/arXiv arXiv 2026

  30. [30]

    Unified multimodal models as auto-encoders

    Zhiyuan Yan, Kaiqing Lin, Zongjian Li, Junyan Ye, Hui Han, Haochen Wang, Zhendong Wang, Bin Lin, Hao Li, Xinyan Xiao, et al. Unified multimodal models as auto-encoders. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 41903–41912, 2026

    2026

  31. [31]

    Reconstruction alignment improves unified multimodal models, 2025a.URL https://arxiv.org/abs/2509.07295, 1:3, 2025

    Ji Xie, Trevor Darrell, Luke Zettlemoyer, and XuDong Wang. Reconstruction alignment improves unified multimodal models, 2025a.URL https://arxiv.org/abs/2509.07295, 1:3, 2025

    Pith/arXiv arXiv 2025

  32. [32]

    Learning to generate via understanding: Understanding-driven intrinsic rewarding for unified multimodal models

    Jiadong Pan, Liang Li, Yuxin Peng, Yu-Ming Tang, Shuohuan Wang, Yu Sun, Hua Wu, Qingming Huang, and Haifeng Wang. Learning to generate via understanding: Understanding-driven intrinsic rewarding for unified multimodal models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22174–22184, 2026

    2026

  33. [33]

    Steer- ing visual generation in unified multimodal models with understanding supervision.arXiv preprint arXiv:2605.05781, 2026

    Zeyu Liu, Zanlin Ni, Yang Yue, Cheng Da, Huan Yang, Di Zhang, Kun Gai, and Gao Huang. Steer- ing visual generation in unified multimodal models with understanding supervision.arXiv preprint arXiv:2605.05781, 2026

    Pith/arXiv arXiv 2026

  34. [34]

    Flow-grpo: Training flow matching models via online rl.Advances in neural information processing systems, 38:40783–40818, 2026

    Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl.Advances in neural information processing systems, 38:40783–40818, 2026

    2026

  35. [35]

    Mixgrpo: Unlocking flow-based grpo efficiency with mixed ode-sde.arXiv preprint arXiv:2507.21802, 2025

    Junzhe Li, Yutao Cui, Tao Huang, Yinping Ma, Chun Fan, Yiming Cheng, Miles Yang, Zhao Zhong, and Liefeng Bo. Mixgrpo: Unlocking flow-based grpo efficiency with mixed ode-sde.arXiv preprint arXiv:2507.21802, 2025

    Pith/arXiv arXiv 2025

  36. [36]

    Editreward: A human- aligned reward model for instruction-guided image editing.arXiv preprint arXiv:2509.26346, 2025

    Keming Wu, Sicong Jiang, Max Ku, Ping Nie, Minghao Liu, and Wenhu Chen. Editreward: A human- aligned reward model for instruction-guided image editing.arXiv preprint arXiv:2509.26346, 2025

    arXiv 2025

  37. [37]

    Thinkrl-edit: Thinking in reinforcement learning for reasoning-centric image editing.arXiv preprint arXiv:2601.03467, 2026

    Hengjia Li, Liming Jiang, Qing Yan, Yizhi Song, Hao Kang, Zichuan Liu, Xin Lu, Boxi Wu, and Deng Cai. Thinkrl-edit: Thinking in reinforcement learning for reasoning-centric image editing.arXiv preprint arXiv:2601.03467, 2026

    arXiv 2026

  38. [38]

    Leveraging verifier-based reinforcement learning in image editing

    Hanzhong Guo, Jie Wu, Jie Liu, Yu Gao, Zilyu Ye, Linxiao Yuan, Xionghui Wang, Yizhou Yu, and Weilin Huang. Leveraging verifier-based reinforcement learning in image editing. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 34343–34352, 2026

    2026

  39. [39]

    Pico-banana-400k: A large-scale dataset for text-guided image editing

    Yusu Qian, Eli Bocek-Rivele, Liangchen Song, Jialing Tong, Yinfei Yang, Jiasen Lu, Wenze Hu, and Zhe Gan. Pico-banana-400k: A large-scale dataset for text-guided image editing. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 37226–37235, 2026

    2026

  40. [40]

    Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  41. [41]

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.Transactions of the association for computational linguistics, 2:67–78, 2014

    2014

  42. [42]

    Editthinker: Unlocking iterative reasoning for any image editor.arXiv preprint arXiv:2512.05965, 2025

    Hongyu Li, Manyuan Zhang, Dian Zheng, Ziyu Guo, Yimeng Jia, Kaituo Feng, Hao Yu, Yexin Liu, Yan Feng, Peng Pei, et al. Editthinker: Unlocking iterative reasoning for any image editor.arXiv preprint arXiv:2512.05965, 2025

    arXiv 2025

  43. [43]

    Blink: Multimodal large language models can see but not perceive

    Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. InEuropean Conference on Computer Vision, pages 148–166. Springer, 2024. 27

    2024

  44. [44]

    Cambrian-1: A fully open, vision-centric exploration of multimodal llms.Advances in Neural Information Processing Systems, 37:87310–87356, 2024

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai C Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms.Advances in Neural Information Processing Systems, 37:87310–87356, 2024

    2024

  45. [45]

    Thinkmorph: Emergent properties in multimodal interleaved chain-of-thought reasoning.arXiv preprint arXiv:2510.27492, 2025

    Jiawei Gu, Yunzhuo Hao, Huichen Will Wang, Linjie Li, Michael Qizhe Shieh, Yejin Choi, Ranjay Krishna, and Yu Cheng. Thinkmorph: Emergent properties in multimodal interleaved chain-of-thought reasoning.arXiv preprint arXiv:2510.27492, 2025

    arXiv 2025

  46. [46]

    Mm-vet: Evaluating large multimodal models for integrated capabilities.arXiv preprint arXiv:2308.02490, 2023

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities.arXiv preprint arXiv:2308.02490, 2023

    Pith/arXiv arXiv 2023

  47. [47]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9568–9578, 2024

    2024

  48. [48]

    Oddgridbench: Expos- ing the lack of fine-grained visual discrepancy sensitivity in multimodal large language models

    Tengjin Weng, Wenhao Jiang, Jingyi Wang, Ming Li, Lin Ma, and Zhong Ming. Oddgridbench: Expos- ing the lack of fine-grained visual discrepancy sensitivity in multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1575–1584, 2026

    2026

  49. [49]

    Wise: A world knowledge-informed semantic evaluation for text-to-image generation.arXiv preprint arXiv:2503.07265, 2025

    Yuwei Niu, Munan Ning, Mengren Zheng, Weiyang Jin, Bin Lin, Peng Jin, Jiaqi Liao, Chaoran Feng, Fanqing Meng, Kunpeng Ning, et al. Wise: A world knowledge-informed semantic evaluation for text-to-image generation.arXiv preprint arXiv:2503.07265, 2025

    Pith/arXiv arXiv 2025

  50. [50]

    Imgedit: A unified image editing dataset and benchmark.Advances in Neural Information Processing Systems, 38, 2026

    Yang Ye, Xianyi He, Zongjian Li, Shenghai Yuan, Zhiyuan Yan, Bohan Hou, Li Yuan, et al. Imgedit: A unified image editing dataset and benchmark.Advances in Neural Information Processing Systems, 38, 2026

    2026

  51. [51]

    Envisioning beyond the pixels: Benchmarking reasoning- informed visual editing.Advances in Neural Information Processing Systems, 38, 2026

    Xiangyu Zhao, Peiyuan Zhang, Kexian Tang, Xiaorong Zhu, Hao Li, Wenhao Chai, Zicheng Zhang, Renqiu Xia, Guangtao Zhai, Junchi Yan, et al. Envisioning beyond the pixels: Benchmarking reasoning- informed visual editing.Advances in Neural Information Processing Systems, 38, 2026. 28

    2026