Pith. sign in

REVIEW 4 major objections 4 minor 58 references

Unified multimodal models do not think in one shared semantic space: understanding-derived steering transfers to generation, while generation-derived steering does not transfer back.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 16:31 UTC pith:5QP53NLB

load-bearing objection The U→G steering result is real and worth a referee; the G→U failure that anchors the 'asymmetric semantic space' claim is unproven because the generation-branch vectors may be extracted from initial noise at an unspecified timestep. the 4 major comments →

arxiv 2607.26411 v1 pith:5QP53NLB submitted 2026-07-29 cs.CV

Do Unified Multimodal Models Think in One Space? A Lens Through Cross-Branch Steering

Yu Wang , Sharon Li This is my paper
classification cs.CV
keywords unified multimodal modelscross-branch steeringsemantic alignmentactivation steeringrepresentation probingtext-to-image generationgeneration hallucinationcontrastive steering dataset
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper asks whether unified multimodal models—systems that both understand images and generate them from text—actually think in one shared semantic space. To answer, it introduces cross-branch semantic steering: learn a direction for a concept such as 'red→pink' from one branch, inject it into the other during inference, and observe whether behavior shifts. The authors report that vectors learned from the understanding branch reliably control image generation (a red ribbon becomes pink, one object becomes three, inside becomes outside), while vectors learned from the generation branch barely move the understanding branch's answers. They trace the asymmetry to a representational mismatch: understanding-derived vectors capture object-centric semantics, whereas generation-derived vectors, under the extraction protocol used, encode low-level appearance statistics. If right, the result turns 'unified' from an architectural claim into a testable representational one, and offers steering as the test.

Core claim

Core discovery: an asymmetry in cross-branch semantic transfer. From 51 contrastive concept pairs, the authors extract steering vectors from answer-token activations in the understanding branch and inject them into VAE-latent hidden states during generation. This reliably changes attributes (red to pink, one to three, inside to outside) while preserving object identity, across three steering estimators. The reverse direction—vectors from generation-branch VAE latents, pooled by mean, object-aware, or attention weighting, applied to understanding text tokens—fails: QA accuracy stays at zero and logit differences stay near zero under dense strength sweeps. Subspace-projection analysis shows un

What carries the argument

The load-bearing instrument is the steering vector: an additive direction in the residual stream estimated from contrastive pairs (e.g., 'blue car' vs. 'red car') by three standard estimators—mean difference (CAA), first principal component of difference vectors (RepE), and normal vector of a linear decision boundary (ITI). Extraction reads residual activations at answer-token positions for the understanding branch and at pooled VAE-latent positions for the generation branch. Transfer adds the vector, scaled by a strength α, to all tokens of the target branch at selected layers. The PCA Subspace Projection (PSP) metric—the fraction of a steering vector's energy lying in the top-20 PCA subspa

Load-bearing premise

The central claim assumes that a concept's semantics are faithfully represented by a linear direction at the particular positions probed—answer-token activations for understanding and pooled VAE-latent activations for generation—so that failure to transfer means the branches genuinely lack shared structure rather than that the probe is looking in the wrong place.

What would settle it

Run the Gen→Und experiment again with generation vectors extracted by training a linear probe on VAE-latent activations to predict object-level attributes (or by decomposing latents into object-centric features with sparse autoencoders) at multiple layers, sweeping α densely. If any such protocol shifts held-out QA predictions above chance on color, counting, style, and appearance, the claimed asymmetry is extraction-dependent rather than representational, and the paper's conclusion would be falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Text-only contrastive QA pairs can be turned into control signals for image synthesis, since understanding-derived vectors change generated attributes while preserving object identity.
  • Steering with understanding-derived vectors improves generation faithfulness on relational concepts (inside/outside, above/below), correcting a class of spatial hallucinations.
  • Generation-branch representations, as extracted in this work, cannot serve as semantic control for understanding tasks; the reverse transfer fails even under exhaustive strength and extraction ablations.
  • Strong performance on understanding and generation benchmarks does not imply aligned representations; cross-branch steering offers a complementary diagnostic.
  • Cross-branch transfer depends on architecture: hybrid AR+diffusion models support it, while two pure autoregressive models degrade or fail, suggesting shared continuous latent structure matters.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the asymmetry may be probe-dependent. The understanding branch is probed at answer tokens where semantics are explicitly decoded, while generation latents are probed at pooled positions; a nonlinear probe, or a probe at concept-specific VAE tokens, might find a shared space the current protocol cannot see.
  • Beyond the paper: cross-branch transferability could become a standard evaluation axis for UMMs—a model can top both understanding and generation benchmarks yet fail a steering test, and such failures may predict where it hallucinates.
  • Beyond the paper: a falsifiable design prediction follows—training generation latents with object-level semantic supervision (e.g., reconstruction or contrastive losses tied to attributes) should raise Gen→Und transfer and shrink the asymmetry.
  • Beyond the paper: the paper's own failure cases (wrong grounding, over-generalized attributes) suggest that stronger object grounding, not just a shared latent space, may be what enables transfer; this predicts that grounding-trained UMMs will show higher cross-branch steering success.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper asks whether the understanding and generation branches of unified multimodal models (UMMs) share a transferable semantic space. It introduces cross-branch semantic steering: for 51 binary concept pairs in the new UMMSteer dataset, contrastive activations are collected from one branch (answer-token residual states for understanding; pooled VAE-latent states for generation), steering vectors are estimated with CAA/RepE/ITI, and the vectors are added to the other branch during inference. On BAGEL, understanding-to-generation (U→G) steering achieves high steering success rates on many concepts (Table 1, Fig. 1), while generation-to-understanding (G→U) steering leaves QA outputs unchanged (Appendix B.3, Table 6). The authors attribute the asymmetry to a representational mismatch: understanding-derived vectors are object-centric and transferable, while generation-derived vectors encode low-level appearance features. They also report U→G results on Janus-Pro, UniPic-1, and Show-o2, propose a PCA subspace projection metric, and demonstrate a hallucination-reduction application on relational prompts.

Significance. If the central asymmetry claim is correct, the paper makes a genuinely useful contribution: it provides a causal, intervention-based probe for semantic alignment in UMMs, introduces a reusable contrastive dataset, and shows that architectural unification does not automatically imply a shared semantic space. The paper also includes several good practices: a human-validated VLM judge (Cohen's κ = 0.894 on 350 images), random- and unrelated-direction baselines (Table 5), and multiple extraction/intervention ablations. However, the conclusion is currently stronger than the evidence. The generation-branch extraction protocol is underspecified in a way that could produce the observed G→U failure as an artifact, the reverse direction is tested on a single model, and no quantitative self-branch positive control for generation-derived vectors is reported. These issues are fixable with additional experiments, but they are load-bearing for the paper's main claim.

major comments (4)
  1. [Appendix A.3, Eq. (4), Fig. 8] The generation-branch steering vectors are extracted from 'all VAE latent token positions' without specifying the denoising timestep at which those latents are read. The paper's own pipeline figures label the inputs as 'Noise VAE token' (Fig. 2, Fig. 8). If the contrastive activations are collected at or near t=0, the extracted vector contains essentially no learned visual-semantic content, so the G→U failure would say only that text-prompt conditioning at pure noise does not linearly encode object semantics. Appendix C.2 changes pooling (global mean, object-aware, attention-weighted) and Appendix C.7 changes intervention positions, but neither varies or even reports the timestep. This is a direct confound for the central asymmetry claim. The authors must specify the timestep and, ideally, extract vectors at multiple timesteps (early/mid/late) to show that the G→U result is not a timing
  2. [Sec. 5, Fig. 5(a), Appendix C.2] There is no quantitative positive control showing that generation-derived vectors can steer the generation branch itself. The only self-branch evidence is qualitative Fig. 5(a), which actually shows that the red→blue generation-derived vector fails to change the object color and instead alters background/illumination. That is evidence that the extraction protocol did not recover the target semantic direction, not evidence that the generation branch lacks transferable semantics. To support the asymmetry interpretation, the authors should demonstrate that some generation-branch extraction (e.g., at a particular timestep, token position, or pooling scheme) yields a vector that quantitatively steers generation in a self-branch setting, and then show that this same vector still fails to transfer to understanding.
  3. [Sec. 4.2 vs. Sec. 6, Table 4] The reverse (G→U) direction is evaluated only on BAGEL. The cross-architecture experiments in Sec. 6 and Table 4 cover only U→G steering. Thus the claim that the asymmetry is a general property of UMMs is not supported: on Janus-Pro, UniPic-1, and Show-o2, the paper does not test whether generation-derived vectors can or cannot steer understanding. At minimum, the G→U experiment should be run on the same additional models, or the conclusion should be explicitly limited to BAGEL.
  4. [Appendix C.6, Table 8] The steering strength α is selected using the SSR metric on the red→blue concept, and Table 8 shows that CAA performance drops from 100% at α=0.1 to 10% at α=0.4. Because this same α is then used for all reported experiments, the headline numbers in Table 1 may be optimistic and concept-specific. The authors should either use a held-out validation split for α selection, report results across a range of α values for all concept categories, or justify why a single α chosen on one color concept is appropriate for all 51 concept pairs.
minor comments (4)
  1. [Appendix A.2, Fig. 7] The test sample reads 'A red apple on the table. What color is this car?' — likely 'car' should be 'apple' or another object. This typo appears in the figure caption and could confuse readers.
  2. [Table 2] Several pairs (e.g., red↔blue, red↔green) appear in both the Color and Text families. If this is intentional, explain that the Text family includes color-word pairs that must be rendered as text; otherwise remove the duplicates.
  3. [Fig. 17 caption] The caption says 'understanding ↔ generation steering' but the appendix shows only U→G examples. The directional arrow should be corrected.
  4. [Appendix C.4] The PSP metric fixes k=20 without sensitivity analysis. Since the 'semantic subspace' is built from 50 contrastive difference vectors, the interpretation of PSP as measuring alignment with the branch's semantic structure should be softened or accompanied by a k-ablation.

Circularity Check

1 steps flagged

Central transfer result is non-circular; only the self-branch PSP metric has a by-construction component, and the diffusion-timestep ambiguity is a validity concern rather than circularity.

specific steps
  1. self definitional [Sec. 5, 'Subspace Alignment Analysis'; Appendix C.4 (Eqs. 6, 11–13); Fig. 4(c)]
    "Eq. 6: "v^{(l)}_{CAA} = 1/N \sum_i \Delta h^{(l)}_i". App. C.4: "d_i = h_{+,i} - h_{-,i}"; "We perform Principal Component Analysis (PCA) on the set {d_i}_{i=1}^N ... Given a steering vector s, we measure its alignment with the semantic subspace via the projection ratio: PSP(s) = |U^\top s|^2 / |s|^2."; Sec. 5: "We compute the PSP for the red–blue semantic direction in both self-branch and cross-branch settings.""

    For a self-branch CAA vector, s is the average of the same difference vectors d_i used to build the PCA subspace U: Eq. 6 defines v_CAA as the mean of Δh_i, while Eq. 11 defines d_i = h_+ - h_- and Eq. 12 builds U from PCA of {d_i}. A vector formed as the mean of a set will generally have high energy in the top-PC subspace of that same set, so high self-branch PSP is partly guaranteed by construction. The paper cites this self-branch alignment as evidence that steering vectors 'follow the model's natural semantic structure.' The cross-branch PSP comparison—understanding vector vs generation subspace, generation vector vs understanding subspace—is not circular, and the central U→G/G→U intervention results do not depend on the self-branch projection. Thus this is a supporting-metric circular

full rationale

The main derivation chain is: UMMSteer contrastive data → CAA/RepE/ITI vector on the source branch → additive intervention on the target branch → measured held-out SSR or ΔmLD. No parameter is fitted to the target branch's outputs: U→G uses held-out prompts with an external Qwen-VL judge, and G→U is evaluated on held-out QA/logits. The headline asymmetry is therefore an independent empirical intervention, not a fitted-input-called-prediction. The only by-construction component I found is the self-branch PSP alignment (Eq. 6 vs Eqs. 11–13), and it is not load-bearing for the asymmetry. The unspecified diffusion timestep at which generation-branch VAE-latent vectors are read (App. A.3, Eq. 4; Fig. 8 labels 'Noise VAE token') is a legitimate threat to the interpretation that the vectors encode the generation branch's semantic content, and Appendix C.2/C.7 ablate pooling and intervention positions but not the timestep. However, that is an extraction-validity/artifact concern, not a reduction of the result to its inputs, so I do not count it as circularity under the hard rules. There are no load-bearing self-citations or imported uniqueness theorems. Score 2 reflects one minor by-construction supporting metric; the central claim retains independent content.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central result carries three hand-set choices (α, k, dataset size) and rests on the linear-intervention assumption and on a VLM judge. The PSP explanation partially reuses the same contrastive data used to build vectors, so it is not a fully independent test of the proposed mechanism.

free parameters (3)
  • steering strength α = α=0.1 (CAA), α=0.2 (RepE, ITI)
    Chosen from ablation on red→blue SSR (Table 8) without a separate validation split; the reported U→G success rates depend on this choice.
  • PCA subspace dimension k = 20
    Set for all PSP analyses; no sensitivity analysis is reported.
  • number of contrastive pairs per concept = 50 train / 50 eval
    Design choice; no evidence that 50 is sufficient to stabilize steering directions across concepts.
axioms (4)
  • domain assumption Linear additive interventions on residual-stream hidden states in one branch are a valid and behaviorally meaningful probe of semantics in the other branch.
    The entire framework (Eq. 1 and Sec. 3) assumes steering vectors learned in one branch can be added to the other branch's hidden states without corrupting computation.
  • ad hoc to paper The 51 hand-picked binary concept pairs and template prompts in UMMSteer are representative enough of semantic structure to support conclusions about the branches' shared subspace.
    Def. 1 and Appendix A.2 define the concept set; the paper offers no evidence that these pairs sample the space broadly.
  • domain assumption Qwen-VL-3 30B's two-stage judgments reliably operationalize 'contains target object' and 'satisfies steered attribute' for all 50 held-out prompts per concept.
    Validated on 350 human-annotated images (Cohen's κ=0.894 overall, but color κ=0.540), then applied to all remaining samples.
  • standard math PCA subspace projection (top-k=20) captures the 'semantic subspace' of a branch.
    Appendix C.4 defines PSP using PCA on contrastive difference vectors; this assumes the relevant semantics are linear and low-dimensional.

pith-pipeline@v1.3.0-daily-deepseek · 23670 in / 14284 out tokens · 119141 ms · 2026-08-01T16:31:22.001157+00:00 · methodology

0 comments
read the original abstract

Unified multimodal models (UMMs) aim to integrate understanding and generation within a single architecture, yet it remains unclear whether these capabilities share a unified and transferable semantic space. This question is fundamentally challenging, as the two branches operate over heterogeneous representations (text tokens vs.\ visual latents) and distinct training objectives, making direct comparison difficult. To address this, we introduce \emph{cross-branch semantic steering}, an intervention-based framework that extracts semantic directions from one branch and applies them to the other. We show that steering vectors learned from the understanding branch can transfer to generation, enabling controllable image synthesis and improved semantic faithfulness. In contrast, the reverse direction consistently shows limited effectiveness. Our analysis suggests that this asymmetry may be related to a practical representational mismatch: understanding-derived vectors capture transferable, object-centric semantics, while generation-derived vectors primarily encode low-level appearance features. Our results reveal that architectural unification does not guarantee semantic alignment, and establish cross-branch steering as a practical tool for probing multimodal representations.

Figures

Figures reproduced from arXiv: 2607.26411 by Sharon Li, Yu Wang.

Figure 1
Figure 1. Figure 1: Qualitative results of understanding → generation steering across semantic types. Each pair shows the original generation (left) and the steered output (right). Steering modifies the target attribute while preserving object identity. More results are provided in the appendix D.1. generation branch to produce a pink object instead of a red one, this constitutes direct, causal evidence of a unified semantic … view at source ↗
Figure 2
Figure 2. Figure 2: Cross-branch steering pipeline (Und → Gen). (a) Contrastive text pairs (e.g., ‘blue’ vs. ‘red’) are fed through the understanding branch. At each layer, we extract residual stream activations from the answer token and compute steering vectors. (b) At inference time, the input text prompt (e.g., ‘one red cup’) is encoded into a prefix KV cache. During denoising, steering vectors are added to VAE token repre… view at source ↗
Figure 3
Figure 3. Figure 3: The steering success rates (SSR) across multiple semantic categories. The effectiveness of CAA-based steering varies within object-level, relational-level, and scene-level semantics. Full per-concept results are provided in Appendix B.2. Steering Effectiveness. We presents the steering success rates across all semantic categories based on BAGEL [1] in Tab. 1. Across all three steering methods, understandin… view at source ↗
Figure 4
Figure 4. Figure 4: Analysis of cross-branch (Gen → Und) steering failure. (a) Setup. Steering vectors learned from the generation branch (e.g., blue vs. red) are applied to the understanding branch, and we evaluate whether QA predictions can be shifted. (b) Layer-wise logit difference. Here, ∆mLD quantifies the effect of steering vectors on model output logits. Understanding-derived vectors consistently shift model preferenc… view at source ↗
Figure 5
Figure 5. Figure 5: (a) Generation-derived vectors fail even in self-branch steering. Steering vectors learned from the generation branch (red → blue) are applied back to the generation branch. Vectors instead alter global appearance (background, illumination) without changing the object. (b) Generalization across UMM architectures. Cross-branch steering fails in pure autoregressive structures (Janus￾Pro [2], UniPic-1 [4]), b… view at source ↗
Figure 6
Figure 6. Figure 6: Understanding-derived steering reduces generation hallucination. (a) Quantitative comparison shows that steering significantly improves generation on relational semantics. (b) Qualitative examples show corrected spatial configurations (e.g., inside, below) that the baseline model hallucinates. autoregressive (AR) models, Janus-Pro [2] and UniPic-1 [4], as well as a hybrid AR+diffusion model, Show-o2 [7], b… view at source ↗
Figure 7
Figure 7. Figure 7: Illustration of cross-branch steering data samples. (a) Understanding-to-generation (U→G): steering vectors are learned from contrastive QA pairs in the understanding branch and applied to image generation, where output image are evaluated using a VQA-based framework. (b) Generation-to-understanding (G→U): steering vectors are learned from contrastive text prompt pairs in the generation branch and applied … view at source ↗
Figure 8
Figure 8. Figure 8: Cross-branch steering pipeline (Gen → Und). First, contrastive text prompt pairs (e.g., ‘blue’ vs. ‘red’) are fed through the generation branch. At each layer, we extract residual-stream activations from the VAE-token hidden states and use them to compute a generation-derived steering vector. At inference time, a test sample is fed into the understanding branch, and the steering vector is added to the text… view at source ↗
Figure 9
Figure 9. Figure 9: Two-stage VQA evaluation prompt for measuring steering success. The judge first verifies [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Qualitative examples of evaluator outputs from Qwen3 VL-30B [53] [PITH_FULL_IMAGE:figures/full_fig_p020_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Steering success rates for each concept across object-level semantics. [PITH_FULL_IMAGE:figures/full_fig_p021_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Steering success rates for each concept across relational semantics. [PITH_FULL_IMAGE:figures/full_fig_p022_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Steering success rates for each concept across scene-level semantics. [PITH_FULL_IMAGE:figures/full_fig_p022_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Additional Layer-wise Logit Difference Analysis. [PITH_FULL_IMAGE:figures/full_fig_p025_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Additional Layer-wise PCA Subspace Projection Analysis. [PITH_FULL_IMAGE:figures/full_fig_p026_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Ablation about the steering layer. Steering is most effective in early-to-middle layers (2–5) and degrades in deeper layers, indicating that transferable semantics are primarily encoded in intermediate representations. C.5 Generalization to Out-of-Template Compositional Prompts Our primary evaluation uses controlled and largely template-based examples to isolate individual semantic factors and reduce pote… view at source ↗
Figure 17
Figure 17. Figure 17: Additional qualitative results of understanding↔generation steering. Each pair shows the original generation (left) and the steered output (right). Steering modifies the target attribute while preserving object identity. strength is varied over a broad range. Therefore, the weak Gen→Und transfer cannot be attributed solely to the use of all text-token hidden states as the intervention target. These result… view at source ↗
Figure 18
Figure 18. Figure 18: Additional qualitative results of understanding↔generation steering. Each pair shows the original generation (left) and the steered output (right). Steering modifies the target attribute while preserving object identity. Overall, these qualitative results provide strong visual evidence that cross-branch steering from understanding to generation enables precise and compositional semantic control. D.2 Failu… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

58 extracted references · 8 linked inside Pith

  1. [1]

    Emerging properties in unified multimodal pretraining, 2025

    Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, Guang Shi, and Haoqi Fan. Emerging properties in unified multimodal pretraining, 2025

    2025

  2. [2]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling.arXiv preprint arXiv:2501.17811, 2025

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling.arXiv preprint arXiv:2501.17811, 2025

    Pith/arXiv arXiv 2025

  3. [3]

    Qwen-image technical report, 2025

    Chenfei Wu, Jiahao Li, Jingren Zhou, Junyang Lin, Kaiyuan Gao, Kun Yan, Sheng ming Yin, Shuai Bai, Xiao Xu, Yilei Chen, Yuxiang Chen, Zecheng Tang, Zekai Zhang, Zhengyi Wang, An Yang, Bowen Yu, Chen Cheng, Dayiheng Liu, Deqing Li, Hang Zhang, Hao Meng, Hu Wei, Jingyuan Ni, Kai Chen, Kuan Cao, Liang Peng, Lin Qu, Minggang Wu, Peng Wang, Shuting Yu, Tingkun...

    2025

  4. [4]

    Skywork unipic: Unified autoregressive modeling for visual understanding and generation, 2025

    Peiyu Wang, Yi Peng, Yimeng Gan, Liang Hu, Tianyidan Xie, Xiaokun Wang, Yichen Wei, Chuanxin Tang, Bo Zhu, Changshi Li, Hongyang Wei, Eric Li, Xuchen Song, Yang Liu, and Yahui Zhou. Skywork unipic: Unified autoregressive modeling for visual understanding and generation, 2025

    2025

  5. [5]

    Unimodel: A visual-only framework for unified multimodal under- standing and generation, 2025

    Chi Zhang, Jiepeng Wang, Youming Wang, Yuanzhi Liang, Xiaoyan Yang, Zuoxin Li, Haibin Huang, and Xuelong Li. Unimodel: A visual-only framework for unified multimodal under- standing and generation, 2025

    2025

  6. [6]

    Uniworld-v1: High-resolution semantic encoders for unified visual understanding and generation, 2025

    Bin Lin, Zongjian Li, Xinhua Cheng, Yuwei Niu, Yang Ye, Xianyi He, Shenghai Yuan, Wangbo Yu, Shaodong Wang, Yunyang Ge, Yatian Pang, and Li Yuan. Uniworld-v1: High-resolution semantic encoders for unified visual understanding and generation, 2025

    2025

  7. [7]

    Show-o2: Improved native unified multimodal models, 2025

    Jinheng Xie, Zhenheng Yang, and Mike Zheng Shou. Show-o2: Improved native unified multimodal models, 2025

    2025

  8. [8]

    Unified multimodal understanding and generation models: Advances, challenges, and opportunities, 2026

    Shanshan Zhao, Xinjie Zhang, Jintao Guo, Jiakui Hu, Lunhao Duan, Minghao Fu, Yong Xien Chng, Guo-Hua Wang, Qing-Guo Chen, Zhao Xu, Weihua Luo, and Kaifu Zhang. Unified multimodal understanding and generation models: Advances, challenges, and opportunities, 2026

    2026

  9. [9]

    Tokenflow: Unified image tokenizer for multimodal understanding and generation

    Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang, Yi Jiang, Yiming Gao, Hu Ye, Daniel K Du, Zehuan Yuan, and Xinglong Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 2545–2555, 2025

    2025

  10. [10]

    Reconstruction alignment improves unified multimodal models, 2025

    Ji Xie, Trevor Darrell, Luke Zettlemoyer, and XuDong Wang. Reconstruction alignment improves unified multimodal models, 2025

    2025

  11. [11]

    Harmonizing visual representations for unified multimodal understanding and generation, 2025

    Size Wu, Wenwei Zhang, Lumin Xu, Sheng Jin, Zhonghua Wu, Qingyi Tao, Wentao Liu, Wei Li, and Chen Change Loy. Harmonizing visual representations for unified multimodal understanding and generation, 2025

    2025

  12. [12]

    Openuni: A simple baseline for unified multimodal understanding and generation, 2025

    Size Wu, Zhonghua Wu, Zerui Gong, Qingyi Tao, Sheng Jin, Qinyue Li, Wei Li, and Chen Change Loy. Openuni: A simple baseline for unified multimodal understanding and generation, 2025. 10

    2025

  13. [13]

    Torchumm: A unified multimodal model codebase for evaluation, analysis, and post-training, 2026

    Yinyi Luo, Wenwen Wang, Hayes Bai, Hongyu Zhu, Hao Chen, Pan He, Marios Savvides, Sharon Li, and Jindong Wang. Torchumm: A unified multimodal model codebase for evaluation, analysis, and post-training, 2026

    2026

  14. [14]

    Unigame: Turning a unified multimodal model into its own adversary, 2026

    Zhaolong Su, Wang Lu, Hao Chen, Sharon Li, and Jindong Wang. Unigame: Turning a unified multimodal model into its own adversary, 2026

    2026

  15. [15]

    Unified multimodal models as auto-encoders, 2026

    Zhiyuan Yan, Kaiqing Lin, Zongjian Li, Junyan Ye, Hui Han, Haochen Wang, Zhendong Wang, Bin Lin, Hao Li, Xinyan Xiao, Jingdong Wang, Haifeng Wang, and Li Yuan. Unified multimodal models as auto-encoders, 2026

    2026

  16. [16]

    Rethinking umm visual generation: Masked modeling for efficient image-only pre-training, 2026

    Peng Sun, Jun Xie, and Tao Lin. Rethinking umm visual generation: Masked modeling for efficient image-only pre-training, 2026

    2026

  17. [17]

    Tuna: Taming unified visual representations for native unified multimodal models, 2025

    Zhiheng Liu, Weiming Ren, Haozhe Liu, Zijian Zhou, Shoufa Chen, Haonan Qiu, Xiaoke Huang, Zhaochong An, Fanny Yang, Aditya Patel, Viktar Atliha, Tony Ng, Xiao Han, Chuyan Zhu, Chenyang Zhang, Ding Liu, Juan-Manuel Perez-Rua, Sen He, Jürgen Schmidhuber, Wenhu Chen, Ping Luo, Wei Liu, Tao Xiang, Jonas Schult, and Yuren Cong. Tuna: Taming unified visual repr...

    2025

  18. [18]

    Omnigen2: Towards instruction-aligned multimodal generation, 2026

    Chenyuan Wu, Pengfei Zheng, Ruiran Yan, Shitao Xiao, Xin Luo, Yueze Wang, Wanli Li, Xiyan Jiang, Yexin Liu, Junjie Zhou, Ze Liu, Ziyi Xia, Chaofan Li, Haoge Deng, Jiahao Wang, Kun Luo, Bo Zhang, Defu Lian, Xinlong Wang, Zhongyuan Wang, Tiejun Huang, and Zheng Liu. Omnigen2: Towards instruction-aligned multimodal generation, 2026

    2026

  19. [19]

    Unitoken: Harmonizing multimodal understanding and generation through unified visual encoding

    Yang Jiao, Haibo Qiu, Zequn Jie, Shaoxiang Chen, Jingjing Chen, Lin Ma, and Yu-Gang Jiang. Unitoken: Harmonizing multimodal understanding and generation through unified visual encoding. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 3600–3610, 2025

    2025

  20. [20]

    Architecture decoupling is not all you need for unified multimodal model.arXiv preprint arXiv:2511.22663, 2025

    Dian Zheng, Manyuan Zhang, Hongyu Li, Kai Zou, Hongbo Liu, Ziyu Guo, Kaituo Feng, Yexin Liu, Ying Luo, Yan Feng, et al. Architecture decoupling is not all you need for unified multimodal model.arXiv preprint arXiv:2511.22663, 2025

    Pith/arXiv arXiv 2025

  21. [21]

    Quantifying the gap between understanding and generation within unified multimodal models.arXiv preprint arXiv:2602.02140, 2026

    Chenlong Wang, Yuhang Chen, Zhihan Hu, Dongping Chen, Wenhu Chen, Sarah Wiegreffe, and Tianyi Zhou. Quantifying the gap between understanding and generation within unified multimodal models.arXiv preprint arXiv:2602.02140, 2026

    arXiv 2026

  22. [22]

    Unieval: Unified holistic evaluation for unified multimodal understanding and generation, 2025

    Yi Li, Haonan Wang, Qixiang Zhang, Boyu Xiao, Chenchang Hu, Hualiang Wang, and Xi- aomeng Li. Unieval: Unified holistic evaluation for unified multimodal understanding and generation, 2025

    2025

  23. [23]

    Steering llama 2 via contrastive activation addition, 2024

    Nina Panickssery, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. Steering llama 2 via contrastive activation addition, 2024

    2024

  24. [24]

    Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J

    Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, J. Zico Kolter, and Dan Hendrycks. Representation engineering: A top-down approach to a...

    2025

  25. [25]

    Inference- time intervention: Eliciting truthful answers from a language model, 2024

    Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference- time intervention: Eliciting truthful answers from a language model, 2024

    2024

  26. [26]

    Unifork: Exploring modality alignment for unified multimodal understanding and generation

    Teng Li, Quanfeng Lu, Lirui Zhao, Hao Li, Xizhou Zhu, Yu Qiao, Jun Zhang, and Wenqi Shao. Unifork: Exploring modality alignment for unified multimodal understanding and generation. arXiv preprint arXiv:2506.17202, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    Uni-cot: Towards unified chain-of-thought reasoning across text and vision, 2026

    Luozheng Qin, Jia Gong, Yuqing Sun, Tianjiao Li, Mengping Yang, Xiaomeng Yang, Chao Qu, Zhiyu Tan, and Hao Li. Uni-cot: Towards unified chain-of-thought reasoning across text and vision, 2026. 11

    2026

  28. [28]

    Mindomni: Unleashing reasoning generation in vision language models with rgpo, 2025

    Yicheng Xiao, Lin Song, Yukang Chen, Yingmin Luo, Yuxin Chen, Yukang Gan, Wei Huang, Xiu Li, Xiaojuan Qi, and Ying Shan. Mindomni: Unleashing reasoning generation in vision language models with rgpo, 2025

    2025

  29. [29]

    Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset, 2025

    Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, Le Xue, Caiming Xiong, and Ran Xu. Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset, 2025

    2025

  30. [30]

    Metamorph: Multimodal under- standing and generation via instruction tuning, 2024

    Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal under- standing and generation via instruction tuning, 2024

    2024

  31. [31]

    Hermes: A unified self-driving world model for simultaneous 3d scene understanding and generation, 2025

    Xin Zhou, Dingkang Liang, Sifan Tu, Xiwu Chen, Yikang Ding, Dingyuan Zhang, Feiyang Tan, Hengshuang Zhao, and Xiang Bai. Hermes: A unified self-driving world model for simultaneous 3d scene understanding and generation, 2025

    2025

  32. [32]

    Generative multimodal models are in-context learners, 2024

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners, 2024

    2024

  33. [33]

    Vazquez, Ulisse Mini, and Monte MacDiarmid

    Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. Steering language models with activation engineering, 2024

    2024

  34. [34]

    Lee, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Erik Miehling, Pierre Dognin, Manish Nagireddy, and Amit Dhurandhar

    Bruce W. Lee, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Erik Miehling, Pierre Dognin, Manish Nagireddy, and Amit Dhurandhar. Programming refusal with conditional activation steering, 2025

    2025

  35. [35]

    Extending activation steering to broad skills and multiple behaviours, 2024

    Teun van der Weij, Massimo Poesio, and Nandi Schoots. Extending activation steering to broad skills and multiple behaviours, 2024

    2024

  36. [36]

    Easysteer: A unified framework for high-performance and extensible llm steering.arXiv preprint arXiv:2509.25175, 2025

    Haolei Xu, Xinyu Mei, Yuchen Yan, Rui Zhou, Wenqi Zhang, Weiming Lu, Yueting Zhuang, and Yongliang Shen. Easysteer: A unified framework for high-performance and extensible llm steering.arXiv preprint arXiv:2509.25175, 2025

    arXiv 2025

  37. [37]

    Enhancing llm steering through sparse autoencoder-based vector refinement.arXiv preprint arXiv:2509.23799, 2025

    Anyi Wang, Xuansheng Wu, Dong Shu, Yunpu Ma, and Ninghao Liu. Enhancing llm steering through sparse autoencoder-based vector refinement.arXiv preprint arXiv:2509.23799, 2025

    arXiv 2025

  38. [38]

    Understanding the theoretical foundations of deep neural networks through differential equations.arXiv preprint arXiv:2603.18331, 2026

    Hongjue Zhao, Yizhuo Chen, Yuchen Wang, Hairong Qi, Lui Sha, Tarek Abdelzaher, and Huajie Shao. Understanding the theoretical foundations of deep neural networks through differential equations.arXiv preprint arXiv:2603.18331, 2026

    arXiv 2026

  39. [39]

    Controllable llm reasoning via sparse autoencoder-based steering.arXiv preprint arXiv:2601.03595, 2026

    Yi Fang, Wenjie Wang, Mingfeng Xue, Boyi Deng, Fengli Xu, Dayiheng Liu, and Fuli Feng. Controllable llm reasoning via sparse autoencoder-based steering.arXiv preprint arXiv:2601.03595, 2026

    Pith/arXiv arXiv 2026

  40. [40]

    Steering llm reasoning through bias-only adaptation, 2025

    Viacheslav Sinii, Alexey Gorbatovski, Artem Cherepanov, Boris Shaposhnikov, Nikita Balagan- sky, and Daniil Gavrilov. Steering llm reasoning through bias-only adaptation, 2025

    2025

  41. [41]

    Steerx: Disentangled steering for llm personalization, 2025

    Xiaoyan Zhao, Ming Yan, Yilun Qiu, Haoting Ni, Yang Zhang, Fuli Feng, Hong Cheng, and Tat-Seng Chua. Steerx: Disentangled steering for llm personalization, 2025

    2025

  42. [42]

    Steering guidance for personalized text-to-image diffusion models, 2025

    Sunghyun Park, Seokeon Choi, Hyoungwoo Park, and Sungrack Yun. Steering guidance for personalized text-to-image diffusion models, 2025

    2025

  43. [43]

    Metaxas, and Yezhou Yang

    Maitreya Patel, Song Wen, Dimitris N. Metaxas, and Yezhou Yang. Steering rectified flow models in the vector field for controlled image generation.arXiv preprint arXiv:2412.00100, 2024

    Pith/arXiv arXiv 2024

  44. [44]

    Steering large reasoning models towards concise reasoning via flow matching, 2026

    Yawei Li, Benjamin Bergner, Yinghan Zhao, Vihang Prakash Patil, Bei Chen, and Cheng Wang. Steering large reasoning models towards concise reasoning via flow matching, 2026

    2026

  45. [45]

    Steering away from harm: An adaptive approach to defending vision language model against jailbreaks

    Han Wang, Gang Wang, and Huan Zhang. Steering away from harm: An adaptive approach to defending vision language model against jailbreaks. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 29947–29957, 2025. 12

    2025

  46. [46]

    Learning to steer: Input-dependent steering for multimodal llms.arXiv preprint arXiv:2508.12815, 2025

    Jayneel Parekh, Pegah Khayatan, Mustafa Shukor, Arnaud Dapogny, Alasdair Newson, and Matthieu Cord. Learning to steer: Input-dependent steering for multimodal llms.arXiv preprint arXiv:2508.12815, 2025

    arXiv 2025

  47. [47]

    Automating steering for safe multimodal large language models

    Lyucheng Wu, Mengru Wang, Ziwen Xu, Tri Cao, Nay Oo, Bryan Hooi, and Shumin Deng. Automating steering for safe multimodal large language models. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 792–814, 2025

    2025

  48. [48]

    Textual steering vectors can improve visual understanding in multimodal large language models.arXiv preprint arXiv:2505.14071, 2025

    Woody Haosheng Gan, Deqing Fu, Julian Asilis, Ollie Liu, Dani Yogatama, Vatsal Sharan, Robin Jia, and Willie Neiswanger. Textual steering vectors can improve visual understanding in multimodal large language models.arXiv preprint arXiv:2505.14071, 2025

    Pith/arXiv arXiv 2025

  49. [49]

    Steering multimodal large language models decoding for context-aware safety.arXiv preprint arXiv:2509.19212, 2025

    Zheyuan Liu, Zhangchen Xu, Guangyao Dou, Xiangchi Yuan, Zhaoxuan Tan, Radha Pooven- dran, and Meng Jiang. Steering multimodal large language models decoding for context-aware safety.arXiv preprint arXiv:2509.19212, 2025

    arXiv 2025

  50. [50]

    Sce2drivex: A generalized mllm framework for scene-to-drive learning.IEEE Robotics and Automation Letters, 2025

    Rui Zhao, Qirui Yuan, Jinyu Li, Haofeng Hu, Yun Li, Zhenhai Gao, and Fei Gao. Sce2drivex: A generalized mllm framework for scene-to-drive learning.IEEE Robotics and Automation Letters, 2025

    2025

  51. [51]

    Evaluating and steering modality preferences in multimodal large language model

    Yu Zhang, Jinlong Ma, Yongshuai Hou, Xuefeng Bai, Kehai Chen, Yang Xiang, Jun Yu, and Min Zhang. Evaluating and steering modality preferences in multimodal large language model. arXiv preprint arXiv:2505.20977, 2025

    arXiv 2025

  52. [52]

    Steering merged llms for multilingual reasoning with coefficient optimization

    Zhuoran Li, Rui Xu, Xiaojie Wang, Junnan Liu, Zhijun Chen, Qianren Mao, Jian Yang, Hongcheng Guo, Likang Xiao, and MING LI. Steering merged llms for multilingual reasoning with coefficient optimization

  53. [53]

    Qwen3-vl technical report, 2025

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan ...

    2025

  54. [54]

    Wise: A world knowledge-informed semantic evaluation for text-to-image generation.arXiv preprint arXiv:2503.07265, 2025

    Yuwei Niu, Munan Ning, Mengren Zheng, Weiyang Jin, Bin Lin, Peng Jin, Jiaqi Liao, Kunpeng Ning, Chaoran Feng, Bin Zhu, and Li Yuan. Wise: A world knowledge-informed semantic evaluation for text-to-image generation.arXiv preprint arXiv:2503.07265, 2025

    Pith/arXiv arXiv 2025

  55. [55]

    Geneval: An object-focused framework for evaluating text-to-image alignment, 2023

    Dhruba Ghosh, Hanna Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment, 2023

    2023

  56. [56]

    Analyzing the generalization and reliability of steering vectors, 2024

    Daniel Tan, David Chanin, Aengus Lynch, Dimitrios Kanoulas, Brooks Paige, Adria Garriga- Alonso, and Robert Kirk. Analyzing the generalization and reliability of steering vectors, 2024

    2024

  57. [57]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection.arXiv preprint arXiv:2303.05499, 2023

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection.arXiv preprint arXiv:2303.05499, 2023

    Pith/arXiv arXiv 2023

  58. [58]

    positive_question

    Xindi Wu, Dingli Yu, Yangsibo Huang, Olga Russakovsky, and Sanjeev Arora. Conceptmix: A compositional image generation benchmark with controllable difficulty, 2024. 13 Appendix Table of Contents A Experiment Details 15 A.1 Implementation Detail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 A.2 Dataset Detail . . . . . . . . ...

    arXiv 2024