pith. sign in

arxiv: 2605.18678 · v1 · pith:WCWSXXVLnew · submitted 2026-05-18 · 💻 cs.CV · cs.AI

Lance: Unified Multimodal Modeling by Multi-Task Synergy

Fengyi Fu , Mengqi Huang , Shaojin Wu , Yunsheng Jiang , Yufei Huo , Hao Li , Yinghang Song , Fei Ding
show 5 more authors
Jianzhu Guo Qian He Zheren Fu Zhendong Mao Yongdong Zhang
This is my paper

Pith reviewed 2026-05-20 11:42 UTC · model grok-4.3

classification 💻 cs.CV cs.AI
keywords unified multimodal modelingmulti-task learningimage video generationmixture of expertsmultimodal understandingrotary positional encoding
0
0 comments X

The pith

Lance establishes that multi-task training on a dual-stream mixture-of-experts architecture enables a unified model to excel at both multimodal understanding and image-video generation.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents Lance as a lightweight model that supports understanding, generating, and editing images and videos all within one system. Instead of scaling model size or favoring one modality, it uses collaborative training across tasks to build capabilities together. The design relies on shared sequences for context but keeps understanding and generation on separate pathways to avoid conflicts. This matters because it provides a practical way to develop versatile multimodal AI that balances seeing and creating without one diminishing the other.

Core claim

Lance establishes that training from scratch with a dual-stream mixture-of-experts architecture on shared interleaved multimodal sequences, modality-aware rotary positional encoding, and a staged multi-task training paradigm with capability-oriented objectives allows for joint context learning while decoupling understanding and generation pathways, leading to superior performance in image and video generation alongside strong understanding.

What carries the argument

dual-stream mixture-of-experts architecture on shared interleaved multimodal sequences combined with modality-aware rotary positional encoding

If this is right

  • Unified models can achieve better generation quality than existing open-source ones without losing understanding abilities.
  • Multi-task synergy enables effective learning across understanding and generation tasks.
  • Staged training with adaptive data scheduling strengthens both semantic comprehension and visual generation.
  • Modality-aware positional encoding mitigates interference among different types of visual tokens.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If this approach generalizes, similar dual-pathway designs could apply to other unified models involving additional modalities like audio or 3D.
  • Future work might test whether removing the dual-stream leads to measurable interference between tasks.
  • The success suggests that focusing on training paradigms rather than architecture scale could be key for efficient multimodal systems.

Load-bearing premise

The dual-stream mixture-of-experts on shared sequences successfully decouples understanding and generation without harmful interference between them.

What would settle it

Demonstrating that a comparable single-stream model achieves similar or better results in both generation and understanding would challenge the necessity of the dual-stream design.

read the original abstract

We present Lance, a lightweight native unified model supporting multimodal understanding, generation, and editing for both images and videos. Rather than relying on model capacity scaling or text-image-dominant designs, Lance explores a practical paradigm for unified multimodal modeling via collaborative multi-task training. It is grounded in two core principles: unified context modeling and decoupled capability pathways. Specifically, Lance is trained from scratch and employs a dual-stream mixture-of-experts architecture on shared interleaved multimodal sequences, enabling joint context learning while decoupling the pathways for understanding and generation. We further introduce modality-aware rotary positional encoding to mitigate interference among heterogeneous visual tokens and boost cross-task alignment. During training, Lance adopts a staged multi-task training paradigm with capability-oriented objectives and adaptive data scheduling to strengthen both semantic comprehension and visual generation performance. Experimental results demonstrate that Lance substantially outperforms existing open-source unified models in image and video generation, while retaining strong multimodal understanding capabilities. The homepage is available at https://lance-project.github.io.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper presents Lance, a lightweight native unified multimodal model for image/video understanding, generation, and editing. It is trained from scratch using a dual-stream mixture-of-experts architecture on shared interleaved multimodal sequences, modality-aware rotary positional encoding, and a staged multi-task training paradigm with capability-oriented objectives and adaptive data scheduling. The central claim is that this design enables joint context learning while decoupling understanding and generation pathways without harmful interference, yielding substantial gains over existing open-source unified models in generation tasks while preserving strong multimodal understanding.

Significance. If the performance claims are supported by rigorous ablations and the decoupling mechanism is validated, the work could advance practical unified multimodal modeling by demonstrating that multi-task synergy on interleaved sequences can outperform capacity-scaling approaches without task interference. The focus on lightweight design and explicit pathway decoupling is a potentially useful contribution to the field.

major comments (2)
  1. [Experimental Results] The manuscript reports final benchmark numbers but supplies no controlled comparison (single-stream vs. dual-stream, with vs. without modality-aware RoPE) and no quantitative interference diagnostic (e.g., understanding accuracy when generation loss weight is varied, or gradient-conflict statistics between heads). Without these, the observed gains could be explained by extra capacity, data schedule, or longer training rather than the claimed architectural decoupling. This directly affects the load-bearing assumption in the abstract and methodology.
  2. [Abstract and Results] The abstract and results sections state that Lance 'substantially outperforms' existing models but provide no quantitative metrics, specific baselines, dataset details, or ablation tables in the summary of findings. This makes it impossible to verify whether the data actually supports the central claim of effective joint context learning with decoupled pathways.
minor comments (1)
  1. [Abstract] The abstract would benefit from including at least one key quantitative result (e.g., FID or accuracy delta) to allow readers to assess the magnitude of the claimed improvements without reading the full experimental section.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. The comments highlight important areas for strengthening the experimental validation of our architectural choices and improving the clarity of our performance claims. We address each point below and will incorporate the suggested changes in the revised manuscript.

read point-by-point responses

  1. Referee: The manuscript reports final benchmark numbers but supplies no controlled comparison (single-stream vs. dual-stream, with vs. without modality-aware RoPE) and no quantitative interference diagnostic (e.g., understanding accuracy when generation loss weight is varied, or gradient-conflict statistics between heads). Without these, the observed gains could be explained by extra capacity, data schedule, or longer training rather than the claimed architectural decoupling. This directly affects the load-bearing assumption in the abstract and methodology.

    Authors: We agree that rigorous controlled ablations are necessary to isolate the contributions of the dual-stream MoE design and modality-aware RoPE from potential confounding factors such as capacity or training schedule. In the revised manuscript we will add a new ablation subsection that directly compares single-stream versus dual-stream variants and models with versus without modality-aware rotary positional encoding, using matched training budgets. We will also report quantitative interference diagnostics, including understanding-task accuracy as a function of generation loss weight and gradient-conflict metrics between the understanding and generation pathways. These additions will provide direct evidence for the claimed decoupling mechanism. revision: yes

  2. Referee: The abstract and results sections state that Lance 'substantially outperforms' existing models but provide no quantitative metrics, specific baselines, dataset details, or ablation tables in the summary of findings. This makes it impossible to verify whether the data actually supports the central claim of effective joint context learning with decoupled pathways.

    Authors: We acknowledge that the abstract would be more informative with explicit quantitative anchors. In the revision we will update the abstract to cite concrete metrics (e.g., FID and CLIP-score improvements on image and video generation benchmarks relative to the strongest open-source unified baselines) while preserving the statement on retained understanding performance. The results section will be expanded to explicitly name the baselines, datasets, and evaluation protocols, and will cross-reference the new ablation tables that validate the joint-context and decoupling claims. revision: yes

Circularity Check

0 steps flagged

No circularity in derivation chain; claims rest on empirical training results

full rationale

The manuscript describes an empirical architecture (dual-stream MoE on interleaved sequences plus modality-aware RoPE) trained from scratch with staged multi-task objectives and reports benchmark outcomes. No equations, uniqueness theorems, or self-citations are invoked to derive performance claims; the results are presented as measured experimental outcomes rather than quantities forced by construction from fitted parameters or prior self-referential premises. The central claims therefore remain independent of the listed circularity patterns.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

Review is limited to the abstract; therefore the ledger records only the high-level design assumptions explicitly stated there. No free parameters or quantitative details are extractable.

axioms (1)
  • domain assumption Unified context modeling and decoupled capability pathways can be realized via a dual-stream mixture-of-experts architecture on shared interleaved multimodal sequences.
    Stated as one of the two core principles grounding the model design.
invented entities (1)
  • modality-aware rotary positional encoding no independent evidence
    purpose: Mitigate interference among heterogeneous visual tokens and boost cross-task alignment.
    Introduced as a specific component to handle mixed visual tokens.

pith-pipeline@v0.9.0 · 5734 in / 1325 out tokens · 49123 ms · 2026-05-20T11:42:45.420648+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

150 extracted references · 150 canonical work pages · 58 internal anchors

  1. [1]

    GPT-4 Technical Report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advancesin neural information processing systems, 35:23716–23736, 2022

    work page 2022

  3. [3]

    Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  4. [4]

    Qwen3-VL Technical Report

    Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, Mei Li, Kaixin Li, Zicheng Lin, Junyang Lin, Xuejing Liu, Jiawei Liu, Chenglong Liu, Yang Liu, Dayiheng Liu, Shixuan ...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  5. [5]

    Qwen2.5-VL Technical Report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report.ar...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  6. [6]

    Improving image generation with better captions.Computer Science

    James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions.Computer Science. https://cdn. openai. com/papers/dall-e-3. pdf, 2(3):8, 2023

    work page 2023

  7. [7]

    Diffusion self-distillation for zero-shot customized image generation.arXiv preprint arXiv:2411.18616, 2024

    Shengqu Cai, Eric Chan, Yunzhi Zhang, Leonidas Guibas, Jiajun Wu, and Gordon Wetzstein. Diffusion self-distillation for zero-shot customized image generation.arXiv preprint arXiv:2411.18616, 2024

    work page arXiv 2024

  8. [8]

    HunyuanImage 3.0 Technical Report

    Siyu Cao, Hangting Chen, Peng Chen, Yiji Cheng, Yutao Cui, Xinchi Deng, Ying Dong, Kipper Gong, Tianpeng Gu, Xiusen Gu, et al. Hunyuanimage 3.0 technical report.arXiv preprint arXiv:2509.23951, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  9. [9]

    Maskgit: Masked generative image transformer

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11315–11325, 2022

    work page 2022

  10. [10]

    Videocrafter2: Overcoming data limitations for high-quality video diffusion models

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7310–7320, 2024

    work page 2024

  11. [11]

    BLIP3-o: A Family of Fully Open Unified Multimodal Models-Architecture, Training and Dataset

    Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie, Silvio Savarese, et al. Blip3-o: A family of fully open unified multimodal models-architecture, training and dataset. arXiv preprint arXiv:2505.09568, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  12. [12]

    Pixart-σ: Weak-to-strong training of diffusion transformer for 4k text-to-image generation

    Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-σ: Weak-to-strong training of diffusion transformer for 4k text-to-image generation. In European Conference on Computer Vision, pages 74–91. Springer, 2024

    work page 2024

  13. [13]

    Timemarker: A versatile video-llm for long and short video understanding with superior temporal localization ability.arXiv preprint arXiv:2411.18211, 2024

    Shimin Chen, Xiaohan Lan, Yitian Yuan, Zequn Jie, and Lin Ma. Timemarker: A versatile video-llm for long and short video understanding with superior temporal localization ability.arXiv preprint arXiv:2411.18211, 2024

    work page arXiv 2024

  14. [14]

    Janus-Pro: Unified Multimodal Understanding and Generation with Data and Model Scaling

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling.arXiv preprint arXiv:2501.17811, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  15. [15]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 67(12):220101, 2024. 26

    work page 2024

  16. [16]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185–24198, 2024

    work page 2024

  17. [17]

    Umo: Scaling multi-identity consistency for image customization via matching reward.arXiv preprint arXiv:2509.06818, 2025

    Yufeng Cheng, Wenxu Wu, Shaojin Wu, Mengqi Huang, Fei Ding, and Qian He. Umo: Scaling multi-identity consistency for image customization via matching reward.arXiv preprint arXiv:2509.06818, 2025

    work page arXiv 2025

  18. [18]

    PaddleOCR 3.0 Technical Report

    Cheng Cui, Ting Sun, Manhui Lin, Tingquan Gao, Yubo Zhang, Jiaxuan Liu, Xueqing Wang, Zelun Zhang, Changda Zhou, Hongen Liu, et al. Paddleocr 3.0 technical report.arXiv preprint arXiv:2507.05595, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  19. [19]

    Yufeng Cui, Honghao Chen, Haoge Deng, Xu Huang, Xinghang Li, Jirong Liu, Yang Liu, Zhuoyan Luo, Jinsheng Wang, Wenxuan Wang, et al. Emu3. 5: Native multimodal models are world learners.arXiv preprint arXiv:2510.26583, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  20. [20]

    Instructblip: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advancesin neural information processing systems, 36:49250–49267, 2023

    work page 2023

  21. [21]

    Chatumm: Robust context tracking for conversational interleaved generation

    Wenxun Dai, Zhiyuan Zhao, Yule Zhong, Yiji Cheng, Jianwei Zhang, Linqing Wang, Shiyi Zhang, Yunlong Lin, Runze He, Fellix Song, et al. Chatumm: Robust context tracking for conversational interleaved generation. arXiv preprint arXiv:2602.06442, 2026

    work page arXiv 2026

  22. [22]

    Emerging Properties in Unified Multimodal Pretraining

    Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, et al. Emerging properties in unified multimodal pretraining.arXiv preprint arXiv:2505.14683, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  23. [23]

    Cogview: Mastering text-to-image generation via transformers.NIPS, 34:19822–19835, 2021

    Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers.NIPS, 34:19822–19835, 2021

    work page 2021

  24. [24]

    Taming transformers for high-resolution image synthesis

    Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873–12883, 2021

    work page 2021

  25. [25]

    Scaling rectified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In ICML, 2024

    work page 2024

  26. [26]

    Unified autoregressive visual generation and understanding with continuous tokens

    Lijie Fan, Luming Tang, Siyang Qin, Tianhong Li, Xuan Yang, Siyuan Qiao, Andreas Steiner, Chen Sun, Yuanzhen Li, Tao Zhu, et al. Unified autoregressive visual generation and understanding with continuous tokens. arXiv preprint arXiv:2503.13436, 2025

    work page arXiv 2025

  27. [27]

    Video-ccam: Enhancingvideo-language understanding with causal cross-attention masks for short and long videos.arXiv preprint arXiv:2408.14023, 2024

    JiajunFei, DianLi, ZhidongDeng, ZekunWang, GangLiu, andHuiWang. Video-ccam: Enhancingvideo-language understanding with causal cross-attention masks for short and long videos.arXiv preprint arXiv:2408.14023, 2024

    work page arXiv 2024

  28. [28]

    Dreamlite: A lightweight on-device unified model for image generation and editing.arXiv preprint arXiv:2603.28713, 2026

    Kailai Feng, Yuxiang Wei, Bo Chen, Yang Pan, Hu Ye, Songwei Liu, Chenqian Yan, and Yuan Gao. Dreamlite: A lightweight on-device unified model for image generation and editing.arXiv preprint arXiv:2603.28713, 2026

    work page arXiv 2026

  29. [29]

    Feededit: Text-based image editing with dynamic feedback regulation

    Fengyi Fu, Lei Zhang, Mengqi Huang, and Zhendong Mao. Feededit: Text-based image editing with dynamic feedback regulation. InProceedings of the Computer Vision and Pattern Recognition Conference, pages 2661– 2670, 2025

    work page 2025

  30. [30]

    Layeredit: Disentangled multi-object editing via conflict-aware multi-layer learning

    Fengyi Fu, Mengqi Huang, Lei Zhang, and Zhendong Mao. Layeredit: Disentangled multi-object editing via conflict-aware multi-layer learning. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 4003–4011, 2026

    work page 2026

  31. [31]

    Mini-internvl: a flexible-transfer pocket multi-modal model with 5% parameters and 90% performance

    Zhangwei Gao, Zhe Chen, Erfei Cui, Yiming Ren, Weiyun Wang, Jinguo Zhu, Hao Tian, Shenglong Ye, Junjun He, Xizhou Zhu, et al. Mini-internvl: a flexible-transfer pocket multi-modal model with 5% parameters and 90% performance. Visual Intelligence, 2(1):1–17, 2024

    work page 2024

  32. [32]

    SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation

    Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granularity comprehension and generation. arXiv preprint arXiv:2404.14396, 2024. 27

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  33. [33]

    Geneval: An object-focused framework for evaluating text-to-image alignment.Advancesin Neural Information Processing Systems, 36:52132–52152, 2023

    Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment.Advancesin Neural Information Processing Systems, 36:52132–52152, 2023

    work page 2023

  34. [34]

    Gemini 3 Pro Image Model Card

    Google DeepMind. Gemini 3 Pro Image Model Card. https://storage.googleapis.com/deepmind-media/ Model-Cards/Gemini-3-Pro-Image-Model-Card.pdf, November 2025. Model card published: November 2025

    work page 2025

  35. [35]

    AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Maneesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  36. [36]

    Tv2tv: A unified framework for interleaved language and video generation

    Xiaochuang Han, Youssef Emad, Melissa Hall, John Nguyen, Karthik Padthe, Liam Robbins, Amir Bar, Delong Chen, Michal Drozdzal, Maha Elbayad, et al. Tv2tv: A unified framework for interleaved language and video generation. arXiv preprint arXiv:2512.05103, 2025

    work page arXiv 2025

  37. [37]

    Emma: Efficient multimodal understanding, generation, and editing with a unified architecture.arXiv preprint arXiv:2512.04810, 2025

    Xin He, Longhui Wei, Jianbo Ouyang, Minghui Liao, Lingxi Xie, and Qi Tian. Emma: Efficient multimodal understanding, generation, and editing with a unified architecture.arXiv preprint arXiv:2512.04810, 2025

    work page arXiv 2025

  38. [38]

    Classifier-Free Diffusion Guidance

    Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance.arXiv preprint arXiv:2207.12598, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  39. [39]

    Denoising diffusion probabilistic models.NIPS, 33:6840–6851, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.NIPS, 33:6840–6851, 2020

    work page 2020

  40. [40]

    CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers.arXiv preprint arXiv:2205.15868, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  41. [41]

    ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment

    Xiwei Hu, Rui Wang, Yixiao Fang, Bin Fu, Pei Cheng, and Gang Yu. Ella: Equip diffusion models with llm for enhanced semantic alignment.arXiv preprint arXiv:2403.05135, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  42. [42]

    Dse-gan: Dynamic semantic evolution generative adversarial network for text-to-image generation

    Mengqi Huang, Zhendong Mao, Penghui Wang, Quan Wang, and Yongdong Zhang. Dse-gan: Dynamic semantic evolution generative adversarial network for text-to-image generation. InProceedings of the 30th ACM International Conference on Multimedia, pages 4345–4354, 2022

    work page 2022

  43. [43]

    Towards accurate image coding: Improved autoregressive image generation with dynamic vector quantization

    Mengqi Huang, Zhendong Mao, Zhuowei Chen, and Yongdong Zhang. Towards accurate image coding: Improved autoregressive image generation with dynamic vector quantization. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22596–22605, 2023

    work page 2023

  44. [44]

    Realcustom: Narrowing real text word for real-time open-domain text-to-image customization

    Mengqi Huang, Zhendong Mao, Mingcong Liu, Qian He, and Yongdong Zhang. Realcustom: Narrowing real text word for real-time open-domain text-to-image customization. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7476–7485, 2024

    work page 2024

  45. [45]

    Self forcing: Bridging the train-test gap in autoregressive video diffusion.Advances in Neural Information Processing Systems, 38:167283–167308, 2026

    Xun Huang, Zhengqi Li, Guande He, Mingyuan Zhou, and Eli Shechtman. Self forcing: Bridging the train-test gap in autoregressive video diffusion.Advances in Neural Information Processing Systems, 38:167283–167308, 2026

    work page 2026

  46. [46]

    Vbench: Comprehensive benchmark suite for video generative models

    Ziqi Huang, Yinan He, Jiashuo Yu, Fan Zhang, Chenyang Si, Yuming Jiang, Yuanhan Zhang, Tianxing Wu, Qingyang Jin, Nattapol Chanpaisit, et al. Vbench: Comprehensive benchmark suite for video generative models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21807–21818, 2024

    work page 2024

  47. [47]

    Vace: All-in-one video creation and editing

    Zeyinzi Jiang, Zhen Han, Chaojie Mao, Jingfeng Zhang, Yulin Pan, and Yu Liu. Vace: All-in-one video creation and editing. InProceedings of the IEEE/CVF InternationalConference on Computer Vision, pages 17191–17202, 2025

    work page 2025

  48. [48]

    EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning

    Xuan Ju, Tianyu Wang, Yuqian Zhou, He Zhang, Qing Liu, Nanxuan Zhao, Zhifei Zhang, Yijun Li, Yuanhao Cai, Shaoteng Liu, et al. Editverse: Unifying image and video editing and generation with in-context learning. arXiv preprint arXiv:2509.20360, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  49. [49]

    Fulldit: Multi-task video generative foundation model with full attention.arXiv preprint arXiv:2503.19907, 2025

    Xuan Ju, Weicai Ye, Quande Liu, Qiulin Wang, Xintao Wang, Pengfei Wan, Di Zhang, Kun Gai, and Qiang Xu. Fulldit: Multi-task video generative foundation model with full attention.arXiv preprint arXiv:2503.19907, 2025

    work page arXiv 2025

  50. [50]

    Kling ai.https://klingai.kuaishou.com/, 2024

    Kling AI. Kling ai.https://klingai.kuaishou.com/, 2024. Accessed: 2024-06-06

    work page 2024

  51. [51]

    VideoPoet: A Large Language Model for Zero-Shot Video Generation

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming-Chang Chiu, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.14125, 2023. 28

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  52. [52]

    Anyv2v: A tuning-free framework for any video-to-video editing tasks.arXiv preprint arXiv:2403.14468, 2024

    Max Ku, Cong Wei, Weiming Ren, Harry Yang, and Wenhu Chen. Anyv2v: A tuning-free framework for any video-to-video editing tasks.arXiv preprint arXiv:2403.14468, 2024

    work page arXiv 2024

  53. [53]

    Flux: Official inference repository for flux.1 models, 2024

    Black Forest Labs. Flux: Official inference repository for flux.1 models, 2024. URLhttps://github.com/ black-forest-labs/flux. Accessed: 2025-02-07

    work page 2024

  54. [54]

    Black Forest Labs, Stephen Batifol, Andreas Blattmann, Frederic Boesel, Saksham Consul, Cyril Diagne, Tim Dockhorn, Jack English, Zion English, Patrick Esser, et al. Flux. 1 kontext: Flow matching for in-context image generation and editing in latent space.arXiv preprint arXiv:2506.15742, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  55. [55]

    Obelics: An open web-scale filtered dataset of interleaved image-text documents.Advancesin Neural Information Processing Systems, 36:71683–71702, 2023

    Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents.Advancesin Neural Information Processing Systems, 36:71683–71702, 2023

    work page 2023

  56. [56]

    LLaVA-OneVision: Easy Visual Task Transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  57. [57]

    Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing

    Dongxu Li, Junnan Li, and Steven Hoi. Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing. Advances in Neural Information Processing Systems, 36:30146–30166, 2023

    work page 2023

  58. [58]

    Onecat: Decoder-only auto-regressive model for unified understanding and generation

    Han Li, Xinyu Peng, Yaoming Wang, Zelin Peng, Xin Chen, Rongxiang Weng, Jingang Wang, Xunliang Cai, Wenrui Dai, and Hongkai Xiong. Onecat: Decoder-only auto-regressive model for unified understanding and generation. arXiv preprint arXiv:2509.03498, 2025

    work page arXiv 2025

  59. [59]

    Mvbench: A comprehensive multi-modal video understanding benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–22206, 2024

    work page 2024

  60. [60]

    Videochat: Chat-centric video understanding.Science China Information Sciences, 68(10):200102, 2025

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding.Science China Information Sciences, 68(10):200102, 2025

    work page 2025

  61. [61]

    Autoregressive image generation without vector quantization.Advancesin Neural Information Processing Systems, 37:56424–56445, 2024

    Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization.Advancesin Neural Information Processing Systems, 37:56424–56445, 2024

    work page 2024

  62. [62]

    Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding

    Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-dit: A powerful multi-resolution diffusion transformer with fine-grained chinese understanding.arXiv preprint arXiv:2405.08748, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  63. [63]

    Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generation

    Chao Liao, Liyang Liu, Xun Wang, Zhengxiong Luo, Xinyu Zhang, Wenliang Zhao, Jie Wu, Liang Li, Zhi Tian, and Weilin Huang. Mogao: An omni foundation model for interleaved multi-modal generation.arXiv preprint arXiv:2505.05472, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  64. [64]

    Video-llava: Learning united visual representation by alignment before projection

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. InProceedings of the 2024 conference on empirical methods in natural language processing, pages 5971–5984, 2024

    work page 2024

  65. [65]

    UniWorld-V1: High-Resolution Semantic Encoders for Unified Visual Understanding and Generation

    Bin Lin, Zongjian Li, Xinhua Cheng, Yuwei Niu, Yang Ye, Xianyi He, Shenghai Yuan, Wangbo Yu, Shaodong Wang, Yunyang Ge, et al. Uniworld: High-resolution semantic encoders for unified visual understanding and generation. arXiv preprint arXiv:2506.03147, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  66. [66]

    Realgeneral: Unifying visual generation via temporal in-context learning with video models

    Yijing Lin, Mengqi Huang, Shuhan Zhuang, and Zhendong Mao. Realgeneral: Unifying visual generation via temporal in-context learning with video models. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 14994–15004, 2025

    work page 2025

  67. [67]

    Flow Matching Guide and Code

    Yaron Lipman, Marton Havasi, Peter Holderrieth, Neta Shaul, Matt Le, Brian Karrer, Ricky TQ Chen, David Lopez-Paz, Heli Ben-Hamu, and Itai Gat. Flow matching guide and code.arXiv preprint arXiv:2412.06264, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  68. [68]

    DeepSeek-V3 Technical Report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  69. [69]

    World Model on Million-Length Video And Language With Blockwise RingAttention

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with blockwise ringattention.arXiv preprint arXiv:2402.08268, 2024. 29

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  70. [70]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    work page 2023

  71. [71]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 26296–26306, 2024

    work page 2024

  72. [72]

    Llavanext: Improved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llavanext: Improved reasoning, ocr, and world knowledge, 2024

    work page 2024

  73. [73]

    Mardini: Masked autoregressive diffusion for video generation at scale,

    Haozhe Liu, Shikun Liu, Zijian Zhou, Mengmeng Xu, Yanping Xie, Xiao Han, Juan C Pérez, Ding Liu, Kumara Kahatapitiya, Menglin Jia, et al. Mardini: Masked autoregressive diffusion for video generation at scale.arXiv preprint arXiv:2410.20280, 2024

    work page arXiv 2024

  74. [74]

    Flow-grpo: Training flow matching models via online rl.Advances in neural information processing systems, 38:40783–40818, 2026

    Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl.Advances in neural information processing systems, 38:40783–40818, 2026

    work page 2026

  75. [75]

    St-llm: Large language models are effective temporal learners

    Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-llm: Large language models are effective temporal learners. InEuropean Conference on Computer Vision, pages 1–18. Springer, 2024

    work page 2024

  76. [76]

    Step1X-Edit: A Practical Framework for General Image Editing

    Shiyu Liu, Yucheng Han, Peng Xing, Fukun Yin, Rui Wang, Wei Cheng, Jiaqi Liao, Yingming Wang, Honghao Fu, ChunruiHan, etal. Step1x-edit: Apracticalframeworkforgeneralimageediting. arXivpreprintarXiv:2504.17761, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  77. [77]

    Tuna: Taming unified visual representations for native unified multimodal models

    Zhiheng Liu, Weiming Ren, Haozhe Liu, Zijian Zhou, Shoufa Chen, Haonan Qiu, Xiaoke Huang, Zhaochong An, Fanny Yang, Aditya Patel, et al. Tuna: Taming unified visual representations for native unified multimodal models. arXiv preprint arXiv:2512.02014, 2025

    work page arXiv 2025

  78. [78]

    Tuna-2: Pixel Embeddings Beat Vision Encoders for Multimodal Understanding and Generation

    Zhiheng Liu, Weiming Ren, Xiaoke Huang, Shoufa Chen, Tianhong Li, Mengzhao Chen, Yatai Ji, Sen He, Jonas Schult, Belinda Zeng, Tao Xiang, Wenhu Chen, Ping Luo, Luke Zettlemoyer, and Yuren Cong. Tuna-2: Pixel embeddings beat vision encoders for multimodal understanding and generation.arXiv preprint arXiv:2604.24763, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  79. [79]

    Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model

    Guoqing Ma, Haoyang Huang, Kun Yan, Liangyu Chen, Nan Duan, Shengming Yin, Changyi Wan, Ranchen Ming, Xiaoniu Song, Xing Chen, et al. Step-video-t2v technical report: The practice, challenges, and future of video foundation model.arXiv preprint arXiv:2502.10248, 2025

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  80. [80]

    Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation

    Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Xingkai Yu, et al. Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7739–7751, 2025

    work page 2025

Showing first 80 references.