Pith. sign in

REVIEW 11 cited by

Orthus: Autoregressive Interleaved Image-Text Generation with Modality-Specific Heads

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2412.00127 v2 pith:MRSCII7A submitted 2024-11-28 cs.CV cs.AIcs.CL

Siqi Kou , Jiachun Jin , Zhihong Liu , Chang Liu , Ye Ma , Jian Jia , Quan Chen , Peng Jiang
show 1 more author
Zhijie Deng
This is my paper
classification cs.CVcs.AIcs.CL
keywords orthusgenerationcontinuousheadimageimagesinterleavedmodel
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We introduce Orthus, an autoregressive (AR) transformer that excels in generating images given textual prompts, answering questions based on visual inputs, and even crafting lengthy image-text interleaved contents. Unlike prior arts on unified multimodal modeling, Orthus simultaneously copes with discrete text tokens and continuous image features under the AR modeling principle. The continuous treatment of visual signals minimizes the information loss for both image understanding and generation while the fully AR formulation renders the characterization of the correlation between modalities straightforward. The key mechanism enabling Orthus to leverage these advantages lies in its modality-specific heads -- one regular language modeling (LM) head predicts discrete text tokens and one diffusion head generates continuous image features conditioning on the output of the backbone. We devise an efficient strategy for building Orthus -- by substituting the Vector Quantization (VQ) operation in the existing unified AR model with a soft alternative, introducing a diffusion head, and tuning the added modules to reconstruct images, we can create an Orthus-base model effortlessly (e.g., within mere 72 A100 GPU hours). Orthus-base can further embrace post-training to better model interleaved images and texts. Empirically, Orthus surpasses competing baselines including Show-o and Chameleon across standard benchmarks, achieving a GenEval score of 0.58 and an MME-P score of 1265.8 using 7B parameters. Orthus also shows exceptional mixed-modality generation capabilities, reflecting the potential for handling intricate practical generation tasks.

Discussion (0). Sign in to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. EduIllustrate: Towards Scalable Automated Generation Of Multimodal Educational Content

    cs.CY 2026-04 unverdicted novelty 7.0 of 10

    EduIllustrate is a benchmark of 230 K-12 STEM problems that evaluates LLMs on interleaved text-diagram generation using sequential anchoring and an 8-dimension rubric, with Gemini 3.0 Pro Preview scoring highest at 87.8%.

  2. WISE: A World Knowledge-Informed Semantic Evaluation for Text-to-Image Generation

    cs.CV 2025-03 unverdicted novelty 7.0 of 10

    Text-to-image models show significant limitations in integrating world knowledge, as measured by the new WISE benchmark and WiScore metric across 20 models.

  3. Twins: Learn to Predict Unified Representations with Focal Loss

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Channel-wise concatenation of SigLIP2 and Flux VAE features into one token, trained with a focal-style flow-matching loss, yields a unified representation with 1.59 gFID on ImageNet 256 and VAE-level reconstruction.

  4. HYDRA-X: Native Unified Multimodal Models with Holistic Visual Tokenizers

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    HYDRA-X presents the first unified multimodal model using a single ViT for holistic image-video tokenization, with ablations on attention and compression plus a latent-level editing improvement.

  5. ProductWebGen: Benchmarking Multimodal Product Webpage Generation

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    Introduces ProductWebGen benchmark for multimodal product webpage generation, compares editing-based vs unified-model workflows on 500 samples, and releases ProductWebGen-1k SFT dataset.

  6. MAR-GRPO: Stabilized GRPO for AR-diffusion Hybrid Image Generation

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    MAR-GRPO stabilizes GRPO for AR-diffusion hybrids via multi-trajectory expectation and uncertainty-based token selection, yielding better visual quality, stability, and spatial understanding than baselines.

  7. X-Omni: Reinforcement Learning Makes Discrete Autoregressive Image Generative Models Great Again

    cs.CV 2025-07 conditional novelty 6.0 of 10

    GRPO reinforcement learning applied to a discrete autoregressive image generator with a diffusion decoder improves instruction following, image quality, and long-text rendering in a unified multimodal model.

  8. LLaDA-V: Large Language Diffusion Models with Visual Instruction Tuning

    cs.LG 2025-05 conditional novelty 6.0 of 10

    LLaDA-V is a diffusion-based multimodal large language model that reaches competitive or state-of-the-art results on visual instruction tasks while using a non-autoregressive architecture.

  9. LongCat-Image Technical Report

    cs.CV 2025-12 unverdicted novelty 5.0 of 10

    LongCat-Image delivers a compact 6B-parameter bilingual image generation model that sets new standards for Chinese character rendering accuracy and photorealism while remaining efficient and fully open-source.

  10. Show-o2: Improved Native Unified Multimodal Models

    cs.CV 2025-06 unverdicted novelty 4.0 of 10

    Show-o2 unifies text, image, and video understanding and generation in a single autoregressive-plus-flow-matching model built on 3D causal VAE representations.

  11. A Survey on Diffusion Language Models

    cs.CL 2025-08 unverdicted novelty 3.0 of 10

    A comprehensive survey of diffusion language models covering taxonomy, training and inference techniques, and comparisons with autoregressive models.

Pith tools