Pith. sign in

REVIEW 24 cited by

AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2402.12226 v5 pith:72JWBN34 submitted 2024-02-19 cs.CL cs.AIcs.CVcs.LG

Jun Zhan , Junqi Dai , Jiasheng Ye , Yunhua Zhou , Dong Zhang , Zhigeng Liu , Xin Zhang , Ruibin Yuan
show 8 more authors
Ge Zhang Linyang Li Hang Yan Jie Fu Tao Gui Tianxiang Sun Yu-Gang Jiang Xipeng Qiu
This is my paper · ORCID
classification cs.CLcs.AIcs.CVcs.LG
keywords multimodalanygptmodalitiesmodelany-to-anydiscretelanguagedataset
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We introduce AnyGPT, an any-to-any multimodal language model that utilizes discrete representations for the unified processing of various modalities, including speech, text, images, and music. AnyGPT can be trained stably without any alterations to the current large language model (LLM) architecture or training paradigms. Instead, it relies exclusively on data-level preprocessing, facilitating the seamless integration of new modalities into LLMs, akin to the incorporation of new languages. We build a multimodal text-centric dataset for multimodal alignment pre-training. Utilizing generative models, we synthesize the first large-scale any-to-any multimodal instruction dataset. It consists of 108k samples of multi-turn conversations that intricately interweave various modalities, thus equipping the model to handle arbitrary combinations of multimodal inputs and outputs. Experimental results demonstrate that AnyGPT is capable of facilitating any-to-any multimodal conversation while achieving performance comparable to specialized models across all modalities, proving that discrete representations can effectively and conveniently unify multiple modalities within a language model. Demos are shown in https://junzhan2000.github.io/AnyGPT.github.io/

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 24 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AV-Odyssey Bench: Can Your Multimodal LLMs Really Understand Audio-Visual Information?

    cs.CV 2024-12 conditional novelty 7.0 of 10

    A new audio-visual benchmark shows current multimodal LLMs perform barely above random guessing, with audio perception errors as the dominant failure mode.

  2. POPS: Recovering Unlearned Multi-Modality Knowledge in MLLMs with Prompt-Optimized Parameter Shaking

    cs.CR 2026-07 conditional novelty 6.0 of 10

    Prompt-optimized suffixes plus synthetic fine-tuning recover ~82% of knowledge that multimodal unlearning methods claim to erase from MLLMs.

  3. Multi-TW: Benchmarking Multimodal Models on Traditional Chinese Question Answering in Taiwan

    cs.AI 2025-08 conditional novelty 6.0 of 10

    Multi-TW is the first Traditional Chinese benchmark to evaluate multimodal models on both image-text and audio-text questions while also measuring inference latency.

  4. ShareGPT-4o-Image: Aligning Multimodal Models with GPT-4o-Level Image Generation

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A 91K GPT-4o-generated image and editing dataset, and a fine-tuned open model Janus-4o, report improved text-to-image scores and new editing ability.

  5. AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMs

    cs.CV 2025-06 reject novelty 6.0 of 10

    A clue-grounded audio-visual counting benchmark over 497 long videos and an RL-trained counting model, whose headline result is undermined by training on the DVD-Counting evaluation benchmark.

  6. Probing the Robustness Properties of Neural Speech Codecs

    eess.AS 2025-05 conditional novelty 6.0 of 10

    DAC is the most noise-robust neural codec at high bitrates, but at 3 kbps EnCodec wins, and measured non-linearity correlates with robustness.

  7. FUDOKI: Discrete Flow-based Unified Understanding and Generation via Kinetic-Optimal Velocities

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A 1.5B unified multimodal model trained with discrete flow matching and metric-induced probability paths matches autoregressive baselines of similar size on generation and understanding benchmarks.

  8. TimeCausality: Evaluating the Causal Ability in Time Dimension for Vision Language Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Introduces a 700-pair benchmark for temporal causal reasoning in VLMs, revealing large open-source vs. closed-source gaps and strong position bias.

  9. Nature Language Model: Deciphering the Language of Nature for Scientific Discovery

    cs.AI 2025-02 conditional novelty 6.0 of 10

    A single sequence-based model, pretrained across molecules, proteins, materials, nucleotides and text, outperforms specialist models on several generation tasks and enables cross-domain design.

  10. Audio Large Language Models Can Be Descriptive Speech Quality Evaluators

    cs.SD 2025-01 conditional novelty 6.0 of 10

    Audio LLMs fine-tuned with token-level distillation against an LLM teacher can predict speech quality scores and generate natural-language descriptions, including A/B comparisons.

  11. CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Models

    cs.CV 2024-12 conditional novelty 6.0 of 10

    CoMT is the first benchmark to ask LVLMs to produce interleaved image and text rationales, and current models perform near random on it.

  12. Olympus: A Universal Task Router for Computer Vision Tasks

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Olympus is a trained MLLM router that delegates 20 vision tasks to specialist models and supports chain-of-action execution of up to five tasks per instruction.

  13. Unleashing In-context Learning of Autoregressive Models for Few-shot Image Manipulation

    cs.CV 2024-12 conditional novelty 6.0 of 10

    An autoregressive model with group self-attention that separates learning from applying achieves state-of-the-art few-shot image manipulation on unseen instructions.

  14. EventGPT: Event Stream Understanding with Multimodal Large Language Models

    cs.CV 2024-12 conditional novelty 6.0 of 10

    EventGPT adapts a LLaVA-style MLLM to event camera streams via three-stage training (image-language, event-language, instruction tuning) and outperforms RGB-based MLLMs on its own benchmark.

  15. SpeechLLM: Unified Speech and Language Model for Enhanced Multi-Task Understanding in Low Resource Settings

    cs.CL 2025-08 conditional novelty 5.0 of 10

    A parameter-efficient adapter bridging Whisper and TinyLlama reports relative improvements in speech recognition, named entity recognition, and sentiment analysis on low-resource benchmarks.

  16. Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model

    cs.AI 2025-06 conditional novelty 5.0 of 10

    Stream-Omni uses CTC-based layer-dimension mapping to align speech with text, achieving vision, speech, and text interaction in one 8B model trained on 23,000 hours of speech.

  17. Uncertainty-o: One Model-agnostic Framework for Unveiling Uncertainty in Large Multimodal Models

    cs.CV 2025-06 conditional novelty 5.0 of 10

    Uncertainty-o estimates uncertainty in large multimodal models by perturbing prompts and computing entropy over semantically clustered answers, improving hallucination detection across five modalities.

  18. ALAS: An Automatic Latent Alignment Score for Audio Language Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    ALAS is a reference-based score for audio-text alignment in speech LLMs, computed from frozen hidden states and a Whisper-derived alignment path, with no training or fitted classifier.

  19. LiSTEN: Learning Soft Token Embeddings for Neural Audio LLMs

    cs.AI 2025-05 conditional novelty 5.0 of 10

    LiSTEN shows that dynamically selecting a few learnable prompt tokens from a shared pool can replace LoRA fine-tuning for audio-language models, matching or beating it with less training data.

  20. EVEv2: Improved Baselines for Encoder-Free Vision-Language Models

    cs.CV 2025-02 conditional novelty 5.0 of 10

    An encoder-free vision-language model using separate attention, normalization, and feed-forward weights for image versus text tokens outperforms earlier encoder-free models and narrows the gap to encoder-based VLMs wi...

  21. Causal Diffusion Transformers for Generative Modeling

    cs.CV 2024-12 conditional novelty 5.0 of 10

    A decoder-only transformer that factors generation over both token order and noise level, coupling autoregressive and diffusion training, achieves competitive ImageNet generation and in-context editing.

  22. ILLUME: Illuminating Your LLMs to See, Draw, and Self-Enhance

    cs.CV 2024-12 conditional novelty 5.0 of 10

    ILLUME unifies visual understanding and generation in one LLM with a semantic vision tokenizer and a self-enhancing alignment scheme, reaching competitive benchmarks with only 15M pretraining pairs.

  23. Panther: Illuminate the Sight of Multimodal LLMs with Instruction-Guided Visual Prompts

    cs.CV 2024-11 conditional novelty 5.0 of 10

    Panther improves multimodal LLMs by converting the user's text question into visual prompts that steer a frozen image encoder toward instruction-relevant regions, gaining about 2 to 3 points on several VQA benchmarks ...

  24. Pisces: An Auto-regressive Foundation Model for Image Understanding and Generation

    cs.CV 2025-06 conditional novelty 4.0 of 10

    A unified image understanding and generation model with decoupled visual encoders achieves competitive benchmark scores on both tasks.

Pith tools