Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Olympus: A Universal Task Router for Computer Vision Tasks

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Olympus claims a 3B MLLM, trained with task-specific routing tokens, can route 20 vision tasks with 94.75% accuracy and chain actions at 91.82% precision.

desk verdict A competent and useful router paper whose headline numbers are in-distribution scores on a GPT-4o-generated benchmark, not evidence of universal routing. read the letter →

arxiv 2412.09612 v3 pith:QKH42KTR submitted 2024-12-12 cs.CV cs.AIcs.CL

Yuanze Lin , Yunsheng Li , Dongdong Chen , Weijian Xu , Ronald Clark , Philip H. S. Torr This is my paper · ORCID
classification cs.CVcs.AIcs.CL
keywords multimodallargelanguagemodelstaskroutingcomputervisionchain-of-actiontokensinstructiontuningOlympusBenchspecialist
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Olympus claims that a small multimodal language model can be turned into a universal task router for computer vision by learning to emit task-specific routing tokens. On a single instruction it answers understanding questions itself and delegates generation, restoration, estimation, and 3D tasks to over 20 specialist modules; on a multi-part instruction it chains up to five such delegations. The paper reports 94.75% average routing accuracy on single tasks and 91.82% precision in chained actions on OlympusBench, beating HuggingGPT running on GPT-4o. These capabilities come at the cost of fine-tuning only a 3B model, with standard multimodal benchmark scores staying roughly at the level of the base model. If the result holds, it offers a modular path to broad vision-task coverage without training a heavy all-in-one generative model.

What carries the argument

The load-bearing mechanism is a set of task-specific routing tokens, markup-like delimiters such as <image_gen>...</image_gen>, <pose_to_image>...</pose_to_image>, and <3D_gen_text>...</3D_gen_text>, learned as normal vocabulary tokens of the MLLM. They convert the problem of deciding which specialist to call, and what prompt to send it, into a next-token prediction objective, so the same cross-entropy loss used for instruction tuning also trains the planner. The supporting infrastructure is OlympusInstruct and OlympusBench, 446.3K and 49.6K GPT-4o-generated instruction-response pairs engineered with varied prefixes, phrases, and three complexity levels; the concise gold responses teach the model to strip a user request down to the essential prompt for the expert module.

What would settle it

Collect a few thousand human-written requests for the same 20 tasks without using GPT-4o templates, run Olympus against HuggingGPT with GPT-4o, and compare routing accuracy and chain precision; if Olympus's lead shrinks or reverses on human prompts, the template-distribution dependence is confirmed. A second check is to build chain-of-action instructions whose task pairs never co-occur in OlympusInstruct and see whether planning precision stays near 91.82%.

Watch

Extended reading notes

Core claim

The central claim is that routing can be learned end-to-end as token prediction. Olympus writes every task delegation as a response wrapped in control tokens, for instance <image_edit>adding green trees</image_edit>, and trains the MLLM with ordinary next-token cross-entropy on 446.3K GPT-4o-generated instruction-response pairs spanning 20 tasks. At inference, the emitted token sequence both identifies the expert module and supplies the refined prompt that module should execute, and chained actions are just concatenated token spans processed in order. On the accompanying OlympusBench set, the 3B model reaches 94.75% routing accuracy and 95.80% precision in the single-task setting and 91.82% precision, 92.75% recall, and 0.18 edit distance in the chain-of-action setting, exceeding the HuggingGPT pipeline driven by GPT-4o on every reported metric. On 11 standard multimodal benchmarks, the same model remains comparable to its Mipha-3B base, losing ground on some tasks and gaining on others.

Load-bearing premise

The load-bearing premise is that the GPT-4o-generated instructions in OlympusBench stand in for real user requests, because the evaluation set is built from the same templates, prefixes, and complexity levels as the training set.

Editorial extensions

If this is right

  • A 3B multimodal model can outperform a frontier LLM used as a prompt-engineered controller on routing accuracy, precision, recall, and F1 over 20 vision tasks, implying that specialized fine-tuning is a competitive way to build task planners.
  • The same framework transfers to different MLLM backbones at 3B, 7B, and 13B scale, and routing accuracy improves with model scale, suggesting the approach is not tied to one architecture.
  • Up to five specialist tasks can be planned within one instruction with 91.82% precision, which would let users express compound requests like generate, segment, then lift to 3D in a single sentence.
  • Because only the router is trained, new expert models can be swapped in without retraining the whole system, and the added training cost over base instruction tuning is about 23.6%.
  • Standard multimodal understanding is largely preserved while the model gains routing, so the router does not appear to sacrifice its original skills for planning ability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The headline numbers are likely optimistic for real-world use because OlympusBench shares GPT-4o's template distribution with the training data; a broader human-written test set would be needed to confirm the margin over HuggingGPT.
  • The routing-token mechanism is a general discrete-planning interface, so the same training recipe could route to non-vision tools or even physical actions, though the paper only demonstrates vision specialists.
  • A confidence-based fallback to a general LLM planner could mitigate out-of-template prompts, since the paper's own 200-instruction human evaluation shows some out-of-distribution transfer but with a small sample.
  • The reported per-task routing accuracy decline from 96.38% with 5 tasks to 94.75% with 20 suggests performance will keep degrading as the task catalog grows, which sets a practical limit on how many specialists a single small router can manage.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Olympus, a framework that turns a small multimodal large language model (MLLM) into a task router for computer vision. A controller MLLM, initialized from Mipha-3B, is fine-tuned on 446.3K GPT-4o-generated instructions (OlympusInstruct) to emit task-specific routing tokens, and at inference time these tokens invoke specialist models for image, video, and 3D tasks. The model is also trained and evaluated on chain-of-action settings, where a single user instruction triggers up to five consecutive routing steps. The authors report an average routing accuracy of 94.75% and chain precision of 91.82% on their own OlympusBench, a 86.5% success rate on 200 human-generated instructions, and comparable performance to Mipha-3B on 11 standard multimodal benchmarks.

Significance. If the routing results are robust, Olympus offers a practical modular alternative to all-in-one generative models: a 3B MLLM can coordinate specialized models for more than 20 vision tasks while retaining standard MLLM performance. The paper's strengths include the scale of the collected instruction data, the use of routing tokens to obtain chain-of-action behavior, the sanity check on standard multimodal benchmarks (Table 1), and the attempt to evaluate on human-written instructions. The main risk is that the central routing claim rests on a benchmark generated from the same GPT-4o prompt template family as the training data; the 200-request human study is the only explicit out-of-distribution evidence, and it conflates routing with downstream specialist success. The chain-of-action evaluation also concatenates independent random task samples rather than testing true sequential dependencies. These issues are load-bearing for the paper's central claims but appear addressable with additional evaluation, so a major revision is warranted.

major comments (4)
  1. [§3.1, §4.2, Appendix E] The evaluation set OlympusBench is generated with the same task-specific GPT-4o prompt scheme as OlympusInstruct, including the same prefixes, phrases, example pairs, and short/moderate/extended complexity levels described in Figure 4 and Appendix E. Tables 2 and 3 therefore primarily measure how well Olympus imitates the template distribution used to create both training and test data; the comparison against prompted HuggingGPT reflects template familiarity as much as routing skill. This is load-bearing for the 'universal task router' claim. Please add an out-of-distribution evaluation (e.g., a benchmark produced by a different generation procedure, or real user requests) and report routing accuracy on that set, and provide per-task or stratified results on OlympusBench. If the evaluation prompts actually differed from the training prompts, the paper should state this explicitly and include the evaluation prompt texts.
  2. [§4.2, Appendix E (Chain-of-Action Samples)] The chain-of-action evaluation is constructed by selecting N random tasks and concatenating one sample from each task. Such chains contain no sequential dependencies: no later instruction refers to the output of an earlier routing step, and downstream failure cannot propagate. Consequently, the 91.82% precision and 0.18 edit distance in Table 3 measure the ability to emit several independent routing tokens in order, not the ability to plan and execute multi-step workflows. Please construct chain evaluations with genuine inter-step dependencies (e.g., 'then add books near the dog' after an image generation step) and report whether intermediate specialist outputs are actually consumed by later steps.
  3. [Table 4, §4.2] The human evaluation 'success rate' conflates routing quality with the fidelity of specialist models: a request can fail because the routed specialist cannot satisfy it even when the routing decision is correct. Since Table 4 is the main out-of-distribution evidence, the paper should also report routing-only accuracy on the 200 human instructions (e.g., comparing predicted routing tokens against gold task labels), together with confidence intervals or annotator agreement. The 200-sample size gives a margin of error of roughly ±4-5 percentage points at 95% confidence when the success rate is near 86.5%, so point estimates alone do not support fine-grained comparisons.
  4. [Tables 2 and 10, §4.2] The headline '94.75% across 20 tasks' is reported only as a pooled aggregate. Table 10 shows large variation in evaluation sample sizes across tasks (e.g., 5,000 Image Generation samples versus 700 Image Deblurring samples), so a pooled accuracy can be dominated by high-frequency tasks and does not demonstrate uniform routing ability 'across 20 tasks.' Please report per-task accuracy and a confusion matrix, and clarify whether 94.75% is a pooled or macro-averaged number.
minor comments (5)
  1. [Throughout] There are several typos and formatting slips that should be corrected: 'concreate' in Figure 4, 'autogressive' in §2.2, 'Image Denosing' in Figure 8, and 'MLLMs Traning' in Table 8.
  2. [§3.1 and Figure 8] The paper says '20 tasks' but the routing token list in Figure 8 and the dataset table count controllable image/video generation conditions as separate token pairs; please clarify how tasks are counted (e.g., whether the six control conditions are grouped as one task or counted individually).
  3. [§4.2] The statement 'we included prompts covering all task types supported by Olympus and excluded prompts for irrelevant tasks' should specify how the subset was chosen for HuggingGPT and whether exactly the same subset is used for Olympus; otherwise the comparison is not fully reproducible.
  4. [§4.2 and Dataset Statistic] The paper reports 200 human-generated instructions but gives no details on how they were collected, who the evaluators were, or the scoring rubric used for the success rate; these details are needed to interpret Table 4.
  5. [Section 5 (Limitation)] The limitation section should also acknowledge the benchmark/training distribution overlap and the random-concatenation construction of chain-of-action samples, as these are the main threats to the routing and chaining claims; the current text only mentions GPT-4o quality and bias.

Circularity Check

2 steps flagged · score 5.0 of 10

OlympusBench is generated from the same GPT-4o prompt templates as OlympusInstruct, so the headline routing scores are in-distribution fit scores; only the 200 human instructions provide an external anchor.

  1. fitted input called prediction [Section 3.1 (Instruction Dataset Collection) and Section 4.2, Tables 2-3]
    "In order to accurately assign user instructions to the appropriate model, we constructed a high-quality and diverse dataset of user instruction–response pairs using GPT-4o. This dataset comprises 446.3K training samples, designated as OlympusInstruct, and 49.6K evaluation samples, designated as OlympusBench, encompassing 20 distinct tasks. For each task, a specialized prompt was developed to align with the specific context of the task."

    The routing claim's headline evidence (94.75% accuracy in Table 2) is a held-out accuracy on OlympusBench, which is produced by the same GPT-4o task-specific prompt pipeline as the OlympusInstruct training pairs (same prefixes, phrases, example pairs, and complexity levels, Figure 4). The model is fitted under the Section 3.3 next-token objective (Eq. 1) to reproduce GPT-4o's routing-token responses on exactly this template family, so scoring on another sample from the same generator measures in-distribution imitation, not universal routing. HuggingGPT is evaluated zero-shot on these templates (Section 4.2), so its 13.4% lower accuracy reflects template unfamiliarity by construction of the benchmark.

  2. other [Appendix E (Chain-of-Action Samples) and Section 4.2, Table 3]
    "Here, N random tasks (ranging from 2 to 5) were selected, and one sample from each task was combined to construct multi-step instruction-response pairs. This approach resulted in 64,800 and 7,200 samples for chain-of-action tasks in OlympusInstruct and OlympusBench, respectively."

    Both training and evaluation chain-of-action pairs are synthesized by the same random-concatenation routine: Appendix E selects N random tasks and combines one sample from each. The model is trained to emit routing-token sequences for such concatenations (Section 3.3) and is then scored (91.82% precision, Table 3) on concatenations built with the identical procedure, so the metric rewards parsing of template-concatenated phrasings rather than genuine sequential dependency handling. The 7,200 evaluation chains come from the same distribution as the 64,800 training chains by design, giving the chain-precision number the same in-distribution character as the single-task score.

full rationale

The central routing claim is anchored on OlympusBench, which is not an independent evaluation: Section 3.1 reports that both OlympusInstruct (446.3K training pairs) and OlympusBench (49.6K evaluation pairs) were collected from GPT-4o using the same task-specific prompts, with Figure 4 showing the shared template of 7 prefixes, 12 phrases, 9 example pairs, and three complexity levels. Because the Section 3.3 training objective (Eq. 1) fits the MLLM to reproduce GPT-4o's routing-token responses on this template family, the 94.75% single-task accuracy and 91.82% chain precision (Tables 2-3) are held-out fit scores to the training generator's distribution, and the gap over zero-shot HuggingGPT measures template familiarity as much as routing skill. Appendix E shows the chain-of-action evaluation is built by the same random task-concatenation procedure as the training set, so the chain metric does not test sequential dependencies. The paper itself acknowledges the dependence in Section 5: 'the quality and diversity of the samples collected directly impact the performance of the generated responses.' The circularity is partial, though: Table 1 uses 11 external multimodal benchmarks, and the 200 human-generated instructions (Table 4) are genuinely out-of-distribution, where Olympus still beats HuggingGPT (86.5% vs 75.2%), albeit on a metric that conflates routing with specialist-model output quality. No load-bearing self-citation or imported uniqueness result appears anywhere in the derivation chain. Score 5 reflects that the headline 'universal router' numbers are in-distribution by construction while independent content remains.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No fitted numeric constants appear in the central claim. The model is trained with standard cross-entropy next-token prediction; routing accuracy is an eval metric, not a fitted parameter. Hand-chosen design elements (number of routing tokens, concise vs coarse responses) are engineering choices, not fitted values. Routing tokens are syntactic markers, not new physical entities, so invented_entities is empty.

assumptions (3)
  • domain assumption The base MLLM (SigLIP-384 + Phi-2) has sufficient inherited capacity for understanding tasks routed to it directly, e.g., VQA.
    Section 3 and 3.4 rely on the MLLM solving understanding tasks internally; if this fails, the 'internal vs external' split collapses. Table 1 provides empirical support for VQA-style tasks.
  • domain assumption GPT-4o-generated instructions and labels are a diverse and accurate population of user requests for the 20 tasks.
    Section 3.1 builds OlympusInstruct and OlympusBench from GPT-4o prompts; Section 4.2 evaluates on this same distribution. This assumption is load-bearing for generalization claims.
  • domain assumption The chosen specialist models (e.g., Stable Diffusion XL, InstructPix2Pix, ControlNet, LGM) produce acceptable outputs for their tasks.
    Appendix D lists these models, but the paper does not quantitatively evaluate final task outputs, so the claim that Olympus 'solves' tasks depends on unmeasured model quality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Olympus: A Universal Task Router for Computer Vision Tasks." pith.science (2026). https://pith.science/paper/QKH42KTR

@misc{pith2026241209612,
  author       = {Pith},
  title        = {Pith review of: Olympus: A Universal Task Router for Computer Vision Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QKH42KTR}},
  note         = {Machine review of arXiv:2412.09612}
}
read the original abstract

We introduce Olympus, a new approach that transforms Multimodal Large Language Models (MLLMs) into a unified framework capable of handling a wide array of computer vision tasks. Utilizing a controller MLLM, Olympus delegates over 20 specialized tasks across images, videos, and 3D objects to dedicated modules. This instruction-based routing enables complex workflows through chained actions without the need for training heavy generative models. Olympus easily integrates with existing MLLMs, expanding their capabilities with comparable performance. Experimental results demonstrate that Olympus achieves an average routing accuracy of 94.75% across 20 tasks and precision of 91.82% in chained action scenarios, showcasing its effectiveness as a universal task router that can solve a diverse range of computer vision tasks. Project page: http://yuanze-lin.me/Olympus_page/

Figures

Figures reproduced from arXiv: 2412.09612 by the authors.

Figure 1
Figure 1. With its versatile capabilities, Olympus addresses a broad spectrum of vision tasks across images, videos, and even 3D content, in fact, it can cover over 20 different tasks. Abstract We introduce Olympus, a new approach that transforms Mul￾timodal Large Language Models (MLLMs) into a unified framework capable of handling a wide array of computer vision tasks. Utilizing a controller MLLM, Olympus dele￾gates over 20 … view at source ↗
Figure 2
Figure 2. Given the user prompts, a trainable MLLM can perform routing across a wide range of specified models. In this concept, MLLMs can solve multimodal understanding tasks (e.g., VQA) with its inherited capacity, while MLLMs can allocate appropriate specialized models to address multimodal generative and classic vision tasks (e.g., image generation and depth estimation), then aggregate the results and deliver a response t… view at source ↗
Figure 3
Figure 3. The framework of Olympus. It can solve those tasks like VQA through the inherited capacities of MLLM directly. For other tasks, e.g., image editing, Olympus can generate the response, which consists of task-specific routing tokens and refined prompts, they are then used to schedule specialist models for addressing diverse user requests. (e.g., image, video, and 3D generation) and classic vision tasks (e.g., image su… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: One example to illustrate the prompt we use to generate the instruction-response pairs for image editing by GPT-4o. costume.</image_gen>, which is designed to effec￾tively address the user’s request. Thus, user instructions and the responses form input-answer pairs for…
Figure 5
Figure 5. Figure 5: The statistic of the collected dataset. Note that CVG and CIG denote controllable video generation and controllable image generation, RVOS represents referring video object segmentation and image SR means image super-resolution in figure (b). Method LM Res. VQAv2 GQA V…
Figure 6
Figure 6. Figure 6: Diverse applications of Olympus. The 1st and 2nd columns show the scenarios for single task, the 3rd column displays the results under multi-turn conversations, while the last (4th) column shows the chain-of-action capacity of Olympus. Method Acc ↑ Pre ↑ Recall ↑ F1 ↑ …
Figure 7
Figure 7. Figure 7: Training time cost for varying numbers of tasks. [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: Task-specific routing tokens for 20 diverse tasks, covering image, video and 3D domains. Note that “RVOS” denotes referring video object segmentation. Specialists for 20 individual computer tasks Task / Model Task / Model Image Generation: Stable Diffusion XL [58] Vide…
Figure 9
Figure 9. Figure 9: The final prompt used to generate user instruction-response pairs for image editing in our experiments [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Diverse applications of Olympus. The first and second columns denote controllable image generation conditioning on depth, pose, scribble and normal. The third column represent the applications of object detection and image deblurring. The last column represent the app…
Figure 11
Figure 11. Figure 11: Diverse applications of Olympus. The first column represents image and video editing, the second column contains the examples of image-to-3D generation and image deraining, the third column denotes image deblurring and depth estimation, and the final column displays t…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. IllumiCraft: Unified Geometry and Illumination Diffusion for Controllable Video Generation

    cs.CV 2025-06 reject novelty 6.0 of 10

    A diffusion video model that jointly uses HDR lighting, relit frames, and 3D point tracks to relight videos from text prompts.

Reference graph

Works this paper leans on

102 extracted references · 27 canonical work pages · cited by 1 Pith paper

  1. [1]

    Jointly training large autoregressive multi- modal models

    Emanuele Aiello, LILI YU, Yixin Nie, Armen Aghajanyan, and Barlas Oguz. Jointly training large autoregressive multi- modal models. In ICLR, 2024. 3

    2024

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736,

  3. [3]

    Gemini: A family of highly capable multimodal models

    Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 1, 2023. 2

    arXiv 2023

  4. [4]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425– 2433, 2015. 1

    2015

  5. [5]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. CoRR, abs/2308.12966, 2023. 2, 6

    arXiv 2023

  6. [6]

    In- structpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. In- structpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392–18402, 2023. 14, 15

    2023

  7. [7]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, Sand- hini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, S...

    1901

  8. [8]

    Shikra: Unleashing multimodal llm’s ref- erential dialogue magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s ref- erential dialogue magic. arXiv preprint arXiv:2306.15195,

    arXiv

Show all 102 references
  1. [9]

    Text-to-3d using gaussian splatting

    Zilong Chen, Feng Wang, Yikai Wang, and Huaping Liu. Text-to-3d using gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21401–21412, 2024. 2

    2024

  2. [10]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. lmsys. org (accessed 14 April 2023), 2023. 6

    2023

  3. [11]

    Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer,...

    2022

  4. [12]

    Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices

    Xiangxiang Chu, Limeng Qiao, Xinyang Lin, Shuang Xu, Yang Yang, Yiming Hu, Fei Wei, Xinyu Zhang, Bo Zhang, Xiaolin Wei, et al. Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886, 2023. 3, 6

    2023 arXiv

  5. [13]

    Mobilevlm v2: Faster and stronger baseline for vision language model.arXiv preprint arXiv:2402.03766,

    Xiangxiang Chu, Limeng Qiao, Xinyu Zhang, Shuang Xu, Fei Wei, Yang Yang, Xiaofei Sun, Yiming Hu, Xinyang Lin, Bo Zhang, et al. Mobilevlm v2: Faster and stronger baseline for vision language model.arXiv preprint arXiv:2402.03766,

    arXiv

  6. [14]

    Swin2sr: Swinv2 transformer for compressed im- age super-resolution and restoration

    Marcos V Conde, Ui-Jin Choi, Maxime Burchi, and Radu Timofte. Swin2sr: Swinv2 transformer for compressed im- age super-resolution and restoration. In European Confer- ence on Computer Vision, pages 669–687. Springer, 2022. 15

    2022

  7. [15]

    High- quality image restoration following human instructions

    Marcos V Conde, Gregor Geigle, and Radu Timofte. High- quality image restoration following human instructions. arXiv preprint arXiv:2401.16468, 2024. 15

    2024 arXiv

  8. [16]

    Instructblip: Towards general- purpose vision-language models with instruction tuning,

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general- purpose vision-language models with instruction tuning,

  9. [17]

    DreamLLM: Synergistic multimodal com- prehension and creation

    Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, and Li Yi. DreamLLM: Synergistic multimodal com- prehension and creation. In ICLR, 2024. 2, 3

    2024

  10. [18]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition ...

    2021

  11. [19]

    Scaling rectified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In ICML, 2024. 2

    2024

  12. [20]

    Mme: A comprehensive evaluation bench- mark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Meng- dan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation bench- mark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. 6, 7, 8

    2023 arXiv

  13. [21]

    Seed-x: Multimodal models with unified multi-granularity compre- hension and generation

    Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granularity compre- hension and generation. arXiv preprint arXiv:2404.14396,

    arXiv

  14. [22]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913...

    2017

  15. [23]

    Visual program- ming: Compositional visual reasoning without training

    Tanmay Gupta and Aniruddha Kembhavi. Visual program- ming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 14953–14962, 2023. 3

    2023

  16. [24]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617,

  17. [25]

    Efficient multimodal learning from data-centric perspective

    Muyang He, Yexin Liu, Boya Wu, Jianhao Yuan, Yueze Wang, Tiejun Huang, and Bo Zhao. Efficient multimodal learning from data-centric perspective. arXiv preprint arXiv:2402.11530, 2024. 6

    2024 arXiv

  18. [26]

    Denoising diffu- sion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models. NeurIPS, pages 6840–6851, 2020. 1

    2020

  19. [27]

    Video diffu- sion models

    Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffu- sion models. NeurIPS, 2022. 2

    2022

  20. [28]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 6700–6709, 2019. 6, 7

    2019

  21. [29]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perel- man, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Weli- hinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. 2, 17

    2024 arXiv

  22. [30]

    Text2video-zero: Text-to- image diffusion models are zero-shot video generators

    Levon Khachatryan, Andranik Movsisyan, Vahram Tade- vosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text-to- image diffusion models are zero-shot video generators. In Proceedings of the IEEE/CVF International Conference on Comput...

    2023

  23. [31]

    Adam: A method for stochastic opti- mization

    Diederik P Kingma. Adam: A method for stochastic opti- mization. arXiv preprint arXiv:1412.6980, 2014. 14

    2014 arXiv

  24. [32]

    Auto-encoding varia- tional bayes

    Diederik P Kingma and Max Welling. Auto-encoding varia- tional bayes. arXiv preprint arXiv:1312.6114, 2013. 3

    2013 arXiv

  25. [33]

    Obelics: An open web-scale filtered dataset of interleaved image-text documents

    Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bek- man, Amanpreet Singh, Anton Lozhkov, Thomas Wang, Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Information ...

    2024

  26. [34]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 2

    2024 arXiv

  27. [35]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, pages 19730–19742, 2023. 2, 3

    2023

  28. [36]

    Evaluating object hallucina- tion in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina- tion in large vision-language models. arXiv preprint arXiv:2305.10355, 2023. 6, 7

    2023 arXiv

  29. [37]

    Mini-gemini: Mining the potential of multi-modality vision language models

    Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403.18814,

    arXiv

  30. [38]

    Taskmatrix

    Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji, Shaoguang Mao, et al. Taskmatrix. ai: Completing tasks by connecting foun- dation models with millions of apis. Intelligent Computing, 3:0063, 2024. 3

    2024

  31. [39]

    Moe-llava: Mixture of experts for large vision-language models

    Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Junwu Zhang, Munan Ning, and Li Yuan. Moe-llava: Mixture of experts for large vision-language models. arXiv preprint arXiv:2401.15947, 2024. 6

    2024 arXiv

  32. [40]

    Revive: Regional visual represen- tation matters in knowledge-based visual question answer- ing

    Yuanze Lin, Yujia Xie, Dongdong Chen, Yichong Xu, Chen- guang Zhu, and Lu Yuan. Revive: Regional visual represen- tation matters in knowledge-based visual question answer- ing. Advances in Neural Information Processing Systems, 35:10560–10571, 2022. 3

    2022

  33. [41]

    Smaug: Sparse masked autoencoder for effi- cient video-language pre-training

    Yuanze Lin, Chen Wei, Huiyu Wang, Alan Yuille, and Ci- hang Xie. Smaug: Sparse masked autoencoder for effi- cient video-language pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2459–2469, 2023. 2

    2023

  34. [42]

    Text-driven image editing via learnable regions

    Yuanze Lin, Yi-Wen Chen, Yi-Hsuan Tsai, Lu Jiang, and Ming-Hsuan Yang. Text-driven image editing via learnable regions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7059–7068,

  35. [43]

    Dreampolisher: Towards high-quality text-to-3d generation via geometric diffusion

    Yuanze Lin, Ronald Clark, and Philip Torr. Dreampolisher: Towards high-quality text-to-3d generation via geometric diffusion. arXiv preprint arXiv:2403.17237, 2024. 2

    2024 arXiv

  36. [44]

    Rethinking visual prompting for multimodal large language models with ex- ternal knowledge

    Yuanze Lin, Yunsheng Li, Dongdong Chen, Weijian Xu, Ronald Clark, Philip Torr, and Lu Yuan. Rethinking visual prompting for multimodal large language models with ex- ternal knowledge. arXiv preprint arXiv:2407.04681, 2024. 2

    2024 arXiv

  37. [45]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, pages 26296–26306, 2024. 6, 8, 14

    2024

  38. [46]

    Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Im- proved reasoning, ocr, and world knowledge, 2024. 2

    2024

  39. [47]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 1

    2024

  40. [48]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 36, 2024. 2, 6, 14

    2024

  41. [49]

    Grounding dino: Marrying dino with grounded pre-training for open-set object detection

    Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499, 2023. 15

    2023 arXiv

  42. [50]

    Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023. 6, 7

    2023 arXiv

  43. [51]

    Wonder3d: Single image to 3d using cross-domain diffusion

    Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Marc Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Patt...

    2024

  44. [52]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507–2521,

  45. [53]

    Computation of normal- ized edit distance and applications

    Andres Marzal and Enrique Vidal. Computation of normal- ized edit distance and applications. IEEE transactions on pattern analysis and machine intelligence, 15(9):926–932,

  46. [54]

    Mm1: Methods, analysis & insights from multimodal llm pre-training

    Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xi- anzhi Du, Futang Peng, Floris Weers, et al. Mm1: Methods, analysis & insights from multimodal llm pre-training. arXiv preprint arXiv:2403.09611, 2024. 2

    2024 arXiv

  47. [55]

    Phi-2: The surprising power of small language models, 2023

    Microsoft. Phi-2: The surprising power of small language models, 2023. 6

    2023

  48. [56]

    Improved denoising diffusion probabilistic models

    Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International conference on machine learning, pages 8162–8171. PMLR,

  49. [57]

    Kosmos-2: Grounding multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. CoRR, abs/2306.14824, 2023. 2

    2023 arXiv

  50. [58]

    Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion mod- els for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023. 1, 2, 14, 15

    2023 arXiv

  51. [59]

    Tool learning with foundation models, 2023

    Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Ye, Bowen Li, Ziwei Tang, ...

    2023

  52. [60]

    Photorealistic text-to-image diffusion models with deep language understanding

    Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- ti...

    2022

  53. [61]

    Toolformer: Lan- guage models can teach themselves to use tools

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Lan- guage models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36, 2024. 3

    2024

  54. [62]

    Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weim- ing Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Systems, 36, 2024. 2, 3, 7, 17

    2024

  55. [63]

    Eagle: Exploring the design space for multimodal llms with mixture of encoders

    Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, et al. Eagle: Exploring the design space for multimodal llms with mixture of encoders. arXiv preprint arXiv:2408.15998, 2024. 2

    2024 arXiv

  56. [64]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792,

    arXiv

  57. [65]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 6, 7

    2019

  58. [66]

    Emu: Generative pretraining in multimodality

    Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. In ICLR, 2023. 3

    2023

  59. [67]

    An empirical study of multimodal model merging

    Yi-Lin Sung, Linjie Li, Kevin Lin, Zhe Gan, Mohit Bansal, and Lijuan Wang. An empirical study of multimodal model merging. arXiv preprint arXiv:2304.14933, 2023. 6

    2023 arXiv

  60. [68]

    Vipergpt: Visual inference via python execution for reasoning

    Dídac Surís, Sachit Menon, and Carl V ondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11888–11898, 2023. 3

    2023

  61. [69]

    Lgm: Large multi-view gaussian model for high-resolution 3d content creation

    Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. In European Conference on Computer Vision, pages 1–18. Springer, 2025. 14, 15

    2025

  62. [70]

    Any-to-any generation via composable diffusion

    Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffusion. NeurIPS, 36, 2024. 2, 3

    2024

  63. [71]

    Chameleon: Mixed-modal early-fusion foundation models

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint arXiv:2405.09818, 2024. 2, 3

    2024 arXiv

  64. [72]

    Cambrian- 1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian- 1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860, 2024. 2

    2024 arXiv

  65. [73]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Mar- tinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Roz- ière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation lang...

    2023 arXiv

  66. [74]

    Emu3: Next-token prediction is all you need

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18869, 2024. 2, 3

    2024 arXiv

  67. [75]

    Visual chatgpt: Talking, drawing and editing with visual foundation models

    Chenfei Wu, Shengming Yin, Weizhen Qi, Xiaodong Wang, Zecheng Tang, and Nan Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671, 2023. 3

    2023 arXiv

  68. [76]

    General object foundation model for images and videos at scale

    Junfeng Wu, Yi Jiang, Qihao Liu, Zehuan Yuan, Xiang Bai, and Song Bai. General object foundation model for images and videos at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3783–3795, 2024. 15

    2024

  69. [77]

    Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation

    Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In ICCV, 2023. 2

    2023

  70. [78]

    Next-gpt: Any-to-any multimodal llm

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519, 2023. 2, 3

    2023 arXiv

  71. [79]

    Omnigen: Unified image generation

    Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xin- grun Xing, Ruiran Yan, Shuting Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation. arXiv preprint arXiv:2409.11340, 2024. 2, 3

    2024 arXiv

  72. [80]

    Segformer: Simple and efficient design for semantic segmentation with transform- ers

    Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transform- ers. Advances in neural information processing systems, 34: 12077–12090, 2021. 15

    2021

  73. [81]

    Show-o: One single transformer to unify multimodal understanding and generation

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408.12528, 2024. 2, 3

    2024 arXiv

  74. [82]

    xgen-mm (blip-3): A family of open large multimodal models

    Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yu- tong Dai, Michael S Ryoo, et al. xgen-mm (blip-3): A family of open large multimodal models. arXiv preprint arXiv:2408.08872, 2024. 2

    2024

  75. [83]

    Depth anything v2

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xi- aogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv preprint arXiv:2406.09414, 2024. 15

    2024 arXiv

  76. [84]

    Seed-story: Multimodal long story generation with large language model

    Shuai Yang, Yuying Ge, Yang Li, Yukang Chen, Yixiao Ge, Ying Shan, and Yingcong Chen. Seed-story: Multimodal long story generation with large language model. arXiv preprint arXiv:2407.08683, 2024. 3

    2024 arXiv

  77. [85]

    Effec- tive whole-body pose estimation with two-stages distillation

    Zhendong Yang, Ailing Zeng, Chun Yuan, and Yu Li. Effec- tive whole-body pose estimation with two-stages distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4210–4220, 2023. 15

    2023

  78. [86]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 15

    2024 arXiv

  79. [87]

    X-vila: Cross-modality alignment for large language model

    Hanrong Ye, De-An Huang, Yao Lu, Zhiding Yu, Wei Ping, Andrew Tao, Jan Kautz, Song Han, Dan Xu, Pavlo Molchanov, et al. X-vila: Cross-modality alignment for large language model. arXiv preprint arXiv:2405.19335,

    arXiv

  80. [88]

    mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Haowei Liu, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. arXiv preprint arXiv:2311.04257, 2023. 6

    2023 arXiv

  81. [89]

    Scaling autoregressive multi- modal models: Pretraining and instruction tuning

    Lili Yu, Bowen Shi, Ramakanth Pasunuru, Benjamin Muller, Olga Golovneva, Tianlu Wang, Arun Babu, Binh Tang, Brian Karrer, Shelly Sheynin, et al. Scaling autoregressive multi- modal models: Pretraining and instruction tuning. arXiv preprint arXiv:2309.02591, 2(3), 2023. 3

    2023 arXiv

  82. [90]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023. 6, 7

    2023 arXiv

  83. [91]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on C...

    2024

  84. [92]

    Sigmoid loss for language image pre-training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975–11986, 2023. 6

    2023

  85. [93]

    Anygpt: Unified multimodal llm with dis- crete sequence modeling

    Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. Anygpt: Unified multimodal llm with dis- crete sequence modeling. arXiv preprint arXiv:2402.12226,

    arXiv

  86. [94]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3836–3847, 2023. 15

    2023

  87. [95]

    Tinyllava: A framework of small-scale large multimodal models

    Baichuan Zhou, Ying Hu, Xi Weng, Junlong Jia, Jie Luo, Xien Liu, Ji Wu, and Lei Huang. Tinyllava: A framework of small-scale large multimodal models. arXiv preprint arXiv:2402.14289, 2024. 6

    2024 arXiv

  88. [96]

    Transfusion: Pre- dict the next token and diffuse images with one multi-modal model

    Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Pre- dict the next token and diffuse images with one multi-modal model. arXiv preprint arXiv:2408.11039, 2024. 2, 3

    2024 arXiv

  89. [97]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. CoRR, abs/2304.10592, 2023. 2

    2023 arXiv

  90. [98]

    VL-GPT: A generative pre-trained transformer for vision and language understanding and generation

    Jinguo Zhu, Xiaohan Ding, Yixiao Ge, Yuying Ge, Si- jie Zhao, Hengshuang Zhao, Xiaohua Wang, and Ying Shan. VL-GPT: A generative pre-trained transformer for vision and language understanding and generation. CoRR, abs/2312.09251, 2023. 3

    2023 arXiv

  91. [99]

    Mipha: A comprehensive overhaul of multimodal assistant with small language models

    Minjie Zhu, Yichen Zhu, Xin Liu, Ning Liu, Zhiyuan Xu, Chaomin Shen, Yaxin Peng, Zhicai Ou, Feifei Feng, and Jian Tang. Mipha: A comprehensive overhaul of multimodal assistant with small language models. CoRR, 2024. 3, 6

    2024

  92. [100]

    Llava-phi: Efficient multi-modal assistant with small language model

    Yichen Zhu, Minjie Zhu, Ning Liu, Zhicai Ou, Xiaofeng Mou, and Jian Tang. Llava-phi: Efficient multi-modal assistant with small language model. arXiv preprint arXiv:2401.02330, 2024. 2, 6

    2024 arXiv

  93. [101]

    <3D_gen_text>ancient library, a repository of books and scrolls from ancient times.</3D_gen_text>

    Zhuofan Zong, Guanglu Song, and Yu Liu. Detrs with collaborative hybrid assignments training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6748–6758, 2023. 15 A. Appendix In the supplementary materials, we provide the following sections: • ...

    2023

  94. [102]

    Com- plexities

    were selected, and one sample from each task was com- bined to construct multi-step instruction-response pairs. This approach resulted in 64,800 and 7,200 samples for chain- of-action tasks in OlympusInstruct and OlympusBench, re- spectively. These chain-of-action samples are ...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.