Pith. sign in

REVIEW 22 cited by

VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.08394 v3 pith:QC7CIPJY submitted 2024-06-12 cs.CV

Jiannan Wu , Muyan Zhong , Sen Xing , Zeqiang Lai , Zhaoyang Liu , Zhe Chen , Wenhai Wang , Xizhou Zhu
show 5 more authors
Lewei Lu Tong Lu Ping Luo Yu Qiao Jifeng Dai
This is my paper · ORCID
classification cs.CV
keywords tasksvisionllmend-to-endhundredsmllmmodelvisiondecoders
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

We present VisionLLM v2, an end-to-end generalist multimodal large model (MLLM) that unifies visual perception, understanding, and generation within a single framework. Unlike traditional MLLMs limited to text output, VisionLLM v2 significantly broadens its application scope. It excels not only in conventional visual question answering (VQA) but also in open-ended, cross-domain vision tasks such as object localization, pose estimation, and image generation and editing. To this end, we propose a new information transmission mechanism termed "super link", as a medium to connect MLLM with task-specific decoders. It not only allows flexible transmission of task information and gradient feedback between the MLLM and multiple downstream decoders but also effectively resolves training conflicts in multi-tasking scenarios. In addition, to support the diverse range of tasks, we carefully collected and combed training data from hundreds of public vision and vision-language tasks. In this way, our model can be joint-trained end-to-end on hundreds of vision language tasks and generalize to these tasks using a set of shared parameters through different user prompts, achieving performance comparable to task-specific models. We believe VisionLLM v2 will offer a new perspective on the generalization of MLLMs.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 22 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Synthetic Visual Genome

    cs.CV 2025-06 conditional novelty 7.0 of 10

    A GPT-4V/GPT-4o pipeline for completing and refining scene graph annotations yields a dense synthetic dataset that, after instruction tuning, gives a 3B model strong relationship understanding and grounding results.

  2. TaskGalaxy: Scaling Multi-modal Instruction Fine-tuning with Tens of Thousands Vision Task Types

    cs.CV 2025-02 conditional novelty 7.0 of 10

    The authors release TaskGalaxy, a 413k-sample multimodal instruction dataset with 19,227 hierarchically organized task types, and report consistent benchmark gains when it is added to LLaVA-v1.5 and InternVL-Chat-v1.0.

  3. HRSeg: High-Resolution Visual Perception and Enhancement for Reasoning Segmentation

    cs.CV 2025-07 conditional novelty 6.0 of 10

    HRSeg combines high-resolution image crops, region attention, and cross-attention mask enhancement, improving reasoning segmentation over LLM-Seg by up to 13.2 gIoU points on LLM-Seg40K.

  4. LIRA: Inferring Segmentation in Large Multi-modal Models with Local Interleaved Region Assistance

    cs.CV 2025-07 conditional novelty 6.0 of 10

    LIRA improves referring segmentation and reduces hallucination in multimodal LLMs by fusing semantic and pixel features and interleaving local image regions with text descriptions.

  5. UrbanLLaVA: A Multi-modal Large Language Model for Urban Intelligence with Spatial Reasoning and Understanding

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A fine-tuned small multimodal LLM outperforms much larger general models on urban tasks in a new benchmark, with caveats about benchmark overlap with training data.

  6. Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Models

    cs.CL 2025-01 conditional novelty 6.0 of 10

    Migician is an instruction-tuned MLLM that performs free-form grounding across multiple images, with a new 630k dataset and a 10-task benchmark, but the evaluation is weakened by source overlap between training and be...

  7. Explanatory Instructions: Towards Unified Vision Tasks Understanding and Zero-shot Generalization

    cs.CV 2024-12 conditional novelty 6.0 of 10

    Explanatory instructions, detailed text descriptions of image-to-image transformations, are introduced with a 12M-pair dataset and show qualitative evidence of zero-shot generalization on unseen vision tasks.

  8. RSUniVLM: A Unified Vision Language Model for Remote Sensing via Granularity-oriented Mixture of Experts

    cs.CV 2024-12 conditional novelty 6.0 of 10

    RSUniVLM is a 1-billion-parameter remote sensing vision-language model that unifies image-, region-, and pixel-level tasks plus multi-image change analysis, achieving state-of-the-art visual grounding on VRSBench and ...

  9. GREAT: Geometry-Intention Collaborative Inference for Open-Vocabulary 3D Object Affordance Grounding

    cs.CV 2024-11 conditional novelty 6.0 of 10

    GREAT combines multimodal language model reasoning with 3D geometry to ground open-vocabulary object affordances, and introduces the large PIADv2 dataset.

  10. Multimodal 3D Reasoning Segmentation with Complex Scenes

    cs.CV 2024-11 conditional novelty 6.0 of 10

    A new benchmark and model enable multi-object 3D reasoning segmentation, where a point-cloud scene and a question produce both explanations and masks for several objects at once.

  11. Advancing Fine-Grained Visual Understanding with Multi-Scale Alignment in Multi-Modal Models

    cs.CV 2024-11 conditional novelty 6.0 of 10

    A three-stage training method that aligns object texts, coordinates, and cropped images, plus a GPT-4V-based data synthesis pipeline, lets 1.5B and 3B models match or exceed larger MLLMs on grounding and VQA benchmarks.

  12. HintEval: A Comprehensive Framework for Hint Generation and Evaluation for Questions

    cs.CL 2025-02 conditional novelty 5.0 of 10

    The paper presents HintEval, an open-source Python framework that unifies hint-generation datasets, model wrappers, and five families of evaluation metrics with fifteen methods for question-answering hints.

  13. Task Preference Optimization: Improving Multimodal Large Language Models with Vision Task Alignment

    cs.CV 2024-12 conditional novelty 5.0 of 10

    Adding task-specific heads for tracking, grounding, and segmentation to multimodal LLMs via a three-stage training recipe improves both fine-grained visual tasks and general video understanding benchmarks.

  14. ChatRex: Taming Multimodal LLM for Joint Perception and Understanding

    cs.CV 2024-11 conditional novelty 5.0 of 10

    ChatRex couples a universal proposal network with an LLM that retrieves box indices, reaching 48.2 mAP on COCO and strong referring and region-level results.

  15. GeoGround: A Unified Large Vision-Language Model for Remote Sensing Visual Grounding

    cs.CV 2024-11 conditional novelty 5.0 of 10

    GeoGround unifies horizontal box, oriented box, and mask visual grounding in a single remote sensing vision-language model using a text-based mask representation.

  16. KptLLM++: Towards Generic Keypoint Comprehension with Large Language Model

    cs.CV 2025-07 conditional novelty 4.0 of 10

    KptLLM++ unifies keypoint semantic understanding, visual-prompt detection, and text-prompt detection in a single multimodal LLM, reporting SOTA accuracy on COCO, AP-10K, Human-Art, and other benchmarks.

  17. Vision Generalist Model: A Survey

    cs.CV 2025-06 conditional novelty 3.0 of 10

    A structured review of vision generalist models, classifying them into encoding-based and sequence-to-sequence frameworks and summarizing datasets, benchmarks, techniques, and open problems.

  18. Visual Large Language Models for Generalized and Specialized Applications

    cs.CV 2025-01 conditional novelty 3.0 of 10

    This paper reviews and taxonomizes VLLM applications into vision-to-text, vision-to-action, and text-to-vision, adding ethics and future-work discussion.

  19. Optimizing Vision-Language Interactions Through Decoder-Only Models

    cs.CV 2024-12 reject novelty 3.0 of 10

    An encoder-free vision-language model that claims state-of-the-art scores; the paper lacks any reproducible evidence.

  20. Dynamic Cross-Modal Alignment for Robust Semantic Location Prediction

    cs.CV 2024-12 reject novelty 3.0 of 10

    CoVLA, built from standard cross-modal attention and gated fusion, is claimed to improve semantic location prediction accuracy by 2.3% and F1 by 2.4% over the SG-MFT baseline, despite missing error bars, code, and key...

  21. Leveraging Retrieval-Augmented Tags for Large Vision-Language Understanding in Complex Scenes

    cs.CV 2024-12 reject novelty 2.0 of 10

    VRAP claims state-of-the-art vision-language results by prepending image-derived object tags to LLM prompts, but the evidence is incomplete and internally inconsistent.

  22. Bridging Vision and Language: Modeling Causality and Temporality in Video Narratives

    cs.CV 2024-12 reject novelty 2.0 of 10

    Injecting self-attention and a transformer encoder into LVLMs is reported to improve video captioning metrics on MSVD and MSR-VTT, but the losses, dataset, and code are not described.

Pith tools