REVIEW 6 cited by
Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling and Visual-Language Co-Referring
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Large Vision Language Models have achieved fine-grained object perception, but the limitation of image resolution remains a significant obstacle to surpassing the performance of task-specific experts in complex and dense scenarios. Such limitation further restricts the model's potential to achieve nuanced visual and language referring in domains such as GUI Agents, counting, \textit{etc}. To address this issue, we introduce a unified high-resolution generalist model, Griffon v2, enabling flexible object referring with visual and textual prompts. To efficiently scale up image resolution, we design a simple and lightweight down-sampling projector to overcome the input tokens constraint in Large Language Models. This design inherently preserves the complete contexts and fine details and significantly improves multimodal perception ability, especially for small objects. Building upon this, we further equip the model with visual-language co-referring capabilities through a plug-and-play visual tokenizer. It enables user-friendly interaction with flexible target images, free-form texts, and even coordinates. Experiments demonstrate that Griffon v2 can localize objects of interest with visual and textual referring, achieve state-of-the-art performance on REC and phrase grounding, and outperform expert models in object detection, object counting, and REG. Data and codes are released at https://github.com/jefferyZhan/Griffon.
Forward citations
Cited by 6 Pith papers
-
Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoning
Training a multimodal LLM on GPT-4o-generated chain-of-thought referring traces, then optimizing with GRPO, improves referring accuracy and abstention on HumanRef.
-
Migician: Revealing the Magic of Free-Form Multi-Image Grounding in Multimodal Large Language Models
Migician is an instruction-tuned MLLM that performs free-form grounding across multiple images, with a new 630k dataset and a 10-task benchmark, but the evaluation is weakened by source overlap between training and be...
-
VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with Video LLM
A dataset-model-benchmark suite that lets video LLMs understand and reason about user-specified objects across time, using mask-based spatial-temporal object tokens and a 700K instruction set.
-
Apollo: An Exploration of Video Understanding in Large Multimodal Models
Design decisions for video-LMMs can be made on 2-4B models and datasets and transfer to larger models, yielding efficient Apollo models, though some SOTA claims are contradicted by the paper's own table.
-
UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learning
UniVG-R1 uses CoT supervised fine-tuning plus GRPO with difficulty-aware reweighting to make Qwen2-VL substantially better at multi-image, reasoning-based visual grounding.
-
ChatRex: Taming Multimodal LLM for Joint Perception and Understanding
ChatRex couples a universal proposal network with an LLM that retrieves box indices, reaching 48.2 mAP on COCO and strong referring and region-level results.
Discussion (0). Continue with ORCID to comment.