Pith. sign in

REVIEW 15 cited by

Perceive Anything: Recognize, Explain, Caption, and Segment Anything in Images and Videos

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2506.05302 v1 pith:EMEKERJM submitted 2025-06-05 cs.CV

Weifeng Lin , Xinyu Wei , Ruichuan An , Tianhe Ren , Tingwei Chen , Renrui Zhang , Ziyu Guo , Wentao Zhang
show 2 more authors
Lei Zhang Hongsheng Li
This is my paper
classification cs.CV
keywords understandinganythingregion-levelsemanticvisualapproachcaptiondata
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We present Perceive Anything Model (PAM), a conceptually straightforward and efficient framework for comprehensive region-level visual understanding in images and videos. Our approach extends the powerful segmentation model SAM 2 by integrating Large Language Models (LLMs), enabling simultaneous object segmentation with the generation of diverse, region-specific semantic outputs, including categories, label definition, functional explanations, and detailed captions. A key component, Semantic Perceiver, is introduced to efficiently transform SAM 2's rich visual features, which inherently carry general vision, localization, and semantic priors into multi-modal tokens for LLM comprehension. To support robust multi-granularity understanding, we also develop a dedicated data refinement and augmentation pipeline, yielding a high-quality dataset of 1.5M image and 0.6M video region-semantic annotations, including novel region-level streaming video caption data. PAM is designed for lightweightness and efficiency, while also demonstrates strong performance across a diverse range of region understanding tasks. It runs 1.2-2.4x faster and consumes less GPU memory than prior approaches, offering a practical solution for real-world applications. We believe that our effective approach will serve as a strong baseline for future research in region-level visual understanding.

Discussion (0). Sign in to comment.

Forward citations

Cited by 15 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. MICo-150K: A Comprehensive Dataset Advancing Multi-Image Composition

    cs.CV 2025-12 unverdicted novelty 7.0 of 10

    MICo-150K is a new 150K-image dataset with 7 tasks, a De&Re real-image subset, MICo-Bench, and Weighted-Ref-VIEScore metric that improves AI models for generating consistent composites from arbitrary numbers of refere...

  2. FRFDet: Efficient UAV Small Object Detection with Symmetric Sampling and Scalable Fusion

    cs.CV 2026-07 accept novelty 6.0 of 10

    FRFDet improves UAV small-object detection with learnable symmetric Inverse Bidirectional Sampling and capacity-dependent Scale-Feature Relationship Cross-Fusion, reaching SOTA among lightweight models.

  3. See What I Mean: Aligning Vision and Language Representations for Video Fine-grained Object Understanding

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    SWIM aligns cross-attention maps from object nouns to ground-truth masks during training on the new NL-Refer dataset to enable text-only fine-grained video object understanding in MLLMs.

  4. WOW-Seg: A Word-free Open World Segmentation Model

    cs.CV 2026-05 conditional novelty 6.0 of 10

    WOW-Seg proposes a word-free open-world segmentation model using Mask2Token and Cascade Attention Mask modules, reporting 89.7 semantic similarity and 82.4 semantic IoU on LVIS with one-eighth the parameters of prior ...

  5. Enhancing Foundation VLM Robustness to Missing Modality: Scalable Diffusion for Bi-directional Feature Restoration

    cs.AI 2026-02 unverdicted novelty 6.0 of 10

    A diffusion model with dynamic modality gating and cross-modal mutual learning restores missing features in VLMs bi-directionally while preserving the original model's generalization.

  6. Scone: Bridging Composition and Distinction in Subject-Driven Image Generation via Unified Understanding-Generation Modeling

    cs.CV 2025-12 conditional novelty 6.0 of 10

    Scone unifies subject understanding and generation in a two-stage trained model to improve both composition and distinction in multi-subject image generation, outperforming prior open-source models on new benchmarks.

  7. Scone: Bridging Composition and Distinction in Subject-Driven Image Generation via Unified Understanding-Generation Modeling

    cs.CV 2025-12 conditional novelty 6.0 of 10

    Scone adds a semantic-bridge attention-masking step to a unified understanding-generation model, improving subject distinction in multi-candidate reference images, and introduces the SconeEval benchmark.

  8. RefBench-PRO: Perceptual and Reasoning Oriented Benchmark for Referring Expression Comprehension

    cs.CV 2025-12 conditional novelty 6.0 of 10

    RefBench-PRO organizes REC into attribute, position, interaction, relation, commonsense, and reject tasks; no tested MLLM exceeds 72%, and Ref-R1 raises Qwen2.5-VL-7B from 57.6 to 69.4 on it.

  9. Strefer: Empowering Video LLMs with Space-Time Referring and Reasoning via Synthetic Instruction Data

    cs.CV 2025-09 conditional novelty 6.0 of 10

    Adding Strefer's synthetic space-time reference questions to video instruction tuning improves mask-referred description/QA, timestamp QA, and temporal reasoning over a video-LLM baseline.

  10. Watch, Remember, Reason: Human-View Video Understanding with MLLMs

    cs.CV 2026-06 unverdicted novelty 4.0 of 10

    This is a survey that frames video MLLM research via a human-view formulation of perceptual representations, memory states, reasoning traces, and predictions, then reviews methods, datasets, benchmarks, and open problems.

  11. OpenWorldLib: A Unified Codebase and Definition of Advanced World Models

    cs.CV 2026-04 conditional novelty 4.0 of 10

    OpenWorldLib defines world models as perception-centered systems with interaction and long-term memory, and provides a modular inference codebase unifying interactive video, 3D, reasoning, and VLA tasks.

  12. OpenWorldLib: A Unified Codebase and Definition of Advanced World Models

    cs.CV 2026-04 unverdicted novelty 4.0 of 10

    OpenWorldLib offers a standardized codebase and definition for world models that combine perception, interaction, and memory to understand and predict the world.

  13. Describe Anything Model for Visual Question Answering on Text-rich Images

    cs.CV 2025-07 conditional novelty 4.0 of 10

    DAM-QA aggregates answers from full-image and sliding-window views of the Describe Anything Model with a weighted vote, improving text-rich VQA on some benchmarks but not all.

  14. LMMs Meet Object-Centric Vision: Understanding, Segmentation, Editing and Generation

    cs.CV 2026-04 unverdicted novelty 3.0 of 10

    This review organizes literature on large multimodal models and object-centric vision into four themes—understanding, referring segmentation, editing, and generation—while summarizing paradigms, strategies, and challe...

  15. Empowering Multimodal LLMs with External Tools: A Comprehensive Survey

    cs.CV 2025-08 unverdicted novelty 2.0 of 10

    A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.

Pith tools