Rea2Seg turns image segmentation into candidate mask discovery from MLLM attention followed by MLLM-based comparative scoring and selection, plus a new multi-dimensional reasoning benchmark ReasonSeg-SGDR.
hub
Personalize segment anything model with one shot
19 Pith papers cite this work. Polarity classification is still indexing.
19
Pith papers citing it
abstract
Driven by large-data pre-training, Segment Anything Model (SAM) has been demonstrated as a powerful and promptable framework, revolutionizing the segmentation models. Despite the generality, customizing SAM for specific visual concepts without man-powered prompting is under explored, e.g., automatically segmenting your pet dog in different images. In this paper, we propose a training-free Personalization approach for SAM, termed as PerSAM. Given only a single image with a reference mask, PerSAM first localizes the target concept by a location prior, and segments it within other images or videos via three techniques: target-guided attention, target-semantic prompting, and cascaded post-refinement. In this way, we effectively adapt SAM for private use without any training. To further alleviate the mask ambiguity, we present an efficient one-shot fine-tuning variant, PerSAM-F. Freezing the entire SAM, we introduce two learnable weights for multi-scale masks, only training 2 parameters within 10 seconds for improved performance. To demonstrate our efficacy, we construct a new segmentation dataset, PerSeg, for personalized evaluation, and test our methods on video object segmentation with competitive performance. Besides, our approach can also enhance DreamBooth to personalize Stable Diffusion for text-to-image generation, which discards the background disturbance for better target appearance learning. Code is released at https://github.com/ZrrSkywalker/Personalize-SAM
hub tools
citation-role summary
background 1
citation-polarity summary
roles
background 1polarities
background 1representative citing papers
FindIt is the first comprehensive benchmark for evaluating generalist MLLMs on promptable object detection, referring expression detection, instance-level detection, and video detection with standardized parsable outputs.
Set-of-Mark prompting marks segmented image regions with alphanumerics and masks to let GPT-4V achieve state-of-the-art zero-shot results on referring expression comprehension and segmentation benchmarks like RefCOCOg.
CLIPix repurposes CLIP by tracing classification activations, applying noise-resistant correction, and localization embedding to reach SOTA zero-shot binary open-set segmentation on PASCAL-5i and COCO-20i.
ExACT combines a Vision Exemplar-based Calibrator and Structure-Aware Refiner to improve training-free visual grounding of language descriptions in remote sensing images using frozen MLLMs and SAM.
M2C turns SAM3 into an auto-promptable annotator for medical few-shot segmentation via test-time concept embedding optimization and uncertainty-driven active refinement.
A training-free CD-FSS framework built on DINOv3 with SAFR, ASE, and HPM modules reports state-of-the-art results on four target-domain datasets without any parameter updates.
DHANet uses multi-scale spatial and channel aggregation with a probabilistic bank to mitigate over-alignment and reports state-of-the-art results on four target datasets for cross-domain few-shot segmentation.
Presents MMIO benchmark and RTVP method achieving state-of-the-art 42.2% AP in zero-shot industrial defect detection.
Presents MMIOC-1M benchmark with 1M+ samples across 14 super-categories and RTVPNet with domain projection, sparse sampling, and bidirectional interaction, claiming SOTA on MMIOC-1M, LVIS, and COCO.
DeCoDrift stabilizes decoder coupling in closed-loop foundation segmentation by constraining prompt updates without retraining or ground truth.
Introduces LCA dual-branch adapter and pairwise loss for lighting-robust SAM instance segmentation, validated on existing benchmarks plus a new Unity synthetic dataset.
SegRAG is a training-free retrieval-augmented framework that extracts class-specific point prompts from a filtered DINOv3 feature bank to boost SAM3 semantic segmentation performance on standard and agricultural benchmarks.
PanoSAM2 adapts SAM2 with a Pano-Aware Decoder, Distortion-Guided Mask Loss, and Long-Short Memory Module to improve 360 video object segmentation, reporting +5.6 and +6.7 gains over base SAM2 on two benchmarks.
RPG-SAM improves one-shot polyp segmentation by weighting high-fidelity support features and dynamically adjusting thresholds via morphological consensus, yielding 5.56% mIoU gain on Kvasir.
AHCQ-SAM introduces ACNR, HLUQ, CAG, and LNQ quantization techniques that deliver 15.2% mAP gain on 4-bit SAM-B and 14.01% J&F gain on 4-bit SAM2-Tiny versus prior PTQ methods.
A two-stage RL method with information gaps and grounding loss trains MLLMs to focus on and precisely crop relevant image regions, yielding SOTA results on high-resolution VQA benchmarks.
OmniCD proposes a multimodal semantic-guided framework for remote sensing change detection supporting binary to zero-shot tasks, plus the RSITCD dataset, with claimed SOTA performance.
citing papers explorer
-
Reason Twice: Segmentation via Candidate Discovery and Comparative Reasoning
Rea2Seg turns image segmentation into candidate mask discovery from MLLM attention followed by MLLM-based comparative scoring and selection, plus a new multi-dimensional reasoning benchmark ReasonSeg-SGDR.
-
FindIt: A Format-Informed Visual Detection Benchmark for Generalist Multimodal LLMs
FindIt is the first comprehensive benchmark for evaluating generalist MLLMs on promptable object detection, referring expression detection, instance-level detection, and video detection with standardized parsable outputs.
-
Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V
Set-of-Mark prompting marks segmented image regions with alphanumerics and masks to let GPT-4V achieve state-of-the-art zero-shot results on referring expression comprehension and segmentation benchmarks like RefCOCOg.
-
Repurposing CLIP to Localize at Pixel Level
CLIPix repurposes CLIP by tracing classification activations, applying noise-resistant correction, and localization embedding to reach SOTA zero-shot binary open-set segmentation on PASCAL-5i and COCO-20i.
-
ExACT: Exemplar-Driven Calibrated Refinement for Training-Free Visual Grounding in Remote Sensing Images
ExACT combines a Vision Exemplar-based Calibrator and Structure-Aware Refiner to improve training-free visual grounding of language descriptions in remote sensing images using frozen MLLMs and SAM.
-
Mask to Concept: Auto-Promptable SAM3 via Efficient Test-Time Concept Embedding Search for Few-Shot Annotation
M2C turns SAM3 into an auto-promptable annotator for medical few-shot segmentation via test-time concept embedding optimization and uncertainty-driven active refinement.
-
Training-free Cross-domain Few-shot Segmentation via Robust Semantic Representation and Matching
A training-free CD-FSS framework built on DINOv3 with SAFR, ASE, and HPM modules reports state-of-the-art results on four target-domain datasets without any parameter updates.
-
Hierarchical Spatial and Channel Aggregation for Cross-domain Few-shot Segmentation
DHANet uses multi-scale spatial and channel aggregation with a probabilistic bank to mitigate over-alignment and reports state-of-the-art results on four target datasets for cross-domain few-shot segmentation.
-
Zero-Shot Learning in Industrial Scenarios: New Large-Scale Benchmark, Challenges and Baseline
Presents MMIO benchmark and RTVP method achieving state-of-the-art 42.2% AP in zero-shot industrial defect detection.
-
Unification of Closed-Open Industrial Detection Scenarios: New Large-Scale Benchmarks,Challenges and Baselines
Presents MMIOC-1M benchmark with 1M+ samples across 14 super-categories and RTVPNet with domain projection, sparse sampling, and bidirectional interaction, claiming SOTA on MMIOC-1M, LVIS, and COCO.
-
DeCoDrift: Stabilizing Decoder Coupling in Closed-Loop Foundation Segmentation
DeCoDrift stabilizes decoder coupling in closed-loop foundation segmentation by constraining prompt updates without retraining or ground truth.
-
Lighting-aware Unified Model for Instance Segmentation
Introduces LCA dual-branch adapter and pairwise loss for lighting-robust SAM instance segmentation, validated on existing benchmarks plus a new Unity synthetic dataset.
-
SegRAG: Training-Free Retrieval-Augmented Semantic Segmentation
SegRAG is a training-free retrieval-augmented framework that extracts class-specific point prompts from a filtered DINOv3 feature bank to boost SAM3 semantic segmentation performance on standard and agricultural benchmarks.
-
PanoSAM2: Lightweight Distortion- and Memory-aware Adaptions of SAM2 for 360 Video Object Segmentation
PanoSAM2 adapts SAM2 with a Pano-Aware Decoder, Distortion-Guided Mask Loss, and Long-Short Memory Module to improve 360 video object segmentation, reporting +5.6 and +6.7 gains over base SAM2 on two benchmarks.
-
RPG-SAM: Reliability-Weighted Prototypes and Geometric Adaptive Threshold Selection for Training-Free One-Shot Polyp Segmentation
RPG-SAM improves one-shot polyp segmentation by weighting high-fidelity support features and dynamically adjusting thresholds via morphological consensus, yielding 5.56% mIoU gain on Kvasir.
-
AHCQ-SAM: Toward Accurate and Hardware-Compatible Post-Training Segment Anything Model Quantization
AHCQ-SAM introduces ACNR, HLUQ, CAG, and LNQ quantization techniques that deliver 15.2% mAP gain on 4-bit SAM-B and 14.01% J&F gain on 4-bit SAM2-Tiny versus prior PTQ methods.
-
Learning to Focus and Precise Cropping: A Reinforcement Learning Framework with Information Gaps and Grounding Loss for MLLMs
A two-stage RL method with information gaps and grounding loss trains MLLMs to focus on and precisely crop relevant image regions, yielding SOTA results on high-resolution VQA benchmarks.
-
OmniCD: A Foundational Framework for Remote Sensing Image Change Detection Guided by Multimodal Semantics
OmniCD proposes a multimodal semantic-guided framework for remote sensing change detection supporting binary to zero-shot tasks, plus the RSITCD dataset, with claimed SOTA performance.
- Functionalization via Structure Completion and Motion Rectification