Masked-attention Mask Transformer for Universal Image Segmentation
Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:2ES3ABWKrecord.jsonopen to challenge →
read the original abstract
Image segmentation is about grouping pixels with different semantics, e.g., category or instance membership, where each choice of semantics defines a task. While only the semantics of each task differ, current research focuses on designing specialized architectures for each task. We present Masked-attention Mask Transformer (Mask2Former), a new architecture capable of addressing any image segmentation task (panoptic, instance or semantic). Its key components include masked attention, which extracts localized features by constraining cross-attention within predicted mask regions. In addition to reducing the research effort by at least three times, it outperforms the best specialized architectures by a significant margin on four popular datasets. Most notably, Mask2Former sets a new state-of-the-art for panoptic segmentation (57.8 PQ on COCO), instance segmentation (50.1 AP on COCO) and semantic segmentation (57.7 mIoU on ADE20K).
This paper has not been read by Pith yet.
Forward citations
Cited by 16 Pith papers
-
ChartREG++: Towards Benchmarking and Improving Chart Referring Expression Grounding under Diverse referring clues and Multi-Target Referring
ChartREG++ creates a new multi-target chart grounding benchmark with diverse cues and a code-driven synthesis pipeline for accurate masks, yielding a model that outperforms baselines and generalizes to real ChartQA charts.
-
Beyond Accuracy: Benchmarking Cross-Task Consistency in Unified Multimodal Models
XTC-Bench reveals that strong performance on generation or understanding tasks in unified multimodal models does not guarantee cross-task semantic consistency, which instead depends on how tightly coupled the learning...
-
Hilti-Trimble-Oxford Dataset: 360 Visual-Inertial Benchmark with Floor Plan Priors for SLAM and Localization
A 30-sequence 360-degree visual-inertial dataset from an active construction site with LiDAR ground truth and floor plans, benchmarked via an open challenge with 84 teams.
-
Beyond Aesthetics: Quantifying Information Loss in Turbid Scenes
Introduces the TUB dataset of 1320 real turbid underwater images and PCD metric showing strong correlation with instance segmentation performance where standard metrics fail.
-
Better Queries, Cheaper Attention: Adapting Transformers for Efficient Sparse Reconstruction
A geometry-aware dynamic-query transformer decoder with Local Strided Cross-Attention raises track reconstruction efficiency from 94.1% to 98.1%, halves latency, and cuts memory use by over 10x versus fixed-query base...
-
A Joint Finite-Sample Certificate for Adaptive Selective Conformal Risk Control
A joint finite-sample certificate for adaptive selective conformal risk control that treats selected risk as a ratio and couples empirical-Bernstein, Clopper-Pearson, and closeness bounds.
-
SegRAG: Training-Free Retrieval-Augmented Semantic Segmentation
SegRAG is a training-free retrieval-augmented framework that extracts class-specific point prompts from a filtered DINOv3 feature bank to boost SAM3 semantic segmentation performance on standard and agricultural benchmarks.
-
SegRAG: Training-Free Retrieval-Augmented Semantic Segmentation
SegRAG augments SAM3 with class-specific point prompts retrieved via DINOv3 features and filtered by ICCD, using TSG at inference to improve open-vocabulary segmentation.
-
ChartREG++: Towards Benchmarking and Improving Chart Referring Expression Grounding under Diverse referring clues and Multi-Target Referring
ChartREG++ benchmark for multi-target chart referring expression grounding with diverse clues plus a plotting-code synthesis pipeline for accurate masks that improves model performance.
-
Zero-shot World Models Are Developmentally Efficient Learners
A zero-shot visual world model trained on one child's experience achieves broad competence on physical understanding benchmarks while matching developmental behavioral patterns.
-
Mogao: An Omni Foundation Model for Interleaved Multi-Modal Generation
Mogao presents a causal unified model with deep fusion, dual encoders, and interleaved position embeddings that achieves strong performance on multi-modal understanding, text-to-image generation, and coherent interlea...
-
Phast: Simultaneous reconstruction of photoelectron count and time profiles from PMT waveforms via machine learning
Phast applies a transformer encoder plus count-conditioned query decoder to reconstruct photoelectron count and time profiles from simulated PMT waveforms on toy Monte Carlo datasets.
-
Vision Transformer-Conditioned UNet for Domain-Adaptive Semantic Segmentation
ViTC-UNet adapts frozen ViT representations to biomedical semantic segmentation by conditioning a UNet via learnable tokens and two-way attention decoding.
-
Revitalizing Dense Material Segmentation: Stabilized Vision Transformers and the Generalization Paradox
Stabilized SegFormer-B5 reaches 0.4572 mIoU SOTA on original Apple DMS split; 80/10/10 split reaches 0.5276 mIoU but degrades real-world OOD performance per qualitative review.
-
KAYRA: A Microservice Architecture for AI-Assisted Karyotyping with Cloud and On-Premise Deployment
KAYRA packages a cascade of EfficientNet-B5 + U-Net, Mask R-CNN, and ResNet-18 models into a microservice architecture that supports both cloud and on-premise deployment and reaches 98.91% segmentation accuracy in a p...
-
Efficiently Linking Real Scenes with Synthetic Data Generation for AI-based Cognitive Robotics and Computer Vision Applications
The paper reviews limits in AI vision for robotics and describes work-in-progress on bridging sim-to-real domain gaps by linking real and synthetic training data.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.