Pith. sign in

REVIEW 4 major objections 6 minor 80 references

PanSt3R: Multi-view Consistent Panoptic Segmentation

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read PanSt3R does 3D geometry plus multi-view panoptic segmentation in one forward pass from unposed images, with no camera parameters or test-time optimization, and beats previous methods on four benchmarks at orders-of-magnitude lower cost.

desk verdict Genuine single-forward-pass multi-view panoptic segmentation from unposed images, with a solid architecture and ablation study, but the SOTA headline is undercut by an evaluation that leans on posed 3DGS rendering and LUDVIG. read the letter →

arxiv 2506.21348 v1 pith:CNPEI6DW submitted 2025-06-26 cs.CV

Lojze Zust , Yohann Cabon , Juliette Marrie , Leonid Antsfeld , Boris Chidlovskii , Jerome Revaud , Gabriela Csurka This is my paper
classification cs.CV
keywords panopticsegmentation3Dsceneunderstandingmulti-viewconsistencyDUSt3RMask2FormerunposedimagesGaussianSplattingquery-based
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

PanSt3R claims that panoptic segmentation of a 3D scene—giving every object instance and background region a class label while keeping one identity per object across all views—can be done in a single network forward pass on an unordered batch of images, with no camera poses, no depth input, and no per-scene optimization. The paper argues that existing methods, which lift noisy per-frame 2D panoptic predictions into a NeRF or a 3D Gaussian field at test time, waste compute and sacrifice accuracy by splitting segmentation from reconstruction. On the PanLift benchmark (ScanNet, Hypersim, Replica) and on ScanNet++, PanSt3R reports panoptic quality above Panoptic Lifting, Contrastive Lift, and PLGS, while running in minutes instead of hours (about 2.3 minutes on ScanNet++ versus roughly 460–500 minutes for the NeRF baselines). The architecture concatenates frozen DINOv2 semantic features and MUSt3R 3D features into multi-view frame tokens, decodes a shared pool of 200 learnable instance queries with a Mask2Former-style transformer, and merges the resulting masks with a quadratic binary optimization that replaces the standard single-image heuristic, which the ablations show is a major source of the gain.

What carries the argument

The load-bearing mechanism is a single shared pool of $Q = 200$ learnable instance queries in a Mask2Former-style transformer decoder. The queries cross-attend simultaneously to frame tokens built from concatenated DINOv2 and MUSt3R encoder/decoder features across all input views, so each query is pressed to represent the same 3D object or stuff region in every frame, yielding instance masks that are globally consistent by construction. A second mechanism carries the final labeling: mask merging as a quadratic unconstrained binary optimization (QUBO), solved by simulated annealing, in which each candidate mask contributes its covered area as a reward and each pair of overlapping masks pays a penalty proportional to their intersection; the solver selects the subset of masks with maximum global score. The paper identifies this merging step, rather than the network itself, as the point where standard methods lose multi-view consistency.

What would settle it

Take a scene with more than 200 clearly separated, visually near-identical objects (a warehouse aisle of stacked boxes, a row of identical chairs) and run PanSt3R on a set of unposed views: if the output contains at most 200 instance masks with two distinct objects sharing one instance ID across views, the fixed query budget is the binding constraint. A complementary check on the feature side: suppress the MUSt3R feature branch at inference on a two-object scene and measure whether the same query still binds to the same object across views—the paper's own ablation predicts a large consistency drop, which would confirm that geometric features, not semantics, carry cross-view identity.

Watch

Extended reading notes

Core claim

The paper's central claim is that 3D reconstruction and panoptic segmentation are one reasoning task—decomposing a scene into geometry and persistent instances—and should be executed by a single network rather than by fusing off-the-shelf 2D predictions into an optimized 3D field. PanSt3R feeds an unordered set of RGB frames through frozen DINOv2 and MUSt3R encoders, concatenates their token maps into compact frame tokens, and decodes a fixed set of 200 learnable queries with a mask transformer that cross-attends to all frames at once. Each query is trained to bind to one physical object or stuff region, so the same instance ID is emitted for that object in every view where it appears; class labels come from cosine similarity between query embeddings and SigLIP text embeddings, giving an open-vocabulary head that can be trained on heterogeneous datasets. The decoded soft masks are merged by a QUBO solver that maximizes covered area while penalizing pairwise overlap, a step the paper shows is far more effective than the confidence-thresholding and voting scheme inherited from single-image segmentation. Optionally, the labeled point cloud can be uplifted into 3D Gaussians with LUDVIG, supervised by an auxiliary rendered-label regularization loss, to render panoptic predictions from novel viewpoints.

Load-bearing premise

The load-bearing premise is that a fixed pool of 200 learnable instance queries, decoded from frozen DINOv2 and MUSt3R features, can represent every object and stuff region in the scene and that those features will bind the same query to the same physical object across all views; the paper sets $Q = 200$ without ablating it and does not test where this binding starts to fail.

Editorial extensions

If this is right

  • Scene-level panoptic segmentation no longer needs camera calibration or per-scene optimization; one forward pass over the image collection produces both geometry and consistent instance labels.
  • Replacing the standard MaskFormer-style mask-selection heuristic with global QUBO selection is worth a large PQ gain (about 5–6 points on Hypersim and ScanNet, more on Replica), so the merging step is where multi-view consistency is won or lost.
  • Because instance IDs are produced directly and views are processed with a shared query set, the output is immediately usable as supervision for downstream tasks like robot interaction, augmented reality, and 3D scene editing, without a separate association step.
  • The open-vocabulary classification head lets one trained model transfer across datasets with different class vocabularies (COCO, ADE20K, ScanNet++, Infinigen), so training data can be pooled from heterogeneous sources.
  • Novel-view panoptic predictions can be obtained either by running the same network on rendered views or by uplifting labels into 3D Gaussians with LUDVIG, with uplifting generally improving quality by aggregating evidence over views.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the query pool is fixed at $Q = 200$ with no ablation reported, a scene containing more than 200 distinct instances would force two objects to share one query; measuring PQ on a deliberately instance-dense scene (a crowd, a warehouse aisle) would map the method's ceiling.
  • The same shared-query architecture could be extended to streaming video by carrying queries across temporal windows, turning the model into an online panoptic tracker with no separate data-association module.
  • The open-vocabulary head suggests a cheap extension: querying the decoder with arbitrary text embeddings (not just class names) could yield 3D grounding from free-form language, since each query already encodes a scene region.
  • The ablations attribute the largest loss (14.7 PQ points) to removing the MUSt3R 3D features and only 4 points to removing DINOv2, implying cross-view identity binding is carried mostly by geometric features—a testable prediction: perturbing MUSt3R features should break instance consistency across views more than perturbing semantic features.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes PanSt3R, a network that combines frozen MUSt3R and DINOv2 feature extractors with a Mask2Former-style mask transformer using a shared set of learnable queries, to jointly predict 3D point maps and multi-view panoptic segmentations from an unconstrained collection of unposed images in a single forward pass. A QUBO-based mask selection step replaces the standard MaskFormer merging procedure, and an optional 3DGS+LUDVIG stage is introduced to produce novel-view panoptic predictions. The method is evaluated on ScanNet, Hypersim, Replica, and ScanNet++, reporting large gains over NeRF/3DGS lifting baselines at substantially lower runtime, with extensive ablations of the feature backbones, QUBO merging, 3DGS regularization, keyframe count, and model resolution.

Significance. If the central claims hold, PanSt3R is a practically significant step: it removes pose estimation and per-scene optimization from panoptic 3D scene understanding, shows that foundation-model features and a shared query set can yield multi-view consistent masks, and provides a simple alternative to the dominant 2D-lifting pipelines. The paper is extensive and well structured, with multiple benchmarks, thorough ablations, pseudo-code, and a comparison against a point-cloud segmentation baseline. The main risks are that the headline 'state-of-the-art' numbers are obtained through a posed 3DGS rendering/optimization pipeline rather than the pose-free single forward pass, and that the QUBO formulation as written is incomplete. These issues are load-bearing but appear fixable with additional experiments and clarification, so the recommendation is major revision rather than rejection.

major comments (4)
  1. [§4.3, §4.4, Tables 2–3] In Sec. 4.3, the PanSt3R rows in Table 2 are obtained by running the network on test views rendered with vanilla 3DGS built from posed training images (footnote 2). The identical values in Table 3 (46.7/43.2/55.8) and Appendix Table 8 (rendered, 512) indicate that the ScanNet++ comparison is also on rendered views. Thus the SOTA numbers for the pose-free method are produced by a pipeline that requires camera poses and a roughly 35-minute 3DGS optimization, which is exactly the kind of test-time overhead the abstract says is eliminated. The only direct forward-pass evaluation on original held-out test images (Appendix Table 8, PanSt3R 512, 57.3 PQ) is reported without PanLift or Contrastive Lift baselines. In addition, the 'orders of magnitude faster' timing in Tables 2–3 compares the roughly 2–4 minute forward pass to the baselines' full optimization, while the rendered-evaluation variant incurs an extra +35 minute 3DGS build (Table 2). Please either add a direct forward-pass comparison on original test views against the same baselines, or restrict the SOTA claim to the rendered-view protocol and include the required 3DGS time in the runtime comparison.
  2. [§3.2, QUBO objective] The QUBO maximization in Sec. 3.2 has no term penalizing the number of selected proposals and no confidence threshold in the objective. For any two disjoint masks with positive areas, selecting both strictly increases the objective, so the global maximizer would select every proposal with positive area unless overlap penalties dominate. This would admit low-confidence spurious masks and is inconsistent with the intended filtering behavior. The paper does not describe a confidence pre-filter, a cardinality penalty, or an equivalent constraint. Please specify the exact candidate set, any thresholding, and the full objective actually optimized, and verify that the reported QUBO gains are not an artifact of an unstated heuristic.
  3. [§3.1 / Appendix D, pseudo-code] The model fixes Q=200 shared instance queries (Sec. 3.1 and pseudo-code) with no ablation of Q and no analysis of scenes containing more than 200 instances or of cross-view query-to-instance binding failures. Since each query must select the same physical object across views through frozen DINOv2/MUSt3R features, the capacity and binding reliability are load-bearing for the multi-view consistency claim. Please add ablations of Q (e.g., 100, 300, 500), report the distribution of instance counts in the evaluation scenes, and provide a quantitative measure of cross-view instance-ID consistency.
  4. [Abstract / §3.3] The abstract and introduction state that PanSt3R 'eliminates the need for test-time optimization' and works 'without requiring any camera parameters.' However, the best benchmark results (Table 2: PanSt3R+LUDVIG, 66.3 PQ on Hypersim and 67.5 PQ on ScanNet) are obtained by optimizing a 3DGS scene with poses and the panoptic regularization loss of Eq. (6) before uplifting labels (Sec. 3.3). The paper does present LUDVIG as an optional novel-view prediction strategy, but the unqualified claims should be revised to separate the pose-free single-forward-pass setting (input-view prediction) from the posed, optimization-based novel-view setting in which the SOTA numbers are obtained.
minor comments (6)
  1. [Table 7] Table 7 cites 'PanoLift [39]' but reference [39] is OpenScene, not Panoptic Lifting; the correct citation appears to be [50]. Please fix the citation and standardize the name ('PanLift' vs 'PanoLift').
  2. [§3.1] The symbol for the joint token dimension is introduced as d_t but then given as d_f = 768; please unify the notation.
  3. [§3.2] In Sec. 3.2, Q_i and Q_{i,j} are used in the displayed objective before their definitions; move the definitions before the display or number the equations for readability.
  4. [§3.2] The simulated-annealing solver for the QUBO is described only as 'simulated annealing'; specify the temperature schedule, number of iterations, and initialization for reproducibility, or state that a black-box solver is used.
  5. [Appendix Table 8 / §4.4] Appendix Table 8 shows PanSt3R(orig) at 512 achieving 57.3 PQ, which is higher than PanSt3R+LUDVIG (54.8); the main text's emphasis on LUDVIG improvements in Sec. 4.4 should acknowledge that on original test views the direct forward pass is actually better.
  6. [Figures 3 and 5] The captions state that colors and their nuances denote different classes and object instances, but no legend or consistent color mapping is provided; this makes the qualitative figures difficult to interpret.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: PanSt3R's predictions are ground-truth supervised and evaluated on held-out images; the gap between the pose-free claim and the posed-3DGS benchmark protocol is an evidence issue, not circularity.

full rationale

PanSt3R is an empirically trained model, not a derivation whose output reduces to its input by construction. The panoptic predictions are supervised by ground-truth labels through L = λc Lcls + λd Ldice + λb Lbce on datasets such as COCO, ADE20K, ScanNet++, and ASE, and the reported scene-PQ scores are computed against held-out ground truth. The QUBO mask merging in Sec. 3.2 and the LUDVIG uplifting in Sec. 3.3 consume the model's own predicted masks, but they are post-processing or label-transfer steps whose outputs are then compared with independent GT; using one's own predictions as input to a later stage is self-referential but not circular. The paper does rely on prior work by overlapping authors (MUSt3R, DUSt3R, LUDVIG), and these components are load-bearing, but they are external published models with publicly available code and are not invoked as a uniqueness theorem or as a way to forbid alternative explanations of the target result. No equation in the paper is equivalent to its own input, and no fitted parameter is renamed as a prediction. The skeptic's concern that Table 2 reports results on posed-3DGS rendered views or with LUDVIG optimization rather than the pure pose-free single forward pass is a mismatch between the claimed evaluation setting and the benchmark protocol; that is a correctness and evidence concern, not a circularity. Direct forward-pass numbers on original test views are only given in App. Tab. 8 without baseline comparisons, which weakens the SOTA claim but does not make the derivation circular. Overall, the central contribution is independently evaluated, so the circularity score is low; the small score reflects only the presence of self-citations that are not load-bearing in a circular sense.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method is an empirical pipeline composed of existing components (MUSt3R, DINOv2, Mask2Former-style decoder, 3DGS, LUDVIG). No new physical entities are postulated. The main tunable components are the QUBO penalty, the 3DGS regularization weight, the query count, and the keyframe count.

free parameters (4)
  • QUBO overlap penalty lambda_p = 2
    Hand-chosen penalty multiplying the overlap cost in the QUBO objective; no ablation is reported for this value. It directly controls how aggressively overlapping masks are suppressed and hence affects the final panoptic masks.
  • 3DGS panoptic regularization weight lambda = 1
    Weight of the L1 panoptic rendering loss during 3DGS optimization; set to 1 in all experiments. Table 5 only ablates the loss on and off, not the value.
  • Number of instance queries Q = 200
    Maximum number of instances that can be represented; scenes with more than 200 things and stuff regions cannot be fully segmented. The value is fixed in the problem statement and not ablated.
  • Number of keyframes = 50
    Keyframes selected via farthest-point sampling on retrieval features; the ablation in Table 6 shows a plateau at 50 for ScanNet++, but larger scenes are not stress-tested.
assumptions (4)
  • domain assumption MUSt3R and DINOv2 features concatenated through an MLP encode globally consistent 3D geometry and semantics across all input views.
    The whole cross-view binding of queries relies on this; Section 3.1 states the features are concatenated and passed to the mask transformer, with no explicit geometric alignment or pose within the network.
  • ad hoc to paper A single set of 200 learnable queries is sufficient to simultaneously represent all object instances and stuff regions in a scene.
    The paper fixes Q=200 in the problem statement without ablating it, assuming scene complexity is below this cap.
  • domain assumption The training datasets (ScanNet++, ASE, Infinigen, COCO, ADE20k) provide enough coverage for generalization to the evaluation benchmarks.
    The method is trained on this mix and evaluated on ScanNet, Hypersim, Replica, and ScanNet++; no zero-shot or small-data analysis is provided.
  • standard math Simulated annealing provides a near-optimal solution to the QUBO problem in practice.
    QUBO is NP-hard; the approximation quality is not analyzed and no annealing schedule or iteration count is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of PanSt3R: Multi-view Consistent Panoptic Segmentation." pith.science (2026). https://pith.science/paper/CNPEI6DW

@misc{pith2026250621348,
  author       = {Pith},
  title        = {Pith review of: PanSt3R: Multi-view Consistent Panoptic Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CNPEI6DW}},
  note         = {Machine review of arXiv:2506.21348}
}
read the original abstract

Panoptic segmentation of 3D scenes, involving the segmentation and classification of object instances in a dense 3D reconstruction of a scene, is a challenging problem, especially when relying solely on unposed 2D images. Existing approaches typically leverage off-the-shelf models to extract per-frame 2D panoptic segmentations, before optimizing an implicit geometric representation (often based on NeRF) to integrate and fuse the 2D predictions. We argue that relying on 2D panoptic segmentation for a problem inherently 3D and multi-view is likely suboptimal as it fails to leverage the full potential of spatial relationships across views. In addition to requiring camera parameters, these approaches also necessitate computationally expensive test-time optimization for each scene. Instead, in this work, we propose a unified and integrated approach PanSt3R, which eliminates the need for test-time optimization by jointly predicting 3D geometry and multi-view panoptic segmentation in a single forward pass. Our approach builds upon recent advances in 3D reconstruction, specifically upon MUSt3R, a scalable multi-view version of DUSt3R, and enhances it with semantic awareness and multi-view panoptic segmentation capabilities. We additionally revisit the standard post-processing mask merging procedure and introduce a more principled approach for multi-view segmentation. We also introduce a simple method for generating novel-view predictions based on the predictions of PanSt3R and vanilla 3DGS. Overall, the proposed PanSt3R is conceptually simple, yet fast and scalable, and achieves state-of-the-art performance on several benchmarks, while being orders of magnitude faster than existing methods.

Figures

Figures reproduced from arXiv: 2506.21348 by the authors.

Figure 1
Figure 1. PanSt3R jointly predicts 3D geometry and panoptic segmentation of a scene in a single forward pass. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Architecture of PanSt3R. First, the input un-posed RGB frames are passed through pretrained DINOv2 [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Qualitative examples of novel-view panoptic [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison of novel-view panop [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Qualitative results of novel-view panoptic segmentation on various [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Visual comparison of the effects of panoptic [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Qualitative panoptic segmentation results [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: Point-cloud labeling: (a) output of PanSt3R, [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Further qualitative example of PanSt3R’s seg [PITH_FULL_IMAGE:figures/full_fig_p015_9.png]
Figure 10
Figure 10. Figure 10: Pseudo-code of PanSt3R. def mask_transformer(in_queries, frame_tokens): queries = in_queries for i in range(num_layers): # 1. Cross attention with frame tokens queries = masked_cross_attention(queries, frame_tokens) # 2. Self attention queries = self_attention(queries…
Figure 11
Figure 11. Figure 11: Pseudo-code of PanSt3R: mask transformer architecture [PITH_FULL_IMAGE:figures/full_fig_p016_11.png]
Figure 12
Figure 12. Figure 12: Pseudo-code of PanSt3R: prediction heads [PITH_FULL_IMAGE:figures/full_fig_p017_12.png]
Figure 13
Figure 13. Figure 13: Pseudo-code of PanSt3R: upscaler architecture [PITH_FULL_IMAGE:figures/full_fig_p017_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

80 extracted references · 73 canonical work pages

  1. [1]

    Frost, Luke Holland, Camp- bell Orme, Jakob Engel, Edward Miller, Richard A

    Armen Avetisyan, Christopher Xie, Henry Howard- Jenkins, Tsun-Yi Yang, Samir Aroudj, Suvam Patra, Fuyang Zhang, Duncan P. Frost, Luke Holland, Camp- bell Orme, Jakob Engel, Edward Miller, Richard A. New- combe, and Vasileios Balntas. SceneScript: Reconstruct- ingScenesWithAnAutoregressiveStructuredLanguage Model. arXiv:2403.13064, 2024. 6

    arXiv 2024

  2. [2]

    Henriques, Andrew Zisserman, and Andrea Vedaldi

    Yash Bhalgat, Iro Laina, João F. Henriques, Andrew Zisserman, and Andrea Vedaldi. Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusion. InNeurIPS, 2024. 2, 3, 5, 7, 8

    work page 2024

  3. [3]

    MUSt3R: Multi-view Network for Stereo 3D Reconstruction

    Yohann Cabon, Lucas Stoffl, Leonid Antsfeld, Gabriela Csurka, Boris Chidlovskii, Jerome Revaud, and Vincent Leroy. MUSt3R: Multi-view Network for Stereo 3D Reconstruction. InCVPR, 2025. 3, 6, 9 Original Image w/o regularization with regularization Figure 6: Visual comparison of the effects of panoptic 3DGS regularization on ScanNet++. Results are shown fo...

    work page 2025

  4. [4]

    End-to-End Object Detection with Trans- formers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-End Object Detection with Trans- formers. InECCV, 2020. 2

    work page 2020

  5. [5]

    TensoRF: Tensorial Radiance Fields

    Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. TensoRF: Tensorial Radiance Fields. InECCV,

    work page

  6. [6]

    Ting Chen, Lala Li, Saurabh Saxena, Geoffrey Hinton, and David J. Fleet. A Generalist Framework for Panoptic Segmentation of Images and Videos. InICCV, 2023. 1, 2

    work page 2023

  7. [7]

    Open-vocabulary Panoptic Seg- mentation with Embedding Modulation

    Xi Chen, Shuang Li, Ser-Nam Lim, Antonio Torralba, and Hengshuang Zhao. Open-vocabulary Panoptic Seg- mentation with Embedding Modulation. InICCV, 2023. 2

    work page 2023

  8. [8]

    Collins, Yukun Zhu, Ting Liu, Thomas S

    Bowen Cheng, Maxwell D. Collins, Yukun Zhu, Ting Liu, Thomas S. Huang, Hartwig Adam, and Liang-Chieh Chen. Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation. In CVPR, 2020. 1, 2

    work page 2020

Show all 80 references
  1. [9]

    Schwing, and Alexander Kirillov

    Bowen Cheng, Alexander G. Schwing, and Alexander Kirillov. Per-Pixel Classification is Not All You Need for Semantic Segmentation. InNeurIPS, 2021. 3, 5, 9

    2021

  2. [10]

    Buol: A bottom-up framework with occupancy-aware lifting for panoptic 3d scene reconstruction from a single image

    TaoChu,PanZhang,QiongLiu,andJiaqiWang. Buol: A bottom-up framework with occupancy-aware lifting for panoptic 3d scene reconstruction from a single image. In CVPR, 2023. 3

    2023

  3. [11]

    Panoptic 3d scene reconstruction from a single rgb image

    Manuel Dahnert, Ji Hou, Matthias Niessner, and Angela Dai. Panoptic 3d scene reconstruction from a single rgb image. InNeurIPS, 2021. 2, 3

    2021

  4. [12]

    Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner

    Angela Dai, Angel X. Chang, Manolis Savva, Maciej Hal- ber, Thomas Funkhouser, and Matthias Nießner. Scan- Net: Richly-Annotated 3D Reconstructions of Indoor Scenes. InCVPR, 2017. 6, 7, 8, 9

    2017

  5. [13]

    Panoptic Segmentation with a Joint Semantic and Instance Segmentation Network

    Daan de Geus, Panagiotis Meletis, and Gijs Dubbel- 10 PanSt3R: Multi-view Consistent Panoptic Segmentation man. Panoptic Segmentation with a Joint Semantic and Instance Segmentation Network. arXiv:1809.02110,

    arXiv

  6. [14]

    Open- Vocabulary Universal Image Segmentation with MaskCLIP

    Zheng Ding, Jieke Wang, and Zhuowen Tu. Open- Vocabulary Universal Image Segmentation with MaskCLIP. InICML, 2023. 1, 2

    2023

  7. [15]

    Masked-attention Mask Transformer for Universal Image Segmentation

    Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Masked-attention Mask Transformer for Universal Image Segmentation. InCVPR, 2022. 1, 2, 3, 4, 5, 8

    2022

  8. [16]

    Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation

    Xiao Fu, Shangzhan Zhang, Tianrun Chen, Yichong Lu, Lanyun Zhu, Xiaowei Zhou, Andreas Geiger, and Yiyi Liao. Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation. In3DV, 2022. 2, 3, 5

    2022

  9. [17]

    SSAP: Single-Shot Instance Segmentation With Affinity Pyramid

    Naiyu Gao, Yanhu Shan, Yupei Wang, Xin Zhao, Yinan Yu, Ming Yang, and Kaiqi Huang. SSAP: Single-Shot Instance Segmentation With Affinity Pyramid. InICCV,

  10. [18]

    Xiuye Gu, Yin Cui, Jonathan Huang, Abdullah Rashwan, Xuan Yang, Xingyi Zhou, Golnaz Ghiasi, Weicheng Kuo, Huizhong Chen, Liang-Chieh Chen, and David A. Ross. 3DaTaSeg: Taming a Universal Multi-Dataset Multi-Task Segmentation Model. InNeurIPS, 2023. 2

    2023

  11. [19]

    Mask R-CNN

    Kaiming He, Georgia Gkioxari, Piotra Dollár, and Ross Girshick. Mask R-CNN. InICCV, 2017. 2

    2017

  12. [20]

    Primitive Generation and Semantic-related Alignment for Univer- sal Zero-Shot Segmentation

    Shuting He, Henghui Ding, and Wei Jiang. Primitive Generation and Semantic-related Alignment for Univer- sal Zero-Shot Segmentation. InCVPR, 2023. 2

    2023

  13. [21]

    LiDAR-based Panoptic Segmentation via Dynamic Shifting Network

    Fangzhou Hong, Hui Zhou, Xinge Zhu, Hongsheng Li, and Ziwei Liu. LiDAR-based Panoptic Segmentation via Dynamic Shifting Network. InCVPR, 2021. 1, 2

    2021

  14. [22]

    3D Gaus- sian Splatting for Real-Time Radiance Field Rendering

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimkuehler, and George Drettakis. 3D Gaus- sian Splatting for Real-Time Radiance Field Rendering. ACM Transactions on Graphics, 42(4):1–14, 2023. 2, 3, 5

    2023

  15. [23]

    Panoptic Feature Pyramid Networks

    Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic Feature Pyramid Networks. InCVPR, 2019. 2

    2019

  16. [24]

    Panoptic Segmentation

    Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic Segmentation. In CVPR, 2019. 1, 2, 7

    2019

  17. [25]

    Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation

    Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas Guibas, Andrea Tagliasac- chi, Frank Dellaert, and Thomas Funkhouser. Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation. InCVPR, 2022. 2, 3, 5, 7, 8

    2022

  18. [26]

    Grounding Image Matching in 3D with MASt3R

    Vincent Leroy, Yohann Cabon, and Jérôme Revaud. Grounding Image Matching in 3D with MASt3R. In ECCV, 2024. 1, 3

    2024

  19. [27]

    Panoptic-PHNet: Towards Real- Time and High-Precision LiDAR Panoptic Segmentation via Clustering Pseudo Heatmap

    Jinke Li, Xiao He, Yang Wen, Yuan Gao, Xiaoqiang Cheng, and Dan Zhang. Panoptic-PHNet: Towards Real- Time and High-Precision LiDAR Panoptic Segmentation via Clustering Pseudo Heatmap. InCVPR, 2022. 1, 2

    2022

  20. [28]

    Generalized Fo- cal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection

    Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang. Generalized Fo- cal Loss: Learning Qualified and Distributed Bounding Boxes for Dense Object Detection. InNeurIPS, 2020. 5

    2020

  21. [29]

    OMG-Seg: Is One Model Good Enough For All Segmentation? In CVPR, 2024

    Xiangtai Li, Haobo Yuan, Wei Li, Henghui Ding, Size Wu, Wenwei Zhang, Yining Li, Kai Chen, and Chen Change Loy. OMG-Seg: Is One Model Good Enough For All Segmentation? In CVPR, 2024. 1, 2

    2024

  22. [30]

    Attention- Guided Unified Network for Panoptic Segmentation

    Yanwei Li, Xinze Chen, Zheng Zhu, Lingxi Xie, Guan Huang, Dalong Du, and Xingang Wang. Attention- Guided Unified Network for Panoptic Segmentation. In CVPR, 2019. 1, 2

    2019

  23. [31]

    Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs

    Chen Liang-Chieh, George Papandreou, Iasonas Kokki- nos, Kevin Murphy, and Alan Yuille. Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFs. InICLR, 2015. 2

    2015

  24. [32]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common Objects in Context. InECCV, 2014. 6, 7

    2014

  25. [33]

    AnEnd-to-EndNetwork for Panoptic Segmentation

    Huanyu Liu, Chao Peng, Changqian Yu, Jingbo Wang, XuLiu,GangYu,andWeiJiang. AnEnd-to-EndNetwork for Panoptic Segmentation. InCVPR, 2019. 1, 2

    2019

  26. [34]

    Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering

    Tao Lu, Mulin Yu, Linning Xu, Yuanbo Xiangli, Limin Wang, Dahua Lin, and Bo Dai. Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering. InCVPR,

  27. [35]

    LUDVIG:Learning- free Uplifting of 2D Visual Features to Gaussian Splat- ting scenes

    Juliette Marrie, Romain Ménégaux, Michael Arbe- landDianeLarlus,andJulienMairal. LUDVIG:Learning- free Uplifting of 2D Visual Features to Gaussian Splat- ting scenes. arXiv:2410.14462, 2024. 2, 6

    2024 arXiv

  28. [36]

    Srinivasan, Matthew Tancik, Jonathan T

    Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. InECCV, 2020. 2, 3

    2020

  29. [37]

    EfficientPS: Efficient Panoptic Segmentation.International Journal of Com- puter Vision, 129:1551–1579, 2021

    Rohit Mohan and Abhinav Valada. EfficientPS: Efficient Panoptic Segmentation.International Journal of Com- puter Vision, 129:1551–1579, 2021. 1, 2

    2021

  30. [38]

    DINOv2: Learning Robust Visual Features without Supervision

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fer- nandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rab...

    2024

  31. [39]

    OpenScene: 3D Scene Understanding with Open Vocabularies

    Songyou Peng and Kyle Genova. OpenScene: 3D Scene Understanding with Open Vocabularies. InCVPR, 2023. 4, 13

    2023

  32. [40]

    Seamless Scene Segmentation

    Lorenzo Porzi, Samuel Rota Buló, Aleksander Colovic, and Peter Kontschieder. Seamless Scene Segmentation. In CVPR, 2019. 2

    2019

  33. [41]

    Qi, Hao Su, Kaichun Mo, and Leonidas J

    Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. InCVPR, 2017. 1 11 PanSt3R: Multi-view Consistent Panoptic Segmentation

    2017

  34. [42]

    Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Guibas. PointNet++: Deep Hierarchical Feature Learn- ing on Point Sets in a Metric Space. InNeurIPS, 2017. 1

    2017

  35. [43]

    FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentation

    Jie Qin, Jie Wu, Pengxiang Yan, Ming Li, Ren Yuxi, Xuefeng Xiao, Yitong Wang, Rui Wang, Shilei Wen, Xin Pan, and Xingang Wang. FreeSeg: Unified, Universal and Open-Vocabulary Image Segmentation. InCVPR,

  36. [44]

    Infinigen Indoors: Photorealistic Indoor Scenes using Procedural Generation

    Alexander Raistrick, Lingjie Mei, Karhan Kayan, David Yan, Yiming Zuo, Beining Han, Hongyu Wen, Meenal Parakh, Stamatis Alexandropoulos, Lahav Lipson, Zeyu Ma, and Jia Deng. Infinigen Indoors: Photorealistic Indoor Scenes using Procedural Generation. InCVPR,

  37. [45]

    GP-S3Net: Graph-based Panop- tic Sparse Semantic Segmentation Network

    Ryan Razani, Ran Cheng, Enxu Li, Ehsan Taghavi, Yuan Ren, and Liu Bingbing. GP-S3Net: Graph-based Panop- tic Sparse Semantic Segmentation Network. InICCV,

  38. [46]

    Susskind

    Mike Roberts, Jason Ramapuram, Anurag Ranjan, At- ulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M. Susskind. Hypersim: A Photoreal- istic Synthetic Dataset for Holistic Indoor Scene Under- standing. InICCV, 2021. 6, 7

    2021

  39. [47]

    S. S. Kirkpatrick, Gelatt C. D. Jr., and M. P. Vecchi. Op- timization by Simulated Annealing.Science, 220, 1983. 5

    1983

  40. [48]

    Structure-from-Motion Revisited

    Johannes Lutz Schönberger and Jan-Michael Frahm. Structure-from-Motion Revisited. InCVPR, 2016. 3

    2016

  41. [49]

    A Benchmark and a Baseline for Robust Multi-view Depth Estimation

    Philipp Schröppel, Jan Bechtold, Artemij Amiranashvili, and Thomas Brox. A Benchmark and a Baseline for Robust Multi-view Depth Estimation. In3DV, 2022. 1

    2022

  42. [50]

    Panoptic Lifting for 3D Scene Understand- ing With Neural Fields

    Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulò, Nor- man Müller, Matthias Nießner, Angela Dai, and Peter Kontschieder. Panoptic Lifting for 3D Scene Understand- ing With Neural Fields. InCVPR, 2023. 2, 3, 5, 7, 8

    2023

  43. [51]

    EfficientLPS: Efficient LiDAR Panoptic Segmentation

    Kshitij Sirohi, Rohit Mohan, Daniel Büscher, Wolfram Burgard, and Abhinav Valada. EfficientLPS: Efficient LiDAR Panoptic Segmentation. IEEE Transactions on Robotics, 38(3):1894–1914, 2021. 1, 2

    1914

  44. [52]

    AdaptIS: Adaptive Instance Selection Network

    Konstantin Sofiiuk, Olga Barinova, and Anton Konushin. AdaptIS: Adaptive Instance Selection Network. InICCV,

  45. [53]

    Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel, Raul Mur- Artal,CarlRen,ShobhitVerma,AntonClarkson,Mingfei Yan, Brian Budge, Yajie Yan, Xiaqing Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingh...

    1906 arXiv

  46. [54]

    Sudre, Wenqi Li, Tom Vercauteren, Sébastien Ourselin, and M

    Carole H. Sudre, Wenqi Li, Tom Vercauteren, Sébastien Ourselin, and M. Jorge Cardoso. Generalised Dice Over- lap as a Deep Learning Loss Function for Highly Unbal- anced Segmentations. InDLMIA, 2017. 5

    2017

  47. [55]

    Mv-dust3r+: Single-stage scene reconstruction from sparse views in 2 seconds

    Zhenggang Tang, Yuchen Fan, Dilin Wang, Hongyu Xu, Rakesh Ranjan, Alexander Schwing, and Zhicheng Yan. Mv-dust3r+: Single-stage scene reconstruction from sparse views in 2 seconds. arXiv:2412.06974, 2024. 1

    2024 arXiv

  48. [56]

    A Sim- ple Latent Diffusion Approach for Panoptic Segmenta- tion and Mask Inpainting

    Wouter Van Gansbeke and Bert De Brabandere. A Sim- ple Latent Diffusion Approach for Panoptic Segmenta- tion and Mask Inpainting. InECCV, 2024. 1, 2

    2024

  49. [57]

    DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Images

    Bing Wang, Lu Chen, and Bo Yang. DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Images. InICLR, 2023. 2, 3, 5, 7, 8

    2023

  50. [58]

    3D Reconstruction with Spatial Memory

    Hengyi Wang and Lourdes Agapito. 3D Reconstruction with Spatial Memory. arXiv:2408.16061, 2024. 1

    2024 arXiv

  51. [59]

    Yuille, and Liang-Chieh Chen

    HuiyuWang,YukunZhu,BradleyGreen,HartwigAdam, Alan L. Yuille, and Liang-Chieh Chen. Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation. In ECCV, 2020. 2

    2020

  52. [60]

    DFormer: Diffusion-guided Transformer for Universal Image Seg- mentation

    Hefeng Wang, Jiale Cao, Rao Muhammad Anwer, Jin Xie, Fahad Shahbaz Khan, and Yanwei Pang. DFormer: Diffusion-guided Transformer for Universal Image Seg- mentation. arXiv:2306.02240, 2023. 1, 2

    2023 arXiv

  53. [61]

    DUSt3R: Geometric 3D Vision Made Easy

    Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jérôme Revaud. DUSt3R: Geometric 3D Vision Made Easy. InCVPR, 2024. 1, 2, 3

    2024

  54. [62]

    PLGS: Robust Panoptic Lifting with 3D Gaussian Splat- ting

    Yu Wang, Xiaobao Wei, Ming Lu, and Guoliang Kang. PLGS: Robust Panoptic Lifting with 3D Gaussian Splat- ting. arXiv:2410.17505, 2024. 2, 3, 5, 7, 8

    2024 arXiv

  55. [63]

    Panorecon: Real- time panoptic 3d reconstruction from monocular video

    Dong Wu, Zike Yan, and Hongbin Zha. Panorecon: Real- time panoptic 3d reconstruction from monocular video. In CVPR, 2024. 2, 3

    2024

  56. [64]

    UPSNet: A Unified Panoptic Segmentation Network

    Yuwen Xiong, Renjie Liao, Hengshuang Zhao, Rui Hu, Min Bai, Ersin Yumer, and Raquel Urtasun. UPSNet: A Unified Panoptic Segmentation Network. InCVPR,

  57. [65]

    Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models

    Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xi- aolong Wang, and Shalini De Mello. Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models. InCVPR, 2023. 1, 2

    2023

  58. [66]

    Sparse Cross-scale Attention Network for Efficient LiDAR Panoptic Segmentation

    Shuangjie Xu, Rui Wan, Maosheng Ye, Xiaoyi Zou, and Tongyi Cao. Sparse Cross-scale Attention Network for Efficient LiDAR Panoptic Segmentation. InAAAI, 2022. 1, 2

    2022

  59. [67]

    MasQCLIP for Open-Vocabulary Universal Image Seg- mentation

    Xin Xu, Tianyi Xiong, Zheng Ding, and Zhuowen Tu. MasQCLIP for Open-Vocabulary Universal Image Seg- mentation. InICCV, 2023. 1, 2

    2023

  60. [68]

    Collins, Yukun Zhu, Jyh-Jing Hwang, Ting Liu, Xiao Zhang, Vivienne Sze, George Papandreou, and Liang-Chieh Chen

    Tien-Ju Yang, Maxwell D. Collins, Yukun Zhu, Jyh-Jing Hwang, Ting Liu, Xiao Zhang, Vivienne Sze, George Papandreou, and Liang-Chieh Chen. DeeperLab: Single- Shot Image Parser. arXiv:1902.05093, 2019. 2

    1902 arXiv

  61. [69]

    SGI- Former: Semantic-Guided and Geometric-Enhanced In- terleaving Transformer for 3D Instance Segmentation

    Lei Yao, Yi Wang, Moyun Liu, and Lap-Pui Chau. SGI- Former: Semantic-Guided and Geometric-Enhanced In- terleaving Transformer for 3D Instance Segmentation. IEEE Transactions on Circuits and Systems for Video Tech- nology, 35(3), 2025. 14

    2025

  62. [70]

    ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes

    Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes. InICCV, 2023. 6, 8 12 PanSt3R: Multi-view Consistent Panoptic Segmentation

    2023

  63. [71]

    Convolutions Die Hard: Open- Vocabulary Segmentation with Single Frozen Convo- lutional CLIP

    Qihang Yu, Ju He, Xueqing Deng, Xiaohui Shen, and Liang-Chieh Chen. Convolutions Die Hard: Open- Vocabulary Segmentation with Single Frozen Convo- lutional CLIP. InNeurIPS, 2023. 2

    2023

  64. [72]

    Object- Contextual Representations for Semantic Segmentation

    Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object- Contextual Representations for Semantic Segmentation. In ECCV, 2020. 2

    2020

  65. [73]

    Sigmoid Loss for Language Image Pre- Training

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid Loss for Language Image Pre- Training. InICCV, 2023. 4

    2023

  66. [74]

    A Simple Frame- work for Open-Vocabulary Segmentation and Detection

    Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chun- yuan Li, Jianwei Yang, and Lei Zhang. A Simple Frame- work for Open-Vocabulary Segmentation and Detection. In ICCV, 2023. 5

    2023

  67. [75]

    Uni-3d: A universal model for panoptic 3d scene reconstruction

    Xiang Zhang, Zeyuan Chen, Fangyin Wei, and Zhuowen Tu. Uni-3d: A universal model for panoptic 3d scene reconstruction. InICCV, 2023. 2, 3

    2023

  68. [76]

    ShuaifengZhi, TristanLaidlow, StefanLeutenegger, and Andrew J. Davison. In-Place Scene Labelling and Under- standing with Implicit Scene Representation. InICCV,

  69. [77]

    Scene Parsing through ADE20K Dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene Parsing through ADE20K Dataset. InCVPR, 2017. 6

    2017

  70. [78]

    Eprecon: An efficient framework for real-time panoptic 3d reconstruction from monocular video

    Zhen Zhou, Yunkai Ma, Junfeng Fan, Shaolin Zhang, Fengshui Jing, and Min Tan. Eprecon: An efficient framework for real-time panoptic 3d reconstruction from monocular video. arXiv:2409.01807, 2024. 3

    2024 arXiv

  71. [79]

    PCF-Lift: Panoptic Lifting by Probabilistic Contrastive Fusion

    Runsong Zhu, Shi Qiu, Qianyi Wu, Ka-Hei Hui, Pheng- Ann Heng, and Chi-Wing Fu. PCF-Lift: Panoptic Lifting by Probabilistic Contrastive Fusion. InECCV, 2024. 3, 7, 8

    2024

  72. [80]

    seen" images is reduced to 50, while PanSt3R, both with and without LUDVIG, is less sensitive to shrinking the set of

    Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, Nanyun Peng, Lijuan Wang, Yong Jae Lee, and Jianfeng Gao. Generalized Decoding for Pixel, Image, and Language. InCVPR, 2023. 1 Appendix A. Further ablative ...

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.