REVIEW 4 major objections 5 minor 28 references
OpenFusion++: An Open-vocabulary Real-time Scene Understanding System
T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read OpenFusion++ sharpens real-time open-vocabulary 3D maps by fusing confidence, area-weighted semantics, and two-stage queries.
desk verdict OpenFusion++ is a real incremental advance with two well-motivated modules, but the adaptive cache's core equation (Eq. 4) has the depth-area relation inverted, so the reported cache gains are not explained as written. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Three concrete mechanisms carry the argument. (1) Confidence-guided point sampling: within each centimeter-scale voxel block, the system assumes one instance and selects the candidate point set with the highest average confidence (Eq. 2–3), using SEEM's per-pixel confidence to resolve instance-boundary conflicts that random sampling would blur. (2) Adaptive semantic cache: a hash-indexed min-heap stores up to three semantic embeddings per instance, ordered by ascending negative physical area; insertion triggers an area-weighted fusion update of the global embedding (Eq. 4–5), so later, more complete views continuously refine the label. (3) Dual-path hierarchical query: the query's core object noun is extracted by a spaCy model, encoded by SEEM's text encoder, and used to score candidate instances by cosine similarity; after a threshold-based candidate set is formed, Alpha-CLIP matches the full query against each candidate's environment features and returns the best instance.
What would settle it
On a controlled sequence with two small, adjacent objects of different categories (e.g., a mug on a bookshelf) whose extent is comparable to one voxel block, run OpenFusion++ end to end; if the reconstructed map omits or mislabels the lower-confidence object only where the two overlap, the single-instance-per-block assumption is the cause. The paper's own Replica ablation, where confidence sampling alone lowers f-mIoU by 1.87 points, provides a baseline expectation for this failure mode.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that a real-time open-vocabulary 3D mapper can be made more accurate at three points in its pipeline without sacrificing responsiveness. First, when a voxel block receives 3D points from more than one 2D instance mask, the system computes the average SEEM confidence of each candidate point set and keeps only the highest-confidence set, giving that block its label (Eq. 2–3). Second, instead of freezing each instance's semantic embedding at its first observation, the mapper inserts each new observation into a per-instance min-heap ordered by ascending physical area, keeps up to three entries, and updates the global embedding as an area-weighted fusion (Eq. 4–5). Third, a query is answered in two stages: SEEM text features select candidate instances whose category is near the query's core noun, then Alpha-CLIP features re-rank those candidates by how well they match the full query, including context like 'near the window.' The paper reports that these mechanisms together raise mean accuracy over the baseline on all evaluated datasets and that the complete system outperforms each module alone in the ablation studies.
Load-bearing premise
The load-bearing premise is that each centimeter-scale voxel block contains exactly one instance, so selecting the highest-average-confidence point set decides the block's label; when two instances occupy the same block, the lower-confidence instance is erased, and the paper acknowledges that confidence sampling then 'dominates insertions, preventing correction.'
Editorial extensions
If this is right
- Open-vocabulary 3D mappers can use confidence signals that already exist in the 2D segmenter to fix boundary conflicts, so no retraining or new sensor is needed.
- Instance semantic labels can be continuously refined from partial views using physical area as a completeness proxy, reducing first-impression semantic drift.
- Complex queries such as 'the cabinet near the window' become answerable by separating category-level selection from environment-level matching.
- The gains hold across different category granularities (20 classes on ScanNet vs 200 on ScanNet++), suggesting the method is not overfit to one benchmark's label set.
- If the reported accuracy gains transfer to deployment, robots and AR devices can name and locate arbitrary objects in a scene in real time without a fixed category list.
Reading between the lines
- The single-instance-per-voxel-block assumption means the method likely struggles when two small objects are closer than one block size; a batch-insertion variant with weighted confidence voting, which the conclusion itself floats as future work, would be a natural remedy.
- The same confidence-guided selection rule could be transferred to any point-cloud-based semantic fusion pipeline that has per-point uncertainty, not just TSDF maps.
- The dual-path query design suggests that open-vocabulary 3D retrieval may generally benefit from splitting the tasks of 'what kind of thing' and 'which one in this scene' rather than forcing both into one embedding space.
- A concrete testable extension: measuring query latency versus accuracy under varying cache sizes, since the min-heap cap of three observations is a hyperparameter with no reported sensitivity analysis.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript presents OpenFusion++, a real-time open-vocabulary 3D scene understanding system built on the OpenFusion TSDF-based mapping framework. It introduces three main contributions: confidence-guided 3D point sampling to improve instance boundaries, an adaptive semantic cache that updates instance embeddings from multi-view observations, and a dual-path query framework combining SEEM object features with Alpha-CLIP spatial context. The paper reports quantitative semantic segmentation improvements over OpenFusion on Replica, ScanNet, and ScanNet++ (Table I), and qualitative query-response results on ICL, Replica, and ScanNet. An ablation study (Table II) attributes gains on Replica to the confidence-guided sampling and adaptive cache modules.
Significance. If the reported results are reliable, OpenFusion++ is a useful incremental contribution to open-vocabulary 3D mapping: it directly addresses semantic drift and instance-boundary errors in incremental TSDF-based mapping, and its dual-path query framework is a practical response to nested queries such as “the cabinet next to the window.” The paper ships no code or machine-checked proofs, but it does include a quantitative comparison against a meaningful external baseline and an ablation study, which are appropriate for this kind of systems paper. However, the central cache mechanism contains a mathematical error that undermines the explanation of the ablation gains, and the quantitative evaluation lacks statistical support and full disclosure of test sequences. These issues must be resolved before the performance claims can be accepted.
major comments (4)
- [Section III-C, Eq. (4)] Equation (4) is not the physical area of the 2D instance mask and has the depth dependence inverted. For a pinhole camera, the physical surface area subtended by one pixel at depth D is proportional to D^2/(f_x f_y), not f_x f_y/D^2. The quantity in Eq. (4) has units of inverse area, and for a fixed object it decreases rather than increases as the viewpoint becomes more global. Since the heap ordering W = -A_t^k and the multi-view fusion weights in Eq. (5) both depend on A_t^k, the adaptive cache mechanism as written is biased toward close-up partial observations. This directly contradicts the claim that the cache preserves complete observations, and it means the +2.29 mAcc and +1.10 f-mIoU gains attributed to the cache in Table II are not explained by the mechanism as described. The authors should correct Eq. (4) to use D^2/(f_x f_y) and re-run the ablation, or clearly explain what quantity Eq. (4) is intended to represent.
- [Section III-C, min-heap eviction] The min-heap eviction rule is internally inconsistent. The text says the heap stores embeddings “in ascending order based on the weight W = -A_t^k” and that when the heap is full, “the smallest area entry is removed.” With W = -A_t^k and a min-heap, the root has the most negative W, i.e., the largest A_t^k, so removing the root removes the largest-area observation, not the smallest. If the intended behavior is to evict the least complete observation, the implementation should use a max-heap on A_t^k or a min-heap with W = A_t^k. The authors need to clarify the actual data structure and, if the deployed code uses the formula as written, report what is really being evicted.
- [Section IV, Table I and Table II] The quantitative evidence for “significantly outperforms” is statistically incomplete. The manuscript does not name the 26 test sequences (8 Replica, 10 ScanNet, 8 ScanNet++), does not report error bars, per-sequence breakdowns, or significance tests, and the hyperparameters alpha = 0.8 and cache size N = 3 are fixed without sensitivity analysis. The abstract and introduction claim experiments on ICL, Replica, ScanNet, and ScanNet++, but ICL appears only in qualitative query tests and not in Table I; the paper should either add ICL results to the quantitative table or remove it from the abstract claim. The ablation in Table II also shows that confidence-guided sampling alone decreases f-mIoU by 1.87, so the statement that the module improves “semantic accuracy” is only true for mAcc. Finally, despite “real-time” in the title, no frame rate, latency, or runtime comparison is reported; the system should be measured or the real-time claim should be qualified.
- [Section III-B and Section V] The confidence-guided sampling strategy assumes that each centimeter-scale voxel block contains exactly one instance, as stated before Eq. (3). When a block straddles two instances, the method forces a single label onto the whole block, and the paper’s own conclusion acknowledges that confidence sampling dominates insertions and can prevent later correction. This is a genuine limitation of the first contribution, not just a presentation issue: it can erase a legitimate instance at object boundaries, which is exactly the scenario the paper claims to fix (e.g., sofa–pillow confusion in Fig. 2). The authors should quantify how often this occurs, for example by reporting a boundary-region ablation or by measuring the effect when the one-instance assumption is violated, and should discuss whether the mAcc gain is partly an artifact of assigning larger instances’ labels to smaller ones.
minor comments (5)
- [Abstract and Section IV] The abstract lists ICL as a dataset for quantitative experiments, but ICL is only used for qualitative query tests; please align the abstract with the actual evaluation.
- [Section III-D] “en core web sm” is a typo for the spaCy model “en_core_web_sm”; also, the spaCy model should be explicitly cited or listed in the implementation details.
- [Section II-C] In Related Works C, “OpenFusion [1] proposed a TSDF-based real-time scene reconstruction framework” cites [1] (Radford et al., CLIP); the correct reference is [4] (Yamazaki et al., OpenFusion).
- [Throughout] The metric notation is inconsistent: the manuscript alternates between “F-mIOU,” “f-mIoU,” and “F-mIoU.” Please standardize to a single notation.
- [Figure 4 caption] The phrase “weights of semantic embeddings are managed by inverting the physical coverage area as a minimum heap” is unclear; after correcting Eq. (4), please rewrite the caption to explain the heap ordering in terms of the corrected area measure.
Circularity Check
No circular reasoning identified: the modules are defined from external inputs, evaluated against an external baseline, and supported by ablations; the Eq. (4) area-formula issue is a correctness risk, not a circular reduction.
full rationale
OpenFusion++'s derivation chain is self-contained. Confidence-guided sampling (Eqs. 2-3) uses SEEM confidence maps and depth as external inputs; the arg-max rule is a data-selection heuristic, and the assigned labels come from SEEM rather than from the reported mAcc/f-mIoU metrics. The adaptive semantic cache (Eq. 5) fuses observed embeddings with weights computed from masks and depth; none of the hyperparameters (alpha=0.8, cache size N=3, point count N=16) is stated to be fitted to the test outcomes, and the cache's contribution is tested by an ablation (Table II) against the same external OpenFusion baseline. The dual-path query framework uses fixed cosine similarity and a threshold alpha=0.8, with no dataset labels encoded in the retrieval equations. Table I compares directly with OpenFusion, and no self-citation or uniqueness theorem is load-bearing. The paper's own limitation statement about confidence sampling preventing later correction is an acknowledged robustness concern, not a circular dependency. The main genuine risk is Eq. (4): for a pinhole camera the physical area subtended by one pixel at depth D is proportional to D^2/(fx*fy), not fx*fy/D^2, so the claimed monotonic increase of A_t^k with global views is mathematically inverted. That is a correctness or implementation error that could invalidate the cache mechanism's explanation of its ablation gain, but it is not a case where the derivation reduces to its own inputs; the reported cache improvement is an empirical result, not a constructional equivalence. The abstract's mention of ICL experiments is not matched by Table I, which is a reporting inconsistency rather than circularity. Overall, no circular step meets the required evidentiary standard.
Assumptions & free parameters
free parameters (4)
- alpha (query candidate threshold) =
0.8
- cache size N (min-heap capacity) =
3
- points per voxel block N =
16
- voxel block size =
8/512 m
assumptions (4)
- domain assumption Each centimeter-scale voxel block contains only one instance.
- domain assumption SEEM pixel-wise confidence maps correctly indicate boundary uncertainty.
- domain assumption Physical area A_t^k monotonically increases as the viewpoint observes more of an instance.
- domain assumption Area-weighted averaging of semantic embeddings improves global semantics without information loss.
Cite this review
Pith. "Pith review of OpenFusion++: An Open-vocabulary Real-time Scene Understanding System." pith.science (2026). https://pith.science/paper/DYKESJ2I
@misc{pith2026250419266,
author = {Pith},
title = {Pith review of: OpenFusion++: An Open-vocabulary Real-time Scene Understanding System},
year = {2026},
howpublished = {\url{https://pith.science/paper/DYKESJ2I}},
note = {Machine review of arXiv:2504.19266}
}
read the original abstract
Real-time open-vocabulary scene understanding is essential for efficient 3D perception in applications such as vision-language navigation, embodied intelligence, and augmented reality. However, existing methods suffer from imprecise instance segmentation, static semantic updates, and limited handling of complex queries. To address these issues, we present OpenFusion++, a TSDF-based real-time 3D semantic-geometric reconstruction system. Our approach refines 3D point clouds by fusing confidence maps from foundational models, dynamically updates global semantic labels via an adaptive cache based on instance area, and employs a dual-path encoding framework that integrates object attributes with environmental context for precise query responses. Experiments on the ICL, Replica, ScanNet, and ScanNet++ datasets demonstrate that OpenFusion++ significantly outperforms the baseline in both semantic accuracy and query responsiveness.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Learning transferable visual models from natural language supervision,
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark et al., “Learning transferable visual models from natural language supervision,” in International conference on machine learning . PMLR, 2021, pp. 8748–8763
2021
-
[2]
Towards Real-Time Open-Vocabulary Video Instance Segmentation
B. Yan, M. Sundermeyer, D. J. Tan, H. Lu, and F. Tombari, “To- wards real-time open-vocabulary video instance segmentation,” arXiv preprint arXiv:2412.04434, 2024
work page Pith review arXiv 2024
-
[3]
Segment anything,
A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo et al. , “Segment anything,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 4015–4026
2023
-
[4]
Open-fusion: Real-time open-vocabulary 3d mapping and queryable scene representation,
K. Yamazaki, T. Hanyu, K. V o, T. Pham, M. Tran, G. Doretto, A. Nguyen, and N. Le, “Open-fusion: Real-time open-vocabulary 3d mapping and queryable scene representation,” in 2024 IEEE Interna- tional Conference on Robotics and Automation (ICRA) . IEEE, 2024, pp. 9411–9417
work page 2024
-
[5]
Segment everything everywhere all at once,
X. Zou, J. Yang, H. Zhang, F. Li, L. Li, J. Wang, L. Wang, J. Gao, and Y . J. Lee, “Segment everything everywhere all at once,”Advances in Neural Information Processing Systems , vol. 36, 2024
work page 2024
-
[6]
Emerging properties in self-supervised vision trans- formers,
M. Caron, H. Touvron, I. Misra, H. J ´egou, J. Mairal, P. Bojanowski, and A. Joulin, “Emerging properties in self-supervised vision trans- formers,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 9650–9660
2021
-
[7]
Dinov2: Learning robust visual features without supervision,
M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khali- dov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby et al., “Dinov2: Learning robust visual features without supervision,” arXiv preprint arXiv:2304.07193, 2023
arXiv 2023
-
[8]
Masked autoencoders are scalable vision learners,
K. He, X. Chen, S. Xie, Y . Li, P. Doll ´ar, and R. Girshick, “Masked autoencoders are scalable vision learners,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 16 000–16 009
2022
Show all 28 references
-
[9]
Groupvit: Semantic segmentation emerges from text su- pervision,
J. Xu, S. De Mello, S. Liu, W. Byeon, T. Breuel, J. Kautz, and X. Wang, “Groupvit: Semantic segmentation emerges from text su- pervision,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 18 134–18 144
2022
-
[10]
Segclip: Patch aggregation with learnable centers for open-vocabulary semantic segmentation,
H. Luo, J. Bao, Y . Wu, X. He, and T. Li, “Segclip: Patch aggregation with learnable centers for open-vocabulary semantic segmentation,” in International Conference on Machine Learning . PMLR, 2023, pp. 23 033–23 044
2023
-
[11]
Conceptfusion: Open-set multimodal 3d mapping,
K. M. Jatavallabhula, A. Kuwajerwala, Q. Gu, M. Omama, T. Chen, A. Maalouf, S. Li, G. Iyer, S. Saryazdi, N. Keetha et al. , “Conceptfusion: Open-set multimodal 3d mapping,” arXiv preprint arXiv:2302.07241, 2023
2023 arXiv
-
[12]
Openmask3d: Open-vocabulary 3d instance segmen- tation,
A. Takmaz, E. Fedele, R. W. Sumner, M. Pollefeys, F. Tombari, and F. Engelmann, “Openmask3d: Open-vocabulary 3d instance segmen- tation,” arXiv preprint arXiv:2306.13631 , 2023
2023 arXiv
-
[13]
Opensu3d: Open world 3d scene understanding using foundation models,
R. Mohiuddin, S. M. Prakhya, F. Collins, Z. Liu, and A. Borrmann, “Opensu3d: Open world 3d scene understanding using foundation models,” arXiv preprint arXiv:2407.14279 , 2024
2024 arXiv
-
[14]
Open3dis: Open-vocabulary 3d instance segmentation with 2d mask guidance,
P. Nguyen, T. D. Ngo, E. Kalogerakis, C. Gan, A. Tran, C. Pham, and K. Nguyen, “Open3dis: Open-vocabulary 3d instance segmentation with 2d mask guidance,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 4018–4028
2024
-
[15]
Openins3d: Snap and lookup for 3d open-vocabulary instance seg- mentation,
Z. Huang, X. Wu, X. Chen, H. Zhao, L. Zhu, and J. Lasenby, “Openins3d: Snap and lookup for 3d open-vocabulary instance seg- mentation,” in European Conference on Computer Vision . Springer, 2024, pp. 169–185
2024
-
[16]
Openscene: 3d scene understanding with open vocabularies,
S. Peng, K. Genova, C. Jiang, A. Tagliasacchi, M. Pollefeys, T. Funkhouser et al. , “Openscene: 3d scene understanding with open vocabularies,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2023, pp. 815–824
2023
-
[17]
Sam3d: Segment anything in 3d scenes,
Y . Yang, X. Wu, T. He, H. Zhao, and X. Liu, “Sam3d: Segment anything in 3d scenes,” arXiv preprint arXiv:2306.03908 , 2023
2023 arXiv
-
[18]
Maskclustering: View consensus based mask graph clustering for open-vocabulary 3d in- stance segmentation,
M. Yan, J. Zhang, Y . Zhu, and H. Wang, “Maskclustering: View consensus based mask graph clustering for open-vocabulary 3d in- stance segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 28 274–28 284
2024
-
[19]
Embod- iedsam: Online segment any 3d thing in real time,
X. Xu, H. Chen, L. Zhao, Z. Wang, J. Zhou, and J. Lu, “Embod- iedsam: Online segment any 3d thing in real time,” arXiv preprint arXiv:2408.11811, 2024
2024 arXiv
-
[20]
Fast segment anything,
X. Zhao, W. Ding, Y . An, Y . Du, T. Yu, M. Li, M. Tang, and J. Wang, “Fast segment anything,” arXiv preprint arXiv:2306.12156 , 2023
2023 arXiv
-
[21]
Panoslam: Panoptic 3d scene reconstruction via gaussian slam,
R. Chen, Z. Wang, J. Wang, Y . Ma, M. Gong, W. Wang, and T. Liu, “Panoslam: Panoptic 3d scene reconstruction via gaussian slam,” arXiv preprint arXiv:2501.00352, 2024
2024 arXiv
-
[22]
3d gaussian splatting for real-time radiance field rendering
B. Kerbl, G. Kopanas, T. Leimk ¨uhler, and G. Drettakis, “3d gaussian splatting for real-time radiance field rendering.” ACM Trans. Graph., vol. 42, no. 4, pp. 139–1, 2023
2023
-
[23]
Ovo-slam: Open- vocabulary online simultaneous localization and mapping,
T. B. Martins, M. R. Oswald, and J. Civera, “Ovo-slam: Open- vocabulary online simultaneous localization and mapping,” arXiv preprint arXiv:2411.15043, 2024
2024
-
[24]
Alpha-clip: A clip model focusing on wherever you want,
Z. Sun, Y . Fang, T. Wu, P. Zhang, Y . Zang, S. Kong, Y . Xiong, D. Lin, and J. Wang, “Alpha-clip: A clip model focusing on wherever you want,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2024, pp. 13 019–13 029
2024
-
[25]
A benchmark for rgb-d visual odometry, 3d reconstruction and slam,
A. Handa, T. Whelan, J. McDonald, and A. J. Davison, “A benchmark for rgb-d visual odometry, 3d reconstruction and slam,” in 2014 IEEE international conference on Robotics and automation (ICRA) . IEEE, 2014, pp. 1524–1531
2014
-
[26]
Scannet: Richly-annotated 3d reconstructions of indoor scenes,
A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner, “Scannet: Richly-annotated 3d reconstructions of indoor scenes,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 5828–5839
2017
-
[27]
Scannet++: A high- fidelity dataset of 3d indoor scenes,
C. Yeshwanth, Y .-C. Liu, M. Nießner, and A. Dai, “Scannet++: A high- fidelity dataset of 3d indoor scenes,” in Proceedings of the IEEE/CVF International Conference on Computer Vision , 2023, pp. 12–22
2023
-
[28]
The replica dataset: A digital replica of indoor spaces,
J. Straub, T. Whelan, L. Ma, Y . Chen, E. Wijmans, S. Green, J. J. Engel, R. Mur-Artal, C. Ren, S. Verma et al. , “The replica dataset: A digital replica of indoor spaces,” arXiv preprint arXiv:1906.05797 , 2019
1906 arXiv
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.