{"id":"02a46f0b-6294-4808-a4d1-32055b945326","arxiv_id":"2504.19266","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"OpenFusion++ upgrades the OpenFusion real-time 3D mapping system with confidence-based boundary refinement, area-weighted semantic caching, and a two-stage query matching that improves semantic accuracy and spatial query response.","lead":"OpenFusion++ is a system that builds a 3D map of a room in real time and can answer questions like \"find the chair next to the sofa\" by combining object recognition with scene context. It improves an earlier system, OpenFusion, by using confidence scores to clean up object boundaries, updating object labels as the camera sees more, and splitting queries into an object part and an environment part.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation (4) inverts the depth-to-area relation, so the adaptive semantic cache's completeness weighting is likely wrong and the reported cache gain is not explained.","rationale":"The reader correctly identified the one-instance-per-voxel assumption and the conclusion's admission that confidence sampling can prevent correction; that is a genuine limitation. However, the more load-bearing concern for the central claim is in the adaptive semantic cache: Eq. (4) uses an inverse-square depth relation that contradicts the physical-area definition and the asserted monotonicity. Since the cache is one of the three main contributions and Table II attributes a measurable gain to it, an error in its weighting formula directly undermines the explanatory mechanism behind that gain. The appropriate response is not rejection but a conditional acceptance: the authors should correct Eq. (4) (or clarify the intended definition) and recompute the cache ablation. If the gain persists with the corrected formula, the central claim is supported; if not, the cache contribution needs to be reassessed. The reader's verdict of CONDITIONAL remains appropriate, though for a partly different reason than the weakest assumption stated in the reader's report.","tokens_in":9561,"tokens_out":8041,"duration_ms":85488,"concrete_test":"Render a planar patch of known physical area (e.g., 1 m^2) at depths 1, 2, and 4 m with fixed intrinsics and compute Eq. (4) for each view. If the value scales as 1/D^2 instead of remaining approximately constant, Eq. (4) is not a physical area. Then re-run the Table II ablation with the corrected area weight A_t^k = sum_p D_t(p)^2/(f_x f_y) and check whether the adaptive-cache gain (+2.29 mAcc, +1.10 f-mIoU) persists; if it reverses or vanishes, the described cache mechanism is not responsible for the claimed improvement.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The adaptive semantic cache is one of the three claimed contributions, and its core quantity is Eq. (4): A_t^k = sum_{p in M} f_x f_y / D_t(p)^2, called 'physical coverage area' and asserted to increase monotonically as views go from local to global. For a pinhole camera with focal lengths in pixels, the physical area subtended by one pixel at depth D is proportional to D^2/(f_x f_y), not f_x f_y/D^2. Eq. (4) therefore decreases as the camera moves farther away, which is the opposite of the claimed monotonic increase and is not a physical area in the stated units. This error propagates directly into the cache: Section III-C orders the min-heap by W = -A_t^k, and Eq. (5) weights multi-view semantic fusion by A_t^k. If the implementation follows Eq. (4), close-up partial views will dominate over complete global views, and the heap may evict the most informative observations. Consequently, the +2.29 mAcc and +1.10 f-mIoU gain attributed to the adaptive cache in Table II is not explained by the mechanism as written, weakening the central claim that the cache improves semantic accuracy. The reader's confidence-sampling concern is valid, but it is acknowledged by the authors and is secondary to this unnoticed mathematical slip.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents OpenFusion++, a real-time open-vocabulary 3D scene understanding system built on the OpenFusion TSDF-based mapping framework. It introduces three main contributions: confidence-guided 3D point sampling to improve instance boundaries, an adaptive semantic cache that updates instance embeddings from multi-view observations, and a dual-path query framework combining SEEM object features with Alpha-CLIP spatial context. The paper reports quantitative semantic segmentation improvements over OpenFusion on Replica, ScanNet, and ScanNet++ (Table I), and qualitative query-response results on ICL, Replica, and ScanNet. An ablation study (Table II) attributes gains on Replica to the confidence-guided sampling and adaptive cache modules.","tokens_in":9795,"tokens_out":5803,"duration_ms":60952,"significance":"If the reported results are reliable, OpenFusion++ is a useful incremental contribution to open-vocabulary 3D mapping: it directly addresses semantic drift and instance-boundary errors in incremental TSDF-based mapping, and its dual-path query framework is a practical response to nested queries such as “the cabinet next to the window.” The paper ships no code or machine-checked proofs, but it does include a quantitative comparison against a meaningful external baseline and an ablation study, which are appropriate for this kind of systems paper. However, the central cache mechanism contains a mathematical error that undermines the explanation of the ablation gains, and the quantitative evaluation lacks statistical support and full disclosure of test sequences. These issues must be resolved before the performance claims can be accepted.","major_comments":[{"comment":"Equation (4) is not the physical area of the 2D instance mask and has the depth dependence inverted. For a pinhole camera, the physical surface area subtended by one pixel at depth D is proportional to D^2/(f_x f_y), not f_x f_y/D^2. The quantity in Eq. (4) has units of inverse area, and for a fixed object it decreases rather than increases as the viewpoint becomes more global. Since the heap ordering W = -A_t^k and the multi-view fusion weights in Eq. (5) both depend on A_t^k, the adaptive cache mechanism as written is biased toward close-up partial observations. This directly contradicts the claim that the cache preserves complete observations, and it means the +2.29 mAcc and +1.10 f-mIoU gains attributed to the cache in Table II are not explained by the mechanism as described. The authors should correct Eq. (4) to use D^2/(f_x f_y) and re-run the ablation, or clearly explain what quantity Eq. (4) is intended to represent.","section":"Section III-C, Eq. (4)"},{"comment":"The min-heap eviction rule is internally inconsistent. The text says the heap stores embeddings “in ascending order based on the weight W = -A_t^k” and that when the heap is full, “the smallest area entry is removed.” With W = -A_t^k and a min-heap, the root has the most negative W, i.e., the largest A_t^k, so removing the root removes the largest-area observation, not the smallest. If the intended behavior is to evict the least complete observation, the implementation should use a max-heap on A_t^k or a min-heap with W = A_t^k. The authors need to clarify the actual data structure and, if the deployed code uses the formula as written, report what is really being evicted.","section":"Section III-C, min-heap eviction"},{"comment":"The quantitative evidence for “significantly outperforms” is statistically incomplete. The manuscript does not name the 26 test sequences (8 Replica, 10 ScanNet, 8 ScanNet++), does not report error bars, per-sequence breakdowns, or significance tests, and the hyperparameters alpha = 0.8 and cache size N = 3 are fixed without sensitivity analysis. The abstract and introduction claim experiments on ICL, Replica, ScanNet, and ScanNet++, but ICL appears only in qualitative query tests and not in Table I; the paper should either add ICL results to the quantitative table or remove it from the abstract claim. The ablation in Table II also shows that confidence-guided sampling alone decreases f-mIoU by 1.87, so the statement that the module improves “semantic accuracy” is only true for mAcc. Finally, despite “real-time” in the title, no frame rate, latency, or runtime comparison is reported; the system should be measured or the real-time claim should be qualified.","section":"Section IV, Table I and Table II"},{"comment":"The confidence-guided sampling strategy assumes that each centimeter-scale voxel block contains exactly one instance, as stated before Eq. (3). When a block straddles two instances, the method forces a single label onto the whole block, and the paper’s own conclusion acknowledges that confidence sampling dominates insertions and can prevent later correction. This is a genuine limitation of the first contribution, not just a presentation issue: it can erase a legitimate instance at object boundaries, which is exactly the scenario the paper claims to fix (e.g., sofa–pillow confusion in Fig. 2). The authors should quantify how often this occurs, for example by reporting a boundary-region ablation or by measuring the effect when the one-instance assumption is violated, and should discuss whether the mAcc gain is partly an artifact of assigning larger instances’ labels to smaller ones.","section":"Section III-B and Section V"}],"minor_comments":[{"comment":"The abstract lists ICL as a dataset for quantitative experiments, but ICL is only used for qualitative query tests; please align the abstract with the actual evaluation.","section":"Abstract and Section IV"},{"comment":"“en core web sm” is a typo for the spaCy model “en_core_web_sm”; also, the spaCy model should be explicitly cited or listed in the implementation details.","section":"Section III-D"},{"comment":"In Related Works C, “OpenFusion [1] proposed a TSDF-based real-time scene reconstruction framework” cites [1] (Radford et al., CLIP); the correct reference is [4] (Yamazaki et al., OpenFusion).","section":"Section II-C"},{"comment":"The metric notation is inconsistent: the manuscript alternates between “F-mIOU,” “f-mIoU,” and “F-mIoU.” Please standardize to a single notation.","section":"Throughout"},{"comment":"The phrase “weights of semantic embeddings are managed by inverting the physical coverage area as a minimum heap” is unclear; after correcting Eq. (4), please rewrite the caption to explain the heap ordering in terms of the corrected area measure.","section":"Figure 4 caption"}],"recommendation":"major_revision","confidential_remarks":"The central idea is promising and the comparison against OpenFusion is a reasonable evaluation strategy, but the cache equation error is load-bearing: it invalidates the mechanism-level explanation of one of the three claimed contributions. I see this as fixable rather than fatal, because the intended formula (D^2/(f_x f_y)) is standard, and the ablation could be rerun. However, the lack of statistical rigor and missing sequence disclosure should be addressed in the same revision. The reference mismatch in Related Works C also suggests the manuscript needs a careful proofread before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a plausible incremental improvement over OpenFusion, and two of the three modules are well motivated and reasonably evaluated. But one module — the adaptive semantic cache — has a mathematical error in its central formula that, as written, inverts the depth-to-area relationship. That makes the ablation gain attributed to the cache unexplained and needs to be fixed before the paper's claims hold up.\n\nWhat is genuinely new: confidence-guided sampling uses SEEM's per-pixel confidence to select which instance's points populate a voxel block, which is a sensible way to handle boundary erosion; the dual-path query (SEEM for object category, Alpha-CLIP for environment context) is a coherent way to handle nested queries; and the idea of maintaining a cache of recent embeddings rather than a single static one is worth exploring. The authors also deserve credit for running a real baseline comparison and for explicitly acknowledging in the conclusion that confidence sampling can lock in wrong labels and prevent later correction — that is honest, and it is a real limitation.\n\nThe soft spots, in order of softness. First, Eq. (4) computes \"physical coverage area\" as sum fx fy / D_t(p)^2. For a pinhole camera, the physical area subtended by one pixel at depth D is D^2/(fx fy), not fx fy/D^2. So as the viewpoint moves from local to global (depth increasing), the paper's quantity decreases instead of increasing. The cache ordering (W = -A) and the fusion weights in Eq. (5) therefore favor close-up, partial views. The +2.29 mAcc and +1.10 f-mIoU gain from the cache in Table II is not explained by the mechanism as described. If the implementation actually uses the correct formula, the paper needs to say so; if it uses Eq. (4) as printed, the module may be doing something different from what is claimed.\n\nSecond, the evaluation has the usual weaknesses: no error bars or significance tests, the exact 26 sequences are not listed, and query responsiveness is only shown qualitatively. The abstract mentions ICL experiments, but Table I omits ICL — ICL appears only in ablation figures, which is an overstatement. There are also citation numbering errors that a careful copyedit would catch.\n\nThe confidence-sampling limitation the reader flagged is real, but it is acknowledged by the authors and is secondary to the Eq. (4) issue.\n\nWho this is for: people building real-time open-vocabulary mapping systems in robotics or AR will find the system description useful, and the dual-path query idea is worth borrowing. But the mathematical slip means the cache contribution needs significant revision or re-derivation.\n\nRecommendation: send it to peer review. A serious referee can push for a corrected area formulation, quantitative query metrics, and an explicit sequence list. The core system is real, and the issues are addressable, but this is not ready as is.","headline":"OpenFusion++ is a real incremental advance with two well-motivated modules, but the adaptive cache's core equation (Eq. 4) has the depth-area relation inverted, so the reported cache gains are not explained as written.","tokens_in":10356,"tokens_out":2549,"would_cite":false,"duration_ms":25508,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OpenFusion++ sharpens real-time open-vocabulary 3D maps by fusing confidence, area-weighted semantics, and two-stage queries.","keywords":["open-vocabulary 3D scene understanding","real-time semantic mapping","TSDF reconstruction","confidence-guided point sampling","adaptive semantic cache","dual-path query","SEEM","RGB-D perception"],"falsifier":"On a controlled sequence with two small, adjacent objects of different categories (e.g., a mug on a bookshelf) whose extent is comparable to one voxel block, run OpenFusion++ end to end; if the reconstructed map omits or mislabels the lower-confidence object only where the two overlap, the single-instance-per-block assumption is the cause. The paper's own Replica ablation, where confidence sampling alone lowers f-mIoU by 1.87 points, provides a baseline expectation for this failure mode.","tokens_in":9325,"feed_emoji":"🗺️","tokens_out":9416,"duration_ms":80194,"temperature":0.7,"pith_summary":"OpenFusion++ is a real-time 3D mapping system that labels and answers questions about objects in an indoor scene as a robot or AR device streams RGB-D video. The paper claims that three additions to the OpenFusion baseline—confidence-guided point sampling, an area-weighted adaptive semantic cache, and a two-stage object-plus-environment query—boost both segmentation accuracy and the ability to resolve context-dependent queries. This claim is supported by quantitative results on Replica, ScanNet, and ScanNet++, where mean accuracy improves by 6.29, 1.94, and 4.64 points respectively, and by qualitative query results on ICL, Replica, and ScanNet. The significance is that open-vocabulary 3D understanding can be improved at run time by using confidence signals and observation completeness that already exist in the 2D segmentation pipeline, without retraining a model.","feed_headline":"Three targeted fixes lift open-vocabulary 3D mapping accuracy","feed_subtitle":"Beats OpenFusion on Replica, ScanNet, and ScanNet++ via sharper boundaries and area-weighted semantic caching","key_machinery":"Three concrete mechanisms carry the argument. (1) Confidence-guided point sampling: within each centimeter-scale voxel block, the system assumes one instance and selects the candidate point set with the highest average confidence (Eq. 2–3), using SEEM's per-pixel confidence to resolve instance-boundary conflicts that random sampling would blur. (2) Adaptive semantic cache: a hash-indexed min-heap stores up to three semantic embeddings per instance, ordered by ascending negative physical area; insertion triggers an area-weighted fusion update of the global embedding (Eq. 4–5), so later, more complete views continuously refine the label. (3) Dual-path hierarchical query: the query's core object noun is extracted by a spaCy model, encoded by SEEM's text encoder, and used to score candidate instances by cosine similarity; after a threshold-based candidate set is formed, Alpha-CLIP matches the full query against each candidate's environment features and returns the best instance.","core_discovery":"On the paper's own terms, the central discovery is that a real-time open-vocabulary 3D mapper can be made more accurate at three points in its pipeline without sacrificing responsiveness. First, when a voxel block receives 3D points from more than one 2D instance mask, the system computes the average SEEM confidence of each candidate point set and keeps only the highest-confidence set, giving that block its label (Eq. 2–3). Second, instead of freezing each instance's semantic embedding at its first observation, the mapper inserts each new observation into a per-instance min-heap ordered by ascending physical area, keeps up to three entries, and updates the global embedding as an area-weighted fusion (Eq. 4–5). Third, a query is answered in two stages: SEEM text features select candidate instances whose category is near the query's core noun, then Alpha-CLIP features re-rank those candidates by how well they match the full query, including context like 'near the window.' The paper reports that these mechanisms together raise mean accuracy over the baseline on all evaluated datasets and that the complete system outperforms each module alone in the ablation studies.","pith_inferences":["The single-instance-per-voxel-block assumption means the method likely struggles when two small objects are closer than one block size; a batch-insertion variant with weighted confidence voting, which the conclusion itself floats as future work, would be a natural remedy.","The same confidence-guided selection rule could be transferred to any point-cloud-based semantic fusion pipeline that has per-point uncertainty, not just TSDF maps.","The dual-path query design suggests that open-vocabulary 3D retrieval may generally benefit from splitting the tasks of 'what kind of thing' and 'which one in this scene' rather than forcing both into one embedding space.","A concrete testable extension: measuring query latency versus accuracy under varying cache sizes, since the min-heap cap of three observations is a hyperparameter with no reported sensitivity analysis."],"forward_implications":["Open-vocabulary 3D mappers can use confidence signals that already exist in the 2D segmenter to fix boundary conflicts, so no retraining or new sensor is needed.","Instance semantic labels can be continuously refined from partial views using physical area as a completeness proxy, reducing first-impression semantic drift.","Complex queries such as 'the cabinet near the window' become answerable by separating category-level selection from environment-level matching.","The gains hold across different category granularities (20 classes on ScanNet vs 200 on ScanNet++), suggesting the method is not overfit to one benchmark's label set.","If the reported accuracy gains transfer to deployment, robots and AR devices can name and locate arbitrary objects in a scene in real time without a fixed category list."],"supporting_citations":[{"why":"the OpenFusion baseline that OpenFusion++ extends; supplies the TSDF mapping architecture and Hungarian matching that all new modules plug into, and is the comparison point for every reported improvement.","marker":"[4]"},{"why":"SEEM, the segmenter that provides instance masks, per-pixel confidence maps, and object-centric semantic embeddings used by confidence-guided sampling and the query's object branch.","marker":"[5]"},{"why":"CLIP, the contrastive vision-language model defining the shared embedding space that open-vocabulary alignment and Alpha-CLIP build on.","marker":"[1]"},{"why":"Alpha-CLIP, the region-focused CLIP variant used in the second query stage to match a query against instance environment features.","marker":"[24]"},{"why":"ICL dataset, used for qualitative query-response tests and the progressive ablation of the query framework.","marker":"[25]"},{"why":"ScanNet dataset, 20-class indoor RGB-D data used in the quantitative comparison of Table I.","marker":"[26]"},{"why":"ScanNet++ dataset, 200-class extension used to test performance at finer semantic granularity in Table I.","marker":"[27]"},{"why":"Replica dataset, photorealistic indoor scenes used for the quantitative comparison and the module ablations in Table II.","marker":"[28]"}],"fun_headline_variants":["Three targeted fixes lift open-vocabulary 3D scene understanding","Real-time 3D mapper sharpens semantics and query responses","Area-weighted cache and dual-path queries boost 3D mapping","OpenFusion++ refines 3D semantics with confidence, cache, and context","Confidence fusion and adaptive cache sharpen open-vocab 3D mapping"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that each centimeter-scale voxel block contains exactly one instance, so selecting the highest-average-confidence point set decides the block's label; when two instances occupy the same block, the lower-confidence instance is erased, and the paper acknowledges that confidence sampling then 'dominates insertions, preventing correction.'","fun_headline_variants_meta":{"raw":{"variants":["Three targeted fixes lift open-vocabulary 3D scene understanding","Real-time 3D mapper sharpens semantics and query responses","Area-weighted cache and dual-path queries boost 3D mapping","OpenFusion++ refines 3D semantics with confidence, cache, and context","Confidence fusion and adaptive cache sharpen open-vocab 3D mapping"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000843,"raw_usage":{"total_tokens":3659,"prompt_tokens":923,"completion_tokens":2736,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":2641}},"tokens_in":539,"tokens_out":2736,"duration_ms":19879,"temperature":1.0,"reasoning_tokens":2641,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:56:13.480831+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a controlled sequence with two small, adjacent objects of different categories (e.g., a mug on a bookshelf) whose extent is comparable to one voxel block, run OpenFusion++ end to end; if the reconstructed map omits or mislabels the lower-confidence object only where the two overlap, the single-instance-per-block assumption is the cause. The paper's own Replica ablation, where confidence sampling alone lowers f-mIoU by 1.87 points, provides a baseline expectation for this failure mode.","supporting_citations":[{"cited_title":"Open-fusion: Real-time open-vocabulary 3d mapping and queryable scene representation,","cited_arxiv_id":null,"evidence_quote":"the OpenFusion baseline that OpenFusion++ extends; supplies the TSDF mapping architecture and Hungarian matching that all new modules plug into, and is the comparison point for every reported improvement."},{"cited_title":"Segment everything everywhere all at once,","cited_arxiv_id":null,"evidence_quote":"SEEM, the segmenter that provides instance masks, per-pixel confidence maps, and object-centric semantic embeddings used by confidence-guided sampling and the query's object branch."},{"cited_title":"Alpha-clip: A clip model focusing on wherever you want,","cited_arxiv_id":null,"evidence_quote":"Alpha-CLIP, the region-focused CLIP variant used in the second query stage to match a query against instance environment features."},{"cited_title":"A benchmark for rgb-d visual odometry, 3d reconstruction and slam,","cited_arxiv_id":null,"evidence_quote":"ICL dataset, used for qualitative query-response tests and the progressive ablation of the query framework."}],"review_version":1}