{"id":"d585388d-577b-4f99-b53a-a85e9c5b4c38","arxiv_id":"2506.03073","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LEG-SLAM is a real-time RGB-D SLAM that jointly renders photorealistic images and open-vocabulary semantic masks by distilling PCA-compressed DINOv2 features into 3D Gaussians.","lead":"LEG-SLAM adds open-vocabulary language features to a real-time 3D Gaussian Splatting SLAM system, using DINOv2 embeddings compressed with PCA and queried with text via Talk2DINO. The authors report 10-18 fps mapping on Replica and ScanNet, roughly 30x faster than slower semantic mapping baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ScanNet mIoU is not a valid open-vocabulary measure: the PCA compressor used in evaluation is trained on the same 20 categories being scored, so the reported 41.4 mIoU may be inflated and the core semantic claim currently rests on an untested transfer assumption.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing concern: the quantitative semantic evaluation is contaminated because the PCA compressor is trained on the 20 ScanNet categories used as benchmark labels. This is the right focal point because the paper's novelty is not merely real-time dense mapping — ORB-SLAM3 and 3DGS already provide that — but the combination of real-time SLAM with open-vocabulary language queries. The speed numbers in Table 1 and Table 2 are internally consistent and give independent support to the real-time claim. The semantic claim, however, rests entirely on the contaminated ScanNet evaluation; the qualitative 'backpack' example is not quantified and is not a substitute. A duplicated OVO-Gaussian-SLAM row in Table 2 also weakens the comparison table, but it does not directly threaten the central semantic argument as much as the PCA leakage does. The proposed leave-one-class-out check is a clean way to decide whether the PCA transfer assumption holds: if held-out-class mIoU collapses, the reported 41.4 is not a valid open-vocabulary metric and the paper should be conditional on re-evaluation with a class-disjoint compressor.","tokens_in":13879,"tokens_out":6859,"duration_ms":83098,"concrete_test":"Retrain the PCA compressor with a leave-one-class-out scheme over the 20 ScanNet evaluation categories: for each class, fit PCA on the remaining 19 categories only, then evaluate LEG-SLAM's mIoU on the held-out class, and report the mean held-out mIoU across all 20 classes. If this held-out mIoU is substantially below the reported 41.4 average, the number is inflated by evaluation-set PCA training, and the open-vocabulary claim lacks quantitative support.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central novelty is 'real-time SLAM with language features in an unified manner, enabling open-vocabulary segmentation' (§1, Abstract). The speed component is independently supported by Table 1 (57.02 ms/frame at 640×480, about 17.5 fps; 95.21 ms at 1200×680, about 10.5 fps) and is plausible. The semantic component, however, is not independently supported. In the Supplementary Material ('ScanNet dataset'), the authors state that 'the same 20 categories were used to train the PCA encoder for ScanNet dataset evaluation' — exactly the categories used to compute the reported 41.4 mIoU and 74.3 mAcc. This contradicts the main-text claim (§3.3) that PCA is trained on 1,000 ImageNet text classes to learn a general projection. Because the same PCA is applied both to DINOv2 visual features and to the text-query embedding (§3.5), fitting it on the evaluation classes can only make dot-product matching easier for those classes, so Table 3's mIoU is not evidence for transfer to unseen categories. The paper's assertion that PCA 'learns a generalized compression strategy' is unsupported by the presented experiments: all quantitative semantic evaluations use the contaminated compressor. Thus the load-bearing part of the novelty (open-vocabulary semantics, not just real-time mapping) currently lacks a clean quantitative test.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"LEG-SLAM proposes a real-time dense RGB-D SLAM system that jointly renders color, depth, and language-semantic features by integrating ORB-SLAM3 tracking, DINOv2 visual features, a PCA-based feature compressor, Talk2DINO text-to-vision embedding alignment, and a 3D Gaussian Splatting mapper. The paper reports per-frame processing times (57.02 ms at 640x480, 95.21 ms at 1200x680), reconstruction quality on Replica, and open-vocabulary segmentation results on ScanNet, claiming more than 10 fps on Replica and 18 fps on ScanNet and up to a 30x speedup over prior language-augmented Gaussian Splatting methods.","tokens_in":14205,"tokens_out":3141,"duration_ms":32532,"significance":"If the quantitative claims hold, the speed component is a useful engineering contribution: the per-stage timing breakdown in Table 1 is internally consistent, and the ablations (Tables 4-5) provide a clear view of the accuracy/speed trade-off from PCA dimensionality. The paper also delivers a concrete system description and comparisons across multiple baselines. However, the central novelty is open-vocabulary semantic SLAM, and the evidence for that claim is weakened by a circularity in the ScanNet evaluation and by an apparent copy-paste error in the comparison table. The contributions are significant if the semantic evaluation is re-done cleanly, but they are not presently established beyond the real-time mapping component.","major_comments":[{"comment":"The ScanNet semantic evaluation uses a PCA compressor trained on exactly the same 20 categories used to compute mIoU and mAcc, as stated in the supplementary material. This contradicts the main-text claim (§3.3) that PCA is trained on 1,000 ImageNet text classes to learn a general projection. Since the same PCA projection is applied to both the visual embeddings and the text-query embedding (§3.5), training on the evaluation classes can only make dot-product matching easier for those classes. The reported 41.4 mIoU and 74.3 mAcc therefore do not measure transfer to unseen categories and cannot support the open-vocabulary claim. The authors must re-run the quantitative evaluation with a PCA trained on the 1,000 ImageNet classes (or another held-out class set) and report the resulting metrics, or clearly label the current numbers as closed-vocabulary results.","section":"Supplementary Material, ScanNet dataset; §4.4; §3.3"},{"comment":"The row for OVO-Gaussian-SLAM is numerically identical to the row for Gaussian-SLAM: ATE RMSE 0.31, Depth L1 0.68, PSNR 42.08, SSIM 0.996, LPIPS 0.018. Because Table 3 lists different semantic metrics for OVO-Gaussian-SLAM and the two methods are distinct, this appears to be a copy-paste error. The comparison is not trustworthy as presented. The authors must provide the correct OVO-Gaussian-SLAM SLAM metrics and clarify the source of these numbers.","section":"Table 2"},{"comment":"The claim that LEG-SLAM is 'the only method in this comparison capable of integrating real-time SLAM with open-vocabulary scene understanding' is not supported by the table as printed because OVO-Gaussian-SLAM is also marked 'Open-Vocabulary'. Depending on the corrected OVO-Gaussian-SLAM row, the 'first real-time' claim may need to be re-scoped. A direct runtime and capability comparison with OVO-Gaussian-SLAM (and OVO-SLAM) under identical hardware and input conditions is needed to substantiate the speed/novelty claim.","section":"§4.5; Table 2"},{"comment":"The 'Training Time' comparison in Table 3 mixes settings: LangSplat, Feature3DGS, and Semantic Gaussians are per-scene offline optimization methods without SLAM tracking, while LEG-SLAM is an online SLAM system. Comparing 1.5 minutes per scene to those methods conflates unrelated workloads. The speed advantage should be reported against methods that operate in the same online SLAM setting, and the exact timing protocol (per-frame from Table 1 vs. total scene processing) should be stated.","section":"§4.4; Table 3"}],"minor_comments":[{"comment":"The abstract describes PCA as a 'learnable feature compressor,' but PCA is normally a fixed linear projection fit once, not learned online. Please adjust the wording to 'learned' or 'fit' and use consistent terminology with §3.3.","section":"Abstract; §3.3"},{"comment":"The text says the DINOv2 embeddings are 'compressed using an autoencoder,' but the actual method uses PCA. Please align the terminology to avoid confusion.","section":"§3.1"},{"comment":"The caption mentions a 'learning-based approach to a language-enhanced rasterizer for Gaussian Splatting,' but no learning of the rasterizer is described in the paper. Please clarify or rephrase.","section":"Figure 2 caption"},{"comment":"The 'Closed-V ocabulary' and 'Open-V ocabulary' entries contain an extra space due to a line break in the source. Please fix the formatting.","section":"Table 2 header"},{"comment":"The timing '33 ms' for DINOv2 + PCA is consistent with Table 1 (31.91 ms feature extraction plus 1.57 ms compression), but the sentence 'including embedding compression' could be clearer; please state the decomposition.","section":"§4.6.1"},{"comment":"The entries 'Counter not presented' and 'Person not presented' are ambiguous: do these classes not appear in the tested scenes, or are they excluded from evaluation due to annotation errors? Please clarify.","section":"Supplementary Material, Table 2"},{"comment":"The description 'the semantic vector H×W×K is assigned zero values' conflates an image-sized feature map with per-Gaussian vectors. Please rephrase to indicate that each Gaussian is initialized with a K-dimensional semantic vector of zeros.","section":"§3.4, step 1"}],"recommendation":"major_revision","confidential_remarks":"The two main issues—PCA training contamination in the ScanNet evaluation and the duplicated OVO-Gaussian-SLAM row in Table 2—are checkable by the editor. The open-vocabulary claim is central to the paper's novelty and currently rests on an evaluation that may be circular. The speed results themselves appear plausible and are not in question."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline is that this paper does deliver a real systems result: a Gaussian Splatting SLAM pipeline that runs at 10+ fps on Replica and 18 fps on ScanNet while producing text-queryable semantic maps. The per-stage timings in Table 1 are internally consistent, the ablation on embedding dimension is sensible, and the comparison to OVO-Gaussian-SLAM's 0.28 fps makes the 'first real-time open-vocabulary' claim plausible. The PCA compression of DINOv2 features down to 64 dimensions so they can be fed into the Gaussian optimizer is a neat trick, and the integration with Talk2DINO is a reasonable way to align text with visual features. This is a useful contribution to the Gaussian SLAM subfield, and I expect it will be cited as a reference point for what is now possible on timing.\n\nThe soft spot is real and it sits right on the main semantic claim. In the supplementary, the authors state that 'the same 20 categories were used to train the PCA encoder for ScanNet dataset evaluation' - exactly the 20 classes scored in Table 3. The main text says the PCA is trained on 1,000 ImageNet text classes to learn a general projection. Those statements contradict each other, and if the evaluation-time PCA has seen the test categories, the reported 41.4 mIoU is not evidence for open-vocabulary transfer. The paper offers a hand-wavy claim that PCA learns a 'generalized compression strategy,' but there is no experiment that tests transfer to held-out classes with a PCA that never saw them. The speed numbers, the rendering quality, and the qualitative segmentation results are unaffected by this, but the quantitative semantic evaluation is compromised. This is a load-bearing flaw for the open-vocabulary novelty, not a minor blemish.\n\nA second, smaller issue: in Table 2, the OVO-Gaussian-SLAM row has exactly the same ATE, Depth L1, PSNR, SSIM, and LPIPS as Gaussian-SLAM (0.31, 0.68, 42.08, 0.996, 0.018) and differs only in FPS. That looks like a copy-paste error and needs correction. The paper also mentions an 'autoencoder' in Section 3.1 but the actual method is PCA, which is a minor inconsistency. No code is released despite a project page.\n\nFor a systems paper, the speed contribution stands. For the semantic claim, the authors need to retrain PCA on held-out categories (or use the ImageNet-trained PCA for evaluation), release code, and fix the table. I'd send it to peer review with that as a required revision. The community interested in real-time language-queryable mapping will get value from the system and the engineering choices, even if the current numbers overstate the semantic quality.","headline":"The speed claim is real and the integration is a genuine systems contribution, but the semantic evaluation is contaminated by PCA trained on the test categories, so the open-vocabulary claim is not yet backed by clean numbers.","tokens_in":782,"tokens_out":1301,"would_cite":true,"duration_ms":44015,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims LEG-SLAM is the first real-time SLAM system that unifies language features, open-vocabulary segmentation, and Gaussian-splatting mapping, running at more than 10 fps on Replica and 18 fps on ScanNet.","keywords":["3D Gaussian Splatting","Simultaneous Localization and Mapping","open-vocabulary segmentation","DINOv2","language-enhanced mapping","PCA feature compression","RGB-D SLAM","real-time semantic mapping"],"falsifier":"Train the PCA projector on a held-out set of categories (for example, only 10 of the 20 ScanNet classes, or ImageNet classes with ScanNet categories removed) and evaluate mIoU on all 20 classes; if held-out categories show a clear drop well below the 41.4 mIoU reported when the compressor saw the evaluation classes, the generalization claim behind the open-vocabulary numbers is falsified. A second check is to run the full pipeline with a 10 Hz RGB-D stream and verify end-to-end per-frame latency under 100 ms including a text query, since the real-time claim depends on that budget.","tokens_in":13719,"feed_emoji":"🗺️","tokens_out":9066,"duration_ms":83161,"temperature":0.7,"pith_summary":"The paper sets out to prove that a single online SLAM system can build a dense 3D Gaussian scene that is both photorealistic and queryable by arbitrary text, with no precomputed semantic map or offline preparation. The central claim is that LEG-SLAM is the first real-time system to do this, running at more than 10 frames per second on Replica and 18 frames per second on ScanNet while segmenting scenes by text queries such as “vase” or “chair.” It attributes this to an efficient combination of DINOv2 feature extraction, PCA-based compression of semantic features, and a Gaussian-splatting mapper that optimizes color, depth, and language features together. If the claim holds, interactive applications such as robots and augmented reality could ask a map where an object is while the map is being built, at sensor frame rates instead of after minutes of offline processing.","feed_headline":"Open-vocabulary 3D mapping runs in real time at 18 fps","feed_subtitle":"A Gaussian-splatting SLAM pipeline answers text queries like 'vase' while mapping, about 30x faster than prior language-aware methods.","key_machinery":"The central object is the semantic 3D Gaussian: each Gaussian’s parameters are augmented with a K-dimensional language-feature vector, initialized to zero and updated during mapping. The shared differentiable rasterizer is the mechanism that carries the argument, rendering color, depth, and semantic features in one pass so that semantic consistency and geometry are optimized on the same cloud. The DINOv2 feature encoder supplies the semantic signal; a PCA projector, trained on CLIP-projected, Talk2DINO-aligned ImageNet classes with noise augmentation, compresses 768-dimensional maps to 64 dimensions with negligible cosine-similarity loss; and Talk2DINO, a learned CLIP-to-DINOv2 bridge, lets arbitrary text queries enter the same compressed space. A coarse-to-fine Gaussian pyramid accelerates convergence, and ORB-SLAM3 provides poses.","core_discovery":"LEG-SLAM promises a unified pipeline in which each 3D Gaussian carries not only position, scale, orientation, and color but also a compressed K-dimensional DINOv2 language-feature vector, with K set to 64 in the final configuration. These semantic vectors are optimized jointly with the geometric and photometric parameters, so a single differentiable rasterizer renders RGB, depth, and a semantic feature map at the same time. At query time, a CLIP text embedding is mapped into DINOv2 space by Talk2DINO, compressed with the same PCA projector, and matched against the rendered feature map by dot product. The authors report that this yields open-vocabulary segmentation on ScanNet at 41.4 mIoU, reconstruction at about 32 dB PSNR on Replica, and a roughly 30x speedup over the compared language-Gaussian baselines, with a full ScanNet scene processed in about 1.5 minutes.","pith_inferences":["If the PCA compressor truly learns a general projection rather than the 20 benchmark classes, the same 64-dimensional features should serve queries outside the training labels; a direct test would train PCA on a disjoint label set and measure held-out mIoU.","The timing profile suggests feature extraction (about 32 ms at 640x480) dominates the 57 ms frame budget, so faster visual backbones or lower-resolution extraction would translate almost directly into higher frame rates or lower-power operation.","The paper’s Figure 2 names ORB-SLAM3 as the tracker, but the bibliography contains no ORB-SLAM3 citation; locating and citing that component matters for reproducing the SLAM accuracy and speed numbers, and the absence is a support gap rather than a refutation of the speed claim.","Because the semantic vectors are optimized online, a natural extension is to refine the map from user text feedback during exploration, for example by using a query heatmap as a pseudo-label, which the paper does not explore."],"forward_implications":["A robot or AR device could query a scene while mapping it, without a pre-built semantic map or precomputed poses, which the paper argues is the first such capability.","Scene-level open-vocabulary analysis that previously took tens of minutes to hours (LERF, LangSplat, Feature3DGS) would drop to about 1.5 minutes per ScanNet scene, making interactive querying practical.","Because semantic features live inside the Gaussian cloud, the same map can be re-queried for any number of text concepts without rerunning segmentation for each query.","The 64-dimensional compressed features keep segmentation near 41.4 mIoU while preserving real-time rendering, so language-enhanced maps can be built at sensor frame rate with modest GPU memory."],"supporting_citations":[{"why":"Supplies the pretrained DINOv2 backbone that produces the 768-dimensional visual-language embeddings distilled into the Gaussians.","marker":"[22]"},{"why":"Provides the Talk2DINO module that maps CLIP text embeddings into DINOv2 space, enabling open-vocabulary queries.","marker":"[1]"},{"why":"Supplies the CLIP text encoder used to produce query embeddings and the ImageNet class text embeddings used to train the PCA compressor.","marker":"[24]"},{"why":"Provides the base 3D Gaussian Splatting representation and differentiable rasterizer that the system extends with semantic features.","marker":"[12]"},{"why":"LangSplat is the main language-Gaussian baseline that LEG-SLAM compares against and accelerates.","marker":"[23]"},{"why":"SplaTAM is a Gaussian-based dense SLAM baseline used for reconstruction and tracking comparison.","marker":"[11]"},{"why":"SGS-SLAM is a closed-vocabulary semantic Gaussian SLAM method that LEG-SLAM extends toward open-vocabulary operation.","marker":"[16]"},{"why":"Photo-SLAM is a real-time SLAM baseline listed in the speed and reconstruction comparisons.","marker":"[7]"}],"fun_headline_variants":["LEG-SLAM: Real-time language-aware Gaussian splatting at 18 fps","Open-vocabulary Gaussian splatting SLAM runs at 18 fps","18 fps language-aware SLAM with Gaussian splatting","Real-time semantic mapping via language-enhanced Gaussians","Gaussian splatting SLAM with language features at 18 fps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the PCA compressor trained to reconstruct the 20 ScanNet semantic categories generalizes to arbitrary text queries, because the reported 41.4 mIoU is measured with a compressor that has already seen those same 20 categories; if that transfer fails, the open-vocabulary accuracy is unmeasured.","fun_headline_variants_meta":{"raw":{"variants":["LEG-SLAM: Real-time language-aware Gaussian splatting at 18 fps","Open-vocabulary Gaussian splatting SLAM runs at 18 fps","18 fps language-aware SLAM with Gaussian splatting","Real-time semantic mapping via language-enhanced Gaussians","Gaussian splatting SLAM with language features at 18 fps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00149,"raw_usage":{"total_tokens":5997,"prompt_tokens":977,"completion_tokens":5020,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":593,"completion_tokens_details":{"reasoning_tokens":4927}},"tokens_in":593,"tokens_out":5020,"duration_ms":36202,"temperature":1.0,"reasoning_tokens":4927,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:09:40.772608+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the PCA projector on a held-out set of categories (for example, only 10 of the 20 ScanNet classes, or ImageNet classes with ScanNet categories removed) and evaluate mIoU on all 20 classes; if held-out categories show a clear drop well below the 41.4 mIoU reported when the compressor saw the evaluation classes, the generalization claim behind the open-vocabulary numbers is falsified. A second check is to run the full pipeline with a 10 Hz RGB-D stream and verify end-to-end per-frame latency under 100 ms including a text query, since the real-time claim depends on that budget.","supporting_citations":[{"cited_title":"DINOv2: Learning Robust Visual Features without Supervision.Transactions on Machine Learning Research Journal, pages 1–31, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained DINOv2 backbone that produces the 768-dimensional visual-language embeddings distilled into the Gaussians."},{"cited_title":"3D Gaussian Splatting for Real-Time Radiance Field Rendering.ACM Transactions on Graphics, 42(4):1–14, 2023","cited_arxiv_id":null,"evidence_quote":"Provides the base 3D Gaussian Splatting representation and differentiable rasterizer that the system extends with semantic features."},{"cited_title":"SplaTAM: Splat, Track & Map 3D Gaus- sians for Dense RGB-D SLAM spla-tam","cited_arxiv_id":null,"evidence_quote":"SplaTAM is a Gaussian-based dense SLAM baseline used for reconstruction and tracking comparison."},{"cited_title":"Photo-SLAM: Real-time Simultaneous Localization and Photorealistic Mapping for Monocular Stereo and RGB- D Cameras","cited_arxiv_id":null,"evidence_quote":"Photo-SLAM is a real-time SLAM baseline listed in the speed and reconstruction comparisons."}],"review_version":1}