Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

This paper claims that query-conditioned novel view synthesis from a 3D Gaussian Splatting scene improves embodied reasoning and 3D grounding over fixed-view memory and language-embedded 3DGS baselines.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 09:36 UTC pith:OASU5O6M

load-bearing objection The paper's core selling point, that novel-view synthesis drives the EM-EQA gains, doesn't survive its own ablation: the headline number comes from a verification step that is orthogonal to view synthesis. the 4 major comments →

arxiv 2601.13132 v2 pith:OASU5O6M submitted 2026-01-19 cs.CV

SplatReasoner: Enhancing Embodied Reasoning and Grounding by Novel View Synthesis

Kim Yu-Ji , Dahye Lee , Kim Jun-Seong , Nam Hyeon-Woo , GeonU Kim , Yongjin Kwon , Yu-Chiang Frank Wang , Jaesung Choe
show 1 more author
Tae-Hyun Oh
This is my paper
classification cs.CV
keywords embodied question answering3D Gaussian Splattingnovel view synthesisvision-language models3D referring segmentationview selectionepisodic memoryspatial reasoning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to show that a Vision-Language Model (VLM) answers spatial and compositional questions about an indoor scene better when it is shown novel camera views synthesized from a 3D Gaussian Splatting reconstruction, rather than only the originally recorded frames. The proposed pipeline finds which 3D Gaussians match the question, groups them into object instances, selects the recorded view where each instance is most visible, and then renders perturbed versions of that view, letting a VLM judge which one best supports an answer. On an episodic-memory question answering benchmark this raises the semantic match score from 54.6 for the best fixed-view memory baseline to 57.8 at the same average number of input frames, and on a new 3D referring segmentation benchmark it raises 3D mIoU from 10.56 to 12.87. This matters because it offers a concrete way to give VLMs spatial memory without requiring the agent to physically move.

Core claim

The central claim is that query-conditioned novel view synthesis is an effective way to extract visual evidence for VLM reasoning in 3D scenes. Starting from a semantic 3D Gaussian scene, the method activates Gaussians whose categories match evidence categories extracted from the question, clusters them into instances, and selects the best initial view per instance via a visibility score based on rendering weights. It then synthesizes candidate novel views by shifting or zooming the camera, and a VLM-as-Judge selects the viewpoint whose rendered answer is most informative; a final verification step compares the refined answer against the initial one and chooses the better. The selected viewp

What carries the argument

Semantic 3D Gaussian Splatting with a visibility-based view scoring function and a VLM-as-Judge novel-view adjustment loop. The visibility score renders an activation map from the target Gaussian cluster and checks, at each pixel, whether the highest-weight Gaussian belongs to the cluster, giving an occlusion-robust view selection criterion. The VLM-as-Judge stage renders four perturbed views per initial view, obtains a per-view VLM answer, uses an LLM to pick the best answer/view, and then verifies against the initial view to decide whether the novel view is truly needed.

Load-bearing premise

The pipeline only activates Gaussians whose semantic category appears in the pre-defined object category list, so any query whose key object or attribute is absent from that list activates no evidence and the reasoning chain has nothing to look at.

What would settle it

Run the pipeline on a set of questions whose key objects are deliberately absent from the closed-set category list (for example 'rug' when the detector has no such class); if activated Gaussians drop to zero and the model cannot answer while a fixed-view VLM baseline can, the claimed query-conditioning advantage fails for out-of-vocabulary queries.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Embodied agents can answer 'where is X' or 'what is the color of the X in the kitchen' by actively rendering better views instead of being locked to whatever the camera happened to record.
  • The VLM-as-Judge selection provides a general mechanism for picking the most informative viewpoint for any view-dependent question, not just the specific benchmarks tested here.
  • The same pipeline yields 3D referring segmentation: once the final answer view is chosen, the activated Gaussians inside its frustum localize the referred instance, so question answering and grounding share one stage.
  • The gain is not bought with more frames: the approach uses about the same average number of input images as the fixed-view memory baseline (2.6 vs 2.7).

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The reported gain may partly reflect VLMs' preference for centered or zoomed-in content rather than true 3D understanding; a control that decouples viewpoint informativeness from VLM composition bias would clarify the mechanism.
  • Because evidence extraction is category-based, the method likely handles compositional queries only within a closed vocabulary; extending it to open-vocabulary evidence retrieval (e.g., via the open-set embeddings the pipeline already uses for grounding) is the natural next step.
  • The VLM-as-Judge score could be reused as a reward for active next-best-view planning, turning this static-memory system into a stepping stone for embodied navigation.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper (listed as arXiv:2601.13132, though the full text is titled 'GaussExplorer') proposes a framework that uses 3D Gaussian Splatting as a semantic episodic memory for embodied question answering and 3D referring segmentation. Given a user query, the method (1) extracts evidence categories via an LLM, (2) activates and clusters semantically relevant Gaussians, (3) selects initial camera views using a visibility score, (4) generates novel candidate views through 3DGS and uses a VLM as a judge to select the most informative view, and (5) optionally verifies the final answer against the initial-view answer. The authors report 57.8 LLM-Match on EM-EQA versus 54.6 for 3D-Mem with comparable average frame count, and 12.87 3D mIoU on a newly curated 49-question 3D referring segmentation benchmark versus 10.56 for Dr. Splat.

Significance. If the central claim were established, the paper would make a useful contribution: it combines 3DGS with VLM-based reasoning in a way that can synthesize query-conditioned novel views and perform fine-grained 3D localization. The visibility-based initial view selection is a reasonable and partially novel idea, and the paper usefully shows that a semantic 3DGS representation can be integrated with a VLM pipeline. The new 3D referring segmentation benchmark, though small, addresses a real gap. However, the central causal claim — that novel-view synthesis is what drives the reported gains — is not currently isolated by the experiments. The largest EM-EQA improvement comes from a verification stage that is orthogonal to novel-view synthesis, and the 3D segmentation gain is 0.41 mIoU on 49 questions. The evidence is suggestive but not yet convincing. A revised version with proper ablations and statistical care could change this assessment.

major comments (4)
  1. [Table 6, Sec. 4.3] The central claim that query-conditioned novel view synthesis improves embodied reasoning is not isolated. In Table 6, the full pipeline with novel views but without verification ('Final view') scores 54.5 LLM-Match, effectively identical to 3D-Mem's 54.6. The reported 57.8 gain appears only after adding a verification stage that compares initial and final-view answers and prefers informative answers over 'can't tell' responses. This verification step is not specific to novel-view synthesis. To support the abstract's claim, the paper must report an initial-view-only condition on the full 1,623 questions using the same prompts and verification, and a final-view condition without novel views, so that the contribution of synthesis is separated from the contribution of verification/prompt design.
  2. [Fig. S6, Sec. 3.3] The VLM-as-Judge/verification loop is circular in a way that biases the result. The view-selection LLM chooses the best answer from candidate views, and the verification prompt explicitly instructs the model to 'prefer the one that gives the information' and to reject answers that say 'can't tell'. This conflates 'answers with more confident/verbose content' with 'answers that are grounded in the visual evidence.' Because the same kind of model is used to select, verify, and evaluate answers, the reported gain may reflect prompt-induced answer-style preferences rather than evidence actually revealed by novel synthesis. A control with blinded view order, or with verification applied to initial views only, is needed to rule out this confound.
  3. [Eq. (1), Sec. 3.2] The method's query grounding is closed-set for the EM-EQA evaluation. Gaussians are activated only when their category exactly matches one of the LLM-extracted evidence categories from ScanNet200/YOLOv8. If the user query refers to an attribute, state, or object category not in the closed inventory — e.g., 'blue mug', 'dusty shelf', 'the chair with a broken leg' — no Gaussians are activated and the pipeline has no evidence to reason over. The paper does not report a sensitivity analysis or an open-vocabulary variant for EM-EQA. This is a modeling assumption that limits the generality of the claimed improvement and should be explicitly tested or acknowledged as a boundary condition.
  4. [Sec. 4.2, Table 2, Sec. C] The 3D referring segmentation evidence is too weak to support the grounding claim. The benchmark contains only 49 human-written questions over 24 instances in 5 scenes. The gain of the full method over Dr. Splat (question) is 12.87 vs 10.56, and the novel-view increment is only 0.41 mIoU (12.46 to 12.87). With this sample size and stochastic LLM-based components, these differences are within plausible noise; no error bars, significance tests, or per-scene breakdowns are provided. Additionally, the ReferSplat comparison uses manually generated training data (described in Sec. 4.2) because no official pipeline exists, which may disadvantage that baseline. The authors should provide variance estimates, a per-question analysis, and a fairer ReferSplat protocol before claiming superiority in 3D grounding.
minor comments (5)
  1. [Title/Abstract] The arXiv metadata and abstract refer to 'SplatReasoner', while the full paper is titled 'GaussExplorer'. This identity mismatch must be fixed before publication.
  2. [Throughout] Typos and formatting errors: 'shwon' (Sec. 4.3), 'baesd' (Fig. 6), 'Gaussain' (Suppl. A.1), 'downstars' (Fig. 8), 'dateset' (Sec. 4.2), 'LLaV A' (Sec. 4.1), and 'ReferSplat [12]' in Supp. B.2 (should be [9]).
  3. [Sec. 4.3] All ablations on view selection and score functions are done on only 184 questions with no error bars or significance testing. For instance, the 48.2 vs 45.4 gain in Table 3 and the 50.5 vs 48.2 gain in Table 4 are reported as deterministic numbers. The small subset and stochastic LLM-Match judge make these differences unreliable; at minimum, report standard errors or repeated evaluations.
  4. [Sec. 3.3, Supp. A.2] The method has several free parameters whose sensitivity is not reported: the number of evidence categories k, the similarity threshold τ in Eq. (2), the HDBSCAN parameters (mc, ms), the merge distance ε, and the number of novel-view perturbations per initial view (K = L×4). Since ε is stated to control the average number of frames, a sensitivity table is needed to show that results are not tuned to a specific operating point.
  5. [Table 1, Sec. 4.1] The 'average frames' comparison is informative, but the footnote defines it as 'the number of initial or final views fed to VLMs.' This excludes frames used in the novel-view adjustment stage and the verification stage, so it does not reflect total VLM inference cost. Please report the actual number of VLM calls or rendering cost for a fair efficiency comparison.

Circularity Check

1 steps flagged

The headline EM-EQA gain (57.8 vs 54.6) is not an isolated novel-view effect: Table 6 shows final-view-only at 54.5, and the 3.3-point gain is added by a verification prompt that by construction prefers informative answers over 'can't tell' responses.

specific steps
  1. fitted input called prediction [Sec. 4.3 'Verification process' (Table 6) and Fig. S6 verification prompt]
    "Method LLM-Match (↑) Average Frames — Final view 54.5 2.6 † — Final view + verification (ours) 57.8 2.6 † ... 'Using only initial or final views yields limited performance, whereas incorporating the full verification step significantly improves LLM-Match accuracy with no increase in rendering cost.' ... 'Reject any that say they can’t tell/are unsure ... If one answer gives information about the object while another says it is not visible, prefer the one that gives the information.'"

    The EM-EQA prediction is forced by the verification selection rule rather than by novel-view synthesis. The isolated novel-view condition (Final view) scores 54.5, essentially tied with the 3D-Mem baseline (54.6); the reported 57.8 appears only after the verification step, whose prompt instructs the judge to reject 'can't tell' answers and prefer answers that give information. Since the final answer is the judge's selected candidate (which may be the initial-view answer), the gain is produced by the selection rule by construction, and the paper never reports an initial-view-only condition with identical prompts/verification on the full 1,623 questions to isolate the novel-view contribution.

full rationale

This is a partial circularity, not a full derivation collapse. The 3D referring segmentation results and the subset ablation of initial-vs-final view selection are separate measurements and are not definitionally forced; no load-bearing self-citation chain is present (Dr. Splat [12] is used as a component and baseline, but the comparison is externally checkable). However, the paper's central EM-EQA claim—that query-conditioned novel-view synthesis beats fixed-view memory—rests on the full-1,623 comparison in Table 1, and Table 6 shows the novel-view-only condition does not beat 3D-Mem. The full-pipeline gain is added by a verification prompt that explicitly prefers informative answers over uncertainty, so the headline number is not a measure of what novel views reveal; it is a measure of the answer-selection rule. A fair reader should therefore treat the EM-EQA headline as not isolating novel-view synthesis, while acknowledging the benchmark and 3D localization components are not circular. The closed-set category limitation is a modeling constraint, not a circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The central claim rests on several domain assumptions about the sufficiency of closed-set semantics, the informativeness of visibility, and the VLM's ability to judge views. These are not derived from first principles and are only weakly validated. The free parameters are mostly tuning choices consistent with prior work (3D-Mem), but several are undisclosed.

free parameters (4)
  • k (number of evidence categories) = 3
    Following 3D-Mem, the LLM extracts the top-3 categories most relevant to the question; this determines which Gaussians are activated and thus all downstream view selection.
  • similarity threshold τ for open-set activation = not reported
    In Eq. (2), Gaussians are activated if their CLIP/text similarity exceeds τ; the threshold is scene-dependent and its value is not given, affecting which objects are retrieved.
  • HDBSCAN parameters (mc, ms) and merge distance ε = not reported
    These control instance clustering of activated Gaussians and the number of selected frames; tuned on the 184-question subset, the chosen values are not disclosed.
  • Number of novel-view perturbations per initial view = 4
    Each initial view is perturbed left/right/zoom in/out to produce four candidates; this choice affects coverage of the view space and compute.
axioms (4)
  • domain assumption The closed-set category inventory (YOLOv8/ScanNet200) covers all query-relevant visual evidence.
    Eq. (1) retrieves Gaussians whose category matches the LLM-extracted evidence categories; if the query references an object outside this set, no Gaussians are activated and the pipeline cannot reason. This is load-bearing and not verified in the paper.
  • domain assumption Visibility score (fraction of visible Gaussians by rendering weight) is a reliable proxy for view informativeness.
    Eq. (5) uses the count of top-weight Gaussians in the target cluster as the score for selecting initial views; the paper tests this only against a volume-based alternative, not against other possible measures.
  • domain assumption A VLM's answer-based judgment of view informativeness improves downstream answer accuracy.
    The VLM-as-Judge selects final views based on which rendered candidate yields the best answer to the query; the entire novel-view adjustment rests on the assumption that the VLM can correctly identify the most informative view.
  • domain assumption Nearby Gaussians of the same category belong to the same object instance.
    Clustering into instances (Sec. 3.2) assumes spatial proximity plus category identity defines an object; this can fail for adjacent same-category objects or fragmented objects.

pith-pipeline@v1.3.0-alltime-deepseek · 16450 in / 14458 out tokens · 131648 ms · 2026-08-03T09:36:23.450896+00:00 · methodology

0 comments
read the original abstract

Vision-Language Models (VLMs) have demonstrated strong reasoning capabilities over images and videos, yet their application to embodied scene understanding often constrained by the fixed viewpoints stored in episodic RGB-D memories. These observations may fail to capture query-relevant evidence due to occlusions, object truncation, restricted fields of view, or suboptimal view composition. We present SplatReasoner, a framework that introduces novel view synthesis into the VLM reasoning process by leveraging 3D Gaussian Splatting (3DGS). Given a user query about a 3D scene, SplatReasoner retrieves relevant observations and synthesizes query-conditioned viewpoints that reveal the visual evidence needed to answer the query and ground the referred entities in 3D. Experiments show that query-conditioned novel view synthesis improves both embodied reasoning and 3D grounding over fixed-view memory and language-embedded 3DGS baselines.

Figures

Figures reproduced from arXiv: 2601.13132 by Dahye Lee, GeonU Kim, Jaesung Choe, Kim Jun-Seong, Kim Yu-Ji, Nam Hyeon-Woo, Tae-Hyun Oh, Yongjin Kwon, Yu-Chiang Frank Wang.

Figure 1
Figure 1. Figure 1: GaussExplorer aims at the embodied exploration and reasoning based on 3D Gaussian Splatting. Given an input question, we first identify initial viewpoints by searching for relevant 3D Gaussians. These viewpoints are then refined into novel-view images with the VLM-as-Judge mechanism, which evaluates rendered views to maximize visual evidence and decide the final viewpoints. Finally, the final views are pro… view at source ↗
Figure 2
Figure 2. Figure 2: Overview. (a) We first build a semantic 3DGS scene, where input views I and their semantic information produced by foundation models are lifted into 3D. (b) In initial view selection, (b-1) the query is first rephrased into relevant semantic categories C evidence by LLM, (b-2) activated 3D Gaussians associated with those categories are grouped into spatial clusters, and (b-3) representative training camera… view at source ↗
Figure 4
Figure 4. Figure 4: EQA pipeline comparison. The competing method [34] attempts to answer queries directly using initial views selected from given images, lacking any viewpoint refinement. In contrast, our method automatically explores nearby novel viewpoints via VLM￾based pose adjustment, maximizing visual evidence more clearly. Finally, a VLM-based verification step compares the initial and refined views to select the one p… view at source ↗
Figure 5
Figure 5. Figure 5: Novel-view adjustment. From an initial camera pose, we generate multiple novel-view candidates and refine them through a novel view adjustment module. Each candidate’s view is rendered and evaluated by a VLM with a visual-QA prompt to obtain initial answer predictions. These answers, together with the question, are then passed to an LLM to select the most informative final viewpoint. 3.3. Novel-view Adjust… view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative results of the 3D referring segmentation. Compared to the competing methods, ours more accurately identifies the fine-grained target locations referenced by the question. Dr. Splat is designed to find objects without distinguishing instances, which limits its ability to localize fine-grained target regions (category) and, consequently, makes it struggle to directly interpret complex language qu… view at source ↗
Figure 7
Figure 7. Figure 7: Initial view selection. Compared to 3D-Mem, our visibility-based policy can select more informative views by accounting for actual object visibility, rather than relying solely on category presence. Initial View Selection Final View Selection Question: Where can I access the attic? Ground truth: Door at the top of the stairs. Question: What is the blue object left of the downstars bed? Ground truth : A hum… view at source ↗
Figure 8
Figure 8. Figure 8: Final view selection. We apply novel view synthesis to refine initial selected views, leveraging a unique capability of 3DGS. Our results show that even when the initial views lack suffi￾cient information to answer the question, the novel-view adjustment process can recover the necessary viewpoints, enabling more accu￾rate question-aligned answers. Results [PITH_FULL_IMAGE:figures/full_fig_p007_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. ABot-AgentOS: A General Robotic Agent OS with Lifelong Multi-modal Memory

    cs.AI 2026-07 conditional novelty 6.0

    A robotic agent operating system with source-grounded graph memory and split-wise self-evolution improves long-horizon embodied task success and memory QA scores over baseline controllers.

  2. ABot-AgentOS: A General Robotic Agent OS with Lifelong Multi-modal Memory

    cs.AI 2026-07 conditional novelty 5.5

    A hierarchical robotic Agent OS with source-grounded multi-modal graph memory and split-gated self-evolution improves long-horizon embodied execution and memory QA over single-controller and prior memory baselines.

Reference graph

Works this paper leans on

40 extracted references · 2 linked inside Pith · cited by 1 Pith paper

  1. [1]

    Videonavqa: Bridging the gap between visual and embodied question answering

    C˘at˘alina Cangea, Eugene Belilovsky, Pietro Li`o, and Aaron Courville. Videonavqa: Bridging the gap between visual and embodied question answering. InBMVC, 2019. 2

    2019

  2. [2]

    All you may need for VQA are image captions

    Soravit Changpinyo, Doron Kukliansy, Idan Szpektor, Xi Chen, Nan Ding, and Radu Soricut. All you may need for VQA are image captions. InNorth American Chapter of the Association for Computational Linguistics, 2022. 2

    2022

  3. [3]

    Scannet: Richly- annotated 3d reconstructions of indoor scenes

    Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly- annotated 3d reconstructions of indoor scenes. InCVPR, pages 5828–5839, 2017. 5, 7, 2

    2017

  4. [4]

    Embodied question answering

    Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answering. InCVPR, 2018. 2

    2018

  5. [5]

    A density-based algorithm for discovering clusters in large spatial databases with noise

    Martin Ester, Hans-Peter Kriegel, J¨org Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. InInternational Confer- ence on Knowledge Discovery and Data Mining (KDD), 1996. 4

    1996

  6. [6]

    Lightgaussian: Unbounded 3d gaussian compression with 15x reduction and 200+ fps

    Zhiwen Fan, Kevin Wang, Kairun Wen, Zehao Zhu, Dejia Xu, Zhangyang Wang, et al. Lightgaussian: Unbounded 3d gaussian compression with 15x reduction and 200+ fps. In NeurIPS, 2024. 8

    2024

  7. [7]

    V2pe: Improving multimodal long- context capability of vision-language models with variable visual position encoding

    Junqi Ge, Ziyi Chen, Jintao Lin, Jinguo Zhu, Xihui Liu, Jifeng Dai, and Xizhou Zhu. V2pe: Improving multimodal long- context capability of vision-language models with variable visual position encoding. InICCV, 2025. 2, 5

    2025

  8. [8]

    Concept- graphs: Open-vocabulary 3d scene graphs for perception and planning

    Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. Concept- graphs: Open-vocabulary 3d scene graphs for perception and planning. InIEEE International Conference on Robotics and Automation, 2024. 2, 6

    2024

  9. [9]

    Refersplat: Referring segmentation in 3d gaussian splatting

    Shuting He, Guangquan Jie, Changshuo Wang, Yun Zhou, Shuming Hu, Guanbin Li, and Henghui Ding. Refersplat: Referring segmentation in 3d gaussian splatting. InICML,

  10. [10]

    Gpt-4o system card, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card, 2024. 1

    2024

  11. [11]

    Fastlgs: Speeding up language em- bedded gaussians with feature grid mapping

    Yuzhou Ji, He Zhu, Junshu Tang, Wuyi Liu, Zhizhong Zhang, Xin Tan, and Yuan Xie. Fastlgs: Speeding up language em- bedded gaussians with feature grid mapping. InAAAI, 2025. 2

    2025

  12. [12]

    Kim Jun-Seong, GeonU Kim, Kim Yu-Ji, Yu-Chiang Frank Wang, Jaesung Choe, and Tae-Hyun Oh. Dr. splat: Directly referring 3d gaussian splatting via direct language embedding registration. InCVPR, 2025. 2, 3, 6, 7, 8, 1, 4

    2025

  13. [13]

    3d gaussian splatting for real-time radiance field rendering.ACM TOG, 2023

    Bernhard Kerbl, Georgios Kopanas, Thomas Leimk¨uhler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering.ACM TOG, 2023. 2, 3, 4, 1

    2023

  14. [14]

    A hierarchical 3d gaussian representation for real-time ren- dering of very large datasets.ACM Transactions on Graphics (TOG), 43(4):1–15, 2024

    Bernhard Kerbl, Andreas Meuleman, Georgios Kopanas, Michael Wimmer, Alexandre Lanvin, and George Drettakis. A hierarchical 3d gaussian representation for real-time ren- dering of very large datasets.ACM Transactions on Graphics (TOG), 43(4):1–15, 2024. 1

    2024

  15. [15]

    Topological semantic graph memory for image-goal navigation

    Nuri Kim, Obin Kwon, Hwiyeon Yoo, Yunho Choi, Jeongho Park, and Songhwai Oh. Topological semantic graph memory for image-goal navigation. InAnnual Conference on Robot Learning, 2022. 2

    2022

  16. [16]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. InICCV, 2023. 3, 7, 2

    2023

  17. [17]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. InNeurIPS, 2023. 2

    2023

  18. [18]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. InCVPR,

  19. [19]

    Openeqa: Embodied question answering in the era of foundation models

    Arjun Majumdar, Anurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mc- vay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering in the era of foundation models. InCVPR, 2024. 2, 5, 6, 3

    2024

  20. [20]

    A hybrid approach to hierarchical density-based cluster selection

    Claudia Malzer and Marcus Baum. A hybrid approach to hierarchical density-based cluster selection. In2020 IEEE In- ternational Conference on multisensor fusion and integration for intelligent systems (MFI). IEEE, 2020. 4

    2020

  21. [21]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. https://arxiv.org/ abs/2303.08774, 2023. 2, 3, 6

    Pith/arXiv arXiv 2023

  22. [22]

    Gpt-4v(ision) system card

    OpenAI. Gpt-4v(ision) system card. https://openai. com/research/gpt-4v-system-card, 2023. 2, 3

    2023

  23. [23]

    Langsplat: 3d language gaussian splatting

    Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. Langsplat: 3d language gaussian splatting. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20051–20060, 2024. 2

    2024

  24. [24]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. InICML, 2021. 2, 3, 7, 1

    2021

  25. [25]

    Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai

    Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wi- jmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojciech Galuba, Andrew West- bury, Angel X Chang, et al. Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. InConference on Neural Information Processing Systems Datasets and Benchmarks T...

    2021

  26. [26]

    Language- grounded indoor 3d semantic segmentation in the wild

    David Rozenberszki, Or Litany, and Angela Dai. Language- grounded indoor 3d semantic segmentation in the wild. In ECCV, 2022. 5, 1, 2, 3

    2022

  27. [27]

    Grapheqa: Using 3d semantic scene graphs for real-time embodied question answering

    Saumya Saxena, Blake Buchanan, Chris Paxton, Peiqi Liu, Bingqing Chen, Narunas Vaskevicius, Luigi Palmieri, Jonathan Francis, and Oliver Kroemer. Grapheqa: Using 3d semantic scene graphs for real-time embodied question answering. InAnnual Conference on Robot Learning, 2025. 2

    2025

  28. [28]

    Losing visual needles in image haystacks: Vision language models are easily distracted in short and long contexts

    Aditya Sharma, Michael Saxon, and William Yang Wang. Losing visual needles in image haystacks: Vision language models are easily distracted in short and long contexts. In Findings of the Association for Computational Linguistics: EMNLP 2024, 2024. 2, 5 9

    2024

  29. [29]

    Language embedded 3d gaussians for open-vocabulary scene understanding

    Jin-Chuan Shi, Miao Wang, Hao-Bin Duan, and Shao-Hua Guan. Language embedded 3d gaussians for open-vocabulary scene understanding. InCVPR, 2024. 2

    2024

  30. [30]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Am- jad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. https: //arxiv.org/abs/2307.09288, 2023. 2

    Pith/arXiv arXiv 2023

  31. [31]

    Yolov8: A novel object de- tection algorithm with enhanced performance and robustness

    Rejin Varghese and M Sambath. Yolov8: A novel object de- tection algorithm with enhanced performance and robustness. In2024 International conference on advances in data engi- neering and intelligent computing systems (ADICS). IEEE,

  32. [32]

    Mm- longbench: Benchmarking long-context vision-language mod- els effectively and thoroughly

    Zhaowei Wang, Wenhao Yu, Xiyu Ren, Jipeng Zhang, Yu Zhao, Rohit Saxena, Liang Cheng, Ginny Wong, Simon See, Pasquale Minervini, Yangqiu Song, and Mark Steedman. Mm- longbench: Benchmarking long-context vision-language mod- els effectively and thoroughly. InNeurIPS, 2025. 2, 5

    2025

  33. [33]

    Opengaussian: Towards point-level 3d gaussian-based open vocabulary understanding

    Yanmin Wu, Jiarui Meng, Haijie Li, Chenming Wu, Yahao Shi, Xinhua Cheng, Chen Zhao, Haocheng Feng, Errui Ding, Jingdong Wang, et al. Opengaussian: Towards point-level 3d gaussian-based open vocabulary understanding. InNeurIPS,

  34. [34]

    3d-mem: 3d scene memory for embodied exploration and reasoning

    Yuncong Yang, Han Yang, Jiachen Zhou, Peihao Chen, Hongxin Zhang, Yilun Du, and Chuang Gan. 3d-mem: 3d scene memory for embodied exploration and reasoning. In CVPR, 2025. 2, 3, 4, 5, 6, 8, 1

    2025

  35. [35]

    Gaus- sian grouping: Segment and edit anything in 3d scenes

    Mingqiao Ye, Martin Danelljan, Fisher Yu, and Lei Ke. Gaus- sian grouping: Segment and edit anything in 3d scenes. In ECCV, 2024. 2

    2024

  36. [36]

    Sg-nav: Online 3d scene graph prompting for llm-based zero-shot object navigation

    Hang Yin, Xiuwei Xu, Zhenyu Wu, Jie Zhou, and Jiwen Lu. Sg-nav: Online 3d scene graph prompting for llm-based zero-shot object navigation. InNeurIPS, 2024. 2

    2024

  37. [37]

    Osprey: Pixel understanding with visual instruction tuning

    Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. InCVPR, 2024. 7, 2

    2024

  38. [38]

    Open-vocabulary functional 3d scene graphs for real-world indoor spaces

    Chenyangguang Zhang, Alexandros Delitzas, Fangjinhua Wang, Ruida Zhang, Xiangyang Ji, Marc Pollefeys, and Fran- cis Engelmann. Open-vocabulary functional 3d scene graphs for real-world indoor spaces. InCVPR, 2025. 2

    2025

  39. [39]

    Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields

    Shijie Zhou, Haoran Chang, Sicheng Jiang, Zhiwen Fan, Ze- hao Zhu, Dejia Xu, Pradyumna Chari, Suya You, Zhangyang Wang, and Achuta Kadambi. Feature 3dgs: Supercharging 3d gaussian splatting to enable distilled feature fields. InCVPR,

  40. [40]

    Ewa volume splatting

    Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. Ewa volume splatting. InProceedings Visual- ization, 2001. VIS’01.IEEE, 2001. 4, 1 10

    2001