{"id":"6eac2a52-cf41-4f4e-ab42-b084742d0b26","arxiv_id":"2505.11383","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Dynam3D represents scenes as patch, instance, and zone tokens that update dynamically, and feeds them to a 3.8B vision-language model to improve action prediction in vision-and-language navigation.","lead":"Dynam3D gives a navigating robot a live, layered 3D memory built from camera images, and a vision-language model reads that memory to follow spoken instructions. The authors report roughly 5% higher success rates on standard benchmarks and demonstrate that the memory can update when objects move.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"NavRAG-CE test episodes may overlap with the NavRAG-generated training data, so the claimed SOTA on that benchmark could reflect distribution overlap rather than generalization.","rationale":"The reader's weakest assumption identifies exactly this: NavRAG-CE is assumed to isolate generalization to previously unseen instructions. I agree. The manuscript's own A.1 text shows NavRAG contributes both training samples and the test benchmark, with no disclosed split or overlap check. Since the abstract bills the result as SOTA on all three benchmarks, one contaminated benchmark is enough to invalidate the breadth of the claim, even though R2R-CE and the controlled REVERIE-CE comparison in Table 2 provide independent support for the underlying representation. I would not reject: the R2R-CE result (SR 52.9, SPL 45.7) is a meaningful improvement over published monocular systems, and Table 2 retrains NaVid and g3D-LF on the same data for REVERIE-CE and NavRAG-CE, which is a fairer comparison than Table 1. However, the lack of split disclosure, single-run metrics, and 20-case real-world tests all argue against unconditional acceptance. The verdict should remain conditional, with the condition being that the authors demonstrate NavRAG-CE is a genuinely held-out split and that the clean-split result still beats the retrained baselines. No formal verification is present, but code release and reproducible benchmark numbers are positive evidence.","tokens_in":14989,"tokens_out":8662,"duration_ms":84555,"concrete_test":"Release the episode IDs used to construct NavRAG-CE and the ScaleVLN/NavRAG samples used in 3D-VLM training, or apply exact and near-duplicate matching on instruction text, source scene IDs, and start/goal waypoint pairs. Re-evaluate Dynam3D on a strict unseen split where no training episode shares a scene or a paraphrased instruction with any test episode. If the SR gap over the retrained g3D-LF baseline on that clean split falls to near zero (e.g., ≤1 point), the NavRAG-CE SOTA claim is not established; if the gap is maintained, the overlap concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The broadest form of the central claim—new state-of-the-art on R2R-CE, REVERIE-CE and NavRAG-CE under monocular settings (abstract, Section 4.1)—rests on NavRAG-CE being a valid unseen generalization test. Supplementary A.1 states that the 3D-VLM is trained on 'model-generated data (ScaleVLN, NavRAG; 4M+)' and that NavRAG-CE is evaluated 'by transferring NavRAG [4] datasets to continuous environments.' The paper does not disclose how NavRAG episodes were partitioned between the 4M+ training samples and the NavRAG-CE test split, nor does it report any overlap analysis. If the same instructions, trajectory fragments, or source scenes appear in both training and test, the reported NavRAG-CE gains (SR 24.7 vs 17.4–21.4 for baselines) partly measure memorization of the training distribution rather than generalization to new user demands. This is not a disagreement about consensus—it is an internal protocol gap that the manuscript itself exposes. The R2R-CE SOTA result (SR 52.9) is not affected by this issue, so the core method may still be sound; but the 'including NavRAG-CE' clause of the SOTA claim is not currently supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Dynam3D, a monocular vision-and-language navigation (VLN) system that replaces raw video frames with a dynamically updated hierarchical 3D representation. Posed RGB-D frames are encoded by CLIP, projected into 3D patch feature points, aggregated into FastSAM-based 3D instance tokens, and further aggregated into zone-level tokens, with a frustum-culling mechanism for removing outdated features and a merging discriminator for online instance updates. These tokens are fed into a 3.8B LLaVA-Phi-3-mini model that predicts atomic navigation actions. The authors report state-of-the-art results on R2R-CE, REVERIE-CE, and NavRAG-CE, and additional experiments on pre-exploration, lifelong memory, and real-world static and dynamic environments.","tokens_in":15184,"tokens_out":8233,"duration_ms":70880,"significance":"If the results hold, Dynam3D would be a meaningful advance over video-frame VLN systems: it demonstrates that language-aligned, dynamically updated 3D tokens can provide better spatial understanding and long-term memory for embodied navigation, with a smaller LLM than video-based counterparts. The core idea is coherent and well motivated, and the paper includes several strengths: the frustum-culling and merging mechanism is a sensible approach to dynamic scene updates, the subspace contrastive distillation is a thoughtful remedy for view-dependent CLIP feature bias, the code is released, and the real-robot experiments, including moving targets, address practical deployment. However, the headline SOTA claim is currently under-supported by the evaluation protocol, most importantly because the NavRAG-CE benchmark shares its data source with the training set and because the R2R-CE comparison is not controlled for training data.","major_comments":[{"comment":"The NavRAG-CE evaluation does not currently establish generalization to unseen instructions. The training set is described as \"model-generated data (ScaleVLN, NavRAG; 4M+)\" and the NavRAG-CE test set is created \"by transferring NavRAG datasets to continuous environments.\" The paper neither reports how NavRAG episodes were partitioned between the 4M+ training samples and the NavRAG-CE test split nor provides any overlap analysis. Since NavRAG is also the authors' own dataset, the reported 24.7 versus 21.4 SR advantage over g3D-LF on NavRAG-CE could reflect memorization of the training distribution rather than generalization to new user demands. The authors should disclose the split, remove any overlapping episodes, and re-report NavRAG-CE, or alternatively drop the NavRAG-CE clause from the SOTA claim.","section":"§4.1, Table 2; Supplementary A.1"},{"comment":"The R2R-CE SOTA comparison is not controlled for training data. All baseline numbers in Table 1 are taken from prior publications and are not retrained under the paper's protocol, while Dynam3D is trained on 4M+ model-generated instruction-trajectory pairs in addition to human-annotated R2R-CE and REVERIE-CE data (Supplementary A.1). The +5.7 SR gain over g3D-LF on Val Unseen may therefore be due to the much larger training set rather than to the proposed patch-instance-zone representation. A controlled comparison with g3D-LF and at least one video-based VLM retrained on the same navigation data is needed to support the R2R-CE SOTA claim.","section":"§4.1, Table 1"},{"comment":"The claim of \"state-of-the-art performance on VLN benchmarks including R2R-CE, REVERIE-CE and NavRAG-CE\" is stronger than the reported evidence. Only R2R-CE is evaluated on a Test split; REVERIE-CE and NavRAG-CE are reported on Val Unseen only, and NavRAG-CE is a self-constructed benchmark without an independent test set. The abstract and Section 4.1 should be revised to say \"validation\" for the latter two benchmarks unless test-split results are provided.","section":"§4.1, Tables 1–2"}],"minor_comments":[{"comment":"The notation in Eq. (2) is inconsistent: the depth map is indexed by row h and column w, but the pixel condition is written with u and v, and the inequalities 0<u<H and 0<v<W appear to swap width and height. Please correct the subscripts and coordinate ranges.","section":"§3.1, Eq. (2)"},{"comment":"The statement in Section 4.1 that NaVid and g3D-LF were retrained on the paper's training dataset should be repeated in the table caption, together with the exact training split and number of optimization steps used for these baselines.","section":"§4.1, Table 2 caption"},{"comment":"All metrics are reported from a single run without error bars or multiple seeds; given the stochasticity of navigation evaluation, reporting variance over at least three seeds would make the SOTA comparisons substantially more convincing.","section":"Tables 1, 2, 3, 4, 5"},{"comment":"The Pre-exploration setting uses navigable viewpoints annotated in Matterport3D; the paper should clarify whether this provides the agent with ground-truth navigability information and how the procedure would be realized in an unseen environment without such annotations.","section":"§4.2, Table 3"},{"comment":"The real-world dynamic experiments are based on only 20 test cases per setting; please state whether the target-move procedure and the success criterion were fixed in advance and report per-case statistics, since the 45% SR in Table 5 rests on a small sample.","section":"§4.3, Tables 4–5"},{"comment":"The efficiency comparison with Uni-NaVid refers to the LLM parameter count (3.8B versus 7B), but the full Dynam3D system also includes CLIP, FastSAM, the generalizable feature field, and the instance and zone encoders; the efficiency claim should be scoped accordingly.","section":"§3.2"}],"recommendation":"major_revision","confidential_remarks":"The NavRAG-CE overlap issue is a substantive correctness risk because the benchmark is derived from the authors' own NavRAG dataset, which also contributes to the training set. I would ask the editor to treat the requested overlap analysis and the controlled retraining comparison as prerequisites for acceptance, rather than as optional presentation fixes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dynam3D is a serious systems contribution to monocular VLN. The core idea—maintaining hierarchical patch-instance-zone 3D tokens, updating them online with frustum culling, and feeding them to a 3.8B VLM—is a legitimate step beyond the video-frame baselines. On R2R-CE, the reported SR of 52.9 vs g3D-LF's 47.2 is a meaningful gain, and the ablations show the instance-zone levels and subspace alignment each matter. The pre-exploration and lifelong-memory results, plus the real-world demo where the target is moved mid-run, are genuinely interesting evidence that the dynamic memory is doing work. The code link is a plus.\n\nThe soft spots are real but mostly concentrated in the evaluation's edge cases. The biggest one: NavRAG-CE is constructed from the authors' own NavRAG dataset, and NavRAG also contributes to the 4M+ training instruction pairs. Without a disclosed split or overlap analysis, the NavRAG-CE success rate partly measures distribution overlap, not generalization. That particular benchmark cannot support the 'SOTA on three benchmarks' headline as written. The R2R-CE and REVERIE-CE results are not affected by this, so the core method stands.\n\nElsewhere, Table 1 uses published baseline numbers rather than retrained comparisons; that is conventional but it leaves the exact margin vulnerable to training-data confounds. There are no error bars or repeated-seed runs anywhere, which matters more on the small real-world set (20 cases per condition). The merging discriminator is trained on static 3RScan/ScanNet rooms and assumed to transfer to moving objects; the real-world results suggest it works, but I would like a closer look at failure modes.\n\nWho is this for? Anyone working on VLN, embodied VLM, or online 3D scene memory. The paper deserves a serious referee. I would send it to review and push for a full disclosure of the NavRAG-CE split, a repeat-seed evaluation, and ideally one retrained baseline. If those come back clean, the R2R-CE claim alone would justify publication.\n\nRecommendation: accept for peer review with a request for revision.","headline":"Dynam3D is a credible dynamic 3D-token system with a real R2R-CE gain, but the NavRAG-CE portion of the SOTA claim is compromised by train/test overlap.","tokens_in":15830,"tokens_out":2344,"would_cite":true,"duration_ms":22603,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Dynam3D replaces video frames with dynamically updated, language-aligned patch-instance-zone 3D tokens and claims new state-of-the-art monocular results on R2R-CE, REVERIE-CE, and NavRAG-CE.","keywords":["vision-and-language navigation","3D vision-language model","hierarchical 3D tokens","dynamic scene updates","monocular navigation","continuous environments","CLIP feature alignment","embodied agents"],"falsifier":"Measure generalization on NavRAG-CE with test instructions generated by a different LLM and prompt template than the training instructions, while keeping the same simulator and environments; if the success-rate margin over video baselines collapses to the R2R-CE margin, the benchmark does not isolate generalization to unseen user demands.","tokens_in":14693,"feed_emoji":"🧭","tokens_out":7202,"duration_ms":65794,"temperature":0.7,"pith_summary":"This paper is trying to establish that a navigation agent perceives better when its visual memory is a structured 3D hierarchy of patch, instance, and zone tokens rather than a stack of video frames. It argues that CLIP features projected into 3D space and aligned with language give a 3.8-billion-parameter vision-language model enough geometry and long-term memory to outperform larger video-based agents in continuous indoor environments, even when objects move. If true, this would mean structured, language-grounded 3D memory is a stronger bottleneck for embodied navigation than raw frames, and that smaller models can win by changing the representation rather than the parameter count. The paper backs the claim with state-of-the-art numbers on R2R-CE, REVERIE-CE, and NavRAG-CE, plus real-robot tests with pre-exploration and moved targets.","feed_headline":"Layered 3D tokens beat raw video for language-guided navigation","feed_subtitle":"Dynam3D claims new bests with a 3.8B model—52.9% success on R2R-CE versus 47% for larger video-based agents.","key_machinery":"The key object is the Dynam3D patch-instance-zone representation, a three-level 3D token hierarchy. Patch features come from CLIP-ViT-L/14@336px projected into world coordinates using depth and pose; instance tokens are formed by aggregating FastSAM 2D mask features through a transformer-based instance encoder; zone tokens aggregate instance features inside cubic cells spanning several meters. The mechanism that makes it dynamic has two parts: frustum culling removes patch points that lie between the camera and a newly observed surface, and the merging discriminator, an MLP trained with binary classification on multi-room 3D instance segmentation data, decides whether a fresh 2D segment should merge into an existing 3D instance or start a new one. A generalizable feature field renders agent-centric panoramic patch tokens from the stored feature points. All three levels are aligned with language through 3D-language contrastive learning and CLIP feature distillation, using subspace contrastive learning to subtract each view's semantic center and reduce view-dependent bias, which is what allows a pretrained 2D VLM to reason about the 3D tokens.","core_discovery":"Dynam3D's central claim is that a navigation agent should perceive through a dynamically updated, language-aligned 3D token hierarchy rather than a sequence of video frames. The hierarchy has three levels: patch feature points from CLIP projected into world coordinates, instance tokens formed by merging FastSAM 2D masks into consistent 3D instances, and zone tokens that summarize several-meter spatial cubes. The dynamic part is a frustum-culling update that deletes stale patch points and a merging discriminator that decides whether a newly observed segment belongs to an existing 3D instance, with updates propagating so zone summaries stay current. After large-scale 3D-language pretraining aligns all three levels to CLIP's semantic space, the representation feeds a 3.8B LLaVA-Phi-3-mini that outputs atomic actions such as turning, moving forward, or stopping. On R2R-CE Val Unseen the agent reaches 52.9% success rate versus 47.2% for g3D-LF and 47.0% for the larger Uni-NaVid, and the paper reports similar advantages on REVERIE-CE and NavRAG-CE plus real-robot success after the target is moved.","pith_inferences":["A clean stress test would regenerate NavRAG-CE test instructions from a fresh prompt distribution or LLM; if the margin over video baselines then shrinks to the R2R-CE margin, part of the claimed NavRAG-CE gain reflects distributional memory rather than generalization to unseen user demands.","Because the 3D tokens live in CLIP's semantic space, the same representation could be plugged into other CLIP-based VLMs or next-step predictors as a drop-in memory module, allowing tests of whether the gains transfer without retraining the representation model.","The paper's Limitations section explicitly says the model does not output the target instance's coordinates and cannot answer questions or update tasks; decoding those coordinates from the instance tokens and adding a dialogue head are the natural next steps toward mobile manipulation and interactive replanning.","The subspace-contrastive alignment is a local fix with wider scope: any online 3D learner that distills CLIP features from multiple views could subtract the view's semantic center before contrastive learning to suppress per-view bias."],"forward_implications":["Pre-exploration and lifelong memory become usable: keeping stored 3D representations across episodes raises success rate on R2R-CE from 52.9% to 58.4% and on REVERIE-CE from 40.1% to 49.5%.","Dynamic scenes are tractable without retraining: when the target object is moved during a real-robot run, Dynam3D still reaches 45% success, against 20% for NaVid and 10% for g3D-LF.","Video frames are not necessary for strong monocular VLN: a 3.8B model with layered 3D tokens beats the 7B video-based Uni-NaVid on R2R-CE Val Unseen (52.9 vs 47.0 SR), so representation quality can substitute for model scale.","Instance-zone tokens carry the long-horizon signal: ablations show dropping them cuts REVERIE-CE success rate by about 15 points, meaning local patch-level rendering alone is insufficient for target localization."],"supporting_citations":[{"why":"Supplies the CLIP patch features that are projected into 3D and the text-embedding space that all 3D tokens are aligned to.","marker":"[11]"},{"why":"Provides the generalizable feature field for rendering panoramic 3D patch tokens and is the prior feature-field baseline on R2R-CE.","marker":"[14]"},{"why":"Video-based monocular VLN baseline whose results Dynam3D compares against in benchmarks and real-robot tests.","marker":"[5]"},{"why":"Larger 7B video-based state-of-the-art baseline that Dynam3D must beat to support the efficiency-and-representation claim.","marker":"[6]"},{"why":"Defines R2R-CE, the continuous-environment benchmark for the main navigation success-rate results.","marker":"[3]"},{"why":"Defines REVERIE-CE, the coarse-instruction benchmark where instance-zone tokens give the largest relative gain.","marker":"[2]"},{"why":"Source of NavRAG instructions used both for NavRAG training data and for the NavRAG-CE test transfer, making the unseen-generalization assumption load-bearing.","marker":"[4]"},{"why":"Supplies model-generated ScaleVLN instruction-trajectory pairs that make up the 4M+ continuous training set.","marker":"[56]"},{"why":"Contributes the large-scale 3D-language scene data used to align instance and zone tokens with CLIP text semantics.","marker":"[46]"},{"why":"FastSAM segments each RGB frame into 2D instance masks that seed the 3D instance tokens.","marker":"[12]"}],"fun_headline_variants":["Dynamic 3D tokens outperform video for language-guided navigation","Layered 3D instances give VLN agents a lasting memory","Dynam3D: 3D token hierarchy sets new VLN benchmarks","From video frames to dynamic 3D zones for navigation","Dynam3D turns RGB-D into language-aligned 3D tokens"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The large NavRAG-CE gain stands on the assumption that those test episodes are genuinely unseen rather than distributionally overlapping with the NavRAG-generated training instructions; the R2R-CE and real-robot results do not depend on this assumption.","fun_headline_variants_meta":{"raw":{"variants":["Dynamic 3D tokens outperform video for language-guided navigation","Layered 3D instances give VLN agents a lasting memory","Dynam3D: 3D token hierarchy sets new VLN benchmarks","From video frames to dynamic 3D zones for navigation","Dynam3D turns RGB-D into language-aligned 3D tokens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000266,"raw_usage":{"total_tokens":1704,"prompt_tokens":1130,"completion_tokens":574,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":746,"completion_tokens_details":{"reasoning_tokens":483}},"tokens_in":746,"tokens_out":574,"duration_ms":5688,"temperature":1.0,"reasoning_tokens":483,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:53:55.191939+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure generalization on NavRAG-CE with test instructions generated by a different LLM and prompt template than the training instructions, while keeping the same simulator and environments; if the success-rate margin over video baselines collapses to the R2R-CE margin, the benchmark does not isolate generalization to unseen user demands.","supporting_citations":[{"cited_title":"g3D-LF: Generalizable 3D-Language Feature Fields for Embodied Tasks","cited_arxiv_id":"2411.17030","evidence_quote":"Provides the generalizable feature field for rendering panoramic 3D patch tokens and is the prior feature-field baseline on R2R-CE."},{"cited_title":"Navid: Video-based vlm plans the next step for vision-and-language navigation","cited_arxiv_id":null,"evidence_quote":"Video-based monocular VLN baseline whose results Dynam3D compares against in benchmarks and real-robot tests."},{"cited_title":"Scaling data generation in vision-and-language navigation","cited_arxiv_id":null,"evidence_quote":"Supplies model-generated ScaleVLN instruction-trajectory pairs that make up the 4M+ continuous training set."},{"cited_title":"Sceneverse: Scaling 3d vision-language learning for grounded scene understanding","cited_arxiv_id":null,"evidence_quote":"Contributes the large-scale 3D-language scene data used to align instance and zone tokens with CLIP text semantics."}],"review_version":1}