{"id":"68f096f1-9f0a-4f27-87bc-6c1fb6d209a9","arxiv_id":"2505.12253","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A vision-language model that embeds 3D coordinates plus time into visual and linguistic tokens beats 3D-only models on dynamic scene captioning, grounding, and QA.","lead":"LLaVA-4D teaches a vision-language model to understand dynamic 3D scenes by adding time coordinates to 3D positions, allowing it to describe where objects are and how they move across multiple camera views. The reason to read it is that it is one of the first attempts at a general-purpose 4D scene understanding model and it comes with a new instruction dataset, which could push robots and autonomous systems toward richer spatial-temporal reasoning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 4D benchmark is built from the same MVS/detection pipeline that supplies the training signal, so the reported SAcc/TAcc gains may measure alignment with self-generated coordinates rather than true dynamic understanding.","rationale":"The reader's weakest assumption about MVS depth on dynamic objects is in the right place, but the more precise load-bearing issue is broader: Chat4D is both the training instruction set and the evaluation benchmark, and its coordinate labels are generated by the same geometry pipeline that the model consumes. This makes the reported 4D numbers vulnerable to circular pseudo-ground-truth bias. The paper has genuine positive evidence—clear architectural description, plausible ablations showing each module contributes, and competitive numbers on standard 3D benchmarks—so I do not see an internal inconsistency that would justify rejection. However, the central first-and-general-LMM claim cannot be assessed without independent validation of the 4D benchmark and release of the code/data. The reader's conditional verdict is therefore appropriate; my concern reinforces it rather than redirecting it to a different verdict.","tokens_in":14974,"tokens_out":3375,"duration_ms":39125,"concrete_test":"Release the Chat4D test split annotations and have independent human annotators label 6-DoF object positions at several sampled timestamps in a random subset of at least 50 dynamic clips. Compare these human labels against the MVS/detection-derived coordinate annotations; if the median error on moving objects exceeds 0.25 m or 10% of object size, recompute SAcc/TAcc with corrected labels and check whether LLaVA-4D still outperforms the 3D baselines. This directly tests whether the benchmark measures real 4D understanding or merely alignment with the annotation pipeline's errors.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—being the first general LMM for 4D scene understanding—rests on the Chat4D results in Table 1. Section 4.1 states that Chat4D annotations are generated by running 3D object detection and GPT-4V on multi-view videos, with spatial coordinates derived from the same SfM/MVS geometry used in Eq. 4. Section 5.1 further says all methods are trained on the same benchmark via instruction fine-tuning, but no separate human-verified or independently reconstructed test set is described. Consequently, both the training targets and the evaluation ground truth share the same potential failure modes: MVS assumes static scenes, so moving-object depth at different timestamps can be inconsistent or wrong, and the detection stage can propagate those errors into the coordinate labels. A model trained on these labels can learn to reproduce the systematic errors of the annotation pipeline, inflating SAcc and TAcc without acquiring genuine 4D understanding. The paper's limitation discussion only mentions motion blur on fast-moving objects, not coordinate-validation error on dynamic regions, so the load-bearing assumption that the Chat4D labels are trustworthy 4D ground truth is unverified. If the coordinate pseudo-ground truth is biased, then the claimed superiority over 3D LMMs is not established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LLaVA-4D, a large multimodal model for 4D scene understanding that embeds a spatiotemporal prompt into visual representations for LMMs. The prompt is constructed by encoding 3D coordinates [x, y, z] and time t with learnable Fourier features and an optical-flow-derived motion term (Eqs. 4-6). Visual features from multi-view videos are disentangled into spatial and temporal components via cross-view and adjacent-time correlations (Eqs. 7-8), then fused with the 4D coordinate embedding through cross-attention before projection into the LLM. The authors also introduce Chat4D, a 2D/3D/4D vision-language dataset for instruction fine-tuning, and report results on Scan2Cap, ScanQA, ScanRef, Multi3DRefer, and their own Chat4D benchmark. They claim state-of-the-art performance, including Chat4D spatial grounding accuracy SAcc 58.9 and temporal accuracy TAcc 54.6, and they state that this is the first general vision-language LMM for 4D scene understanding.","tokens_in":15264,"tokens_out":5087,"duration_ms":50269,"significance":"The architecture is coherent, and the ablation tables are internally consistent with the design narrative. The idea of using time-indexed coordinates as a prompt and explicitly separating spatial and temporal visual features is a plausible and potentially transferable design. If the 4D evaluation were independently validated, the paper would represent a significant step: it is one of the first attempts to give an LMM temporally grounded spatial reasoning, and it reports consistent gains on external 3D benchmarks (Scan2Cap, ScanQA, ScanRef, Multi3DRefer), which partly support the static/spatial component. However, the headline 4D claims rest entirely on a self-created benchmark whose annotations come from the same geometric pipeline used for training, and the paper reports no human validation, no independent reconstruction of test scenes, and no variance estimates. The external 3D results are real but do not validate temporal/dynamic understanding. The central novelty therefore needs stronger evaluation evidence. The paper does not mention code, model weights, or dataset release, which limits reproducibility for the new benchmark.","major_comments":[{"comment":"The headline 4D results (SAcc 58.9, TAcc 54.6) are measured on Chat4D, a benchmark the authors construct from the same data distribution and the same SfM/MVS geometry used to create the training targets in Eq. (4). Because the annotation pipeline is also the evaluation pipeline, the model can learn to reproduce systematic errors of that pipeline and appear to gain 4D understanding without actually having it. The paper reports no human validation of the generated 4D annotations, no independent reconstruction of the test scenes, and no error bars across random seeds. Please add a human-evaluated subset or an independently annotated test set, and report variance over multiple training runs.","section":"Sec. 4.1 and Sec. 5.1, Table 1"},{"comment":"The coordinate construction treats MVS depth as per-frame ground truth, but MVS assumes a static scene and is known to be unreliable on moving objects. Since both the training labels and the S/TAcc spatial ground truth are derived from this same depth, errors on dynamic regions propagate directly into the claims about temporal understanding. The Limitation paragraph only mentions motion blur on fast-moving objects and does not quantify depth or pose errors on dynamic regions. Please report depth error and coordinate consistency separately for static and dynamic regions, or provide an alternative validation of moving-object coordinates.","section":"Sec. 3.1, Eq. (4)"},{"comment":"The statement that all methods are trained on the same evaluation benchmark via instruction fine-tuning is underspecified for the 4D columns. The 3D baseline methods have no spatiotemporal prompt and no time-indexed coordinate encoding, so it is unclear how they were instructed to produce answers at multiple timestamps and how TAcc was computed for them. Without a precise protocol for adapting each baseline to the 4D task, the large TAcc gap may reflect an unfair comparison rather than genuine 4D understanding. Please specify the exact instruction format, output parsing, and training recipe used for every baseline, and consider adding a strong baseline that receives the same 4D coordinate prompt.","section":"Table 1 and Sec. 5.1"}],"minor_comments":[{"comment":"The model name is rendered inconsistently ('LLaV A-4D' in the title and header, 'LLaVA-4D' in the abstract), and Section 3.1 contains the typo 'positon'.","section":"Throughout"},{"comment":"Equation (6) uses softmax over optical flow, Φ(vel), but it is not specified over which dimension the softmax is applied or how a vector-valued flow modulates the scalar time encoding. Equation (9) uses MLPobj without defining it.","section":"Eqs. (6) and (9)"},{"comment":"The clustering evidence for the discriminability of spatial and temporal features is qualitative; please supplement it with a quantitative measure such as cluster separation or a classification accuracy of the disentangled features.","section":"Figs. 3 and 6"},{"comment":"The related work cites LLaVA-ST but does not compare with it or discuss how the proposed temporal encoding differs from that line of work.","section":"Sec. 2"},{"comment":"No code, model weights, or dataset release is mentioned; given that the benchmark is new, releasing the annotation pipeline or a sample of annotated examples would substantially improve reproducibility.","section":"Sec. 4.1"}],"recommendation":"major_revision","confidential_remarks":"The core architecture is plausible and the external 3D results are encouraging, but the primary evidence for the paper's central claim is an entirely self-created benchmark built from the same geometric pipeline that generates the training targets. The editor should ask the authors for independent validation of the 4D evaluation, such as a human-verified subset or an externally reconstructed test set, and for protocol details on how 3D baselines were adapted to the 4D task. I do not see evidence of bad faith; the issue is that the evaluation as it stands cannot distinguish genuine 4D understanding from learning the systematic errors of the annotation pipeline."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear Colleague,\n\nThe key thing to know: LLaVA-4D is a plausible architecture for giving LMMs temporal grounding in dynamic scenes, and the internal ablations are consistent with the design story. But the headline 4D numbers are measured on Chat4D, a benchmark the authors built from the same geometry pipeline that produces the training coordinates. Until that loop is broken, the quantitative claim of \"understanding\" is not established.\n\nWhat's actually new: a 4D coordinate prompt that encodes 3D position plus time, with optical flow modulating the temporal encoding; a spatial/temporal disentanglement of visual features before cross-attention fusion; and Chat4D, the first 4D vision-language instruction dataset. The three-stage training is sensible. On the external 3D benchmarks (Scan2Cap, ScanQA, ScanRef, Multi3DRefer) LLaVA-4D is competitive or stronger than prior 3D LMMs, which does give some independent evidence that the spatial side works.\n\nThe soft spot is not minor. Eq. 4 unprojects pixels using SfM poses and MVS depth. MVS is built on static-scene assumptions, so depth on moving objects is suspect. The same geometry is then used to generate the Chat4D annotations via 3D detectors and GPT-4V. No human validation of those labels is reported, and there are no error bars or random seeds. All compared models are trained on the same benchmark, so the SAcc/TAcc gains may reflect the model learning the systematic biases of the annotation pipeline rather than genuine dynamic understanding. The limitation section mentions motion blur but not depth error on dynamic regions.\n\nThat said, the paper is not sloppy on its own terms. The ablations (Tables 2-5) show each component contributes. The 3D benchmark results are a legitimate partial check. The \"first general 4D LMM\" claim is plausible, but it needs support from data the authors did not generate, or at least a human-verified subset of Chat4D.\n\nI would send this to peer review, but with conditions: request code and dataset release, an independent or at least human-validated 4D test set, and an analysis of coordinate error on dynamic objects. Without those, the temporal claims should be treated as suggestive, not demonstrated.\n\nBest.","headline":"A clean architecture for 4D scene understanding in LMMs, but the headline temporal results rest on a self-built benchmark and unvalidated dynamic depth; deserves peer review with conditions.","tokens_in":15818,"tokens_out":2180,"would_cite":false,"duration_ms":21404,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that adding explicit time coordinates to a visual prompt lets a large multimodal model understand dynamic 3D scenes instead of only static ones.","keywords":["4D scene understanding","large multimodal model","spatiotemporal prompt","4D coordinate encoding","vision-language alignment","dynamic scene understanding","visual grounding","instruction tuning"],"falsifier":"On a dynamic multi-view video with independently measured object trajectories, compare the model's per-timestamp coordinates for fast-moving objects against the measured positions; if the reported temporal coordinates deviate by roughly the size of the MVS depth error on those regions, the dynamic-understanding claim fails because it is the static-scene geometry doing the work rather than the 4D prompt.","tokens_in":14757,"feed_emoji":"🕓","tokens_out":6901,"duration_ms":65806,"temperature":0.7,"pith_summary":"The paper tries to show that a large multimodal model can understand dynamic 3D scenes when its visual representation carries a spatiotemporal prompt: a world coordinate plus a timestamp for every observed point. To build that prompt, LLaVA-4D Fourier-encodes the 3D position, scales the time encoding by optical flow, and fuses the result into visual features that are disentangled into spatial and temporal streams. On the new Chat4D benchmark, the model reports object coordinates at multiple timestamps and reasons about likely future motion, attaining 58.9 spatial accuracy and 54.6 temporal accuracy against 3D LMM baselines. The paper also introduces Chat4D, a 2D/3D/4D instruction-tuning dataset. If the argument is right, explicit time in the coordinate prompt, not more training data, is the step that takes LMMs from static scene description to dynamic scene reasoning.","feed_headline":"Time-aware prompt lets AI models see objects move in 3D","feed_subtitle":"A 4D coordinate prompt gives LMMs per-timestamp object positions and future-motion reasoning beyond 3D baselines.","key_machinery":"The load-bearing object is the spatiotemporal prompt $p_{4D} = w_p \\cdot [\\mathrm{PE}(x,y,z) \\| \\mathrm{TE}(t)\\cdot\\beta]$, built from Fourier position encoding and motion-scaled Fourier time encoding; it is the only mechanism that differentiates a moving object from a static background at the same 3D location. It is injected through a spatiotemporal-disentangled vision embedding, where cross-view correlations form spatial features and adjacent-frame correlations form temporal features, and through cross-attention fusion that learns per-coordinate weights. The same encoding is applied to textual coordinate tokens to align language with the 4D-aware visual tokens. Together these components carry the argument that time must be embedded as a first-class coordinate, not inferred from captions.","core_discovery":"The central claim is that existing 3D LMMs fail on dynamic scenes because they embed fixed 3D positions as spatial prompts, while LLaVA-4D embeds a dynamic-aware 4D coordinate tensor $[x,y,z,t]$ as a spatiotemporal prompt and disentangles multi-view video features into spatial and temporal components before fusion. Coordinates are obtained by unprojecting each frame with SfM camera poses and MVS depth; position uses learnable Fourier features and time uses Fourier features scaled by a softmax of optical flow, so moving objects receive a different temporal signature from the static background. Cross-attention fuses the prompt into the disentangled features, the same position/time encoding is applied to textual coordinates, and the resulting tokens are aligned for the LLM. The paper reports that this design outperforms all compared 3D LMMs on 3D benchmarks and on Chat4D, and shows that the spatiotemporal prompt extends to semantic and action prompts.","pith_inferences":["A direct test the paper does not run is to swap MVS depth for an object-aware depth source on moving regions; if temporal accuracy survives the swap, the prompt mechanism is robust, and if not, the dynamic-world gains are partly an artifact of static-scene geometry.","The same recipe could transfer to embodied settings where time is action time and the motion field is the agent's own trajectory; the extensibility experiment with action masks is a step in that direction.","The motion-scaled time encoding predicts that temporal grounding should degrade smoothly with object speed; this is measurable on a speed-sorted subset of any dynamic benchmark."],"forward_implications":["A 4D-prompted LMM can answer time-indexed questions, such as where an object was at $t_1$ and $t_2$, and can report both positions in one response.","Because the prompt separates spatial and temporal streams, dynamic objects are represented distinctly from backgrounds, which is what yields the reported gains on grounding and captioning.","The coordinate-aligned language embedding reduces the chance the LLM misreads coordinate values in instructions, improving spatial accuracy even on static 3D benchmarks.","The prompt plug-in is extensible: adding semantic and action masks as spatiotemporal attributes supports segmentation and action-understanding tasks."],"supporting_citations":[{"why":"Defines the 3D-LLM paradigm of embedding 3D positions as spatial prompts, which this paper extends and compares against.","marker":"[10]"},{"why":"Image-based 3D LMM baseline whose static spatial prompt motivates the proposed 4D spatiotemporal prompt.","marker":"[11]"},{"why":"Video-3D LLM is the strongest compared 3D baseline and the method this work must beat on 3D and 4D benchmarks.","marker":"[23]"},{"why":"Structure-from-motion supplies the camera poses used in Eq. 4 to unproject 2D pixels into world coordinates.","marker":"[29]"},{"why":"Multi-view stereo supplies the depth maps used in the same unprojection to build the 4D coordinate tensor.","marker":"[30]"},{"why":"Learnable Fourier features provide the position and time encoding recipe for the spatiotemporal prompt.","marker":"[33]"},{"why":"The transformer attention mechanism is used for cross-attention fusion of 4D coordinates with visual features.","marker":"[34]"},{"why":"One of the multi-view dynamic scene datasets merged to construct the Chat4D instruction-tuning data.","marker":"[46]"},{"why":"A second dynamic scene dataset used for building the 4D vision-language training set.","marker":"[47]"},{"why":"A third multi-view dynamic video dataset used to create Chat4D's 4D instruction data.","marker":"[48]"}],"fun_headline_variants":["4D prompt helps language models track moving objects in 3D","Time-embedded prompts give AI models 4D scene understanding","Spatiotemporal prompt teaches AI models to track moving 3D objects","Adding time to spatial prompts boosts AI's 4D scene understanding"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 4D coordinates are produced by unprojecting each frame with structure-from-motion camera poses and multi-view stereo depth, and multi-view stereo assumes a static world, so depth on moving objects is likely wrong and every coordinate claim built on it inherits that error.","fun_headline_variants_meta":{"raw":{"variants":["4D prompt helps language models track moving objects in 3D","Time-embedded prompts give AI models 4D scene understanding","Spatiotemporal prompt teaches AI models to track moving 3D objects","Adding time to spatial prompts boosts AI's 4D scene understanding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001203,"raw_usage":{"total_tokens":4980,"prompt_tokens":994,"completion_tokens":3986,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":3910}},"tokens_in":610,"tokens_out":3986,"duration_ms":24408,"temperature":1.0,"reasoning_tokens":3910,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:37:41.902159+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a dynamic multi-view video with independently measured object trajectories, compare the model's per-timestamp coordinates for fast-moving objects against the measured positions; if the reported temporal coordinates deviate by roughly the size of the MVS depth error on those regions, the dynamic-understanding claim fails because it is the static-scene geometry doing the work rather than the 4D prompt.","supporting_citations":[{"cited_title":"Structure-from-motion revisited","cited_arxiv_id":null,"evidence_quote":"Structure-from-motion supplies the camera poses used in Eq. 4 to unproject 2D pixels into world coordinates."},{"cited_title":"A comparison and evaluation of multi-view stereo reconstruction algorithms","cited_arxiv_id":null,"evidence_quote":"Multi-view stereo supplies the depth maps used in the same unprojection to build the 4D coordinate tensor."},{"cited_title":"Learnable fourier features for multi-dimensional spatial positional encoding","cited_arxiv_id":null,"evidence_quote":"Learnable Fourier features provide the position and time encoding recipe for the spatiotemporal prompt."},{"cited_title":"Monocular dynamic view synthesis: A reality check","cited_arxiv_id":null,"evidence_quote":"One of the multi-view dynamic scene datasets merged to construct the Chat4D instruction-tuning data."},{"cited_title":"Neural 3d video synthesis from multi-view video","cited_arxiv_id":null,"evidence_quote":"A third multi-view dynamic video dataset used to create Chat4D's 4D instruction data."}],"review_version":1}