{"id":"f1f6aebf-0ff8-4a38-a0bb-d6e77baa20b2","arxiv_id":"2412.09608","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A hierarchical temporal arrangement of 4D Gaussian primitives achieves near-constant GPU memory, compact storage, and real-time rendering for long volumetric videos.","lead":"This paper introduces a multi-level hierarchy of 4D Gaussian primitives that lets one model reconstruct and render minutes-long volumetric videos from multi-view cameras, using a nearly constant amount of GPU memory. A smart generalist should read it because it targets the main scaling bottleneck in free-viewpoint video, which could make long-form AR/VR and telepresence content practical.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Constant-memory claim hinges on the global segment staying small; a moving camera through new static content makes the global segment grow with video length, violating the claim.","rationale":"The reader's weakest assumption identifies the same load-bearing premise: the global segment and coarse levels are only small because the evaluated scenes have static backgrounds with dynamic foregrounds. My concern sharpens this into a concrete failure mechanism. Eq. (6) can always place a Gaussian into some segment, but the single global segment has no natural bound: any content that persists across the entire video is accumulated there. Since the global segment is loaded at every timestamp (Sec. 3.2.2), its size contributes directly to per-timestamp memory. For a moving camera through a large environment, persistent content grows with T, making the central memory claim false in a natural regime the paper explicitly claims to accommodate ('even if the cameras are moving'). This is not an internal inconsistency of the representation for static-camera scenes; the method is well engineered and the hierarchy mechanism genuinely bounds the active set when the content seen at different times is mostly the same. The issue is an overclaim about generality, which the authors themselves partially acknowledge in Sec. 5.1.1 by noting the datasets' static-background nature. Thus the correct verdict remains conditional acceptance: the core idea is sound and supported for the evaluated domain, but the headline claim should be qualified to scenes whose persistent content is bounded, and a moving-camera experiment or an explicit statement of this limitation is needed before the 'regardless of video length' claim can stand. Other concerns raised by the reader (VRAM summation over segments, non-uniform quality gains, missing quantitative baselines on minute-scale videos) are secondary and addressable without changing the verdict.","tokens_in":23378,"tokens_out":4974,"duration_ms":55257,"concrete_test":"Render a synthetic long multi-view video of a camera translating through a large static scene (e.g., a textured room or city model) at increasing lengths (300, 1200, 4800, 19200 frames). Train the TGH model at each length and record training VRAM and the number of Gaussians in the global segment. If VRAM or global-segment Gaussian count grows roughly linearly with frame count, the near-constant-memory claim fails for this regime. A lighter variant is to measure global-segment Gaussian count as a function of covered path length on any existing long moving-camera sequence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The near-constant GPU memory guarantee depends on the placement rule in Eq. (6): every 4D Gaussian is assigned to the shortest segment that contains its temporal influence range, and Sec. 3.2.1 appends a single global segment of infinite length for static scene content. In Sec. 3.2.2 the renderer always loads one segment per level, including this global segment. If the camera moves through a large static environment, newly revealed static content has a temporal influence spanning the whole video, so it is placed in the global segment. The global segment therefore grows with the total spatial extent visited, i.e., with video length T, and because it is active at every timestamp, per-timestamp VRAM grows with T. This directly contradicts the abstract's claim of 'nearly constant GPU memory usage ... regardless of the video length.' The paper's Sec. 5.1.1 defense that world-coordinate placement handles moving cameras is insufficient: world coordinates do not bound the number of persistent Gaussians when new regions are observed. All evaluations use synchronized static camera arrays over scenes with largely static backgrounds, so this failure mode is never tested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Temporal Gaussian Hierarchy (TGH), a hierarchical 4D representation for long multi-view volumetric video. TGH organizes 4D Gaussian primitives into temporal segments of exponentially decreasing length, plus one global infinite-length segment for static content, and loads only the segments active at a given timestamp. The authors claim this yields nearly constant GPU memory during training and rendering regardless of video length, compact storage, and state-of-the-art quality, and they evaluate on Neural3DV, ENeRF-Outdoor, MobileStage, CMU-Panoptic, and a newly collected SelfCap dataset with sequences up to 6,000 frames.","tokens_in":23632,"tokens_out":5617,"duration_ms":52528,"significance":"If the claims hold, the work is significant: it would move dynamic view synthesis from 1-2 second clips to minute-scale volumetric video with bounded GPU memory and real-time rendering. The temporal-hierarchy idea is coherent, the ablations in Fig. 9(c) show roughly constant VRAM as frame count grows in the tested setting, the sparse-SH appearance model gives substantial storage savings with minimal quality loss (Tab. 2), and the hardware rasterizer provides a clear speed improvement. However, the constant-memory claim is conditional on the global segment staying small and on static-background scenes, and the headline comparisons against baselines use summed VRAM rather than peak per-segment VRAM, which overstates the baselines' memory disadvantage. The contribution is real but the paper currently overclaims its scope and its baseline accounting needs correction.","major_comments":[{"comment":"The VRAM values marked '*' sum per-segment training memory over all segments (e.g., 84 GB for four 300-frame segments on Neural3DV, and 450 GB for 60 segments in Fig. 1), whereas the baselines were actually trained segment-by-segment. Their peak GPU memory during any single training run is therefore approximately the per-segment value, not the sum. This makes the 'constant GPU memory' comparison against baselines misleading. Please report the per-segment peak VRAM alongside the summed total, and base claims of memory advantage over baselines on the per-segment peak.","section":"Sec. 5.1.3, Tab. 1, Figs. 1, 4-6"},{"comment":"The near-constant GPU memory claim relies on the global segment (length infinity) remaining small. For a moving camera or a scene whose static content is progressively revealed, newly observed static Gaussians have temporal influence ranges spanning the whole video and are placed in the global segment by Eq. (6); because the global segment is loaded at every timestamp, per-timestamp VRAM grows with the total observed spatial extent, i.e., with video length. The statement in Sec. 5.1.1 that world-coordinate placement handles moving cameras is not sufficient: world coordinates do not bound the number of persistent Gaussians when new regions are observed. All evaluated datasets use synchronized static camera arrays with largely static backgrounds, so this failure mode is untested. Please add a moving-camera or expanding-scene experiment measuring VRAM versus sequence length, or explicitly restrict the constant-memory claim to scenes with bounded spatial extent.","section":"Sec. 3.2.1, Eq. (6), and Sec. 5.1.1"},{"comment":"The scalability plot that supports constant VRAM ends at 1,200 frames. The SelfCap results (6,000 and 3,540 frames) give single VRAM values (12.9 GB and 9.4 GB) but no VRAM-versus-frame-count curve and no comparison with a baseline under the same accounting. Consequently, the paper's central claim of constant memory at minute scale is not quantitatively demonstrated. Add a VRAM-versus-length sweep on a long sequence and report both peak and total memory for all methods.","section":"Fig. 9(c) and Sec. 5.3"}],"minor_comments":[{"comment":"The notation for the temporal influence interval is inconsistent: the displayed condition in Eq. (6) has duplicate and unclear interval symbols. Please rewrite the interval endpoints explicitly, for example as [tau_min, tau_max].","section":"Sec. 3.2.1, Eqs. (4) and (6)"},{"comment":"The storage row for 'Ours' reads '0. 92 GB', which conflicts with the text and figure captions that report 92 MB; the decimal point appears to be misplaced.","section":"Tab. 2"},{"comment":"The sentence 'no explicit temporal consistency is enforced other than sharing the camera parameters' is confusing, because sharing camera parameters is not a temporal consistency mechanism; please rephrase.","section":"Sec. 5.1.1"},{"comment":"For 4K4D and ENeRF the reported storage includes video-encoded source images, while the proposed method's storage does not include source images; the storage comparison is therefore not on equal footing and this should be stated in the table or main text.","section":"Sec. 5.1.2"},{"comment":"The statement that copying segments to GPU memory 'can be parallelized with the rendering' needs implementation detail, such as asynchronous copy streams, to support the claim that there is no speed downgrade.","section":"Sec. 3.2.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is likely to interest the SIGGRAPH/TOG audience, and the core temporal-hierarchy mechanism is a genuine step toward long volumetric video. My main concerns are fairness of the baseline memory accounting and overgeneralization of the constant-memory claim beyond static-background, bounded-scene settings. The authors' own limitation statement in Sec. 5.1.1 should be addressed head-on, and the baseline VRAM numbers should be corrected. I do not see evidence of circularity; the empirical claims are evaluated against external benchmarks."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper introduces Temporal Gaussian Hierarchy, a multi-level temporal segmentation of 4D Gaussians where each Gaussian lives in the shortest segment containing its temporal influence range, plus a global segment for static content. That is a genuine, non-obvious representation, and the paper shows it working: roughly constant VRAM as video length grows on Neural3DV, and an 18,000-frame demo on one RTX 4090. The ablations in Fig. 9 actually support the mechanism—without the hierarchy, memory climbs with frame count; with it, it plateaus. The compact appearance model and hardware rasterizer are useful supporting contributions. The writing is clear and the experiments are mostly well designed.\n\nNow the soft spots, in order of severity.\n\nFirst, the constant-memory claim is too broad as stated. It holds when static content is bounded, as in the static-camera, static-background datasets used here. But the global segment is loaded at every timestamp. If the camera moves through a large environment, newly revealed static content has an influence range spanning the whole video, so it lands in the global segment, and per-timestamp memory grows with how much new content you expose. The paper's assertion that world-coordinate placement makes moving cameras harmless is not right. The honest claim is \\\"constant memory for fixed spatial extent\\\" or \\\"constant with respect to temporal redundancy\\\", and that is still valuable.\n\nSecond, the baseline memory comparison sums VRAM over training segments (e.g., 450 GB for 4K4D on 18,000 frames). If you train segments sequentially, peak VRAM is what matters; the sum inflates the contrast. Addressable presentation issue, but readers will notice.\n\nThird, the minute-scale SelfCap results have no quantitative baselines, so the flagship \\\"10-minute volumetric video\\\" comparison rests on qualitative figures and summed estimates in Fig. 1. A few numbers against a segment-wise baseline would solidify the claim.\n\nMinor: on ENeRF-Outdoor the PSNR is below 4K4D, so \\\"state-of-the-art quality\\\" is dataset-dependent. And hyperparameters tuned on Neural3DV then applied elsewhere is disclosed and fine.\n\nNone of these are load-bearing flaws in the central idea. The hierarchy is a real contribution, and with code and dataset release it will be useful to people working on long dynamic capture.\n\nRecommendation: send to peer review. It is novel, clearly explained, and the weaknesses are fixable with better reporting. I'd bring it to a reading group and would cite it.","headline":"A genuinely new temporal hierarchy for 4D Gaussians that delivers near-constant memory on static-camera captures; the constant-memory claim overreaches for moving cameras, but the core contribution is solid.","tokens_in":24215,"tokens_out":3455,"would_cite":true,"duration_ms":31992,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that long volumetric video can be represented with a multi-level hierarchy of 4D Gaussian primitives in which only a small subset of primitives is active at any moment, giving near-constant GPU memory and compact storage…","keywords":["volumetric video","dynamic view synthesis","4D Gaussian splatting","temporal hierarchy","multi-view video","real-time rendering","novel view synthesis","compact representation"],"falsifier":"Render a synthetic multi-view sequence of fixed length where the entire visible scene changes color every frame, so no Gaussian influence range can extend across segments and the global segment absorbs almost nothing; then train the method on progressively longer versions of this sequence and measure GPU memory and storage. If VRAM or disk use grows with frame count rather than staying flat, the central constant-memory claim would be falsified.","tokens_in":23176,"feed_emoji":"🎥","tokens_out":10062,"duration_ms":86100,"temperature":0.7,"pith_summary":"This paper aims to make long volumetric video practical: instead of modeling every frame with its own set of 4D Gaussians, it arranges Gaussians into a temporal hierarchy whose coarse levels share primitives across long stretches of time and whose fine levels handle fast motion. The payoff would be that GPU memory and storage stop growing with video length, so minutes of multi-view dynamic footage can be reconstructed and played back in real time on a single GPU. The paper claims the first demonstration of this for sequences up to 18,000 frames, with rendering around 450 FPS at 1080p. It evaluates on several public datasets and a newly captured long-sequence dataset, reporting quality comparable to or better than methods that only handle a few hundred frames.","feed_headline":"18,000 frames of 4D video on one GPU, rendered live","feed_subtitle":"A multi-level temporal hierarchy shares static content across time, so GPU memory and storage stay flat as videos grow to minutes.","key_machinery":"The load-bearing object is the Temporal Gaussian Hierarchy, a tree of temporal segments: a root global segment holds static content, and level $l$ contains non-overlapping segments of length $S/2^l$. Each segment stores 4D Gaussian primitives. The placement rule of Eq. (6) assigns every Gaussian to the shortest segment whose temporal range fully contains its influence range, and the corrected offset $\\tau_l = -S/2^{l+2}$ staggers segment boundaries so that Gaussians are not systematically pushed into longer segments. This makes segment lookup $O(\\log N)$ and keeps the active Gaussian set per timestamp bounded. Two supporting mechanisms reduce storage and speed up rendering: a Compact Appearance Model that applies spherical harmonics only to Gaussians whose gradients exceed a threshold and then entropy-codes the coefficients, and a hardware-accelerated rasterizer that sorts Gaussians on the GPU and draws them as quads.","core_discovery":"The central claim is that temporal redundancy can be turned into a structural guarantee. The method builds a multi-level hierarchy in which level $l$ partitions time into segments of length $S/2^l$, plus one global segment of infinite length for static content. Each 4D Gaussian has a temporal opacity profile with a finite influence range, and the placement rule puts every Gaussian into the shortest segment whose boundaries fully contain that range. Because the levels form a tree, a given timestamp activates exactly one segment per level, so the working set of Gaussians is determined by the hierarchy depth and the scene's distribution of motion, not by the video's frame count. The hierarchy update reassigns Gaussians to levels as their opacity profiles change during optimization, which keeps the constant-memory property throughout training. The paper reports that this representation trains with roughly constant VRAM, stores long videos compactly, and renders in real time.","pith_inferences":["If the same temporal-sharing principle transfers to other 4D representations, feature grids and point-cloud sequences could be given the same long-video scalability by partitioning them into this kind of segment tree.","Because each segment holds a self-contained Gaussian set, one testable extension is streaming playback: a client could download only the active segments for the current timestamp, which would make minute-scale volumetric video deliverable over limited bandwidth.","The hierarchy also separates content by temporal frequency, so after training one could selectively edit, slow down, or remove fast-moving layers without retraining the whole sequence.","The paper does not run a long take with a moving camera; the representation lives in world coordinates, but camera motion would put visible regions into flux and is a natural test of where the constant-memory guarantee saturates."],"forward_implications":["A single model can reconstruct and render a volumetric video of arbitrary length with the same per-frame GPU working set, because only one segment per hierarchy level is active at any timestamp.","Storage stays compact as videos grow: static and slow content is shared across long segments, and the sparse appearance model removes most spherical-harmonic coefficients; the paper reports 2.2 GB for an 18,000-frame sequence.","Real-time playback at 1080p is feasible because rendering touches only the active Gaussian subset and uses hardware rasterization; the reported speed is about 450 FPS.","Training no longer runs out of memory on videos several minutes long; previous 4D methods evaluated in the paper could only train on 300-frame segments on the same GPU hardware.","The hierarchy assigns every Gaussian to a temporal-frequency band, so fast and slow scene regions become separable objects that can be handled differently in rendering or editing."],"supporting_citations":[{"why":"Supplies the 4D Gaussian primitive, including its temporal Gaussian opacity profile, that the hierarchy stores in each segment.","marker":"Yang et al. 2023b"},{"why":"Supplies the 3D Gaussian splatting basis, adaptive density control, and the software rasterizer that this work extends and accelerates.","marker":"Kerbl et al. 2023"},{"why":"Is the primary comparison baseline and the source of the MobileStage dataset used for long-video evaluation.","marker":"Xu et al. 2024b"},{"why":"Provides the Neural3DV multi-view dynamic dataset that anchors the main quality and ablation experiments.","marker":"Li et al. 2022"},{"why":"Provides the ENeRF baseline and the ENeRF-Outdoor sequences for comparing on outdoor long videos.","marker":"Lin et al. 2022"},{"why":"Provides the Dy3DGS per-frame tracking baseline used on the CMU-Panoptic subsequences.","marker":"Luiten et al. 2024"},{"why":"Supplies the entropy coding used to compress the grouped spherical-harmonic coefficients in the Compact Appearance Model.","marker":"Huffman 1952"}],"fun_headline_variants":["Temporal Gaussian hierarchy scales 4D video to minutes","Flat GPU memory for minutes of volumetric video","Long video, constant memory: hierarchical 4D Gaussians","Tree-structured Gaussians render long video in real time","Video length no longer limits memory with Gaussian hierarchy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a scene can be split into time windows so that every moving piece's effect on the video fits inside one window, and that most of the scene barely changes over long stretches; if everything changes everywhere at once, the parts of the model that share content across time would be overwhelmed and the memory and storage savings would disappear.","fun_headline_variants_meta":{"raw":{"variants":["Temporal Gaussian hierarchy scales 4D video to minutes","Flat GPU memory for minutes of volumetric video","Long video, constant memory: hierarchical 4D Gaussians","Tree-structured Gaussians render long video in real time","Video length no longer limits memory with Gaussian hierarchy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000165,"raw_usage":{"total_tokens":1281,"prompt_tokens":1009,"completion_tokens":272,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":625,"completion_tokens_details":{"reasoning_tokens":196}},"tokens_in":625,"tokens_out":272,"duration_ms":3157,"temperature":1.0,"reasoning_tokens":196,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:52:37.565423+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Render a synthetic multi-view sequence of fixed length where the entire visible scene changes color every frame, so no Gaussian influence range can extend across segments and the global segment absorbs almost nothing; then train the method on progressively longer versions of this sequence and measure GPU memory and storage. If VRAM or disk use grows with frame count rather than staying flat, the central constant-memory claim would be falsified.","supporting_citations":[],"review_version":1}