{"id":"7a9b0125-07ca-4f37-9ecf-94af96340cb8","arxiv_id":"2412.02493","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"RelayGS improves dynamic 3D Gaussian reconstruction of large-scale motions by decoupling foreground from background with a learnable mask and decomposing trajectories into per-segment Relay Gaussians, gaining about 1 dB PSNR over prior art.","lead":"This paper presents RelayGS, a three-stage Gaussian Splatting method that reconstructs dynamic scenes with large and complex motions by separating moving foreground from static background and breaking motion trajectories into per-segment copies. It reports about 1 dB average PSNR gain over prior methods on a sports dataset and more coherent basketball reconstructions, which matter for free-viewpoint sports video and VR.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Mask decoupling is the load-bearing step, but it is only supervised at frame 1; any foreground absent or static at frame 1 is unrecoverable, and the paper provides no test of this dependency.","rationale":"The reader's weakest-assumption diagnosis is the same one I find most load-bearing: the Stage-1 learnable mask is the gate through which all foreground Relay Gaussians must pass. If the mask mislabels a dynamic object as background, that object is absent from Stage 2 and cannot benefit from trajectory decomposition; if it mislabels static content as foreground, capacity is wasted and the pseudo-view optimization is polluted. My sharper formulation is that the mask is trained almost exclusively against the first frame for foreground content, so the object must be visible and moving in frame 1. The paper gives no experiment that varies first-frame content, reports mask IoU, or sweeps epsilon, so the failure regime is uncharacterized. I considered the pseudo-view averaging in Eq. (9) as an alternative concern: averaging three dynamic frames produces ghosted targets, and the paper does not prove why this is a principled supervisor. However, ablation #4 shows that removing pseudo-views costs 0.20-0.26 dB, so the design choice is at least empirically supported, whereas the mask has no direct validation. I also weighed the imported/estimated baseline numbers for Dynamic3DGS and D-MiSo on PanopticSports; this affects the exact 1.02 dB margin but not the architectural dependence on the mask, and it is already a reason for the reader's conditional verdict. The promised code and missing hyperparameters, including epsilon, are reproducibility concerns that reinforce CONDITIONAL but do not move the verdict. Overall, the paper is internally coherent and the three-stage pipeline is clearly described; the concern is that its central claim inherits an unvalidated first-frame dependency, so the reader's conditional posture is the right one.","tokens_in":15347,"tokens_out":5103,"duration_ms":59544,"concrete_test":"Run RelayGS on PanopticSports (or a controlled synthetic multi-view scene with the same capture setup) under two conditions: (A) the standard first frame, and (B) an otherwise identical sequence whose first frame is replaced by a frame with the moving actor removed or held fully static for the first k=16 frames. Compare foreground-region PSNR/SSIM on the held-out test views. If condition (B) degrades substantially relative to (A), the first-frame dependency of the mask is confirmed and the central claim is conditional on favorable initialization; if (B) is robust, the concern is resolved. Independently, report the learned mask's IoU against ground-truth foreground maps on a few frames to verify decoupling quality directly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mechanism of RelayGS is the Stage-1 binary mask (Eqs. 7-8): foreground Gaussians are identified by fitting a static 3DGS to all frames while rendering only the first frame with the full set, so moving content is pushed to mask=0 and static content to mask=1. Stage 2 then replicates only the mask=1 foreground Gaussians. This makes the whole Relay-Gaussian decomposition contingent on two unexamined facts: (i) the dynamic object is present and reasonably well reconstructed in frame 1, and (ii) the mask assignment is correct. If a subject enters after frame 1, is occluded in frame 1, or is stationary during the initial frames, its Gaussians will be classified as background or pruned, and no Relay Gaussians will ever be created for it; the error is then inherited by Stage 3. The paper explicitly notes that on PanopticSports foreground colors are 'quite similar to the background' (Sec. 9), which raises the misclassification risk, yet no mask-accuracy measurement (e.g., IoU against known foreground) or first-frame-sensitivity analysis is reported. The mask threshold epsilon is also not disclosed, so the failure boundary cannot be assessed. The final PSNR gains are averaged over the whole frame and a static background dominates the VRU scenes, so a partially failed mask can be masked by the background floor.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RelayGS, a three-stage 3D Gaussian Splatting method for reconstructing dynamic scenes with large and complex motions. In Stage 1, a static 3DGS is trained on all frames together with a learnable per-Gaussian binary mask that separates static background (mask=1) from dynamic foreground (mask=0); all Gaussians render the first frame, while only background Gaussians render later frames. In Stage 2, copies of the foreground Gaussians are created for each temporal segment and optimized against pseudo-views formed by pixel-wise averaging of three frames per segment; these copies are called Relay Gaussians. In Stage 3, the canonical Gaussians and a HexPlane-plus-MLP motion field are jointly optimized, with separate MLPs for foreground and background and a learnable per-Gaussian scaling factor for foreground position deformation. Experiments are reported on the PanopticSports and VRU Basketball Games datasets, with quantitative tables, ablations, and qualitative comparisons.","tokens_in":15536,"tokens_out":9028,"duration_ms":93246,"significance":"If the central claims hold, the idea of temporal densification through Relay Gaussians is a useful and novel contribution to dynamic 3DGS reconstruction. The method avoids pre-trained segmentation or optical-flow priors, and the paper includes ablations for the mask, relay copies, pseudo-views, foreground-background MLP separation, and the scaling factor. The evaluation on held-out test cameras and public datasets is a positive feature. However, the headline PSNR improvements rest on partially estimated baseline numbers, the mask decoupling is not quantitatively validated, and the pseudo-view supervision is theoretically questionable. These issues currently prevent the paper from fully supporting its state-of-the-art claim.","major_comments":[{"comment":"The binary mask is the keystone of the pipeline: Stage 2 replicates only the mask=0 foreground Gaussians, so any error in mask assignment propagates to the Relay Gaussians. The mask is trained only through the first-frame render plus the static-objective gradients on other frames, and the paper provides no quantitative check of mask quality (e.g., IoU against a known foreground segmentation) and does not report the threshold epsilon. Moreover, because the point cloud is initialized from the first frame, any dynamic object absent or heavily occluded in frame 1 has no Gaussians to be replicated in Stage 2. Please add a mask-accuracy measurement, a sensitivity study around epsilon, and at least one experiment or explicit discussion of the first-frame dependence.","section":"Sec. 4.1, Eqs. (7)-(8)"},{"comment":"The pseudo-view used as supervision for Relay Gaussians is a pixel-wise average of three frames at different instants. No real camera observes this image, and a static set of Gaussians cannot simultaneously reproduce three different object poses; the optimum of the pseudo-view loss may be a ghosted or averaged position rather than a point on the actual motion trajectory. The paper does not justify why this composite target is appropriate for decomposing large motions, and the ablation in Table 3 only removes pseudo-views entirely. Please provide a principled justification or an ablation comparing Eq. (9) with alternative supervision, such as separate per-frame supervision for each Relay Gaussian copy.","section":"Sec. 4.2, Eq. (9)"},{"comment":"The average 1.02 dB gain on PanopticSports is the paper's headline quantitative claim, but for two of the four competing methods (Dynamic3DGS and D-MiSo) the table reports numbers 'partially taken directly from their original papers or estimated based on the paper and available code.' It is not demonstrated that these numbers were produced under the same resolution, test cameras, frame range, and initialization as RelayGS. The comparison against those baselines is therefore not apples-to-apples and cannot fully support the state-of-the-art claim. Please re-run all baselines under the same protocol, or explicitly restrict the SOTA claim to the baselines that were re-run.","section":"Table 2 and Sec. 5.2"},{"comment":"No variance or multiple-seed results are reported, although several per-scene gains are small (0.27 dB on Juggle, 0.45 dB average on VRU) and the method involves random initialization, stochastic densification, and several hand-set hyperparameters. Without error bars or a paired significance test over the held-out test cameras, the reader cannot assess whether the improvements are robust. Please report at least mean plus/minus standard deviation over three runs for the main tables.","section":"Sec. 5.2, Tables 1-2"}],"minor_comments":[{"comment":"The word 'strightforward' should be 'straightforward'.","section":"Sec. 4.2"},{"comment":"The word 'Gaaussians' should be 'Gaussians'.","section":"Fig. 1 caption"},{"comment":"The phrase 'reduces the rendering speed in some tend' appears to be a typo for 'in some extent'; please rephrase.","section":"Sec. 5.2"},{"comment":"The final training loss of RelayGS is never written explicitly; Eq. (3) and Eq. (6) are the losses of 3DGS and 4D-GS, and it is unclear whether the third stage uses L1+D-SSIM, L1+Ltv, or a combination. Please state the exact loss.","section":"Sec. 4.3 and Sec. 5.1"},{"comment":"D-MiSo has no storage or training time entries; please complete these entries or state why they are unavailable.","section":"Table 2"},{"comment":"The statement that ST-GS is not applicable to PanopticSports because per-frame point clouds cannot be obtained would benefit from a brief explanation, since ST-GS is otherwise a strong baseline.","section":"Sec. 5.2"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of the journal, and I see no citation or novelty-disclosure issues beyond the estimated-baseline concern already raised in the major comments. The abstract's 'more than 1 dB' claim is stronger than the evidence currently supports because some baseline numbers are not from a controlled re-run."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing worth reading this paper for is the relay idea. Splitting a long, large-amplitude motion trajectory into per-segment Gaussian copies and supervising them with pseudo-views is a clean, practical contribution, and the paper gives it a nice name (temporal densification). The three-stage pipeline is clearly described, the ablations are directed at the right components, and the reported gains are consistent across two real multi-view datasets. On PanopticSports the average gain over the previous best is about 1 dB; on VRU it is smaller, about 0.45 dB over ST-GS, with the useful caveat that the static background dominates the full-frame metric. The authors are honest that several baseline numbers are imported from other papers and that ST-GS cannot run on PanopticSports with its default point-cloud initialization. I believe the central claim: this method reconstructs fast, non-rigid foreground more completely than the baselines, and the qualitative videos would likely show it.\n\nThe soft spots are real but not fatal. The load-bearing piece is the Stage-1 mask, which is learned only by fitting a static 3DGS to all frames while supervising frame 1 with all Gaussians and other frames with masked ones. The stress-test concern about this is valid: any dynamic content that is absent, occluded, or stationary at frame 1 will not be labeled foreground, so no Relay Gaussians will be created for it, and the error propagates through Stages 2 and 3. The paper does not report mask accuracy or any first-frame sensitivity analysis, and the threshold epsilon is not disclosed. On PanopticSports, where the authors themselves note foreground colors resemble the background, the misclassification risk is not negligible. That said, the experimental design is not circular: PSNR is measured on held-out test cameras, and the mask is a model-internal choice, not a fitted target. The lack of error bars is a moderate concern, and the tuning of segment length k on scenes that also appear in the final benchmark is a mild leakage issue.\n\nWho is this for? Anyone working on dynamic 3DGS, especially on sports or other large-motion scenes. The paper deserves a serious referee. My recommendation: engage with it, but push the authors to release code and masks, report the epsilon value, and add an experiment where the first frame has the actor occluded or entering late. That one experiment would either validate the design or reveal its boundary.","headline":"A solid empirical 3DGS paper whose relay-Gaussian idea is genuinely useful, but the mask-first design is fragile at exactly the point the paper does not test.","tokens_in":16223,"tokens_out":658,"would_cite":true,"duration_ms":9391,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RelayGS claims that breaking long object trajectories into shorter per-segment Gaussian nodes, trained against pseudo-views, is what lets 3D Gaussian Splatting reconstruct large-scale complex motions, and reports average PSNR gains of…","keywords":["dynamic scene reconstruction","3D Gaussian Splatting","4D representation","motion field","foreground-background decoupling","temporal densification","multi-view video","novel view synthesis"],"falsifier":"A decisive test would be to render a synthetic multi-view scene with known ground-truth foreground masks and large motions, run RelayGS, and compare the learned Stage 1 mask with the ground truth; if forcing the mask to the ground-truth labels changes the PSNR gain, then the decoupling step, not the relay decomposition, is carrying the reported improvement.","tokens_in":14979,"feed_emoji":"🏀","tokens_out":8109,"duration_ms":74185,"temperature":0.7,"pith_summary":"RelayGS is a 3D Gaussian Splatting method for reconstructing dynamic scenes in which objects move far and in complex ways. The paper's central claim is that a long motion trajectory is easier to reconstruct once broken into shorter pieces, each represented by its own set of foreground Gaussians called Relay Gaussians. The method first learns a static scene from all frames and uses a learnable binary mask to separate fast-moving foreground Gaussians from a nearly static background. It then copies the foreground Gaussians once per temporal segment and trains the copies against pseudo-views formed by blending frames inside that segment, so the copies settle at different points along the true motion path. A final stage learns a compact motion field over these nodes, and the paper reports an average gain of 1.02 dB in PSNR over prior best methods on PanopticSports plus more complete reconstructions of real basketball game scenes.","feed_headline":"Splitting motion into relay segments lifts dynamic-scene PSNR by 1 dB","feed_subtitle":"Decomposing long motion into relay steps lets 4D Gaussian splatting reconstruct fast-moving players and balls.","key_machinery":"The central object is the Relay Gaussian, a per-segment copy of a foreground Gaussian that serves as an explicit node along the object's motion trajectory. Stage 1 learns a static 3DGS from all frames and uses a straight-through-estimator binary mask to decouple high-dynamic foreground and low-dynamic background; Stage 2 replicates foreground Gaussians once per $k$-frame segment and optimizes each copy against pseudo-views, which are uniform blends of three frames from that segment, an operation the paper analyzes as temporal densification; Stage 3 fits a HexPlane-plus-MLP motion field with separate foreground and background MLPs and a learnable scale factor on foreground position offsets. Together these parts turn one long trajectory into several short ones, which is what the paper claims makes large motion learnable.","core_discovery":"The authors' claim, on their own terms, is that explicit temporal decomposition of foreground motion is what allows a canonical-Gaussian-plus-motion-field representation to handle large-scale, complex dynamics. Their three-stage pipeline yields canonical Gaussians that already contain separated foreground and background, plus per-segment Relay Gaussians that approximate an ideal continuous trajectory by discrete, optimized nodes. With this foundation, the compact motion field only needs to predict short displacements between relay nodes rather than one giant leap, and the reported result is an average 1.02 dB PSNR improvement over the previous best on the six PanopticSports scenes, a 0.45 dB gain over the prior best on the VRU basketball scenes, and visibly more complete and temporally coherent players in rendered video.","pith_inferences":["A direct extension the paper leaves implicit is adaptive temporal segmentation: instead of a fixed 16-frame window, segment boundaries could be placed where motion speed changes, which the $k$-ablation suggests would improve on the fixed choice.","Because the mask is trained without motion priors, the method should be tested on a synthetic scene with known foreground identities and foreground/background colors made nearly identical; if the separation then fails, the reported gain would be partly attributable to the datasets' mask learnability rather than the relay mechanism itself.","The authors' stated limitation about small fast objects such as the ball indicates that temporal densification alone does not solve spatial sparsity; combining relay copies with targeted spatial densification along the ball's path is a natural next experiment.","Pseudo-views are currently simple uniform blends of three frames; learned per-pixel blending or frame selection guided by motion magnitude could strengthen the supervisory signal, since the paper's own ablation shows pseudo-views contribute about 0.20 to 0.26 dB."],"forward_implications":["If the central claim holds, dynamic-scene reconstruction methods can address large motions by temporally densifying the trajectory instead of only enlarging the deformation network.","The Stage 1 mask result implies that foreground and background Gaussians can be separated without optical flow or semantic segmentation, which would let the pipeline run on datasets where such priors are unavailable.","The per-scene PSNR gains imply that most of the improvement is concentrated in the moving foreground, so metrics dominated by static background understate the method's effect.","The $k=16$ segment-length ablation implies that segment length is a real hyperparameter: too coarse a decomposition shortens the advantage, while too fine a decomposition increases training cost without further quality gains."],"supporting_citations":[{"why":"Supplies the baseline 4D-GS architecture and the HexPlane plus MLP motion field that RelayGS modifies.","marker":"[36]"},{"why":"Provides the 3D Gaussian primitive, differentiable rasterization, and density control used throughout.","marker":"[14]"},{"why":"Provides the multi-resolution HexPlane spatiotemporal encoding used in the final motion field.","marker":"[3]"},{"why":"Supplies the straight-through estimator that makes the binary foreground/background mask learnable.","marker":"[2]"},{"why":"Provides the PanopticSports multi-view dataset used for the main quantitative comparison.","marker":"[12]"},{"why":"Provides the VRU basketball game sequences used for the second evaluation.","marker":"[34]"},{"why":"Serves as a comparison baseline and the source of the per-camera color tuning used on PanopticSports.","marker":"[24]"},{"why":"Serves as the previous best baseline on VRU and motivates the point-cloud initialization comparison.","marker":"[19]"}],"fun_headline_variants":["Relay Gaussians break big motions into small hops for better dynamic scenes","Relay nodes split complex motion into simple steps, beating SOTA by 1 dB","Segmented relay motion representation lifts dynamic scene PSNR by 1 dB","Basketball players rendered coherently with relay-segmented motion"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that Stage 1's learnable binary mask can tell moving foreground Gaussians from static background Gaussians using only how well each Gaussian explains the first frame versus all other frames; if the mask mislabels a dynamic object as background, those Gaussians are never copied into Relay Gaussians and the later stages inherit the error.","fun_headline_variants_meta":{"raw":{"variants":["Relay Gaussians break big motions into small hops for better dynamic scenes","Relay nodes split complex motion into simple steps, beating SOTA by 1 dB","Segmented relay motion representation lifts dynamic scene PSNR by 1 dB","Basketball players rendered coherently with relay-segmented motion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000877,"raw_usage":{"total_tokens":3819,"prompt_tokens":999,"completion_tokens":2820,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":2738}},"tokens_in":615,"tokens_out":2820,"duration_ms":21434,"temperature":1.0,"reasoning_tokens":2738,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:24:15.884523+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive test would be to render a synthetic multi-view scene with known ground-truth foreground masks and large motions, run RelayGS, and compare the learned Stage 1 mask with the ground truth; if forcing the mask to the ground-truth labels changes the PSNR gain, then the decoupling step, not the relay decomposition, is carrying the reported improvement.","supporting_citations":[{"cited_title":"4d gaussian splatting for real-time dynamic scene rendering","cited_arxiv_id":null,"evidence_quote":"Supplies the baseline 4D-GS architecture and the HexPlane plus MLP motion field that RelayGS modifies."},{"cited_title":"3d gaussian splatting for real-time radiance field rendering","cited_arxiv_id":null,"evidence_quote":"Provides the 3D Gaussian primitive, differentiable rasterization, and density control used throughout."},{"cited_title":"Hexplane: A fast representation for dynamic scenes","cited_arxiv_id":null,"evidence_quote":"Provides the multi-resolution HexPlane spatiotemporal encoding used in the final motion field."},{"cited_title":"Panoptic studio: A massively multiview system for social motion capture","cited_arxiv_id":null,"evidence_quote":"Provides the PanopticSports multi-view dataset used for the main quantitative comparison."},{"cited_title":"Vru-sequence, 2024","cited_arxiv_id":null,"evidence_quote":"Provides the VRU basketball game sequences used for the second evaluation."},{"cited_title":"Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis","cited_arxiv_id":null,"evidence_quote":"Serves as a comparison baseline and the source of the per-camera color tuning used on PanopticSports."},{"cited_title":"Spacetime gaus- sian feature splatting for real-time dynamic view synthesis","cited_arxiv_id":null,"evidence_quote":"Serves as the previous best baseline on VRU and motivates the point-cloud initialization comparison."}],"review_version":1}