{"id":"59134e69-e63d-456b-921f-5da248d22504","arxiv_id":"2607.29284","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"FillGS actively selects spatiotemporal virtual viewpoints using rendering sensitivity and motion-aware observation density, then fine-tunes 4D Gaussian Splatting with reliability-masked generated images, improving sparse-view dynamic rendering.","lead":"4D Gaussian Splatting can render moving scenes, but when only a few cameras cover a scene, fast-moving objects often come out blurry or broken. This paper's method actively chooses which missing viewpoints and moments to ask an image-generation model to fill in, then uses only the trustworthy filled-in pixels to refine the 3D model.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The deficiency score in Eq. (4) is never validated against actual reconstruction gain, and because it also drives fine-tuning weights, the selection ablation is confounded; this leaves the central active-selection claim under-supported.","rationale":"The reader's weakest-assumption analysis correctly identifies Eq. (4) as the load-bearing element of the paper's contribution. My stress-test confirms this: the deficiency score is the core of both selection and fine-tuning, and the paper's evidence is indirect. The ablations do show that components of D matter and that inverted selection is worse, but these comparisons are all within the same pipeline and dataset, without error bars, and the score itself is never checked against a direct measure of reconstruction benefit. The use of the same D in the fine-tuning weight is a genuine confound: Table 4 shows that the weighting scheme itself has a large effect (w/o weight drops to 20.05 PSNR), so selection gains could be partly or wholly due to weighting. I also note a small inconsistency in Table 4: removing the co-visibility mask improves PSNR (21.96 vs 21.73) while worsening LPIPS/FID, which suggests the reported 'consistent improvements' are metric-dependent. However, this is secondary. The overall empirical pattern is still broadly supportive of the pipeline, so a conditional verdict is appropriate rather than rejection. A direct correlation test between D and actual per-trajectory gains would settle whether the active-selection claim is real.","tokens_in":18468,"tokens_out":7152,"duration_ms":81837,"concrete_test":"For each of the 20 virtual trajectories used in FillGS on a subset of Neural 3D Video scenes, record the mean deficiency score D and the change in held-out test PSNR (or LPIPS) caused by fine-tuning with only that trajectory, keeping the diffusion model and consistency mask fixed and using uniform co-visibility weights. Compute the Spearman rank correlation between D and the actual gain across trajectories. If the correlation is not significantly positive (e.g., ρ > 0.3, p < 0.05), Eq. (4) is not a valid proxy for the informative value of a virtual viewpoint, and the central active-selection claim collapses.","verdict_should_be":"UNCHANGED","load_bearing_attack":"FillGS's central claim is that actively selecting virtual viewpoints using the motion-aware deficiency score D_gi(t)=s_gi(t)/(1+C_gi(t)) (Eq. 4) improves 4DGS reconstruction under sparse observations. This score is used both to select viewpoints (Sec. 3.3) and to weight the fine-tuning loss (Sec. 3.5). The paper never validates D against actual reconstruction error or information gain on held-out views. The ablations in Table 3 show that removing speed or observation-count components, or inverting the score, hurts performance, but this only shows D is better than a few fixed alternatives on one dataset; it does not demonstrate that D faithfully identifies the most informative viewpoints. Moreover, because D is also used in the fine-tuning mask, the selection ablations are confounded: the measured gains could come primarily from the weighting scheme, not from choosing informative viewpoints. The paper also reports no error bars or multiple seeds, so the margins (e.g., 21.73 vs 21.15 dB PSNR) may not be significant. If D is not a valid proxy, the proposed active-selection mechanism—the paper's main novelty—would be unsupported, even though the overall pipeline might still help through diffusion-based refinement and consistency masking.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FillGS, a pipeline for 4D Gaussian Splatting (4DGS) under sparse multi-view dynamic captures. It actively selects spatiotemporal virtual viewpoints using a motion-aware observation-deficiency score D_gi(t) = s_gi(t)/(1 + C_gi(t)), where s is Gaussian speed and C is the observation count (Eq. 4). Rendered images from these viewpoints are refined by a video diffusion model, and the 4DGS model is fine-tuned using consistency masks and co-visibility masks that down-weight unreliable or already well-constrained regions. The paper introduces new sparse train/test splits on Neural 3D Video, Technicolor, and Nvidia datasets and reports consistent improvements over E-D3DGS, interpolation-based selection, FisherRF-based selection, and coverage-based selection, with ablations on the viewpoint-selection components and fine-tuning strategy.","tokens_in":18844,"tokens_out":5006,"duration_ms":49868,"significance":"If validated, FillGS addresses a real gap: prior diffusion-based 4DGS refinement methods use heuristic viewpoint selection (random, interpolation, or spatial coverage) and do not account for the spatiotemporal nature of observation sparsity. The paper has several strengths: the experimental protocol controls the backbone and generative model across baselines; the ablations examine both viewpoint selection and fine-tuning; and the supplementary provides experiments with a different backbone (Ex4DGS), a different diffusion model (Difix3D+), a monocular setting, computational cost, and failure cases. The main weakness is that the load-bearing deficiency score in Eq. (4) is never validated against actual reconstruction gain, and because the same score is used in the fine-tuning weights, the selection ablations are confounded. The paper is promising but the central active-selection claim is not yet fully supported.","major_comments":[{"comment":"D_gi(t)=s_gi(t)/(1+C_gi(t)) is presented as the central novelty, but the paper never validates that this score is a proxy for information gain or reconstruction improvement. The only quantitative support is Table 3, where removing speed or count, or inverting the score, degrades results. However, the same D is used as the co-visibility weight in fine-tuning (Sec. 3.5), so the selection ablations are confounded: a change in selection also changes the loss weighting. To support the active-selection claim, the authors should (i) fix the fine-tuning weighting while varying only the selection score, and (ii) provide a direct validation of D against actual per-pixel reconstruction error or information gain, e.g., by comparing candidate viewpoints with high vs. low D after adding a generated view.","section":"Sec. 3.3 and Sec. 3.5, Eq. (4)"},{"comment":"All results are reported as single runs with no error bars, confidence intervals, or significance tests. In sparse-view settings, FID is especially noisy when computed over a small number of test views; the reported margins (e.g., Table 1 interpolation PSNR 21.73 vs. 21.15 for interpolation-based selection, SSIM 0.792 vs. 0.776) may be within run-to-run variation. Please report at least three seeds with mean and standard deviation, or paired significance tests.","section":"Tables 1-4 and supplementary tables"},{"comment":"The evaluation uses new train/test splits that are not released. Without the exact camera/time indices or code to generate them, other researchers cannot reproduce or compare against the numbers. Releasing the splits (and the evaluation protocol) is necessary for the empirical claim to be verifiable.","section":"Sec. 4.1, Fig. 3"},{"comment":"The claim that FillGS improves rendering 'without relying on hand-crafted heuristics' is overstated. Eq. (4) is itself a hand-specified heuristic, and the pipeline has several manually chosen constants: the normalization constant L-bar in Eq. (2), perturbation weights w_delta and perturbation magnitudes in Eq. (3) and the supplementary, and candidate-generation hyperparameters (e.g., 12 local candidates, 10-degree rotations). Please either soften the claim or provide evidence that the results are robust to reasonable variation in these choices.","section":"Abstract and Sec. 1"}],"minor_comments":[{"comment":"The normalization constants are described verbally ('dataset-level mean is 1', 'averages to 1'); please give explicit formulas or a clear pointer to where they are defined.","section":"Sec. 3.2, Eqs. (2)-(3)"},{"comment":"The row 'Negative selection' should state explicitly whether it inverts the score only for viewpoint selection or also for the fine-tuning co-visibility weights, since both use D.","section":"Table 3"},{"comment":"The labels 'GS0.1 m/s GS1 m/s' near the density color bar are confusing; rephrase to clarify what the color scale represents.","section":"Fig. 2"},{"comment":"The patch construction (3 frames x 4 views) and conditioning are only sketched. Please provide precise implementation details in the main text or a more explicit pointer to the supplementary, as this is essential for reproducing the diffusion model.","section":"Sec. 3.4"},{"comment":"The text says the sensitivity score is 'analyzed on synthetic data in the supplementary,' but Supp. B.3 only visualizes the score over space and time; it does not quantitatively correlate the score with reconstruction error. The wording should be adjusted to match what is actually shown.","section":"Sec. 3.2 and Supp. B.3"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline major revision. The controlled comparison is a strength, but the confound between the selection score and the fine-tuning weights is the main technical barrier. I would be supportive if the authors add a decoupled selection experiment, validate D against held-out reconstruction error, report multiple seeds, and release the splits. The 'no hand-crafted heuristics' claim should also be toned down or substantiated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real engineering contribution and it deserves serious refereeing. The new thing is the motion-aware observation-deficiency score D = s/(1+C), used to pick virtual camera-time trajectories for diffusion-based refinement of 4DGS, plus sensitivity-based starting points and reliability masking during fine-tuning. The evaluation is mostly well-controlled: same backbone, same diffusion model, same number of generated images across baselines; the gains are consistent across two multi-view datasets and two train/test splits; the ablations check each component; and the supplementary includes a Blender experiment for the starting-point selection. I'd give credit for the cost analysis and the monocular extension.\n\nSoft spots, in order of importance. First, the D score is asserted, not validated. Eq. (4) is never checked against actual reconstruction error or information gain on held-out views; the synthetic analysis covers starting-point selection, not the deficiency score itself. Second, D is used both to select viewpoints and to weight the fine-tuning loss, so in the selection ablations you can't tell how much of the gain comes from choosing informative views versus from reweighting the same generated content. The ablation would be cleaner if the weighting were fixed to a neutral scheme while varying selection. Third, no error bars or multiple seeds; on small test sets, gaps like 21.73 vs 21.15 PSNR (Table 3) might not be significant, and FID on these small sets is noisy. Fourth, the intro's claim that the method works 'without relying on hand-crafted heuristics' overstates it: the score itself is a hand-specified surrogate, with normalization constants and perturbation magnitudes chosen manually. The paper acknowledges this in the supplementary, but the abstract says otherwise.\n\nNone of this sinks the paper. The core pipeline is coherent, the baselines are fair, and the qualitative figures show real artifact reduction. The main risk is that the specific form of D is arbitrary; a reviewer should push for direct validation of the score (e.g., correlation between D and per-region reconstruction error) and for code/data release so the new splits can be sanity-checked.\n\nWho this is for: anyone working on sparse/dynamic view synthesis or active view selection for generative refinement. It deserves a serious referee; I'd send it out with a request for seeds, error bars, and a direct check of the deficiency score.","headline":"A solid engineering pipeline for active sparse-view 4DGS refinement; the central deficiency score is plausible but under-validated and the selection ablation is confounded with the weighting scheme.","tokens_in":19267,"tokens_out":2329,"would_cite":true,"duration_ms":26079,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FillGS argues that spatiotemporal observation gaps in 4D Gaussian splatting should be located by a motion-aware deficiency score and filled with diffusion-refined views, improving rendering fidelity under sparse view coverage.","keywords":["4D Gaussian Splatting","sparse-view dynamic reconstruction","virtual viewpoint selection","generative refinement","motion-aware observation density","video diffusion model","novel view synthesis","observation gap filling"],"falsifier":"Train a 4DGS model on a sparse two-camera split, compute D for each Gaussian, and render several candidate virtual viewpoints with known deficiency scores. For each candidate, generate a diffusion-refined image, fine-tune the model using only that image (with the proposed masks), and measure the actual PSNR or LPIPS improvement on held-out views. If the rank correlation between the deficiency score and the achieved improvement is not positive, the selection signal is not doing the claimed work.","tokens_in":18370,"feed_emoji":"🎥","tokens_out":2759,"duration_ms":34468,"temperature":0.7,"pith_summary":"The paper claims that under sparse camera coverage, dynamic scenes rendered with 4D Gaussian Splatting suffer artifacts because some spatiotemporal regions are barely observed, and existing generative-refinement methods pick virtual viewpoints by heuristics like interpolation or spatial coverage that ignore where the gaps actually are. FillGS instead actively selects virtual viewpoints by scoring candidates according to a motion-aware deficiency signal per Gaussian, then refines the rendered views with a video diffusion model and fine-tunes the 4D model using only pixels that are consistent with real observations. The central claim is that this active spatiotemporal selection and reliability-weighted fine-tuning consistently improve rendering quality over prior viewpoint-selection and fine-tuning strategies. The authors introduce new sparse train/test splits with only two training cameras and report gains across PSNR, SSIM, LPIPS, FID, and DINOv2 similarity in both interpolation and extrapolation settings.","feed_headline":"Sparse 4D video renders sharper with motion-aware view selection","feed_subtitle":"A deficiency score sends a generative model to spatiotemporal regions real cameras barely observe, improving fidelity.","key_machinery":"The motion-aware observation deficiency score D_gi(t) = s_gi(t) / (1 + C_gi(t)), where s_gi(t) is the speed of Gaussian g_i at time t and C_gi(t) is its observation count. This single scalar plays two roles: it selects virtual viewpoints by ranking candidates that observe many high-deficiency Gaussians, and it weights the fine-tuning loss so that sparsely observed, fast-moving regions receive stronger supervision from generated images while already-constrained regions are left mostly unchanged. A sensitivity-based starting-point selection (a finite-difference approximation to local rendering instability) seeds the exploration, and a consistency mask from feature matching filters unreliable g","core_discovery":"The paper's central claim is that the effectiveness of generative refinement for 4D Gaussian Splatting depends on which virtual viewpoints are chosen, and that the right criterion is spatiotemporal observation sparsity rather than spatial coverage or interpolation. It defines a per-Gaussian, per-time deficiency score D = speed / (1 + observation count), which marks fast-moving and rarely seen Gaussians as under-constrained. Candidate virtual viewpoints are scored by rendering this deficiency field and averaging it over the view, and the highest-scoring candidates are refined with a video diffusion model. During fine-tuning, the same deficiency score down-weights already well-observed regions","pith_inferences":["If the deficiency score is a valid proxy for information gain, the same selection criterion could be reused for physical next-best-view planning in dynamic capture, since it targets under-constrained Gaussians rather than spatial coverage and could guide where to point real cameras.","The consistency-masked fine-tuning suggests a general recipe for any generative-depth or diffusion-based supervision: only back-propagate pixels that agree with real observations, treating generated content as a constraint on under-observed regions rather than as ground truth.","A direct test of the score's validity would be to compare D_gi(t) against the actual reduction in reconstruction error obtained by adding a generated view; if the correlation is weak, the selection stage may be succeeding for other reasons, such as the global candidate sampling or the fine-tuning masks.","The method's dependence on the diffusion model's generation quality is acknowledged in the paper; as generators improve, the same selection-and-filtering pipeline could be expected to yield larger gains without architectural changes."],"forward_implications":["Under two-camera training splits, the full pipeline outperforms interpolation-, Fisher-information-, and coverage-based selection on PSNR, SSIM, LPIPS, FID, and DINOv2 similarity in both interpolation and extrapolation test settings.","Ablations show that each component—starting-point selection, candidate scoring, the consistency mask, and the co-visibility mask—contributes to the final gain; removing any one degrades results.","The method improves fine-tuning over prior weighting schemes when the 4DGS backbone and the generative model are held fixed, indicating that the gains come from selection and weighting rather than from a stronger generator.","The approach transfers to a different 4DGS backbone and a different diffusion generator, and also improves monocular capture settings, suggesting the mechanism is not tied to one architecture.","The method is sample-efficient: even a single virtual trajectory yields competitive results, and ten trajectories outperform twenty-trajectory baselines at lower generation cost."],"fun_headline_variants":["Fill spatiotemporal blind spots in 4DGS","Deficiency score picks views to fill 4DGS gaps","Motion-aware view choice cuts artifacts in dynamic 4D scenes","Sparsity-aware viewpoint picking sharpens 4D Gaussian splatting"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that a Gaussian's deficiency, computed as speed divided by (1 + observation count), faithfully measures how much a new virtual observation of that Gaussian would improve the reconstruction; if this score misranks viewpoints, both the selection of where to generate and the weighting of what to fine-tune are pointed at the wrong places.","fun_headline_variants_meta":{"raw":{"variants":["Fill spatiotemporal blind spots in 4DGS","Deficiency score picks views to fill 4DGS gaps","Motion-aware view choice cuts artifacts in dynamic 4D scenes","Sparsity-aware viewpoint picking sharpens 4D Gaussian splatting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001089,"raw_usage":{"total_tokens":4369,"prompt_tokens":710,"completion_tokens":3659,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":3585}},"tokens_in":454,"tokens_out":3659,"duration_ms":26775,"temperature":1.0,"reasoning_tokens":3585,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T09:58:51.972542+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a 4DGS model on a sparse two-camera split, compute D for each Gaussian, and render several candidate virtual viewpoints with known deficiency scores. For each candidate, generate a diffusion-refined image, fine-tune the model using only that image (with the proposed masks), and measure the actual PSNR or LPIPS improvement on held-out views. If the rank correlation between the deficiency score and the achieved improvement is not positive, the selection signal is not doing the claimed work.","supporting_citations":[],"review_version":1}