{"id":"a87efabe-970d-42b4-bffa-7c97d9aad9a5","arxiv_id":"2412.05256","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new benchmark with 90,810 frames from public AV datasets quantifies a large performance drop of neural rendering methods on extrapolated urban views.","lead":"This paper introduces a benchmark for testing how well neural rendering methods can synthesize views far from the cameras they were trained on, using public driving footage from multiple passes and vehicles. It finds that current 3D Gaussian Splatting methods drop sharply in quality when asked to render extrapolated views, suggesting they overfit to training viewpoints.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's own limitation section states test views were selected to be 'well covered' by training trajectories, so the reported 'extrapolation' may be sparse interpolation; the magnitude of the drop is then partly determined by this coverage choice.","rationale":"The reader's weakest_assumption identified both the appearance confound and the manual 'well covered' selection. I agree that the coverage issue is load-bearing, but I do not think the appearance confound is the single most decisive threat: Setting 2 uses multiple cameras on the same vehicle at the same time, so appearance and scene content are controlled there, and the phenomenon still appears. The coverage issue, however, cuts at the definition of extrapolation itself. The paper explicitly states that test viewpoints were selected to be well covered, which makes the benchmark's central construct (extrapolation) suspect. A concrete quantification of the training-test pose gap and an ablation that removes nearby training views would settle whether the reported drops genuinely reflect extrapolation difficulty or are an artifact of the manual selection. This does not overturn the reader's conditional verdict; it sharpens the condition that must be met. I therefore keep the verdict unchanged and acknowledge partial agreement with the reader's analysis.","tokens_in":17830,"tokens_out":6767,"duration_ms":73784,"concrete_test":"For a random sample of 100 test views per setting, use the released COLMAP poses to compute the minimum SE(3) distance to the closest training camera pose, reporting translation in meters and rotation in degrees. Compare this distribution to the pairwise distances among training poses. Then run an ablation on Setting 1: remove all training traversals whose poses are within 2 m / 5 deg of any test pose, retrain 3DGS, and measure the extrapolated PSNR. If the drop increases sharply relative to the reported 23.4%, the current benchmark underestimates extrapolation difficulty and the 'well covered' selection is a key confound. If the drop is similar, the coverage statement is not driving the result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that current NVS methods overfit and degrade substantially under extrapolated views. This requires the test views to be genuine extrapolations, i.e., far outside the training pose distribution. The paper's own text contradicts this. Section 6 (Limitations) says: 'we carefully manually selected and ensured that the test trajectory viewpoints are well covered by training trajectories.' Section 4.2 for Setting 1 similarly says 'training views fully cover test views with moderate translational changes.' If test viewpoints are well covered by nearby training viewpoints, the benchmark evaluates non-uniform or sparse interpolation rather than large-view extrapolation. The abstract's description of test views that 'largely deviate from training views' is therefore not tied to a quantified pose gap. Moreover, the manual coverage selection acts as a dial: choosing more or less covered test trajectories directly influences the reported performance drop (e.g., 3DGS PSNR 21.36 to 16.37 in Setting 1; 21.22 to 14.99 in Setting 3). Without a quantitative definition of extrapolation and a coverage criterion, the headline finding that current methods overfit under extrapolated views is not a well-posed empirical claim about a specific extrapolation regime.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces EUVS, a benchmark for evaluating novel view synthesis (NVS) under extrapolated viewpoints in urban driving scenes. The authors assemble data from nuPlan, Argoverse 2, and MARS, exploiting multi-traversal, multi-agent, and multi-camera recordings to define three evaluation settings: translation-only, rotation-only, and translation+rotation. They train several state-of-the-art Gaussian Splatting and NeRF methods and report a consistent, substantial drop in rendering metrics (PSNR, SSIM, LPIPS, feature cosine similarity) when moving from interpolated to extrapolated test views, concluding that current methods overfit to training views and that diffusion priors or geometry refinement do not fundamentally solve the problem. The paper also includes perceptual (FID) and depth evaluations, a dynamic-scene baseline, and a per-image appearance model (GS-W) to mitigate lighting inconsistencies.","tokens_in":18094,"tokens_out":6172,"duration_ms":57946,"significance":"If the extrapolation claim is valid, this is a valuable contribution: it addresses a real gap in quantitative evaluation for a capability that matters for AV simulators, and the multi-traversal/multi-agent data curation is nontrivial. The breadth of baselines (3DGS, 3DGM, GaussianPro, VEGS, PGSR, 2DGS, Feature 3DGS, Zip-NeRF, Instant-NGP, GS-W, OmniRe) and the plan to release data and code are strengths. However, the central finding is only as strong as the definition of extrapolation, and the manuscript currently contains statements that undercut that definition; the lack of a quantitative pose-gap measure and the potential appearance confound are load-bearing concerns. If those are resolved, the benchmark would be a useful community resource.","major_comments":[{"comment":"Section 6 (Conclusions, Limitations and Future Work) states: \"we carefully manually selected and ensured that the test trajectory viewpoints are well covered by training trajectories,\" and Section 4.2, Setting 1, states: \"training views fully cover test views with moderate translational changes.\" These statements contradict the abstract's characterization of extrapolation as test views that \"largely deviate from training views.\" The manuscript does not provide a quantitative definition of extrapolation (e.g., nearest-neighbor camera-pose distance, fraction of test views outside the training pose distribution, or angular/translational thresholds). As a result, the reported drops in Table 1 (e.g., 3DGS PSNR 21.36 to 16.37 in Setting 1) cannot be attributed to viewpoint extrapolation; they may reflect the authors' coverage-selection choice. The authors should add a quantitative extrapolation metric and demonstrate that test views lie outside a defined extrapolation regime, or substantially revise the claims and title to describe sparsely covered novel-view synthesis.","section":"Sec. 6 and Sec. 4.2"},{"comment":"Section 5 (Lighting Inconsistency Handling) and Table 4 show that per-image appearance embeddings (GS-W) produce large absolute gains: extrapolated PSNR rises from 16.37 to 20.22 in Setting 1 and from 14.99 to 19.36 in Setting 3, bringing the extrapolated numbers close to the vanilla 3DGS interpolated values. This indicates that appearance variation across traversals is a substantial confound for the extrapolation comparison. The manual curation \"to the eye\" is not a quantitative control. The authors should quantify appearance shift (e.g., mean color statistics or a learned appearance-distance measure) between training and test traversals, and ideally provide an appearance-controlled comparison (e.g., GS-W with and without extrapolated poses) to isolate the geometric extrapolation effect from appearance differences.","section":"Sec. 5 and Table 4"},{"comment":"Section 4.1 states that Grounded-SAM-2 is used to mask potentially movable objects in both training and evaluation, so the reported metrics are computed on static background only. This is disclosed, but the paper does not quantify the masked fraction or discuss how the masking affects the comparison across methods. The authors should report the average masked area and state explicitly that conclusions about \"overfitting to training views\" apply to static scene content; the current wording in the abstract could be read as applying to full scenes.","section":"Sec. 4.1"}],"minor_comments":[{"comment":"In the second paragraph of Related Works, the sentence \"a lack of real data for quantitative evaluation, which confines them to quantitative analysis\" should read \"qualitative analysis\".","section":"Sec. 2"},{"comment":"The header \"A VERAGE\" should be \"AVERAGE\".","section":"Table 1"},{"comment":"The text states \"PSNR dropping by 22.75%,\" but the numbers in Table 1 show an average drop of 25.6% and a 3DGS drop of 24.2%; please reconcile the stated value.","section":"Sec. 4.2, Setting 2"},{"comment":"The caption misspells \"inpainting ability\" as \"impainting ability.\"","section":"Table 2 caption"},{"comment":"The percentage labels (77%, 79%, etc.) are not explained; please clarify which metrics they correspond to.","section":"Figure 1"},{"comment":"For direct comparability with Table 1, consider reporting the interpolation-to-extrapolation drop percentages for GS-W in each setting.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The dataset and evaluation effort are substantial, and I think the paper could become a useful benchmark after revision. My main concern is the mismatch between the 'extrapolation' framing and the authors' own admission that test views are 'well covered' by training views. I would not recommend acceptance until the extrapolation regime is defined quantitatively and the appearance confound is addressed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is a genuinely useful benchmark contribution, but its headline claim—that current NVS methods fail at extrapolation—rests on a definition of extrapolation that is softer than the abstract suggests. The authors assemble the first unified benchmark for extrapolated urban view synthesis by combining multi-traversal, multi-agent, and multi-camera data from nuPlan, Argoverse 2, and MARS, with three evaluation settings (translation, rotation, translation+rotation) and a broad set of 3DGS and NeRF baselines. That is new and worth having.\n\nWhat the paper does well: it processes 90k frames with COLMAP, evaluates a wide range of methods, and reports consistent performance drops from interpolated to \"extrapolated\" test sets. The extra metrics (FID, depth, feature cosine similarity) and the GS-W lighting-robustness experiment make this more thorough than most benchmark papers. The authors also state their limitations openly, which is honest and helps the reader calibrate.\n\nThe soft spot is real and central: the extrapolation is not what it appears. Section 6 says test viewpoints were \"carefully manually selected and ensured... well covered by training trajectories,\" and Section 4.2 says Setting 1's training views \"fully cover\" test views. So the benchmark largely measures non-uniform or sparse interpolation, not large-view extrapolation. The magnitude of the reported drop is partly a function of the coverage choice—a dial the authors control. There is no quantitative measure of pose gap or coverage, so \"extrapolation\" is not a well-defined regime. The appearance difference across traversals is a genuine confound, though the authors attempt to mitigate it and include a GS-W experiment. Also, no error bars are reported, and the data is not yet public.\n\nThat said, the core finding is not a mirage: even in Setting 3, where view changes are genuinely larger, all methods drop substantially. The benchmark would be much stronger with quantified pose gaps, per-scene variances, and a released dataset. The concept is solid enough to be useful to the NVS and AV simulation community.\n\nRecommendation: send it to peer review. It deserves a serious referee; the issues are addressable with more explicit definitions of extrapolation, coverage statistics, and a public release.","headline":"Useful benchmark for extrapolated urban NVS, but the 'extrapolation' is weaker than claimed because test views are manually selected to be well covered by training trajectories.","tokens_in":18574,"tokens_out":1862,"would_cite":true,"duration_ms":19024,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current urban view-synthesis models overfit their training views, and a new extrapolated-view benchmark exposes the failure.","keywords":["extrapolated view synthesis","novel view synthesis","3D Gaussian Splatting","autonomous driving simulation","urban scenes","radiance fields","multi-traversal data","benchmark"],"falsifier":"Render the same extrapolated test poses while holding appearance constant, for example by taking multiple passes over the same route under matched lighting or by applying a strong per-image appearance model, and compare the metric drop. If the interpolation-to-extrapolation gap shrinks to near zero under matched appearance, the benchmark would be measuring appearance shift rather than pose extrapolation; if the gap persists, the overfitting-to-training-views claim is supported.","tokens_in":17676,"feed_emoji":"🚗","tokens_out":8065,"duration_ms":75458,"temperature":0.7,"pith_summary":"This paper introduces the first benchmark built to measure how well novel-view-synthesis models render urban driving scenes from viewpoints that lie outside the training set. The benchmark assembles 90,810 frames from public autonomous-driving recordings that revisit the same places from different lanes, different vehicles, and different cameras, then splits them into interpolated and extrapolated test views across three settings: translation only, rotation only, and translation plus rotation. Across every tested method, rendering quality falls sharply when the test viewpoint extrapolates, while interpolation metrics stay close to training values. The paper reads this as evidence that current scene-fitting radiance-field methods overfit their training views. The benchmark matters because autonomous-vehicle simulators need exactly this ability to generate new viewpoints, not just nearby ones.","feed_headline":"Benchmark shows urban view-synthesis models overfit training views","feed_subtitle":"Across translation, rotation, and combined shifts, PSNR drops up to 30% and perceptual error can more than double.","key_machinery":"The load-bearing mechanism is the benchmark's train/test construction. Instead of sampling test cameras near training cameras, it draws them from separate traversals, agents, or cameras that visit the same scene from shifted poses, so the evaluation measures extrapolation rather than interpolation. Three settings isolate the type of pose change: pure translation, pure rotation, and combined translation plus rotation. Sparse structure-from-motion points initialize all Gaussian methods, movable objects are masked out of training and evaluation, and a suite of image-quality, perceptual, feature-similarity, and depth metrics records the drop. A per-image appearance model is included as a control, showing that part of the extrapolation gap can be absorbed by lighting compensation but a large residual remains.","core_discovery":"The central discovery is that contemporary radiance-field methods, especially 3D Gaussian Splatting, do not generalize to extrapolated urban viewpoints. On the benchmark, vanilla 3D Gaussian Splatting loses 23.4% PSNR in the translation-only setting, 24.2% in the rotation-only setting, and 29.4% when both translation and rotation change; average PSNR drops across all baselines reach 24.6%, 25.6%, and 30.6% respectively. Perceptual similarity degrades even more sharply, with learned perceptual error rising by over 170% for one baseline in the hardest setting. Methods that add diffusion priors, planar geometry, or depth regularization improve isolated cases but do not close the gap, and a per-image appearance model raises absolute quality while leaving a large interpolation-to-extrapolation drop. The paper concludes that the bottleneck is structural: fitted scenes lack the geometry and appearance coverage needed for views that deviate substantially from training trajectories.","pith_inferences":["If missing geometry rather than missing appearance drives the failure, then scaling the number of distinct viewpoints per scene may matter more than adding generative priors, and the plateau seen when adding traversals hints at an upper bound from current per-scene fitting.","A natural next experiment is to train a feed-forward generalizable model on many EUVS scenes and test zero-shot extrapolation; the benchmark's static-scene protocol would make that comparison straightforward.","The benchmark's residual gap after appearance compensation suggests future versions should record lighting and weather metadata per traversal, letting the community separate pose extrapolation from appearance shift instead of controlling only by manual curation.","Extending the evaluation to foreground objects in the translation and combined settings, using multi-agent views of the same dynamic objects, would test whether per-object appearance models can rescue extrapolated dynamic scenes."],"forward_implications":["Researchers can now compare extrapolated view-synthesis methods on a fixed protocol with ground-truth images instead of relying on qualitative demos.","Any method that claims to generalize for driving simulation should be expected to report extrapolated urban view-synthesis metrics, because interpolation scores can remain high while true generalization fails.","Adding diffusion priors or geometric regularization is not sufficient by itself; the benchmark's results suggest the community needs representations trained or regularized explicitly for unseen viewpoints.","More traversal data helps: increasing the number of traversals improves extrapolated metrics up to a plateau, so data diversity is a measurable lever.","Dynamic-scene reconstruction methods show the same failure, so the overfitting pattern is not limited to static background models."],"supporting_citations":[{"why":"It is the primary scene-fitting baseline whose interpolation-to-extrapolation drop anchors the overfitting claim.","marker":"[26]"},{"why":"It is one of three public driving datasets contributing multi-traversal and multi-camera footage.","marker":"[4]"},{"why":"It contributes multi-traversal and multi-agent data used in the translation-only and combined settings.","marker":"[47]"},{"why":"It contributes multi-agent and multi-traversal data for the combined translation-plus-rotation setting.","marker":"[31]"},{"why":"It is the diffusion-prior baseline whose rotation-only gains show priors help only in specific settings.","marker":"[24]"},{"why":"It is the geometry-refined baseline used to test whether densification closes the extrapolation gap.","marker":"[11]"},{"why":"It is the per-image appearance model that quantifies the lighting-confounding portion of the gap.","marker":"[61]"},{"why":"It is the dynamic-scene baseline showing that the extrapolation failure is not limited to static methods.","marker":"[10]"},{"why":"It supplies the sparse structure-from-motion initialization used by all Gaussian-splatting baselines.","marker":"[40]"}],"fun_headline_variants":["First extrapolated urban view benchmark reveals overfitting","Urban view synthesis fails on extrapolated viewpoints: new benchmark","First benchmark shows radiance fields can't handle unseen urban views","Extrapolated views drop PSNR up to 30%: new benchmark exposes limits","Urban view synthesis benchmark: extrapolation is the hard part"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the test views are true pose extrapolations and that the measured quality drop comes from novel viewpoint geometry rather than from lighting, weather, or time-of-day differences between the recording traversals.","fun_headline_variants_meta":{"raw":{"variants":["First extrapolated urban view benchmark reveals overfitting","Urban view synthesis fails on extrapolated viewpoints: new benchmark","First benchmark shows radiance fields can't handle unseen urban views","Extrapolated views drop PSNR up to 30%: new benchmark exposes limits","Urban view synthesis benchmark: extrapolation is the hard part"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000694,"raw_usage":{"total_tokens":3156,"prompt_tokens":979,"completion_tokens":2177,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":2090}},"tokens_in":595,"tokens_out":2177,"duration_ms":14582,"temperature":1.0,"reasoning_tokens":2090,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:49:09.952620+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Render the same extrapolated test poses while holding appearance constant, for example by taking multiple passes over the same route under matched lighting or by applying a strong per-image appearance model, and compare the metric drop. If the interpolation-to-extrapolation gap shrinks to near zero under matched appearance, the benchmark would be measuring appearance shift rather than pose extrapolation; if the gap persists, the overfitting-to-training-views claim is supported.","supporting_citations":[{"cited_title":"3d gaussian splatting for real-time radiance field rendering","cited_arxiv_id":null,"evidence_quote":"It is the primary scene-fitting baseline whose interpolation-to-extrapolation drop anchors the overfitting claim."},{"cited_title":"Multiagent multitraversal multimodal self-driving: Open mars dataset","cited_arxiv_id":null,"evidence_quote":"It contributes multi-agent and multi-traversal data for the combined translation-plus-rotation setting."},{"cited_title":"Gaussian- pro: 3d gaussian splatting with progressive propagation","cited_arxiv_id":null,"evidence_quote":"It is the geometry-refined baseline used to test whether densification closes the extrapolation gap."},{"cited_title":"Structure-from-motion revisited","cited_arxiv_id":null,"evidence_quote":"It supplies the sparse structure-from-motion initialization used by all Gaussian-splatting baselines."}],"review_version":1}