{"id":"165b3214-6798-46b6-a403-b8cbf26620c7","arxiv_id":"2411.10546","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"The Oxford Spires dataset provides outdoor LiDAR-camera-inertial data with TLS ground truth and benchmarks showing radiance field methods overfit training poses and underperform MVS geometry.","lead":"This paper releases a large outdoor dataset of camera, LiDAR and motion data from six Oxford landmarks, with high-precision laser-scanned reference models for testing localisation, 3D reconstruction and radiance fields. Its evaluation finds that modern radiance field methods overfit to their training camera paths and fall behind classical reconstruction on geometry.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The out-of-sequence NVS evaluation conflates pose generalisation with appearance shift and unseen scene content; the central radiance-field claim is not yet established.","rationale":"The reader's weakest assumption about the 1-2 cm ICP ground-truth trajectories is legitimate: Sec. 5.1.7 validates the trajectories by reprojecting the same LiDAR scans into the same TLS map, which is a self-consistency check, and VILENS supplies both the motion-corrected scans used to build the ground truth and one of the evaluated SLAM systems. However, that issue affects the localisation leaderboard, not the paper's primary radiance-field message. The more load-bearing gap is in the NVS protocol itself. The out-of-sequence test set is a separate, manually chosen trajectory; no statistics show that the test images depict surfaces already seen in training. The paper even acknowledges the per-sequence appearance problem in Fig. 12, so the observed metric drops are ambiguous. Because the abstract claims that radiance fields 'do not generalise well to out-of-sequence poses', this ambiguity directly undermines the central contribution. The 3D reconstruction comparison with MVS provides some independent support for geometric weakness, but it extracts Nerfacto depth along training rays, which is a different and arguably unfavourable protocol; I did not make that the primary concern because the NVS issue is more central and more easily settled. The verdict remains CONDITIONAL: the dataset is valuable and the qualitative observations are suggestive, but the headline generalisation claim needs a coverage-controlled analysis and error bars before the benchmark can be used as strong evidence for it.","tokens_in":22029,"tokens_out":6567,"duration_ms":72497,"concrete_test":"Using the released TLS map and ground-truth trajectories, compute per-pixel coverage for each out-of-sequence test image: raycast the TLS depth map from the test pose and check whether any training camera frustum observes the same surface within a few centimetres. Then recompute Table 5 separately for covered and uncovered pixels. If the in-sequence vs out-of-sequence gap largely disappears on covered pixels, the pose-overfitting claim is confounded by content novelty; if it persists on covered pixels, the claim survives. As a second control, hold out a contiguous block of poses from the same training trajectory as the test set instead of a whole second trajectory, so that appearance and scene content are matched while pose extrapolation is still required; compare the resulting metric drop with Table 5.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's headline contribution is the radiance-field finding: state-of-the-art NeRF and 3D Gaussian Splatting methods overfit to training poses/images and underperform in 3D reconstruction compared to MVS. The load-bearing evidence for the generalisation half of this claim is the NVS benchmark of Sec. 6.3. The out-of-sequence test images are taken from a separate trajectory registered with COLMAP and 'manually selected' to be far from the training trajectory or to have very different viewing directions (Sec. 6.3.2). This protocol does not separate 'images from a pose the model has not seen' from 'images of surfaces the model has never observed'. For Radcliffe Observatory Quarter and Blenheim Palace the test trajectories are in different directions, so a test image can easily show facades, sky regions, or shadowed geometry that no training ray ever intersected; any method without prior knowledge of that content would fail there. The paper reports no visibility or coverage statistics of test pixels against the training frustums, and no repeated splits or error bars, so the drop in Table 5 cannot be attributed to pose overfitting. The appearance confound is acknowledged in Fig. 12, where auto-exposure changes per-frame brightness and affects PSNR, but LPIPS and SSIM are also sensitive to lighting and appearance shifts; Nerfacto's test-time averaged appearance embedding is a known source of colour mismatch. Thus the central quantitative claim is not yet established, even though the qualitative artefacts in Fig. 11 and the 3D reconstruction results in Sec. 6.2 independently suggest that radiance-field geometry is imperfect.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the Oxford Spires Dataset, a large-scale outdoor multi-modal dataset captured at six Oxford heritage sites with three synchronised fisheye RGB cameras, a 64-beam LiDAR, and an IMU, together with millimetre-accurate ground-truth 3D models from a Terrestrial LiDAR Scanner. It describes sensor calibration, data formats, processed SLAM/SfM outputs, and three benchmarks: LiDAR-visual localisation, 3D reconstruction, and novel-view synthesis. The paper evaluates several LiDAR-inertial SLAM systems, COLMAP, OpenMVS, and two radiance-field families (Nerfacto and Splatfacto), and reports two headline findings: radiance-field methods overfit to training poses/images and do not generalise to out-of-sequence viewpoints, and radiance-field reconstructions underperform MVS when given the same visual inputs.","tokens_in":22318,"tokens_out":6074,"duration_ms":80517,"significance":"If the dataset is released as described, it is a valuable community resource: it provides outdoor colour multi-camera imagery, LiDAR and inertial data with TLS-based ground truth for both trajectories and dense 3D models, and it ships parsing and evaluation software. The three-camera configuration and the out-of-sequence NVS benchmark are useful additions beyond existing outdoor SLAM datasets. The paper also gives concrete multi-method baselines across three tasks, which is helpful for future benchmarking. However, the two headline claims about radiance-field methods are currently supported by experiments that do not fully isolate pose generalisation from appearance and scene-content confounds, and the localisation ground truth is generated with the same system that is then evaluated. These issues are fixable with additional analyses, but they affect the paper's central conclusions.","major_comments":[{"comment":"The localisation ground truth is computed by ICP-registering LiDAR point clouds that were motion-undistorted with VILENS-SLAM (Sec. 5.1.4) to the TLS map (Sec. 5.1.7). VILENS-SLAM and HBA (which consumes VILENS-SLAM's output, Sec. 6.1) are then evaluated against this ground truth in Table 3. This gives VILENS-SLAM an advantage because its own motion estimate is already embedded in the input point clouds used to create the ground truth, while other LiDAR SLAM systems must undistort with their own odometry. The stated validation, projecting the LiDAR scans into the map and comparing to the TLS map, is a self-consistency check against the same map used for the registrations and does not independently establish the claimed 1-2 cm accuracy. Please provide an independent accuracy assessment (e.g., surveyed checkpoints, manual feature correspondences, or cross-validation with COLMAP poses) and report whether the Table 3 ranking is robust to an alternative ground-truth generation protocol.","section":"Sec. 5.1.7, Sec. 5.1.4, Table 3"},{"comment":"The out-of-sequence NVS benchmark selects test images from a separate trajectory that are \"far away apart or have very different view directions\" (Sec. 6.3.2). This protocol confounds pose extrapolation with appearance shifts caused by auto-exposure and different capture times, which the paper itself acknowledges in Fig. 12, and with scene content (facades, sky, shadowed surfaces) that no training ray observed. PSNR, SSIM, and LPIPS are all sensitive to these confounds, and the paper gives no per-pixel visibility or coverage statistics of test images against training frustums. The manual test selection is also not described with enough precision for exact reproduction. Consequently, the drop in Table 5 does not, by itself, establish the claim that radiance-field methods overfit to training poses/images. Please report coverage statistics, repeat the split multiple times with error bars, add an appearance-normalised control (e.g., exposure-corrected images or same-surface/different-pose held-out frames), and clarify how the conclusion would be affected if the test set were restricted to pixels whose underlying surfaces are visible in the training set. The evidence is also based on only two method families (Nerfacto and Splatfacto), which is a narrow basis for the abstract's general statement about state-of-the-art radiance-field methods.","section":"Sec. 6.3.2, Table 5, Fig. 12"},{"comment":"The reconstruction comparison for Nerfacto uses point clouds generated by calculating expected depth and colour along training rays (Sec. 6.2). This extraction procedure may not be the most representative way to obtain a geometric reconstruction from a radiance field, and it can penalise Nerfacto relative to OpenMVS in ways that are unrelated to the underlying scene representation. Please justify this choice more explicitly, or additionally report results with an alternative extraction (e.g., density-thresholded marching cubes or a mesh extracted from the learned field), so that the conclusion \"radiance-field methods underperform MVS\" is not an artefact of the chosen point-cloud extraction.","section":"Sec. 6.2, Table 4"}],"minor_comments":[{"comment":"The symbol ✗ is used for both SC-LIO-SAM failures and a Fast-LIVO2 missing entry (e.g., Christ Church College 03), but no legend explains the difference between \"failed\", \"not run\", and \"did not converge\"; please add one.","section":"Table 3"},{"comment":"The sentence \"LPIPS increased by 9.6% and SSIM by 2% on average\" is ambiguous because an increase in LPIPS is a degradation; please state the sign of the change relative to the metric's semantics.","section":"Sec. 6.3.3"},{"comment":"The reference list contains two entries for the same paper (Matsuki et al. 2024a and 2024b, both titled \"Gaussian Splatting SLAM\"); these should be merged into a single reference.","section":"References"},{"comment":"The phrase \"far away apart\" should be \"far apart\".","section":"Sec. 6.3.2"},{"comment":"The figure caption does not specify which panel is the evaluation image and which is the rendered image, nor how the per-frame appearance-averaging was performed; please make the comparison easier to interpret.","section":"Fig. 12"}],"recommendation":"major_revision","confidential_remarks":"The dataset itself is a genuinely useful resource, and the engineering effort behind calibration, TLS processing, and released evaluation software is evident. The main risk is that the paper's most prominent claim, that radiance-field methods overfit to training poses and underperform against MVS, is currently supported by experiments that do not isolate pose generalisation from appearance and scene-content confounds. I would advise the editor to require the additional NVS controls and an independent validation of the localisation ground truth before acceptance. The localisation circularity concern, in particular, affects the interpretation of Table 3 and should be addressed head-on rather than by a short paragraph."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The Oxford Spires dataset is a real contribution. Six sites, three synchronized global-shutter colour cameras, a 64-beam LiDAR, IMU, and millimetre TLS ground truth, with processed outputs and evaluation scripts released. The three-camera setup and the out-of-sequence novel-view benchmark are genuinely new relative to Newer College, BotanicGarden, and MCD. The 3D reconstruction comparison (Tab. 4) is also useful: even with the same visual inputs, OpenMVS clearly beats Nerfacto in F-score on most sequences, which supports the claim that radiance fields are not yet competitive for geometry. That result does not depend on the questionable NVS protocol.\n\nThe soft spots are real, and they are concentrated in the two headline claims. First, the localisation ground truth (Sec. 5.1.7) is built by ICP-registering LiDAR point clouds that were motion-undistorted with VILENS-SLAM, and VILENS-SLAM is then evaluated against that GT. The validation described is a reprojection into the same TLS map, which is a self-consistency check, not an independent error measure. This does not invalidate the benchmark, but it means VILENS's small ATE numbers should be read with a grain of salt, and the rankings of the other methods (HBA, COLMAP) could shift if the GT were built differently.\n\nSecond, and more important, the out-of-sequence NVS evaluation (Sec. 6.3) conflates pose overfitting with appearance shift and unseen scene content. The test images are manually selected from a distant trajectory; they are likely to contain facades, sky, or shadowed regions that no training ray intersected, and auto-exposure changes per-frame brightness. The paper acknowledges the appearance issue in Fig. 12 but then says LPIPS and SSIM are more invariant - they are not fully invariant. With only two method families, no repeated splits, and no error bars or visibility statistics, the drop in Tab. 5 cannot be attributed to pose generalisation alone. The qualitative artifacts in Fig. 11 (elongated Gaussians, black ground) do suggest genuine overfitting, so I think the claim is plausible, but the quantitative evidence as presented does not establish it.\n\nThe paper is worth serious refereeing and, after revision, will likely become a standard benchmark reference. The authors should be asked to (1) decouple the localisation GT from VILENS, or at least quantify the bias, (2) report NVS results with multiple train/test splits and error bars, and (3) compute coverage/visibility maps for the out-of-sequence test images. A commit hash for the released code would also help. This is a paper for the SLAM and radiance-field communities; the dataset itself is the primary value, not the specific verdicts in the benchmarks.","headline":"A valuable new dataset with a useful 3D reconstruction comparison, but the headline NVS claim is confounded by appearance and coverage effects, and the localisation GT has a circularity worth fixing.","tokens_in":22946,"tokens_out":3399,"would_cite":true,"duration_ms":33473,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"On a new LiDAR-verified outdoor benchmark, state-of-the-art radiance field methods overfit their training camera poses and underperform multi-view stereo in 3D reconstruction.","keywords":["dataset","LiDAR-visual SLAM","terrestrial LiDAR scanner ground truth","localisation benchmark","3D reconstruction benchmark","novel-view synthesis","neural radiance fields","3D Gaussian splatting"],"falsifier":"An independent survey of one site, such as total-station or GPS ground-control markers placed in the scene, that reveals trajectory errors significantly above 2 cm would change the localisation rankings; conversely, a depth-regularised radiance field that closes the out-of-sequence PSNR and reconstruction-F-score gap would directly weaken the paper's claim that current radiance field representations cannot generalise.","tokens_in":21828,"feed_emoji":"🏛️","tokens_out":8937,"duration_ms":75405,"temperature":0.7,"pith_summary":"This paper introduces the Oxford Spires Dataset, a large outdoor multi-sensor collection from six historic sites, built around three synchronised colour cameras, a 64-beam LiDAR, and an IMU, with millimetre-accurate ground-truth 3D models obtained from a terrestrial LiDAR scanner (TLS). The authors use it to construct three benchmarks—localisation, 3D reconstruction, and novel-view synthesis—and report how current SLAM, SfM, MVS, and radiance field methods perform. Their main finding is that state-of-the-art radiance field methods (NeRF-based and 3D Gaussian Splatting-based) overfit the poses and images they train on: they render well from near-training viewpoints but degrade sharply when rendered from distant trajectories, and they reconstruct less accurate 3D geometry than classical multi-view stereo using the same photographs. If this holds, radiance field methods will need explicit geometry or depth regularisation before they can be used as the map representation in outdoor SLAM, and this dataset provides the measurement tool to track that progress.","feed_headline":"New dataset shows radiance fields fail on distant camera poses","feed_subtitle":"Millimetre-accurate ground truth shows radiance fields need explicit geometry to generalise beyond their training poses.","key_machinery":"The object that carries the argument is the dataset and its ground-truth pipeline: a handheld unit with three synchronised fisheye colour cameras, a wide field-of-view 64-beam LiDAR, and an IMU, paired with survey-grade TLS scans used as reference 3D models. Ground-truth trajectories are produced by ICP-registering motion-undistorted LiDAR scans to the TLS map, following the methodology of the Newer College dataset. The decisive evaluation mechanism is the novel-view synthesis protocol, which distinguishes in-sequence test images (held-out frames from the training trajectory) from out-of-sequence test images (frames from a separate trajectory with viewpoints far from training); this separation is what exposes the overfitting that standard benchmarks miss. Reconstruction quality is scored by F-score at 5 cm and 10 cm thresholds against the TLS map.","core_discovery":"The central claim is that, given the same visual inputs and ground truth from a millimetre-accurate TLS map, state-of-the-art radiance field methods do not generalise beyond their training trajectory: they overfit to the training poses/images, and their 3D geometry is less accurate than that of multi-view stereo systems. The paper demonstrates this with an out-of-sequence evaluation in which test images come from trajectories far from the training trajectory, causing visible artefacts such as elongated Gaussians along training view directions and incorrect view-dependent colours. A further claim is that the three-camera wide field-of-view configuration provides view constraints that make vision-only SfM (COLMAP) competitive with LiDAR SLAM in several sequences.","pith_inferences":["A testable extension the paper does not run: train radiance fields with the provided LiDAR-aligned depth images and measure whether the in-sequence/out-of-sequence gap shrinks; if it does, depth supervision is the direct remedy.","The auto-exposure limitation the authors flag suggests that appearance-embedding techniques (per-frame appearance codes) will be needed for colour-consistent merged reconstructions, but the paper does not evaluate them.","The out-of-sequence protocol could be applied to the learning-based radiance SLAM systems the paper lists as future work (for example Gaussian Splatting SLAM); those systems may show even stronger overfitting because they optimise geometry online."],"forward_implications":["Radiance field methods intended for outdoor SLAM must incorporate explicit geometry or depth supervision rather than relying on photometric training alone.","Novel-view-synthesis benchmarks should include test viewpoints distant from the training trajectory, not only held-out frames sampled from it.","Classical multi-view stereo pipelines remain a stronger baseline than current NeRF and 3D Gaussian Splatting methods for metric 3D reconstruction in large outdoor scenes.","The dataset offers the first outdoor benchmark combining three colour cameras, LiDAR, IMU, and millimetre-accurate TLS ground truth in a single evaluation suite for SLAM, SfM, MVS, and radiance field methods."],"supporting_citations":[{"why":"supplies the ICP-registration-to-TLS method used to generate dense ground-truth trajectories.","marker":"(Ramezani et al. 2020b)"},{"why":"establishes the millimetre-accurate benchmark approach and the reference-target validation the authors compare against.","marker":"(Zhang et al. 2022)"},{"why":"provides the COLMAP SfM pipeline that supplies poses for MVS and radiance fields and is itself the strongest vision-only localisation baseline.","marker":"(Schönberger and Frahm 2016)"},{"why":"defines Nerfacto, the NeRF baseline evaluated, and the Nerfstudio transforms.json format used for processed data.","marker":"(Tancik et al. 2023)"},{"why":"defines Splatfacto, the 3D Gaussian Splatting baseline that shows the largest out-of-sequence degradation.","marker":"(Ye et al. 2024)"},{"why":"introduces the 3D Gaussian Splatting representation whose elongated-Gaussian artefacts are discussed.","marker":"(Kerbl et al. 2023)"},{"why":"provides VILENS, used both as a LiDAR SLAM baseline and for motion-undistorting LiDAR scans before ground truth ICP registration.","marker":"(Wisth et al. 2023)"},{"why":"the NeRF formulation whose view-dependent colour modelling the paper identifies as a source of overfitting.","marker":"(Mildenhall et al. 2021)"}],"fun_headline_variants":["Radiance fields overfit training poses, fail on new views","Oxford dataset shows radiance fields miss out-of-sequence poses","MVS beats NeRF and 3DGS on geometry, new benchmark finds","Radiance fields need explicit geometry to generalize beyond training","New Oxford dataset: radiance fields overfit, MVS wins in 3D"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's localisation rankings assume the ICP-registered ground-truth trajectories are accurate to about 1–2 cm, which is validated by reprojecting the LiDAR scans into the TLS map—a self-consistency check rather than an independent survey.","fun_headline_variants_meta":{"raw":{"variants":["Radiance fields overfit training poses, fail on new views","Oxford dataset shows radiance fields miss out-of-sequence poses","MVS beats NeRF and 3DGS on geometry, new benchmark finds","Radiance fields need explicit geometry to generalize beyond training","New Oxford dataset: radiance fields overfit, MVS wins in 3D"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000389,"raw_usage":{"total_tokens":2087,"prompt_tokens":1018,"completion_tokens":1069,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":977}},"tokens_in":634,"tokens_out":1069,"duration_ms":10394,"temperature":1.0,"reasoning_tokens":977,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:34:29.166438+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An independent survey of one site, such as total-station or GPS ground-control markers placed in the scene, that reveals trajectory errors significantly above 2 cm would change the localisation rankings; conversely, a depth-regularised radiance field that closes the out-of-sequence PSNR and reconstruction-F-score gap would directly weaken the paper's claim that current radiance field representations cannot generalise.","supporting_citations":[{"cited_title":"In: IEEE Int","cited_arxiv_id":null,"evidence_quote":"provides the COLMAP SfM pipeline that supplies poses for MVS and radiance fields and is itself the strongest vision-only localisation baseline."},{"cited_title":"In: SIGGRAPH","cited_arxiv_id":null,"evidence_quote":"defines Nerfacto, the NeRF baseline evaluated, and the Nerfstudio transforms.json format used for processed data."},{"cited_title":"ACM Transactions on Graphics (TOG) 42(4): 1--14","cited_arxiv_id":null,"evidence_quote":"introduces the 3D Gaussian Splatting representation whose elongated-Gaussian artefacts are discussed."},{"cited_title":"IEEE Trans","cited_arxiv_id":null,"evidence_quote":"provides VILENS, used both as a LiDAR SLAM baseline and for motion-undistorting LiDAR scans before ground truth ICP registration."},{"cited_title":"Communications of the ACM 65(1): 99--106","cited_arxiv_id":null,"evidence_quote":"the NeRF formulation whose view-dependent colour modelling the paper identifies as a source of overfitting."}],"review_version":1}