{"id":"b8ae9129-185e-4937-af39-7871feddcac9","arxiv_id":"2412.15550","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"EGSRAL synthesizes labeled novel driving views by enhancing 3D Gaussian Splatting with deformation and opacity modules, a grouping strategy, and a coordinate adaptor, improving rendering and 2D/3D detection.","lead":"EGSRAL is a 3D Gaussian Splatting system for driving footage that renders new camera views and, at the same time, attaches matching 2D and 3D labels to them. If the results hold, it could let autonomous-driving teams produce extra labeled training images from data they already have.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The auto-labeling claim rests on an unvalidated adaptor: Section 3.3's L3D/Lproj pseudo-ground truth assumes SfM and real intrinsics are near-identical, but no direct held-out pose or label-accuracy test is reported.","rationale":"The central claim has two parts: state-of-the-art novel-view rendering and automatic labeling for novel views that improves downstream detection. The rendering part is supported by ablations and comparisons, though the abstract's 'without any extra annotation' is overstated because the nuScenes-D result in Table 2 uses LiDAR initialization. The more load-bearing part is the adaptor, because it produces the labels that are claimed to improve 2D/3D detection. The reader's weakest assumption correctly identifies the adaptor's projection-consistency training as the fragile link: Eq. 11 assumes that real and SfM intrinsics are similar enough that 3D camera-coordinate positions are identical, and the paper offers no direct evidence that this assumption holds for RPT-sampled novel poses. Detection improvements could come from extra image diversity or implicit regularization even if the generated boxes are partly misaligned. The paper does include a direct adaptor-versus-matrix comparison (Table 15), but that evaluates transformation of existing frame poses in OWCS, not novel-view pose prediction. A held-out pose prediction test with projected box IoU would settle whether the adaptor generalizes. Since the reader already issued a CONDITIONAL verdict, this concern reinforces the conditions rather than moving the verdict to a new category.","tokens_in":16692,"tokens_out":15457,"duration_ms":149752,"concrete_test":"Hold out every 5th frame in one nuScenes scene from adaptor training. For each held-out frame, input its OWCS pose to the trained adaptor and compare the predicted EWCS pose with the SfM EWCS pose that was not used in training. Then project the dataset 3D boxes into that held-out image using both poses and report median 2D IoU and 3D center distance. If the median 3D center error exceeds roughly 0.5 m or the 2D IoU is substantially below the Table 15 adaptor level (72.5% AP, 0.605 m AD), the Eq. 11 intrinsic-similarity assumption does not hold for novel poses and the auto-labeling claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"EGSRAL's central novelty is the adaptor that converts existing 3D boxes into labels for novel views. Its training uses three losses (Eqs. 6, 10, 11). Lp fits known OWCS/EWCS pose pairs, but the two novelty losses Lproj and L3D rely on a pseudo-ground truth for RPT-sampled novel poses: subsequent camera poses are projected into the novel camera, and Eq. 11 explicitly assumes 'identical position information in the camera coordinate system' because real and SfM intrinsics are similar. If the SfM reconstruction's focal length or scale deviates from the dataset calibration, the pseudo-ground-truth 3D coordinates are wrong, so the adaptor learns a biased OWCS-to-EWCS map for novel poses. The generated 3D boxes would then be misaligned in the rendered novel views, breaking the downstream detection claim. The paper reports detection AP gains and an OWCS box-propagation comparison (Table 15), but it never directly measures adaptor pose error or box IoU on held-out novel poses; AP gains alone cannot isolate label correctness from image diversity or regularization. The load-bearing assumption in Section 3.3 is therefore untested as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes EGSRAL, an enhanced 3D Gaussian Splatting renderer for large-scale driving scenes that combines a deformation enhancement module, an opacity enhancement module, and a grouping strategy, together with an adaptor network that maps camera poses from the original world coordinate system (OWCS) to the SfM coordinate system (EWCS) so that existing 3D annotations can be transformed to novel views. The adaptor is trained with three losses: pose matching on existing pairs, projection consistency, and a 3D coordinate constraint for randomly sampled nearby poses. Experiments on KITTI and nuScenes report improved rendering metrics over prior 3D GS methods, and detection experiments show that augmenting training data with synthesized labeled views improves Co-DETR and MonoLSS performance.","tokens_in":16941,"tokens_out":7084,"duration_ms":56343,"significance":"If the rendering claims hold, the proposed modules and grouping strategy are simple and effective improvements for dynamic driving scenes, and single-model training without per-object models is a practical advantage. The auto-labeling adaptor is the most novel component and has the potential to turn 3D GS renderers into data generators for perception tasks. The paper ships a code link, reports per-scene results, and includes ablations for module contributions and group counts. However, the central auto-labeling claim is validated only indirectly, as detailed below; the significance of the contribution depends on whether the adaptor truly generalizes to novel poses.","major_comments":[{"comment":"The L3D loss for novel poses assumes that the real and SfM-estimated camera intrinsics are similar enough to treat the 3D positions of the following camera poses in the camera coordinate system as identical (text before Eq. (11)). This assumption is load-bearing because it defines the pseudo-ground truth for the novel-pose constraint, yet no experiment directly measures adaptor pose error or label accuracy on the RPT-sampled novel poses that are used for auto-labeling. Table 15 evaluates the adaptor against Umeyama-based matrix estimation on a held-out subset of existing poses transformed to OWCS, which does not test generalization to nearby novel poses; the detection gains in Tables 4, 5, 16, and 17 cannot isolate label correctness from image diversity or model regularization. Please add a direct evaluation of novel-pose prediction accuracy (e.g., comparing RPT-sampled poses to SfM-reconstructed poses for a held-out frame, or measuring 3D box center error/IoU on manually labeled novel views) and clarify whether the L3D weight w2 is scheduled as the text suggests or fixed at 0.1 throughout.","section":"Section 3.3, Eq. (11); Section 8.3, Table 15"},{"comment":"The abstract and introduction claim the method 'relies solely on training images without extra annotations,' but the nuScenes-D benchmark explicitly initializes 3D Gaussians from LiDAR point clouds ('we utilize LiDAR point clouds to initialize the 3D Gaussians'). This contradiction affects a headline claim and the fairness of the comparison in Table 2, where DrivingGaussian also uses LiDAR. Please either revise the claim to state that the method requires only images and can optionally use LiDAR when available, or repeat the nuScenes-D experiment with SfM initialization and report both configurations.","section":"Section 7.1 (NuScenes-D); Abstract"},{"comment":"The adaptor training protocol is not specified clearly: the text in Section 4.2 can be read as training one adaptor per scene (17 scenes, each with ~230 pose pairs, 1000 epochs each), which would be a significant practical limitation for auto-labeling at scale, whereas a single shared adaptor would make the method more generally applicable. The 34-scene extension in Section 8.3 should state whether additional adaptors are trained for the new scenes and how the per-scene SfM is obtained. Please specify the protocol, the total training cost, and whether the adaptor is applied to scenes outside its training distribution.","section":"Section 4.2; Section 8.3"}],"minor_comments":[{"comment":"The notation 'd ∈ R^{d×1}' uses d for both the state attribute and its dimensionality; rename the dimension, e.g., D, to avoid confusion.","section":"Section 3.2, Eq. (3)"},{"comment":"The 'Supervision' column labels 3D GS, Mip-splatting, Deformable 3D GS, and EGSRAL as 'RGB', but Section 3.1 states that the method takes SfM point clouds as input; the column should read 'RGB + SfM points' or be annotated to distinguish annotation supervision from input modalities, for consistency with Table 2.","section":"Table 1"},{"comment":"There is a typo: 'f ramen+i' should be 'frame n+i'.","section":"Section 3.3, Eq. (7)"},{"comment":"The column header 'Total amount' is ambiguous because the dataset type (sample/all) and the augmentation multiplier (1×/2×/3×) are conflated; please split these into separate columns or use clearer labels.","section":"Tables 4 and 5"},{"comment":"The statement that 'the SfM method used in 3D GS is monocular' is confusing in the context of the nuScenes-D experiment, which uses synchronized 6-camera input; clarify which SfM configuration is used for each dataset.","section":"Section 8.3"},{"comment":"The limitations paragraph only mentions dependence on SfM point cloud quality; it should also acknowledge the adaptor's reliance on the intrinsics-similarity assumption and the lack of direct label-accuracy evaluation.","section":"Appendix Section 8.4 (Limitations)"},{"comment":"The reference list contains duplicate entries for Zong et al. 2023 (2023a and 2023b with different title capitalization); please unify.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The rendering contributions appear solid and the ablation is internally consistent, but the auto-labeling claim is the most novel and least directly validated part of the paper. The use of LiDAR in the main nuScenes comparison should be prominently disclosed, and the adaptor's generalization to novel poses needs a dedicated quantitative evaluation. If the authors can provide that evidence, the paper could be suitable for a journal; in its current form, the central contribution is under-supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know about EGSRAL: the genuinely new piece is the adaptor that converts dataset annotations (OWCS) into SfM coordinates (EWCS) so you can render novel views with 2D/3D boxes attached. The rendering modules (DEM, OEM, grouping) are pragmatic MLP tweaks on Deformable 3D GS; they give consistent but modest gains across KITTI and nuScenes. The ablation is clean and each module contributes. The detection augmentation experiments show real gains for Co-DETR and MonoLSS, and Table 15 gives a direct comparison of the adaptor against an Umeyama matrix baseline on held-out poses: AP 72.5 vs 21.7, AD 0.605 m vs 1.867 m. That is meaningful validation of the coordinate transformation.\n\nSoft spots: the abstract says the method relies solely on training images, but the nuScenes-D comparison uses LiDAR initialization (Section 7.1). That is an overstatement, though the KITTI and nuScenes-S runs are image-only. More importantly, the adaptor's L3D and Lproj losses assume SfM intrinsics are close enough to the dataset intrinsics to treat camera-coordinate positions as identical. The paper acknowledges this but only tests the adaptor via downstream 3D box accuracy in OWCS, not via direct novel-view box IoU or pose error. That leaves a real but narrow gap. Also, key hyperparameters (group number, adaptor N, loss weights) are tuned on the evaluation sets, and there are no error bars. Missing Street Gaussians as a baseline is a minor omission, since that method also handles dynamic urban scenes.\n\nWho this is for: people working on neural rendering for autonomous driving and on using synthetic data for detection. It won't reshape how you build a renderer, but the auto-labeling idea is worth knowing.\n\nRecommendation: send it to review. It is not a home run, but the adaptor is novel, the evaluation is mostly careful, and the direct adaptor test addresses the main risk. A referee should ask for a sharper test of the intrinsic-similarity assumption and for the LiDAR caveat to be fixed.","headline":"A solid incremental contribution: the adaptor is genuinely new, rendering gains are consistent, but the 'images-only' claim is overstated and the adaptor's key assumption deserves a sharper test.","tokens_in":17533,"tokens_out":3120,"would_cite":true,"duration_ms":26872,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EGSRAL claims a single 3D Gaussian Splatting renderer can synthesize photorealistic novel views of driving scenes and simultaneously generate the 2D/3D detection labels those views need, using only images and existing dataset annotations.","keywords":["3D Gaussian Splatting","novel view synthesis","autonomous driving","auto labeling","data augmentation","object detection","scene reconstruction","coordinate transformation"],"falsifier":"Measure, per nuScenes scene, the pixel reprojection error between the SfM-estimated intrinsics and the dataset's reported intrinsics on known 3D points; if that error exceeds a few pixels, the projection-consistency constraint used to train the adaptor is violated and auto-labeled boxes should drift. A direct experiment: deliberately perturb the SfM intrinsics by a fixed amount, retrain the adaptor, and check whether downstream 3D detection AP falls in proportion to the perturbation.","tokens_in":16448,"feed_emoji":"🚗","tokens_out":7829,"duration_ms":56847,"temperature":0.7,"pith_summary":"EGSRAL aims to prove that a single 3D Gaussian Splatting renderer can synthesize photorealistic novel views of large-scale driving scenes and, at the same time, generate the 2D and 3D detection annotations those views need, using only the input images and the dataset's existing annotations. This matters because prior driving-scene renderers require extra supervision—depth maps, LiDAR, object trajectories, or per-object models—and the frames they synthesize lack labels, so they cannot directly feed downstream detectors. The paper reports the highest rendering quality among the methods it compares against, with PSNR 29.04 on the nuScenes scenes used by DrivingGaussian, and shows that augmenting a detector's training data with its auto-labeled novel views improves both 2D and 3D detection accuracy. The load-bearing novelty is a trainable adaptor that learns the coordinate transformation between the dataset's world frame and the structure-from-motion frame, letting existing boxes be re-projected onto any novel pose.","feed_headline":"Auto-labeling 3D renderer hits 29.04 PSNR on nuScenes","feed_subtitle":"Synthesized views come with matching 2D/3D boxes, boosting downstream detectors without added annotation.","key_machinery":"The machinery that carries the argument is the adaptor trained under three losses enforcing the same projective geometry in two coordinate systems: a pose loss aligning known original-world to SfM pose pairs, a 3D relative-position loss that leverages the similarity of real and SfM intrinsics to pin down scale, and a pixel-projection loss that keeps novel poses consistent when neighboring frames are projected onto their image planes. Supporting it are the deformation enhancement and opacity enhancement modules, which use the Gaussian's state attribute and time encoding to refine dynamic-object modeling, and the grouping strategy, which labels each Gaussian with a group id so that rendering uses only nearby primitives.","core_discovery":"The central claim is that an enhanced 3D Gaussian Splatting model, trained from images alone, can serve as a labeled data factory for autonomous driving: every novel view it renders comes with correct 3D bounding boxes in the dataset's original coordinate system. On the rendering side, the paper augments the Deformable 3D GS baseline with a per-Gaussian state attribute that tells the deformation network whether a primitive is dynamic or static, a deformation enhancement module and an opacity enhancement module that refine the deformation field and opacity, and a grouping strategy that assigns each Gaussian a group id so rendering excludes distant primitives outside the current view's field. On the labeling side, it introduces an adaptor—an 8-layer MLP that maps camera poses from the original world coordinate system to the SfM coordinate system—trained with three constraints: pose matching on known frame pairs, 3D relative-pose consistency, and pixel projection consistency for novel poses. The paper argues that this single-model design not only beats composite approaches like DrivingGaussian on rendering metrics but also produces annotations accurate enough to improve 2D and 3D detectors when used as data augmentation.","pith_inferences":["Beyond the paper: if the adaptor's projection-consistency training generalizes, the same approach could replace hand-tuned coordinate alignment in other reconstruction pipelines that need to fuse dataset annotations with SfM geometry.","Beyond the paper: a testable extension is to feed the adaptor camera poses from fisheye or non-pinhole cameras; the paper's assumption of similar intrinsics would break, and the projection constraints would need a different formulation.","Beyond the paper: the reported detection gains compare auto-labeled images against the baseline dataset; isolating labeling error from rendering quality would require comparing against manually labeled versions of the same novel views."],"forward_implications":["Synthetic frames for autonomous driving can be produced on demand with labels, removing the need to annotate every rendered image by hand.","A single model replaces the composite foreground/background pipelines of earlier driving 3D GS methods, cutting training and rendering overhead.","Downstream 2D and 3D detectors improve when auto-labeled novel views are added to their training data, suggesting synthetic data from the renderer can substitute for additional real labeled frames.","The grouping strategy extends 3D Gaussian Splatting to long, large-scale sequences by bounding each view's field of view, which should carry over to other outdoor reconstruction tasks.","Because the method requires only images plus existing 3D annotations and poses, it applies to any driving dataset that already carries such annotations."],"supporting_citations":[{"why":"Defines 3D Gaussian Splatting and the tile-based rasterizer that all rendering in this paper is built on.","marker":"Kerbl et al. 2023"},{"why":"Provides the Deformable 3D GS baseline that the renderer extends with state attributes, DEM, and OEM.","marker":"Yang et al. 2024"},{"why":"Introduces DrivingGaussian, the composite 3D GS driving-scene method this work compares against and whose nuScenes scene split is used for PSNR 29.04.","marker":"Zhou et al. 2024"},{"why":"Supplies the nuScenes dataset with camera poses, intrinsic parameters, and 3D bounding boxes used for training the adaptor and evaluating detection.","marker":"Caesar et al. 2020"},{"why":"Is the structure-from-motion method used to estimate camera poses and intrinsics in the SfM coordinate system.","marker":"Schonberger and Frahm 2016"},{"why":"Is the 2D detector whose performance is measured when trained with EGSRAL's auto-labeled augmentation.","marker":"Zong, Song, and Liu 2023a"},{"why":"Is the monocular 3D detector used to evaluate the 3D auto-labeling data.","marker":"Li, Jia, and Shi 2024"}],"fun_headline_variants":["Image-only 3D GS labeling for driving scenes hits 29 PSNR","3D GS renderer auto-labels driving scenes, no extra annotations","Automated labeling from images alone boosts driving detectors","No depth, no boxes: 3D GS renders driving scenes with labels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The adaptor's training assumes the camera intrinsics estimated by structure-from-motion are close enough to the dataset's real intrinsics that the same 3D points project to the same pixels in both coordinate systems; if that similarity fails, the novel-pose labels will be misaligned.","fun_headline_variants_meta":{"raw":{"variants":["Image-only 3D GS labeling for driving scenes hits 29 PSNR","3D GS renderer auto-labels driving scenes, no extra annotations","Automated labeling from images alone boosts driving detectors","No depth, no boxes: 3D GS renders driving scenes with labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000708,"raw_usage":{"total_tokens":3227,"prompt_tokens":1017,"completion_tokens":2210,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":633,"completion_tokens_details":{"reasoning_tokens":2133}},"tokens_in":633,"tokens_out":2210,"duration_ms":12705,"temperature":1.0,"reasoning_tokens":2133,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:19:50.681642+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure, per nuScenes scene, the pixel reprojection error between the SfM-estimated intrinsics and the dataset's reported intrinsics on known 3D points; if that error exceeds a few pixels, the projection-consistency constraint used to train the adaptor is violated and auto-labeled boxes should drift. A direct experiment: deliberately perturb the SfM intrinsics by a fixed amount, retrain the adaptor, and check whether downstream 3D detection AP falls in proportion to the perturbation.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines 3D Gaussian Splatting and the tile-based rasterizer that all rendering in this paper is built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Deformable 3D GS baseline that the renderer extends with state attributes, DEM, and OEM."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces DrivingGaussian, the composite 3D GS driving-scene method this work compares against and whose nuScenes scene split is used for PSNR 29.04."},{"cited_title":"L.; and Frahm, J.-M","cited_arxiv_id":null,"evidence_quote":"Is the structure-from-motion method used to estimate camera poses and intrinsics in the SfM coordinate system."}],"review_version":1}