{"id":"3a37a1f5-c4fc-423b-9bb1-aae29b8f5272","arxiv_id":"2608.13147","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A single transformer latent pretrained for 3D reconstruction, adapted with camera calibration and split attention, drives depth, detection, and occupancy for self-driving cameras.","lead":"GeoUP takes a pretrained 3D reconstruction model and adapts it to a car's surround cameras, so one shared representation outputs metric depth, 3D object boxes, and occupancy. It reports state-of-the-art results on several driving benchmarks, but the claims need code and stronger baseline comparisons before they can be adopted.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Depth-SOTA claim compares only against VGGT-family methods; but the deeper problem is that the multi-dataset gains are confounded by longer schedule and bigger batch, so the headline unified-representation claim is not yet isolatable.","rationale":"The reader identifies the weakest assumption as the VGGT-12 transfer premise. I agree that this is a plausible weak spot, but the sharper concern is that the paper's own experimental setup cannot isolate the transfer premise from large confounds: the multi-dataset model is trained with twice the epochs, batch 64 versus 16, and Muon instead of AdamW, so the GeoUP† gains that the paper repeatedly attributes to the representation are equally explained by a better optimizer and more compute. Table 9 partially supports the transfer claim (VGGT-12 beats DINOv2-T by 2.5 mAP and 1.7 mIoU), but the DINOv2-T baseline is a weak control because it adds 12 randomly initialized transformer blocks to DINOv2-L; the comparison does not hold compute or parameter count fixed, and it does not include a no-VGGT-initialization control with the full GeoUP architecture. Another omission is that depth SOTA in Table 5 uses only VGGT-family or reconstruction baselines; task-specific metric-depth models are absent, so the claim of SOTA across depth is not established. None of this makes the central claim false; in fact the paper contains several honest disclosures, including the compute-heavy limitation, task-specific heads, and the corrected-evaluator note in the planning table. The conditional verdict is appropriate: the paper should be published only with the missing controls or with the claims scaled back. My stress-test reinforces a specific confound that should be explicitly resolved before the unified-representation attribution is taken at face value.","tokens_in":21613,"tokens_out":4230,"duration_ms":34836,"concrete_test":"Run two controls. First, train GeoUP with all architecture components (factorized self/temporal/view attention, ray-map injection, same heads, same 4-frame window) but initialize the 12-block transformer randomly, and train an identical model initialized from VGGT weights, under identical schedule, batch size, and optimizer (24 epochs, batch 16, AdamW on nuScenes). If the VGGT-init gap in mAP/NDS and mIoU/RayIoU matches the Table 9 VGGT-init gap, the geometry-pretraining claim is supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that geometry grounding of the VGGT-12 latent is what delivers the gains. The evidence most directly tied to this claim is the backbone comparison in Table 9: DINOv2-L versus DINOv2-T (randomly initialized 12 VGGT-style blocks) versus VGGT-12 initialization versus GeoUP. The crucial missing control is a GeoUP architecture with all driving-oriented adaptations (self/temporal/view attention and ray-map injection) but without VGGT initialization, trained under the same schedule and compute. Table 9 shows VGGT-12 beats DINOv2-T by 2.5 mAP and 1.7 mIoU, which supports geometry initialization, but the DINOv2-T baseline uses a randomly initialized 12-block transformer and lacks the GeoUP adaptations, so the comparison conflates architecture, adaptation, and initialization. More importantly, all reported multi-dataset models (GeoUP†, Tables 1-5) are trained with twice the training epochs and batch size 64 versus 16, plus a different optimizer (Muon vs AdamW), as stated in Section 4.2. Therefore the consistent GeoUP† gains that the paper attributes to heterogeneous supervision and a more robust geometry-grounded representation are equally explained by more compute and a stronger optimizer. The depth SOTA claim is likewise weakened because Table 5 omits task-specific metric-depth models such as Metric3D or metric Depth Anything variants; the best depth baselines listed are reconstruction-oriented VGGT/StreamVGGT/MapAnything/DVGT, so the comparison demonstrates improvement over reconstruction-family models, not SOTA among metric-depth methods. These are not internal inconsistencies, but they mean the headline claim that the geometry-grounded latent itself yields SOTA unified perception is underdetermined by the presented experiments.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GeoUP, a unified camera-only 3D perception framework that adapts the reconstruction-pretrained VGGT transformer to calibrated streaming multi-camera driving scenes. It factorizes cross-image attention into self, temporal, and view attention, injects calibration-aware Plücker raymap embeddings, and decodes the resulting shared latent with task-specific heads for metric depth estimation, 3D object detection, and semantic occupancy prediction. The authors report state-of-the-art results on nuScenes, Argoverse 2, Waymo, Occ3D-nuScenes, KITTI, and DDAD, and a NAVSIMv2 planning-transfer experiment, with component ablations over attention factorization, temporal context, backbone initialization, and multi-task learning.","tokens_in":21831,"tokens_out":7270,"duration_ms":59042,"significance":"If the central claim holds, the paper would be a strong demonstration that reconstruction-oriented latents from visual geometry foundation models can serve as a unified backbone for multiple driving perception tasks, rather than requiring task-specific geometry modules on top of recognition backbones. The paper contains useful independent evidence for this view, particularly the VGGT-initialization-versus-random-initialization comparison in Table 9 and the component ablations in Tables 7 and 10. The multi-dataset results in Table 13 and the planning-transfer experiment add breadth. However, the headline claims currently exceed the evidence: the multi-dataset gains are confounded by changes in training schedule, batch size, and optimizer, and the depth comparison omits standard task-specific metric-depth baselines. The paper is a solid systems contribution whose main claims require additional controlled experiments to be fully supported.","major_comments":[{"comment":"The multi-dataset model GeoUP† is trained with twice the epochs, batch size 64 versus 16, and the Muon optimizer instead of AdamW relative to single-dataset GeoUP. The consistent gains attributed to heterogeneous supervision and multi-dataset training are therefore confounded with additional compute and a stronger optimizer. Please provide a same-schedule control, such as multi-dataset training with batch size 16 and AdamW or single-dataset training with batch size 64 and Muon, so that the effect of multi-dataset supervision is isolated; without this control, the conclusion that heterogeneous supervision strengthens the geometry-grounded representation is not established.","section":"§4.2, Tables 1-5, Table 13"},{"comment":"The depth comparison considers only reconstruction-oriented VGGT-family baselines (StreamVGGT, VGGT, MapAnything, DVGT) and omits task-specific metric depth models such as Metric3D or metric Depth Anything variants. The claim of state-of-the-art depth estimation is therefore not supported. Please add standard metric-depth baselines on KITTI and DDAD, or revise the claim to state superiority over visual-geometry reconstruction models only.","section":"Table 5, §4.3"},{"comment":"The backbone ablation does not isolate the VGGT initialization for the full GeoUP architecture. The DINOv2-T row uses randomly initialized VGGT-style blocks without the driving-oriented adaptations, so the comparison between GeoUP and DINOv2-T conflates architecture, adaptation, and initialization. Please add a control consisting of the full GeoUP architecture (self/temporal/view attention plus raymap injection) with random initialization, trained under the same schedule; this is the missing experiment that directly tests whether geometry pretraining, rather than the larger ViT-L backbone and multi-task training, is responsible for the reported gains.","section":"Table 9, §4.4"},{"comment":"In the frame-count ablation, the 1-frame row shows a large gap between current-only heads and temporal heads (48.7 vs 55.1 mAP), although with a single input frame the two settings should coincide. Please clarify what the temporal heads do with one frame or correct the table; as printed, this inconsistency prevents a clean interpretation of the temporal-modeling contribution.","section":"Table 8, §4.4"}],"minor_comments":[{"comment":"On KITTI, GeoUP† (0.075 Abs Rel) is worse than single-dataset GeoUP (0.072), which undercuts the phrase \"consistent gains\" when applied to depth; please discuss this exception or explain why multi-dataset training slightly degrades KITTI depth.","section":"Table 5, §4.3"},{"comment":"The \"Global\" row in Table 7 (53.8 mAP, 61.7 NDS) appears to describe the same VGGT-12 global-attention configuration as the VGGT-12 row in Table 9 (54.6 mAP, 62.0 NDS), but the numbers differ; please clarify whether these are different settings or explain the discrepancy.","section":"Table 7 vs Table 9"},{"comment":"The depth loss is described as combining regression and gradient-based regularization, but Eq. (10) gives equal weights of 1.0 for both terms; please state whether these weights match the main experiments and define the loss weights precisely for reproducibility.","section":"Section 3.3, Appendix A"},{"comment":"The row name \"DINOv2-T\" is confusing; please specify that it denotes a DINOv2 encoder augmented with 12 randomly initialized VGGT-style transformer blocks, and clarify which DINOv2 encoder variant is used.","section":"Table 9"},{"comment":"The construction of VGGT-12 by selecting every other layer from the pretrained VGGT checkpoint is presented without justification or sensitivity analysis; please report whether other layer subsets give similar results, or note this as a limitation.","section":"§4.2"},{"comment":"The unified depth scale of 90 m used to normalize depth targets across datasets is mentioned only in Appendix A; please state it in Section 3.2 or in the implementation details for reproducibility.","section":"Appendix A, §3.2"},{"comment":"The EPDMS results are reported as \"original/corrected\" in Table 6, and the text refers to \"original and corrected evaluators\" without defining which entry corresponds to which; please state the correspondence explicitly.","section":"Table 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid systems contribution and the component ablations are informative, but the main risk is the compute/optimizer confound in the multi-dataset experiments. If the same-schedule control cannot be run, the multi-dataset claims should be substantially softened. The depth claim also needs standard metric-depth baselines. I believe major revision is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know about this paper: it's the first driving adaptation of VGGT I've seen that treats the reconstruction latent as a backbone for detection, occupancy, and depth readouts rather than as a reconstruction engine. That framing is genuinely new relative to DriveVGGT and DVGT, and the factorized temporal/view attention and raymap injection are sensible engineering choices. The paper also has real evidence: Table 9 shows VGGT-12 initialization beats a randomly initialized VGGT-style transformer by 2.5 mAP and 1.7 mIoU with architecture basically fixed, which is the best support for the geometry-grounding claim. Single-dataset gains over RayDN, Far3D, and OPUS-V2 are consistent across three task families.\n\nNow the soft spots, in order of seriousness.\n\nThe multi-dataset GeoUP† results, which anchor the headline SOTA claims, are trained with twice the epochs, batch size 64 instead of 16, and a different optimizer (Muon vs AdamW). That is a textbook confound. The paper attributes the consistent multi-dataset gains to heterogeneous supervision and a more robust latent; equal compute, schedule, and optimizer could explain a good chunk of it. Table 13 does compare single vs multi at low resolution but still uses the doubled schedule for multi. The cleanest missing experiment is a GeoUP† trained with the same epochs and batch as the single-dataset baseline. Without that, the unified-representation claim is underdetermined.\n\nSecond, the depth SOTA claim in Table 5 compares only against reconstruction-family models (VGGT, StreamVGGT, MapAnything, DVGT). There are no metric-depth baselines like Metric3D or metric Depth Anything variants. That's not fatal to the perception story, but it does mean 'SOTA depth' is not shown.\n\nThird, there are no error bars anywhere, and no code or weights. For an empirical systems paper of this scale, that's a reproducibility gap, though the authors do transparently disclose the heavy compute (~0.8 FPS with 4 frames) and the task-specific heads as limitations.\n\nNone of this makes the central idea wrong. I think the premise—that a reconstruction-pretrained latent can be adapted into a metric, multi-view, temporally consistent driving perception backbone—is credible and worth testing. But the paper as written does not isolate the mechanism behind the headline numbers. A serious referee should ask for the missing control, the extra baselines, and a variance estimate before accepting the SOTA language.\n\nWho benefits: anyone working on geometry foundation models for driving, or on unified perception backbones. It deserves a real peer review, not a desk reject.\n\nBest,","headline":"A serious and well-executed attempt to make a reconstruction-pretrained latent serve as a shared 3D perception backbone, but the headline SOTA claims are underdetermined by confounded multi-dataset training and an unclosed ablation gap.","tokens_in":22517,"tokens_out":2247,"would_cite":true,"duration_ms":19087,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One shared geometry latent powers state-of-the-art 3D detection, occupancy, and depth for camera-only driving perception.","keywords":["geometry-grounded representation","unified 3D perception","autonomous driving","metric depth estimation","3D object detection","semantic occupancy prediction","visual geometry foundation model","multi-camera calibration"],"falsifier":"Train the same GeoUP architecture with a randomly initialized VGGT-12 block and no reconstruction pretraining while keeping everything else identical, then compare detection mAP and occupancy mIoU on the nuScenes validation set: if the gap over the DINOv2-L baseline collapses, the claimed geometry transfer from reconstruction pretraining is not the driving factor.","tokens_in":21314,"feed_emoji":"🚗","tokens_out":2801,"duration_ms":19397,"temperature":0.7,"pith_summary":"The paper tries to establish that a single shared 3D scene latent, pretrained for multi-view reconstruction rather than semantic recognition, can serve as the backbone for all major camera-only autonomous driving perception tasks at once. It argues that depth, 3D object detection, and occupancy prediction are different readouts of the same metric scene, and that current backbones fail to encode that geometry explicitly. To that end, it adapts the reconstruction-pretrained VGGT model to streaming multi-camera driving input, and reports state-of-the-art results across all three task families on five benchmarks. A sympathetic reader would care because this points to a path where geometry, not task-specific lifting modules, becomes the shared currency of driving perception.","feed_headline":"One geometry latent powers detection, occupancy, and depth","feed_subtitle":"A reconstruction-pretrained backbone adapted for driving beats task-specific models on all three camera-only tasks at once.","key_machinery":"The central object is a 12-block subset of VGGT's reconstruction-pretrained transformer backbone, called VGGT-12. It receives image patch tokens from a DINOv2 encoder, augmented with Plücker raymap embeddings derived from camera calibration and camera tokens, then processes them with factorized self, temporal, and view attention. The work it does is to provide a shared latent in which surface-level depth, instance-level 3D boxes, and volume-level occupancy can be decoded from the same features.","core_discovery":"GeoUP claims that a reconstruction-oriented latent from VGGT, adapted to calibrated streaming multi-camera driving scenes, yields a shared representation that simultaneously supports metric depth estimation, 3D object detection, and semantic occupancy prediction at state-of-the-art accuracy. The adaptation consists of factorizing cross-image attention into self, temporal, and view attention, plus injecting calibration-aware Plücker raymap encodings for metric scale. The paper reports that this geometry-grounded latent outperforms prior task-specific approaches on nuScenes, Argoverse 2, Waymo, KITTI, and DDAD, and that multi-task joint training across datasets further improves all tasks.","pith_inferences":["A testable extension is to probe whether the latent's geometry comes from the VGGT pretraining objective or simply from multi-view temporal data at training time, by training VGGT-12 without the reconstruction checkpoint on the same tasks.","The factorization into self/temporal/view attention suggests that deploying GeoUP in a streaming setting could cache and recycle attention states to amortize the cost of the slow multi-frame backbone.","Because the paper reports detection results only on nuScenes-style category spaces, the cross-dataset sharing of the geometry latent may not yet demonstrate open-vocabulary or unseen-category detection, which would be a natural next test."],"forward_implications":["Camera-only 3D detection, occupancy, and depth could be unified under one pretrained geometry backbone instead of separate task-specific modules.","Joint training on datasets with heterogeneous annotations becomes a way to strengthen the shared 3D latent, since supervision for any task feeds the same representation.","End-to-end planners can inherit a geometry-grounded perception backbone and improve their closed-loop safety metrics without retraining the planner.","The geometry-grounded latent preserves reconstruction capability, so it can produce consistent point maps from predicted depth on driving scenes."],"supporting_citations":[{"why":"VGGT is the reconstruction-pretrained model whose latent GeoUP adapts; its weights initialize the backbone and its camera branch is retained.","marker":"[63]"},{"why":"DINOv2 provides the image encoder that produces semantically rich patch tokens fed into the geometry backbone.","marker":"[46]"},{"why":"RayDN provides the detection head and its loss, one of the three task-specific readouts.","marker":"[36]"},{"why":"OPUS-V2 provides the occupancy head and its loss, the volume-level readout.","marker":"[61]"},{"why":"DPT is the dense prediction head used for metric depth decoding.","marker":"[50]"},{"why":"nuScenes is the primary benchmark for detection and occupancy evaluation; its coordinate convention is also used as the standard for multi-dataset boxes.","marker":"[3]"},{"why":"Argoverse 2 is the second detection benchmark, providing a different camera rig and longer perception range.","marker":"[72]"},{"why":"Waymo is the third detection benchmark, serving as a large-scale diverse test that this paper claims to outperform prior camera-only detectors on.","marker":"[58]"},{"why":"KITTI supplies the forward-view depth evaluation used to test cross-dataset geometric generalization.","marker":"[15]"},{"why":"DDAD supplies the surround-view metric depth evaluation, the harder case for geometry models.","marker":"[17]"}],"fun_headline_variants":["Geometry latent unifies depth, detection, occupancy","Adapted reconstruction latent tops driving 3D tasks","GeoUP: shared 3D latents beat task-specific models","One geometry-grounded latent for all driving 3D tasks","Calibration-aware latent excels across three perception tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that a 12-block subset of VGGT, initialized from the reconstruction-pretrained checkpoint and fine-tuned with depth and camera losses, actually transfers reconstruction-oriented geometry to a metric driving perception latent, rather than the gains coming mostly from its larger ViT-L backbone, multi-task training, or longer schedules.","fun_headline_variants_meta":{"raw":{"variants":["Geometry latent unifies depth, detection, occupancy","Adapted reconstruction latent tops driving 3D tasks","GeoUP: shared 3D latents beat task-specific models","One geometry-grounded latent for all driving 3D tasks","Calibration-aware latent excels across three perception tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000177,"raw_usage":{"total_tokens":1288,"prompt_tokens":932,"completion_tokens":356,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":291}},"tokens_in":548,"tokens_out":356,"duration_ms":3430,"temperature":1.0,"reasoning_tokens":291,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:44:42.358328+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same GeoUP architecture with a randomly initialized VGGT-12 block and no reconstruction pretraining while keeping everything else identical, then compare detection mAP and occupancy mIoU on the nuScenes validation set: if the gap over the DINOv2-L baseline collapses, the claimed geometry transfer from reconstruction pretraining is not the driving factor.","supporting_citations":[{"cited_title":"Vggt: Visual geometry grounded transformer","cited_arxiv_id":null,"evidence_quote":"VGGT is the reconstruction-pretrained model whose latent GeoUP adapts; its weights initialize the backbone and its camera branch is retained."},{"cited_title":"Ray denoising: Depth-aware hard negative sampling for multi-view 3d object detection","cited_arxiv_id":null,"evidence_quote":"RayDN provides the detection head and its loss, one of the three task-specific readouts."},{"cited_title":"Opus: occupancy prediction using a sparse set","cited_arxiv_id":null,"evidence_quote":"OPUS-V2 provides the occupancy head and its loss, the volume-level readout."},{"cited_title":"Vision meets robotics: The kitti dataset.The international journal of robotics research, 32(11): 1231–1237, 2013","cited_arxiv_id":null,"evidence_quote":"KITTI supplies the forward-view depth evaluation used to test cross-dataset geometric generalization."}],"review_version":1}