{"id":"81c2e25b-6136-4429-bc0f-b40a9989fe0c","arxiv_id":"2506.01933","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"E3D-Bench compares 16 3D geometric foundation models on depth, reconstruction, pose, and view-synthesis tasks with a unified evaluation toolkit.","lead":"This paper builds a common test for 3D AI models that turn ordinary photos into 3D shapes and camera positions. It compares 16 such models on five tasks and many scene types, and reports which ones generalize best and run fastest.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The metric-scale block of Table 2 is near-duplicated for MASt3R and CUT3R, so the benchmark's reliability — not protocol fairness — is the central unresolved risk.","rationale":"I chose this as the load-bearing concern because the paper's value depends on the accuracy of its published numbers, and the near-duplicate rows are the most concrete, falsifiable red flag. The reader's weaker assumption about protocol fairness is real, but it is a design judgment the paper discloses; the near-duplicate Table 2 rows are either an error or an undeclared degenerate protocol and must be settled before the benchmark can serve as a reference. This does not change the conditional verdict: the paper should be accepted only after the check and the code release, and the metric-scale finding may need revision if the rows are wrong.","tokens_in":28245,"tokens_out":11254,"duration_ms":112347,"concrete_test":"Obtain the official MASt3R and CUT3R checkpoints and rerun the Section 3.1 / Appendix B.1 sparse-view depth protocol exactly, printing per-scene AbsRel before averaging. If the published five-dataset rows are not reproduced, correct the table; if they are reproduced, instrument the pipeline to log which checkpoint and alignment path each method actually uses — the normalized rows prove the two models are not identical, so identical metric failures would indicate a shared bug in the metric-scale evaluation branch rather than genuine model behavior.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that E3D-Bench is the first comprehensive, fair, reproducible benchmark for 3D GFMs. For that claim, the tables must be actual outputs of the stated methods. The most direct threat is in Table 2 (Sparse-View Depth Estimation): in the metric-scale block, MASt3R and CUT3R report essentially the same numbers on every dataset — 84.904/0.000 on DTU, 93.584/0.000 on ScanNet, 99.069/0.000 on KITTI, 97.021 vs 97.022/0.000 on ETH3D, and 98.234/0.000 on Tanks and Temples. These are two different networks with different normalized-scale results in the same table (DTU AbsRel 3.343 vs 6.200; KITTI 9.542 vs 23.849), so the near-identical metric failures are not explained by behavioral equivalence. The plausible explanations are a copy-paste/transcription artifact or a protocol bug in which both methods were scored through the same failed alignment. Because the paper's metric-scale finding (Section 4.1, 'metric-scale depth estimation remains a key challenge') rests largely on these rows, this is a load-bearing data-integrity issue rather than a cosmetic discrepancy. The absence of error bars and unreleased code compounds the problem, because the reader cannot check whether other close rankings are similarly affected.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces E3D-Bench, a benchmark for end-to-end 3D geometric foundation models (GFMs). It evaluates 16 models across five tasks (sparse-view depth estimation, video depth estimation, multi-view relative pose estimation, multi-view 3D reconstruction, and novel view synthesis), plus an inference-efficiency analysis, using both standard and out-of-distribution datasets. The authors propose standardized protocols for dataset handling, alignment, and metric computation, and derive findings on task difficulty, domain generalization, architecture choice, and efficiency. They claim this is the first systematic benchmark spanning these tasks and scenarios.","tokens_in":28532,"tokens_out":4954,"duration_ms":44033,"significance":"If the results are reliable, E3D-Bench would fill a real gap: the 3D GFM literature has grown rapidly but lacks a common evaluation platform. The paper's strengths include the breadth of tasks and datasets, the inclusion of both metric-scale and normalized models, explicit efficiency benchmarking, and transparent disclosure of protocol choices in the appendices. The reported rankings do not favor the authors' own LSM model, which suggests that the evaluation is not cherry-picked. However, the data-integrity issue in Table 2, the absence of error bars, and the lack of sensitivity analysis currently prevent the benchmark from serving as a trustworthy reference; these issues are fixable but require substantial additional work.","major_comments":[{"comment":"In the metric-scale block of Table 2, the rows for MASt3R and CUT3R are identical on every dataset (e.g., DTU 84.904/0.000, ScanNet 93.584/0.000, KITTI 99.069/0.000, ETH3D 97.021 vs. 97.022, and Tanks and Temples 98.234/0.000). Since the normalized-scale rows for these two models differ substantially (e.g., DTU AbsRel 3.343 vs. 6.200 and KITTI 9.542 vs. 23.849), the identical metric failures are not plausible as genuine model behavior and appear to be a transcription or scoring bug. The finding in Section 4.1 that 'metric-scale depth estimation remains a key challenge' rests heavily on these rows, so the authors must re-run the evaluation and correct the table or provide a valid explanation. The absence of error bars makes it impossible to know whether other close rankings are similarly affected.","section":"Table 2"},{"comment":"The paper reports no error bars, confidence intervals, or multiple-run statistics for any of the quantitative comparisons. Comparative claims such as 'VGGT consistently achieves the best performance' (Section 3.2) and the architecture findings in Section 4.3 rely on small numeric differences (e.g., Video Depth on Bonn: CUT3R 0.068 vs. VGGT 0.056; Align3R 0.062 vs. MonST3R 0.061). Without variance estimates, the reader cannot assess whether these differences are meaningful. The benchmark should report per-scene standard errors or per-run statistics, and ideally a statistical test for the headline comparisons, especially because the paper's goal is to be the reference for comparing GFMs.","section":"Tables 2-5 and Figure 2"},{"comment":"The evaluation protocol makes several consequential choices: per-view median alignment for normalized sparse-view depth, sequence-level scale-and-shift alignment for video depth, Sim(3) Umeyama alignment for pose trajectories, quasi-optimal source-view selection from RobustMVD, and test-time pose optimization for novel view synthesis. These choices are disclosed, but the paper does not test whether rankings are robust to them. Since the central claim is 'fair, reproducible comparisons,' the authors should provide a sensitivity analysis for at least the most consequential choices: per-frame vs. per-sequence alignment, nearest-neighbor vs. quasi-optimal source views, and evaluation with vs. without ground-truth intrinsics and poses. Without such evidence, findings like 'no single backbone type dominates' could be an artifact of the specific protocol rather than of model capability.","section":"Appendices B.1, B.2, C.2 and Section 3.3"}],"minor_comments":[{"comment":"The sentence 'MonST3R ranks among the top-3 in video depth estimation (Tab.3) but performs poorly in 3D reconstruction (Tab.2)' should reference Table 5 for 3D reconstruction; Table 2 reports sparse-view depth estimation, not 3D reconstruction.","section":"Section 4.1"},{"comment":"The header 'KITTI Ordometry' should be 'KITTI Odometry'.","section":"Table 4"},{"comment":"The abstract and Section 1 state 'five core tasks,' while Section 5 states 'six core tasks'; the count should be reconciled.","section":"Abstract and Section 5"},{"comment":"The website and code URLs in the header appear garbled (e.g., 'g♀be' and '/gtb'); please provide clean, correct links.","section":"Header/links"},{"comment":"The paper promises that 'all code, evaluation scripts, and processed data will be publicly released' but the release is not yet available; to substantiate the reproducibility claim, the toolkit should be made available at least to reviewers before final acceptance.","section":"Code release"}],"recommendation":"major_revision","confidential_remarks":"The Table 2 issue is serious, but it appears fixable by re-running the affected evaluations. I did not find evidence of bias toward the authors' own models; LSM is often reported jointly with DUSt3R and is not the top performer. The main risks are data integrity and insufficient uncertainty quantification. I recommend major revision, with re-review of the corrected tables and the addition of robustness analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I'd send this to referees, but only with major-revision expectations. The paper does what the field needs: it puts 16 end-to-end 3D GFMs on a common set of tasks — sparse-view depth, video depth, pose, reconstruction, NVS — and adds OOD domains and latency/memory numbers. The appendix protocol is careful: per-view median alignment for normalized models, sequence-level scale-shift for video depth, Sim(3) Umeyama for trajectories, and quasi-optimal source views. That is real work, and the discussion of when GFMs beat specialist baselines is useful.\n\nNow the problem. In Table 2, metric-scale block, MASt3R and CUT3R report essentially identical numbers on every dataset: 84.904 vs 84.904 on DTU, 93.584 vs 93.584 on ScanNet, 99.069 vs 99.069 on KITTI, 97.021 vs 97.022 on ETH3D, 98.234 vs 98.234 on T&T. These are different networks, and the same table shows their normalized-scale results are very different (e.g., KITTI AbsRel 9.542 vs 23.849). So this is not behavioral equivalence; it is either a copy-paste error or a failed alignment that scored both methods through the same broken scale. Either way, Section 4.1's point about metric-scale depth failure rests partly on these rows. The stress-test concern is right, and I don't think it can be waved away.\n\nOther soft spots are minor-to-moderate by comparison. No error bars or significance tests, and the code is not yet released, so independent checking isn't possible. The protocol choices are disclosed, but there is no robustness analysis showing rankings survive alternative alignments or view selections. That matters for a benchmark whose whole point is fair comparison. Author overlap with LSM doesn't bother me: LSM results are not favorable in their own tables.\n\nTake it to a reading group if you want to see the benchmark design; cite it only after the duplicate rows are explained and code ships. Who gets value: anyone choosing among DUSt3R-style or diffusion GFMs and anyone designing the next benchmark. Recommendation: peer review, major revision — Table 2 must be fixed, error bars or per-seed variance added, code released before publication.","headline":"A genuinely useful benchmark protocol with a serious data-integrity problem in Table 2 that must be fixed before the results can be trusted.","tokens_in":29045,"tokens_out":3719,"would_cite":false,"duration_ms":40062,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces E3D-Bench, the first systematic benchmark for end-to-end 3D geometric foundation models, covering 16 models, five geometry tasks, and inference efficiency.","keywords":["3D geometric foundation models","benchmark","multi-view 3D reconstruction","depth estimation","camera pose estimation","novel view synthesis","inference efficiency","end-to-end 3D perception"],"falsifier":"Re-run the leaderboards under a no-frills protocol—predicted (not ground-truth) poses and intrinsics, no median alignment, a single fixed random source-view schedule—and check whether VGGT, CUT3R, and the top diffusion models keep their relative order; if the order flips, the evaluation protocol, not model capability, drove the benchmark's findings.","tokens_in":1638,"feed_emoji":"🧊","tokens_out":2633,"duration_ms":91535,"temperature":0.7,"pith_summary":"E3D-Bench is a benchmark built to answer whether end-to-end 3D geometric foundation models—networks that predict point clouds, depth, and camera poses directly from images—can serve as reliable, efficient building blocks for spatial intelligence. The paper evaluates 16 such models on five geometry tasks (sparse-view depth, video depth, multi-view reconstruction, relative pose, and novel view synthesis) plus inference efficiency, across indoor, outdoor, drone, dynamic, and air-ground datasets. Its central claim is that current models excel on simple sub-tasks and in-domain data but degrade with task complexity, extreme distribution shifts, and metric-scale demands, and that no single architecture family wins across the board. If right, the benchmark becomes the shared reference for comparing GFMs and guides where scaling and training data should go next.","feed_headline":"Benchmark of 16 3D models finds no universal winner","feed_subtitle":"Across five geometry tasks, feed-forward and diffusion models trade wins while metric scale and real-time speed remain open","key_machinery":"The carrying object is the benchmark itself: a standardized evaluation toolkit that automates dataset handling, evaluation protocols, and metric computation, applied uniformly to 16 GFMs. The comparison is made fair across heterogeneous models by extracting depth from the $z$-coordinate of predicted point maps, using ground-truth intrinsics and poses for depth and reconstruction tasks, applying per-view median alignment for normalized models and sequence-level scale-and-shift alignment for video depth, aligning trajectories with $\\mathrm{Sim}(3)$ least-squares alignment, selecting quasi-optimal source views following RobustMVD, and using test-time photometric pose optimization for novel-view synthesis.","core_discovery":"On the paper's own terms, the discovery is that systematic comparison changes the field's self-assessment: GFMs are strong enough to beat task-specific specialists on depth and pose, yet none is a plug-and-play solution. Pair-view geometry outperforms true multi-view inference; single-attribute predictions (depth or pose) are more reliable than full scene reconstruction; relative, normalized outputs are far more accurate than metric-scale ones; generalization holds for aerial and driving shifts but collapses on air-ground pairs; feed-forward ViTs and diffusion models each win some tasks; models built on strong 2D features, exemplified by VGGT's DINO-initialized backbone, lead the pack; and none of the models reaches real-time latency, with even efficient ones taking tens of seconds for hundreds of views.","pith_inferences":["Editorial extension: the headline rankings are only as robust as the alignment protocol; re-running with a no-alignment metric-scale protocol, or with predicted rather than ground-truth poses, could reshuffle the order of VGGT, CUT3R, and the diffusion models.","Editorial extension: isolating geometry with ground-truth poses means real-world performance is likely worse across the board, so a joint pose-and-geometry leaderboard would better predict deployment behavior.","Editorial extension: the DINO-backbone correlation suggests a controlled ablation—keep the GFM architecture and training data fixed, swap the 2D backbone, and measure the shift in 3D metrics."],"forward_implications":["New GFM papers can be placed against a single reported table instead of per-task, per-dataset numbers, making capability claims easier to check.","Because joint prediction of pose, depth, and matching correlates with stronger generalization, decomposing hard tasks into sub-problems is a promising path for improving learning under limited 3D data.","The benchmark's generalization failures point to missing aerial-ground and metric-depth supervision, so training on more diverse, metric-scale-annotated data is the concrete lever for robustness.","Efficiency is a first-class axis: online-registration models such as Spann3R and CUT3R balance accuracy against latency, while global-alignment models risk out-of-memory failures as views grow.","Backbone choice should be task-driven: no family dominates, and upgrading the 2D feature extractor is a concrete lever for 3D gains."],"supporting_citations":[{"why":"Defines the feed-forward pointmap paradigm and the DUSt3R-family baseline evaluated throughout.","marker":"[13]"},{"why":"Metric-scale variant of DUSt3R whose failures mark metric-scale depth as an open problem.","marker":"[14]"},{"why":"Monocular-video adaptation whose sliding-window global alignment and sequence-level evaluation protocol inform the video-depth protocol.","marker":"[16]"},{"why":"The strongest feed-forward multi-view model in the comparison; the benchmark's headline results largely track its performance.","marker":"[17]"},{"why":"Online-registration sequence model that leads long-sequence pose and sparse-view reconstruction among the tested GFMs.","marker":"[19]"},{"why":"Diffusion-based GFM, second-strongest on video depth, representing the denoiser family in the architecture comparison.","marker":"[21]"},{"why":"Online-registration model whose latency and memory efficiency anchor the efficiency analysis.","marker":"[26]"},{"why":"Supplies the quasi-optimal source view selection used to control input bias in sparse-view depth.","marker":"[38]"},{"why":"Supplies the least-squares alignment used for trajectory and point-cloud metrics across all evaluations.","marker":"[54]"},{"why":"Air-ground fine-tuning baseline showing domain-specific data can fix the benchmark's hardest generalization failure.","marker":"[55]"}],"fun_headline_variants":["No universal 3D model: benchmark maps task-specific wins","3D GFM benchmark: depth and pose beat full reconstruction","Pair-view beats multi-view in 3D geometric models","16 models, 5 tasks, no real-time solution yet","3D models trade wins; none are plug-and-play"],"cache_read_input_tokens":31232,"weakest_assumption_plain":"The rankings assume the benchmark's alignment and pose-handling choices treat normalized and metric-scale, feed-forward and diffusion, and pair-view and multi-view models even-handedly; if those choices favor one family, the headline conclusions would be artifacts of the protocol.","fun_headline_variants_meta":{"raw":{"variants":["No universal 3D model: benchmark maps task-specific wins","3D GFM benchmark: depth and pose beat full reconstruction","Pair-view beats multi-view in 3D geometric models","16 models, 5 tasks, no real-time solution yet","3D models trade wins; none are plug-and-play"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000281,"raw_usage":{"total_tokens":1676,"prompt_tokens":965,"completion_tokens":711,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":627}},"tokens_in":581,"tokens_out":711,"duration_ms":7184,"temperature":1.0,"reasoning_tokens":627,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:30:25.760931+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the leaderboards under a no-frills protocol—predicted (not ground-truth) poses and intrinsics, no median alignment, a single fixed random source-view schedule—and check whether VGGT, CUT3R, and the top diffusion models keep their relative order; if the order flips, the evaluation protocol, not model capability, drove the benchmark's findings.","supporting_citations":[],"review_version":1}