{"id":"999cf232-2d24-4d0d-b78c-1293b0bc72c8","arxiv_id":"2608.12179","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Using a pretrained metric reconstruction model as the detector encoder, with an up-to-scale 3D box head scaled by the model's predicted scale factor, gives stronger online monocular 3D detection and transfer than 2D-to-3D lifting baselines.","lead":"Map-Det3D repurposes a feed-forward metric 3D reconstruction model as the encoder of a detection transformer, so streaming monocular RGB can directly produce metric 3D boxes in reconstructed space. It reports state-of-the-art class-agnostic indoor results on CA-1M and zero-shot ScanNet, pointing to a more robust alternative to 2D-to-3D lifting for RGB-only detection.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ScanNet zero-shot comparison does not control for MapAnything's pretraining corpus; if ScanNet appears in reconstruction pretraining, the headline 'robust transfer' claim is inflated by data overlap rather than architecture.","rationale":"The reader's concern about per-window scale factor rho is legitimate: all metric outputs are multiplied by rho in Sec. 3.4, and no direct calibration curve on ScanNet is given. However, end-to-end zero-shot AP already provides indirect evidence that rho is not catastrophically biased, and an oracle-scale ablation would resolve that cleanly. The pretraining-corpus issue is more fundamental because it affects the interpretation of the entire ScanNet comparison, not just the scale factor. If MapAnything has seen ScanNet during reconstruction pretraining, the paper's claim of 'robust transfer without adaptation' is overstated regardless of rho. The paper does provide real evidence for the architecture (Tab. 1 ablations show design choices matter; Tab. 2 shows the direct 3D head helps; external benchmark comparisons are meaningful), so the issue is not sufficient to reject. It should keep the CONDITIONAL verdict, with the added explicit condition that the authors disclose/control MapAnything pretraining data. This is why I set verdict_should_be to UNCHANGED relative to the reader, while disagreeing with the specific weakest assumption selected.","tokens_in":14093,"tokens_out":9991,"duration_ms":94646,"concrete_test":"Ask the authors to disclose the full MapAnything pretraining dataset list. If ScanNet/ScanNet200 is included, re-run the zero-shot evaluation of Map-Det3D using a MapAnything checkpoint trained without those scenes (or on a disjoint indoor benchmark such as ARKitScenes) and compare AP15/AP25 on the same ScanNet200 split. If the gap to CuTR largely persists, the zero-shot claim stands; if AP drops toward CuTR's 4.3, the headline 'zero-shot without adaptation' must be weakened to 'zero-shot for detection supervision.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central 'zero-shot transfer to ScanNet200' claim (Sec. 4.4, Tab. 5) is interpreted as evidence that the FF3R prior generalizes across indoor domains. The control shown is CuTR trained on the same CA-1M detection data, which rules out detection-data overlap. It does not rule out encoder-data overlap: Map-Det3D's backbone is initialized from MapAnything, and Sec. 3.2 never states which datasets were used for MapAnything's reconstruction pretraining. MapAnything is a same-group model; standard FF3R training mixtures (e.g., VGGT-style corpora) commonly include ScanNet. If ScanNet or ScanNet200 imagery/geometry was part of MapAnything pretraining, then the ScanNet evaluation scenes are not unseen to the geometric encoder, and the 15.2 vs 4.3 AP15 gap over CuTR could reflect pretraining overlap rather than the proposed online multi-view architecture or its scale handling. The abstract's phrase 'robust transfer without adaptation' therefore rests on an unstated exclusion. This is a missing-control/missing-disclosure issue, not an internal inconsistency; it is directly checkable because MapAnything's training corpus is known to the authors.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Map-Det3D, an online multi-view 3D object detection framework that repurposes a feed-forward metric 3D reconstruction model (MapAnything) as the encoder of a DETR-style detection transformer. A short temporal window of RGB frames is treated as multi-view input, and an up-to-scale 3D bounding box head recovers metric coordinates and dimensions by multiplying the backbone's predicted per-window scale factor. The method is trained class-agnostically on CA-1M and evaluated on held-out CA-1M scenes, zero-shot on ScanNet200, and per-scene on ScanNetV2 with a simple tracking-by-detection module. The reported results show improvements over monocular baselines such as CuTR and Cube R-CNN, and competitive zero-shot performance on ScanNet.","tokens_in":14211,"tokens_out":4421,"duration_ms":36863,"significance":"If the results hold, this is a valuable architectural insight: rather than regressing depth or scale from 2D detections, a feed-forward reconstruction model's learned scale prior can be directly converted into metric 3D boxes. The ablations in Tab. 1 provide controlled evidence for the main design decisions (unfreezing the multi-view transformer, camera conditioning, and the direct 3D head), and the release of code and models supports reproducibility. The significance is somewhat tempered by the fact that the method is evaluated only on indoor class-agnostic detection, and by unresolved questions about the zero-shot comparison's control for backbone pretraining data and about the statistical reliability of the single-run results.","major_comments":[{"comment":"The zero-shot ScanNet200 comparison does not disclose the pretraining corpus of the MapAnything backbone, despite the central claim of 'robust transfer without adaptation' (abstract, Sec. 4.4). MapAnything is a same-group model, and common FF3R training mixtures include ScanNet. If ScanNet validation imagery or geometry was part of the reconstruction pretraining, the large gap over CuTR (15.2 vs 4.3 AP15) could reflect encoder-data overlap rather than the proposed online multi-view architecture or scale handling. Please provide the complete list of datasets used in MapAnything's reconstruction pretraining and, if ScanNet is included, assess the impact of overlap (e.g., by reporting results with a backbone variant that was not trained on ScanNet, or by stratifying performance across scenes with different overlap levels).","section":"Sec. 4.4, Tab. 5"},{"comment":"The metric outputs of the model depend entirely on the predicted scale factor rho (x = rho * x_tilde, z = rho * exp(d_tilde), w = rho * exp(s_w_tilde), etc.). The paper does not directly validate whether rho remains metrically calibrated on the zero-shot benchmark, even though a biased rho would corrupt every metric box regardless of the up-to-scale geometry. Please report the scale factor's accuracy (e.g., median absolute relative error of rho with respect to scene scale) on both CA-1M and ScanNet validation, and, ideally, ablate the downstream detection AP when rho is replaced by an oracle or ground-truth scale factor.","section":"Sec. 3.4, Eqs. (3)–(4) and Tab. 5"},{"comment":"All experimental tables report single-run results without error bars or significance tests. Several decisive comparisons involve modest margins (e.g., 17.2 vs 17.3 in Tab. 1 rows 5 and 7, and 21.1 vs 21.2 in Tab. 3), so the reader cannot judge whether the reported improvements are stable or within run-to-run noise. Please provide at least three independent runs for the main results and report mean and standard deviation, especially for the SOTA comparisons in Tabs. 4 and 5.","section":"Sec. 4.3–4.5, Tabs. 1–6"},{"comment":"The evaluation section states that, unlike prior benchmarks, all ground-truth boxes are evaluated regardless of visibility and truncation. If this protocol is applied to Map-Det3D but the baseline numbers in Tabs. 4 and 5 are taken from prior publications that used a different protocol (e.g., only visible boxes), the comparison is not apples-to-apples. Please state explicitly whether all baselines were re-evaluated under the same protocol, and, if so, provide the re-evaluated numbers; otherwise, the SOTA claims in Secs. 4.4 and 4.5 are not properly supported.","section":"Sec. 4.2, Tabs. 4–5"}],"minor_comments":[{"comment":"The phrase 'and adapts it for 3D object detection' should read 'and adapts it for 3D object detection' or 'and adapts it for 3D object detection'; the verb form is inconsistent with the rest of the caption.","section":"Fig. 1 caption"},{"comment":"The symbol T is reused for the temporal window size (Sec. 3.1) and for the track memory in the per-scene evaluation (Sec. 4.5). Please rename one of these to avoid confusion.","section":"Sec. 4.5 vs Sec. 3.1"},{"comment":"The row configuration in Tab. 1 is not fully self-explanatory; for example, row 4 has checkmarks for 'Unfreeze Scale Head' and 'Unfreeze MV Transformer' but no 'Multi-View' checkbox, which may make the incremental contribution hard to parse. Consider adding a short textual description of each configuration or explicitly labeling the row conditions.","section":"Sec. 4.3, Tab. 1"},{"comment":"The method relies heavily on reference [11] (MapAnything), which appears to be a same-group publication. Please confirm that this work is publicly available or accepted, since the paper's reproducibility depends on the released checkpoints.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The concerns are addressable: the missing disclosure of MapAnything's pretraining corpus is directly checkable by the authors, and the scale-calibration and error-bar requests are standard empirical additions. I recommend major revision rather than rejection because the core architecture is sensible and the ablations are informative; the central claims are currently not fully supported by the evidence as presented."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look. This is a solid systems integration: Map-Det3D takes MapAnything's feed-forward metric reconstruction model, turns it into the encoder of a deformable DETR-style detector, and adds an up-to-scale 3D head that converts unscaled regressions to metric boxes via the backbone's predicted scale factor. The design is clearly motivated, the ablations in Table 1 show that unfreezing the multi-view transformer, adding camera pose, and using temporal context each matter, and the in-domain numbers on CA-1M beat the monocular baselines by a real margin. The class-agnostic framing is honest, and the limitations section does not oversell the indoor-only scope.\n\nThe main soft spot is the zero-shot ScanNet story. The comparison to CuTR trained on the same CA-1M detection data rules out detection-data overlap, but it does not rule out encoder-pretraining overlap. Map-Det3D's backbone is initialized from MapAnything, and the paper never states which datasets were used for reconstruction pretraining. If ScanNet or ScanNet200 was part of that pretraining, then the 15.2 vs 4.3 AP15 gap is partly explained by the encoder having already seen the evaluation scenes, not by the proposed architecture. This is a missing disclosure, not an internal inconsistency, and it is directly checkable. The authors should state MapAnything's training corpus or rerun the zero-shot comparison with a backbone whose pretraining data excludes ScanNet.\n\nThe other issues are more routine. All results are single-run with no error bars, which matters when the headline gains are a few AP points. The description of how each baseline was re-evaluated under the class-agnostic protocol is thin. And the whole metric-output pipeline depends on the per-window scale factor rho_t staying calibrated on unseen scenes; the paper offers no independent check of that on ScanNet. If rho_t is biased, every box is wrong regardless of the up-to-scale geometry.\n\nI do not see circularity: the detection benchmarks are external, and the scale factor is a learned output, not fitted to the reported metrics. The citation pattern is fine, with same-group work clearly identified. Overall, this is a useful direction and the empirical evidence is mostly coherent, but the robustness-transfer claim is less clean than the abstract implies.\n\nI would send it to a serious referee, mainly to force the pretraining disclosure, error bars, and a clearer baseline-evaluation protocol. It is not a desk reject; it is a conditional accept that needs tightening.","headline":"A competent systems paper with real gains from repurposing an FF3R backbone into a DETR-style detector, but the zero-shot ScanNet claim needs a pretraining-overlap disclosure before I would call it robust transfer.","tokens_in":14904,"tokens_out":1506,"would_cite":true,"duration_ms":15628,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Map-Det3D claims that a feed-forward metric reconstruction model can be the geometric backbone of an online detection transformer, decoding metric 3D boxes without depth sensors or 2D-to-3D lifting.","keywords":["Multi-view 3D object detection","Feed-forward metric reconstruction","Monocular 3D detection","Up-to-scale bounding box head","Online perception","Zero-shot transfer","Class-agnostic detection"],"falsifier":"Multiply every camera's focal length in the input metadata by a constant factor (e.g., 1.2) while keeping the images and scene geometry unchanged, and check whether predicted box depths and sizes scale accordingly; if the metric outputs shift with the camera parameters, the scale factor is not a true metric prior.","tokens_in":13791,"feed_emoji":"📐","tokens_out":8877,"duration_ms":69627,"temperature":0.7,"pith_summary":"Map-Det3D tries to solve the scale problem in monocular 3D detection by moving detection into a metric 3D space built by a feed-forward reconstruction network. Instead of detecting in 2D and then regressing depth and size, the model decodes bounding boxes in an up-to-scale coordinate system and converts them to meters with a per-window scale factor predicted by the reconstruction backbone. The paper reports that this design reaches 16.9 AP25 on held-out CA-1M scenes, beating the monocular baseline CuTR by 3.4 points, and transfers zero-shot to ScanNet200 with 15.2 AP15 versus CuTR's 4.3. If this holds, it would mean that learned reconstruction priors are a practical source of metric scale for image-only, online object detection.","feed_headline":"A reconstruction prior lifts 3D detection without depth sensors","feed_subtitle":"Using a reconstruction net's scale factor, Map-Det3D beats lifting baselines and transfers zero-shot to ScanNet.","key_machinery":"The mechanism that carries the argument is the per-window metric scale factor $\\rho_t$ produced by the MapAnything backbone together with an up-to-scale 3D bounding-box head. The head predicts all geometric attributes in a scale-free representation, such as up-to-scale center offsets, log-depth, and log-dimensions, and then converts them to metric units using $\\rho_t$, so that the network never regresses absolute meters directly. A second load-bearing component is the 16-layer multi-view transformer that fuses features across the temporal window, with the scale token updated jointly; ablations show this is the part whose fine-tuning produces the largest gains, and adding camera intrinsics and poses further stabilizes geometry.","core_discovery":"The central discovery is that a metric feed-forward reconstruction model, designed for camera pose and dense geometry, can be repurposed as the encoder of a DETR-style detector and supply the absolute scale that monocular 3D object detection lacks. Map-Det3D feeds a sliding window of frames through MapAnything's multi-view transformer, obtains fused multi-scale features and a scale token, and decodes boxes in an up-to-scale parameterization: center offsets $\\tilde{x}, \\tilde{y}$, log-depth $\\tilde{d}$, and log-dimensions $\\tilde{s}_w, \\tilde{s}_l, \\tilde{s}_h$, then rescales via $x = \\rho\\tilde{x}$, $y = \\rho\\tilde{y}$, $z = \\rho\\exp(\\tilde{d})$, $w = \\rho\\exp(\\tilde{s}_w)$, and so on. This factorization keeps geometric reasoning in the well-posed up-to-scale space while the reconstruction prior owns metric scale. The paper's experiments attribute the gains to unfreezing the multi-view transformer, adding temporal context, and camera conditioning, and show the top image-only results on held-out CA-1M and zero-shot ScanNet200 while remaining competitive with online depth-free per-scene methods.","pith_inferences":["If the scale factor stays calibrated under camera and scene shift, the same recipe should extend to outdoor settings by retraining or swapping the reconstruction backbone; a clean test would be evaluating on a driving dataset with known metric boxes.","Compare Map-Det3D against a variant that predicts metric depth with a monocular depth foundation model instead of the reconstruction backbone; this would isolate whether the reconstruction's multi-view fusion or just its scale token is doing the work.","Because the boxes are class-agnostic and metric, the output could serve as generic 3D proposals for open-vocabulary detection or segmentation by matching them to text queries through 2D grounding models, an extension the paper names as future work but does not demonstrate.","The up-to-scale head could be detached from detection and used as a calibration probe: the distribution of predicted scale factors over windows might flag when the reconstruction prior is operating out of domain, giving a self-diagnostic for deployment."],"forward_implications":["Depth sensors and 2D-to-3D lifting are not necessary for competitive indoor metric 3D detection: with only RGB video and camera metadata, Map-Det3D reports 16.9 AP25 on held-out CA-1M and outperforms monocular baselines.","The same geometric backbone can serve both reconstruction and detection; adapting it with object-aware losses preserves its transfer ability, as shown by zero-shot ScanNet200 results of 15.2 AP15.","Temporal windows function as additional views: increasing the window from T=1 to T=5 raises CA-1M AP15 from 14.0 to 21.2, so causal multi-view aggregation is a direct source of geometric evidence.","Camera metadata matters: adding intrinsics improves AP15 by 0.8 points and poses by a further 4.7 points in the ablation, so the design can consume whatever geometry information is available.","The class-agnostic detector can be combined with simple tracking-by-detection to produce per-scene results on ScanNetV2, reaching 27.6 AP15 without ground-truth depth."],"supporting_citations":[{"why":"Supplies the feed-forward metric reconstruction backbone, including the multi-view transformer, scale token, and predicted per-window scale factor that Map-Det3D repurposes as its encoder.","marker":"[11]"},{"why":"Provides the CA-1M dataset and the CuTR monocular baseline; the held-out CA-1M and ScanNet200 comparisons against CuTR are the paper's main evidence.","marker":"[16]"},{"why":"The visual geometry transformer architecture whose multi-view fusion and view-count randomization the backbone builds on.","marker":"[40]"},{"why":"Supplies the 6D rotation representation, the disentangled corner loss, and the Omni3D monocular baselines used in the zero-shot comparison.","marker":"[2]"},{"why":"Provides the deformable DETR decoder, query refinement, and bipartite matching that form the detection head on top of the reconstruction features.","marker":"[50]"},{"why":"Defines the per-scene ScanNetV2 evaluation protocol and the online depth-free comparison setting in which Map-Det3D reports 27.6 AP15.","marker":"[15]"},{"why":"Provides the ScanNetV2 benchmark used for zero-shot transfer evaluation and its class-agnostic per-frame and per-scene protocols.","marker":"[5]"}],"fun_headline_variants":["Reconstruction prior gives monocular 3D detection metric scale","No depth sensors: Map-Det3D uses a reconstruction prior for scale","Metric 3D boxes from RGB video via a reconstruction prior","Zero-shot 3D detection on new scenes via reconstruction prior","Reuse a reconstruction model to get 3D detection right"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the per-window scale factor predicted by the reconstruction backbone is metrically correct on scenes the model has never seen, and that fine-tuning the detector does not destroy that calibration.","fun_headline_variants_meta":{"raw":{"variants":["Reconstruction prior gives monocular 3D detection metric scale","No depth sensors: Map-Det3D uses a reconstruction prior for scale","Metric 3D boxes from RGB video via a reconstruction prior","Zero-shot 3D detection on new scenes via reconstruction prior","Reuse a reconstruction model to get 3D detection right"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000725,"raw_usage":{"total_tokens":3314,"prompt_tokens":1072,"completion_tokens":2242,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":688,"completion_tokens_details":{"reasoning_tokens":2153}},"tokens_in":688,"tokens_out":2242,"duration_ms":15437,"temperature":1.0,"reasoning_tokens":2153,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:13:49.864158+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Multiply every camera's focal length in the input metadata by a constant factor (e.g., 1.2) while keeping the images and scene geometry unchanged, and check whether predicted box depths and sizes scale accordingly; if the metric outputs shift with the camera parameters, the scale factor is not a true metric prior.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the feed-forward metric reconstruction backbone, including the multi-view transformer, scale token, and predicted per-window scale factor that Map-Det3D repurposes as its encoder."},{"cited_title":"In: Proceedings of the Computer Vision and Pattern Recognition Conference","cited_arxiv_id":null,"evidence_quote":"Provides the CA-1M dataset and the CuTR monocular baseline; the held-out CA-1M and ScanNet200 comparisons against CuTR are the paper's main evidence."},{"cited_title":"In: Computer Graphics Forum","cited_arxiv_id":null,"evidence_quote":"Defines the per-scene ScanNetV2 evaluation protocol and the online depth-free comparison setting in which Map-Det3D reports 27.6 AP15."}],"review_version":1}