{"id":"01ce803a-c227-4643-8d38-196c36b1a3f0","arxiv_id":"2412.11412","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"V-MIND lifts 2D instance masks to pseudo 3D boxes with monocular depth and camera intrinsics, then trains an indoor 3D detector with self-calibration and ambiguity losses, improving Omni3D indoor detection and enabling detection of new classes.","lead":"This paper shows how an indoor 3D object detector can learn many more object types by converting 2D photos and labels from a large web dataset into rough 3D training boxes, without any new 3D annotations. A generalist reader might care because it points to a cheaper path for scaling the object vocabulary of AR/VR and robot perception.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pseudo-box quality is the load-bearing risk: Table 1 reports only 29.72% precision for lifted boxes, and Eq. (3) corrects only a per-image global scale on the center, so depth/intrinsic errors that are not a global scale directly corrupt the only 3D supervision for new classes.","rationale":"The reader's weakest assumption is the same one I consider load-bearing: monocular lifting errors are assumed to be approximately corrected by a per-image global scale. I sharpen this with two internal pieces of evidence: Table 1's 29.72% precision shows the pseudo boxes are mostly imprecise, and Eq. (3) only rescales the box center by one scalar per image. Since new classes have no other 3D supervision, their AP3D rests on this fragile assumption. The self-calibration network is absent at inference, raising the additional risk that the training loss is satisfied by the calibration rather than by the detector learning correct depth. The proposed check, residual error after per-image scale fitting plus a stated-IoU precision, would directly settle whether this concern lands. I do not move the verdict because the current CONDITIONAL already captures this risk; the paper would need to provide the residual-error analysis and strict-IoU numbers before the central claim can be accepted.","tokens_in":12605,"tokens_out":7638,"duration_ms":71695,"concrete_test":"On ScanNet200 or Omni3DIN images with ground-truth depth and intrinsics, run ZoeDepth and Wild Camera, lift instance masks to pseudo boxes, fit the best per-image global scale exp(s) to the box centers, and recompute pseudo-box precision/recall at a stated IoU (e.g., 0.25). If precision after scale correction remains below roughly 50%, or if the residual per-object center error exceeds about 50 cm or 30% of object depth, then the global-scale self-calibration in Eq. (3) cannot denoise the lifted boxes and the new-class 3D supervision is unreliable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim depends on lifted LVIS boxes being accurate enough to teach 3D geometry. For the 38 new classes, these pseudo boxes are the sole 3D supervision. Table 1 shows the pipeline achieves only 29.72% precision and 16.53% recall on ScanNet200 (with no IoU threshold stated), so most generated boxes do not match a ground-truth box. The self-calibration loss in Eq. (3) multiplies only the predicted box center by a single per-image scalar exp(s); it cannot correct per-object depth errors, spatially varying depth bias, or errors in box size and orientation. ZoeDepth and Wild Camera errors are not known to be a global multiplicative scale per image. Moreover, the calibration network is applied only during training, so if it absorbs a per-image depth bias, the detector at inference may not actually predict calibrated depths. Under these conditions, the reported APnew3D = 2.66 (computed over IoU thresholds 0.05 to 0.50) may reflect permissive scoring rather than genuine 3D localization. The paper does not report residual depth errors after per-image scale correction or an explicit IoU threshold for Table 1, so the accuracy of the pseudo labels is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes V-MIND, a monocular indoor 3D object detector that augments the Cube R-CNN architecture with pseudo 3D training data lifted from the large-vocabulary 2D dataset LVIS. Lifting uses an off-the-shelf metric depth estimator (ZoeDepth) and a camera intrinsic predictor (Wild Camera), followed by point-cloud denoising, per-category point thresholds, and oriented 3D box fitting. The detector is trained jointly on Omni3DIN and the lifted boxes using three additional components: a CLIP embedding classifier, a self-calibration loss that applies a per-image scalar scale to predicted box centers when supervising on pseudo boxes, and an ambiguity loss for new classes that are present in images of the 3D dataset but unannotated there. On Omni3DIN, the method reports AP2D 12.98 vs. 9.64 and AP3D 8.52 vs. 7.52 over Cube R-CNN, with nonzero APnew3D 2.66 for new classes.","tokens_in":12863,"tokens_out":4527,"duration_ms":41271,"significance":"If the reported results hold, the paper offers a practical recipe for expanding the class vocabulary of monocular 3D detectors without additional 3D annotation, which is a meaningful step toward scalable indoor 3D perception. The paper has several strengths: the ablations in Table 3 isolate the contributions of the two proposed losses, the pseudo-box quality is explicitly evaluated on ScanNet200 (Table 1), and the per-category filtering thresholds are derived from ScanNet200 and CLIP embeddings rather than from the target Omni3D evaluation set, which reduces circularity. The CLIP-based classification is well motivated for the long-tailed class distribution. However, the strength of the empirical claims is tempered by the low reported pseudo-box precision (29.72%), the training-only use of the self-calibration network, and the absence of comparisons to state-of-the-art monocular 3D detectors other than Cube R-CNN variants.","major_comments":[{"comment":"The pseudo-box evaluation on ScanNet200 reports precision 29.72% and recall 16.53% without stating the matching protocol: it is unclear whether these numbers use 3D IoU, which IoU threshold (if any), and how predicted boxes are matched to ground-truth boxes. Since new classes receive all of their 3D supervision from these pseudo boxes (Section 3.1, Table 3), this omission is load-bearing. Please specify the protocol and report precision/recall at several 3D IoU thresholds (e.g., 0.1, 0.25, 0.5), ideally per class for the 38 new classes, so the reader can judge how many pseudo boxes are usable for 3D localization rather than only for 2D region supervision.","section":"§4.2, Table 1"},{"comment":"The self-calibration mechanism applies a single per-image scalar exp(s) only to the predicted box center x; it cannot correct spatially varying depth errors, errors in box dimensions w,h,l, or orientation R(p). Moreover, the text states that the calibration network is applied only for pseudo 3D data during training, so at inference the detector outputs the uncalibrated x while the training loss optimizes exp(s)x. This creates a potential train/test mismatch, and the concern is directly relevant to the Table 3 ablation where the self-calibration loss raises APnew3D from 1.31 to 2.66. Please clarify whether the calibration network is used at inference, or provide evidence that the training-only use does not harm inference, e.g., by reporting the learned distribution of s, the residual depth error after calibration, or an experiment that applies the calibration network at inference.","section":"§3.3, Eq. (3)"},{"comment":"The paper claims state-of-the-art performance, but the only comparisons are Cube R-CNN and Cube R-CNN with CLIP embeddings. Section 2.2 acknowledges that UniMode outperforms Cube R-CNN on Omni3D, yet no quantitative comparison to UniMode or to any other published monocular 3D detector on Omni3DIN is provided. Please either add comparisons to published Omni3DIN results or soften the claim to 'improves over Cube R-CNN' throughout the abstract and conclusion.","section":"§4.2, Table 2"},{"comment":"The selection of the 38 new classes is described only as 'excluding those with relatively few instances in the LVIS dataset.' The paper does not report which classes are excluded, the LVIS instance counts per new class, or per-class AP. The aggregate APnew3D of 2.66 could be driven by a small number of easy classes. Please provide the full list of new classes, their LVIS instance counts, and per-class AP2D/AP3D, or at least a histogram of per-class AP, so the versatility claim is supported.","section":"§4.1, Original class and New class"},{"comment":"All results appear to be single runs without error bars or multiple seeds. Given that the AP3D improvement is 8.52 vs. 7.52 and APoriginal3D drops by 4.3%, the paper should report variance over at least three seeds or state that the differences are stable across runs; without this, the reader cannot assess whether the headline gains are within noise.","section":"§4.2, Table 2"}],"minor_comments":[{"comment":"The caption contains a typo: 'Quantitaive' should be 'Quantitative'.","section":"Table 1 caption"},{"comment":"The phrase 'state-of-the-art' appears in the abstract and introduction, but the experimental section compares only against Cube R-CNN baselines; this should be reconciled with comment 3 above.","section":"Abstract and §1"},{"comment":"The paper states that Omni3DIN contains 84 classes but only 38 are used as original classes and 38 as new classes, totaling 76; please clarify the relationship between the 84 classes and the 76 classes used in training.","section":"§4.1"},{"comment":"The use of '∞' for relative improvement over a zero baseline is unconventional and potentially misleading; please define the notation more explicitly or report absolute improvements.","section":"§4.2, Table 2"},{"comment":"The ambiguity loss sums probabilities over all new classes plus background for every background-assigned proposal; this may over-encourage new-class probability for proposals that are truly background. Please discuss this trade-off or report an ablation with a limited candidate set.","section":"§3.4, Eq. (5)"},{"comment":"The caption claims the lifted boxes are 'accurate in terms of categories, locations, and orientations,' which is in tension with the 29.72% precision in Table 1; please soften the caption or add quantitative support.","section":"§4.2, Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely problem and the proposed pipeline is sensible, but the assessment hinges on the quality of the pseudo 3D boxes and on the train/test behavior of the self-calibration network. The missing details about the ScanNet200 matching protocol and the training-only calibration are fixable with additional experiments and should be addressed before the paper can be accepted. The lack of comparison with UniMode or other recent Omni3DIN methods is also a significant gap for a 'state-of-the-art' claim, though it can be remedied by rephrasing and adding available published numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper to know: V-MIND lifts 2D instance masks from LVIS into pseudo 3D boxes using off-the-shelf monocular depth and intrinsic predictors, then trains Cube R-CNN with a CLIP classifier, a per-image self-calibration loss, and an ambiguity loss. It reports gains on Omni3D indoor: AP3D from 7.52 to 8.52, AP2D from 9.64 to 12.98, and positive AP for 38 new classes without any new 3D annotations. The core idea is not brand-new—OV-3DET does pseudo-lifting for point-cloud detectors—but applying it to monocular detection and handling the two failure modes (depth error and background ambiguity) is a legitimate step forward. The ambiguity loss is well-motivated: new-class objects in the 3D dataset get labeled as background, and maximizing the union probability of new classes plus background is a clean fix. The ablations back up both proposed losses.\n\nThe soft spots are real but not fatal. The pseudo-box quality is weak: 29.72% precision on ScanNet200 means most lifted boxes are wrong, and the evaluation uses a loose IoU range (0.05–0.50). The self-calibration loss multiplies the box center by a single per-image scale during training only. If the learned scale absorbs a depth bias, the model at inference may not actually predict calibrated coordinates; new classes, trained only on pseudo data, are most exposed. The authors are transparent about the original-class AP3D drop (4.3%), but they don't report residual depth errors or precision at a concrete IoU threshold. The SOTA claim is also not well-supported: they only compare to Cube R-CNN and a CLIP variant, and UniMode is mentioned but not compared. No code, no data, no error bars, and the new-class set is curated by excluding classes with few LVIS instances, which is reasonable but should be justified.\n\nOn balance, the central claim—that monocular 3D detection can be extended to new classes via lifted 2D data—holds up better than the stress-test worries suggest. The method is plausible and the ablations give real evidence. The issues are matters of evidence and comparison, not a load-bearing flaw. This deserves a serious referee; with code, broader comparisons, and tighter analysis of pseudo-box quality, it could be a solid contribution.\n\nRecommendation: send to peer review, ask for major revision.","headline":"V-MIND's lifted-2D-data approach to monocular indoor 3D detection is plausible and worth a serious look; pseudo-box quality and narrow comparisons need work, not rejection.","tokens_in":13437,"tokens_out":3928,"would_cite":true,"duration_ms":34533,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"V-MIND shows that large-scale 2D datasets, lifted into 3D with monocular depth and camera calibration, can train a versatile indoor 3D object detector that detects new classes without any 3D labels.","keywords":["monocular 3D object detection","indoor scene understanding","pseudo-label generation","2D-to-3D lifting","self-calibration loss","ambiguity loss","large-vocabulary detection","CLIP embeddings"],"falsifier":"On ScanNet200, compute the ratio of predicted to ground-truth depth for each lifted object; if the within-image variance of these ratios is comparable to the between-image variance, a single per-image scale cannot absorb the errors, and the pseudo-box signal for new classes would be unreliable.","tokens_in":12407,"feed_emoji":"📦","tokens_out":7725,"duration_ms":58547,"temperature":0.7,"pith_summary":"The paper claims that the scarcity of 3D annotations, not the detector architecture, is the main bottleneck for versatile monocular indoor 3D detection. To remove it, V-MIND converts large-scale 2D instance masks from LVIS into pseudo 3D bounding boxes by lifting pixels with a monocular metric depth estimator and a camera intrinsic predictor. A self-calibration loss lets the detector learn a per-image scale correction for the noisy lifted boxes, and an ambiguity loss prevents the original 3D dataset from penalizing correct detections of classes it never annotated. Trained jointly on real 3D data and the lifted pseudo data, V-MIND improves AP2D from 9.64 to 12.98 and AP3D from 7.52 to 8.52 on Omni3DIN, and it detects 38 new classes that the baseline cannot see at all. This points to a path for scaling 3D detectors to many object classes without additional 3D annotation.","feed_headline":"2D labels alone train a 3D detector for new indoor classes","feed_subtitle":"V-MIND converts LVIS 2D masks into pseudo 3D boxes, raising Omni3D indoor AP2D from 9.64 to 12.98.","key_machinery":"The load-bearing pieces are a pixel lifting pipeline, a self-calibration loss, and an ambiguity loss. The lifting pipeline back-projects each 2D instance mask into a 3D point cloud via $[x,y,z]^\\top = ZK^{-1}[u,v,1]^\\top$ using predicted depth $Z$ and predicted camera intrinsics $K$, then fits a tight oriented 3D bounding box after outlier removal and per-class point-count filtering. The self-calibration loss applies a learnable per-image scale $\\exp(s)$ to the predicted box center before computing Chamfer distance against the pseudo box, regularized by $\\|s\\|_1$, which lets the detector absorb global depth errors during training. The ambiguity loss $-\\log \\sum_{c \\in \\Omega_{new} \\cup \\Omega_{bkg}} \\Pr(c)$ treats the union of new classes and background as one group, avoiding incorrect penalties when the 3D dataset lacks annotations for objects that the 2D data says are real. Classification is performed in a pre-trained CLIP embedding space so semantically related class names share feature structure.","core_discovery":"On the paper's own terms, the discovery is that pseudo 3D boxes lifted from 2D-only annotations are a viable training signal for monocular 3D object detection, provided the detector can correct a per-image scale error and is shielded from missing-annotation ambiguity. On the Omni3D indoor benchmark, adding lifted LVIS data raises overall AP2D from 9.64 to 12.98 and AP3D from 7.52 to 8.52, with new classes reaching AP2D 5.89 and AP3D 2.66 despite having no 3D ground truth. The small drop in original-class AP3D (14.39 versus the baseline 15.04) is the price paid for this broader vocabulary, and the paper argues the trade is favorable overall.","pith_inferences":["The single per-image scale $\\exp(s)$ is a coarse model of depth error; a natural extension would be per-object or spatially varying scale corrections, which could recover some of the 4.3% drop on original classes.","The ambiguity loss is a general remedy for partially labeled detection data and could transfer to 2D open-vocabulary detection where background proposals may contain unannotated novel objects.","The low pseudo-box precision on ScanNet200 (29.72%) suggests that filtering by predicted confidence or by agreement between multiple depth models could yield a higher-quality training set with little recall loss.","Since new-class detection relies entirely on pseudo boxes, the method's ceiling for novel classes is set by the depth and intrinsic models; combining it with self-supervised depth finetuning on the target domain is a testable way to push APnew3D higher."],"forward_implications":["Adding more 2D labeled images directly enlarges the pseudo 3D training set, so the detector should keep improving as 2D data scales up.","Better monocular depth estimators and camera intrinsic predictors would translate into better pseudo boxes and higher AP3D, especially for new classes.","The self-calibration and ambiguity losses are detector-agnostic, so the same recipe could be applied to other monocular 3D detectors such as BEV-based ones.","Classifying in a pre-trained vision-language space becomes more important as the vocabulary grows, and should help with long-tailed and semantically related classes."],"supporting_citations":[{"why":"Supplies the monocular metric depth estimate Z used to lift 2D pixels into 3D point clouds.","marker":"[2]"},{"why":"Predicts camera intrinsics K so the lifting step can back-project pixels into metric camera coordinates.","marker":"[47]"},{"why":"Provides the large-scale 2D instance masks and 1,200+ classes that become pseudo 3D training data.","marker":"[11]"},{"why":"Defines the Omni3D indoor benchmark, the Cube R-CNN baseline, and the 38 original classes used for evaluation.","marker":"[3]"},{"why":"Gives the pre-trained vision-language embedding space in which V-MIND classifies proposals, improving long-tailed and new classes.","marker":"[24]"},{"why":"Supplies ScanNet200 ground-truth boxes and per-category point counts used both to set point thresholds and to validate pseudo-box precision.","marker":"[30]"}],"fun_headline_variants":["2D labels lift indoor 3D detection: AP2D 9.64 to 12.98","V-MIND: pseudo 3D boxes from 2D annotations train versatile detector","Scale error corrected: 2D-only data teaches 3D indoor detection","Indoor 3D detector learns from 2D masks, surpasses baselines","From 2D masks to 3D boxes: versatile indoor detector"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The depth errors in the lifted pseudo boxes are assumed to be approximately correctable by a single per-image scale, so that after calibration the boxes are a trustworthy training signal for classes that have no real 3D labels.","fun_headline_variants_meta":{"raw":{"variants":["2D labels lift indoor 3D detection: AP2D 9.64 to 12.98","V-MIND: pseudo 3D boxes from 2D annotations train versatile detector","Scale error corrected: 2D-only data teaches 3D indoor detection","Indoor 3D detector learns from 2D masks, surpasses baselines","From 2D masks to 3D boxes: versatile indoor detector"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000742,"raw_usage":{"total_tokens":3323,"prompt_tokens":972,"completion_tokens":2351,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":588,"completion_tokens_details":{"reasoning_tokens":2240}},"tokens_in":588,"tokens_out":2351,"duration_ms":14987,"temperature":1.0,"reasoning_tokens":2240,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:57:26.660554+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On ScanNet200, compute the ratio of predicted to ground-truth depth for each lifted object; if the within-image variance of these ratios is comparable to the between-image variance, a single per-image scale cannot absorb the errors, and the pseudo-box signal for new classes would be unreliable.","supporting_citations":[{"cited_title":"Tame a wild camera: in-the-wild monocular camera calibra- tion","cited_arxiv_id":null,"evidence_quote":"Predicts camera intrinsics K so the lifting step can back-project pixels into metric camera coordinates."},{"cited_title":"LVIS: A dataset for large vocabulary instance segmentation","cited_arxiv_id":null,"evidence_quote":"Provides the large-scale 2D instance masks and 1,200+ classes that become pseudo 3D training data."},{"cited_title":"Omni3d: A large benchmark and model for 3d object detection in the wild","cited_arxiv_id":null,"evidence_quote":"Defines the Omni3D indoor benchmark, the Cube R-CNN baseline, and the 38 original classes used for evaluation."},{"cited_title":"Language- grounded indoor 3d semantic segmentation in the wild","cited_arxiv_id":null,"evidence_quote":"Supplies ScanNet200 ground-truth boxes and per-category point counts used both to set point thresholds and to validate pseudo-box precision."}],"review_version":1}