Pith. sign in

REVIEW 3 major objections 5 minor 15 references

ObitoNet: Multimodal High-Resolution Point Cloud Reconstruction

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Cross-attention fusion of image tokens with point-cloud tokens reconstructs point clouds at Chamfer 1.36, undercutting PointMAE's 1.53.

desk verdict The core idea is plausible but nothing in the experiments supports the headline claim; the CLIP-based pairing is unvalidated and Table 4 is the only evidence. read the letter →

arxiv 2412.18775 v1 pith:QWMNZRHT submitted 2024-12-25 cs.CV cs.AIcs.LG

Apoorv Thapliyal , Vinay Lanka , Swathi Baskaran This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords pointcloudreconstructionmultimodalfusioncross-attentionVisionTransformerfarthestsamplingK-nearestneighborsChamferdistanceTanksandTemples
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ObitoNet claims that a transformer-based point cloud reconstructor performs better when it can also see an RGB image of the same scene, with point cloud tokens attending to image tokens through cross-attention. On paired scenes from the Tanks and Temples dataset, the paper reports Chamfer distances of 1.36 for the base model and 1.20 for the large model, compared with 1.53 for the point-only PointMAE baseline. If the claim is right, it means sparse or noisy 3D scans can be densified and completed using ordinary images as guidance, which are cheaper to acquire than dense scans. The paper further claims that a three-stage training schedule, point-only completion followed by image-token adaptation and then joint fine-tuning, keeps each component reusable for downstream tasks.

What carries the argument

The load-bearing object is the cross-attention module: point cloud tokens become the queries $Q$, image tokens become the keys $K$ and values $V$, so each geometric token retrieves semantic context from the image to fill missing geometry. The point cloud tokens are formed by FPS anchor selection and KNN grouping into clusters $N \in \mathbb{R}^{B \times G \times M \times 3}$, then projected to token dimension; the image tokens are ViT patch embeddings with the CLS token removed. The fused token stream is refined by a transformer decoder with residual connections, and the reconstruction head maps features to 3D coordinates with two 1D convolutional layers and ReLU. The three-stage training order is part of the machinery: it first teaches point-only gap filling, then adapts the image encoder to the frozen point-cloud branch, and finally fine-tunes all modules jointly.

What would settle it

Use the Tanks and Temples camera poses to project each training point cloud into its paired RGB image and measure the median reprojection error; if a large share of the CLIP-matched pairs are misaligned, or if retraining with randomly shuffled image-point pairs reproduces the same Chamfer losses, the reported gain is not coming from genuine cross-modal fusion.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that geometry completion can be driven by a cross-attention block in which point cloud tokens act as queries and image tokens act as keys and values. The point cloud tokenizer selects anchor centers with farthest point sampling and groups local neighborhoods with k-nearest neighbors, while the image branch embeds non-overlapping patches with a pretrained Vision Transformer and discards the CLS token. Fused tokens pass through a transformer decoder and a two-layer 1D convolution head that outputs three times as many points as were input. Trained in three stages on CLIP-matched image-point-cloud pairs, the base model reaches Chamfer distance 1.36, the large variant 1.20, and the ViTMAE variant 1.41, against 1.53 for the PointMAE baseline. The paper takes these numbers to show that multimodal fusion improves high-resolution reconstruction over point-cloud-only processing.

Load-bearing premise

The load-bearing premise is that each CLIP-matched image really shows the same scene from the same viewpoint as its paired point cloud; the paper calls the matching threshold predefined but never states it and reports no check of the pairs.

Editorial extensions

If this is right

  • If the reported Chamfer numbers transfer beyond the tested pairs, image-guided cross-attention gives lower reconstruction error than the point-only PointMAE pipeline.
  • The 3x upsampling objective means the same fusion can densify sparse scans, not just reconstruct at the input density.
  • The modular tokenizers and cross-attention block can be reused for tasks the paper lists, including object classification, semantic segmentation, point cloud registration, and multimodal spatial reasoning.
  • The three-stage schedule allows components to be frozen and reused independently, lowering the barrier to adapting the model to new tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The Table 4 comparison is on CLIP-matched Tanks and Temples pairs, not on the ShapeNet benchmark named in the abstract; reproducing the method on ShapeNet or another standard split would test how far the numbers generalize.
  • A direct ablation that feeds random image-point pairs through the same cross-attention architecture would isolate whether the gain comes from genuine geometric fusion or from the point-cloud transformer alone.
  • The CLIP pairing threshold is left unspecified; publishing it and the pairwise similarity distribution would let others judge whether the training set is truly viewpoint-aligned.
  • The same cross-attention design is a natural fit for image-guided completion of partial LiDAR or depth scans in robotics, provided a calibration-based pairing replaces the similarity matching.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ObitoNet, a multimodal point cloud reconstruction framework that fuses Vision Transformer (ViT) image tokens with point cloud tokens derived from Farthest Point Sampling (FPS) and K-Nearest Neighbors (KNN) through a cross-attention module. The authors describe dataset construction on Tanks and Temples, where CLIP is used to pair images with point clouds, and report Chamfer distances for three ObitoNet variants and a PointMAE baseline (Table 4), claiming comparable or better performance. The paper also includes model configuration tables and qualitative visualizations.

Significance. If the central empirical claim were properly supported, the cross-attention fusion of image and point cloud tokens for point cloud completion and upsampling would be a modest but useful contribution to multimodal 3D reconstruction. The paper gives credit for a modular architecture, explicit model configuration tables, a GitHub repository link, and an honest acknowledgment of the POV-alignment difficulty in Section 3.2. However, the evaluation as presented cannot sustain the claim of comparable performance: the CLIP-based pairing is unvalidated, the baseline comparison is minimal, and the experiments lack error bars, dataset splits, and the ShapeNet evaluation promised in the abstract. These are load-bearing weaknesses, not presentation issues.

major comments (3)
  1. [§3.2 / Table 4] The CLIP-based image–point-cloud pairing is load-bearing for the multimodal fusion claim, but it is never validated. Section 3.2 says images and point clouds are paired when a similarity score exceeds 'a predefined threshold' without reporting the threshold, describing the projection method beyond 'projected onto a 2D plane', or checking geometric/pose consistency. Since Section 3.3 uses point cloud tokens as queries and image tokens as keys and values, any misaligned pairs would train the cross-attention module on spurious correspondences. Until the threshold and a validation of POV alignment are provided, the Chamfer values in Table 4 cannot be interpreted as evidence for genuine multimodal reconstruction.
  2. [§4 / Table 4] The central claim of 'comparable performance to state-of-the-art' is unsupported by the experiments. Table 4 reports single Chamfer loss values for ObitoNet/Base, ObitoNet/Large, ObitoNet/ViTMAE, and PointMAE, but gives no error bars, no train/validation/test split, no number of test samples, and no source for the PointMAE number. The abstract claims evaluation on ShapeNet, but Section 4 uses only Tanks and Temples. With a single unquantified baseline, the reported margin (1.36 vs. 1.53) could be noise or a dataset artifact, so the comparison does not meet the standard needed to support the paper's headline claim.
  3. [§3.5 / §4.2] Experimental reproducibility is not met because key training details are missing. The paper gives some architectural parameters (token size, number of tokens, MAE decoder depth) but does not report learning rate, batch size, optimizer, masking ratio, KNN group size M, training epochs for the Base and ViTMAE variants, or how the test set is constructed. Without these details, the reported Chamfer losses are not reproducible, which further weakens the empirical contribution.
minor comments (5)
  1. [References] References [5] and [6] are both assigned arXiv:2111.14819, but [5] is VoxelGridNet and [6] is PointBERT; at least one identifier is incorrect.
  2. [Abstract vs. §4] The abstract states that experimental evaluations were performed on ShapeNet, but Section 4 uses only the Tanks and Temples dataset; the paper should clarify which dataset was actually used.
  3. [Equation (1)] Equation (1) defines a squared L2 Chamfer loss, while the abstract mentions 'Chamfer Distance (L1/L2)'; please clarify whether the loss uses L2 squared, L1, or both.
  4. [Figure 1] Figure 1 is captioned 'PointMAE pointcloud reconstruction sample outputs' but is not referenced in the text and its relationship to the proposed method is not explained.
  5. [Table 2 caption] The caption of Table 2 reads 'The results of this experiment are highlighted in Table 2,' but Table 2 lists model configuration parameters; the results appear in Table 4.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical architecture comparison; its lost function and evaluation metric coincide only in the standard, non-circular sense, and no load-bearing claim reduces to its own inputs.

full rationale

ObitoNet makes no closed-form derivation and invokes no uniqueness theorem or author self-citation chain. The central comparison (Table 4) is an empirical measurement of Chamfer distance after training, and using Chamfer distance as both the training loss and the evaluation metric is standard practice in point-cloud reconstruction; it does not make the reported numbers true by construction. The unvalidated CLIP-based image-point-cloud pairing in Section 3.2 is a reproducibility and soundness concern (the threshold is called 'predefined' but not specified, and no pose-alignment check is reported), not a circular step: the pairing is an input-construction procedure, not a prediction derived from the model. The duplicate arXiv identifier in references [5] and [6] is a citation error, not evidence of circularity. No step in the paper reduces, by the paper's own equations or by self-citation, to its own inputs, so the appropriate circularity score is 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim relies on several domain assumptions: correct CLIP-based pairing, transferability of pretrained image features, and the adequacy of Chamfer distance. Architecture choices such as token count, decoder depth, and KNN group size are hand-picked, with key values unreported. No new physical entities are introduced.

free parameters (4)
  • Number of tokens = 64 (Base, ViTMAE), 256 (Large)
    Hand-chosen architectural capacity; controls the number of point-cloud groups and image patches.
  • MAE decoder depth = 4 (Base, ViTMAE), 12 (Large)
    Hand-chosen depth of the transformer decoder.
  • CLIP pairing threshold = not reported
    Dataset generation pairs images and point clouds when CLIP similarity exceeds a predefined threshold; the threshold value is not specified.
  • KNN group size M = not reported
    The number of points per KNN group is not given, though it determines the granularity of the point cloud tokens.
assumptions (4)
  • domain assumption CLIP similarity scores identify correctly aligned image-point cloud pairs
    Section 3.2 constructs the training set with CLIP matching; wrong alignment would invalidate cross-modal learning.
  • standard math Chamfer distance is a sufficient training objective and evaluation metric
    Chamfer distance (Eq. 1) is used for both training and evaluation, a standard practice in point cloud generation.
  • domain assumption Pretrained ViT and ViTMAE features transfer to 3D reconstruction
    The image encoders are pretrained for other tasks and are assumed to provide useful semantic features for geometry completion.
  • domain assumption FPS and KNN preserve the local geometry needed for reconstruction
    The tokenizer is built on FPS and KNN; the paper provides no evidence about the optimal group size.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ObitoNet: Multimodal High-Resolution Point Cloud Reconstruction." pith.science (2026). https://pith.science/paper/QWMNZRHT

@misc{pith2026241218775,
  author       = {Pith},
  title        = {Pith review of: ObitoNet: Multimodal High-Resolution Point Cloud Reconstruction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QWMNZRHT}},
  note         = {Machine review of arXiv:2412.18775}
}
read the original abstract

ObitoNet employs a Cross Attention mechanism to integrate multimodal inputs, where Vision Transformers (ViT) extract semantic features from images and a point cloud tokenizer processes geometric information using Farthest Point Sampling (FPS) and K Nearest Neighbors (KNN) for spatial structure capture. The learned multimodal features are fed into a transformer-based decoder for high-resolution point cloud reconstruction. This approach leverages the complementary strengths of both modalities rich image features and precise geometric details ensuring robust point cloud generation even in challenging conditions such as sparse or noisy data.

Figures

Figures reproduced from arXiv: 2412.18775 by the authors.

Figure 1
Figure 1. PointMAE pointcloud reconstruction sample outputs [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the proposed pipeline. ObitoNet integrates image features with [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Side-by-side comparison of (a) Point Cloud Visualizations and (b) Cross [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Side-by-side comparison of (a) Point Cloud Visualizations and (b) Cross [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Side-by-side comparison of (a) Point Cloud Visualizations and (b) Cross [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Further ObitoNet/Base model sample outputs. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: Further ObitoNet/ViTMAE model sample outputs. [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: Further ObitoNet/Large model sample outputs. [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 2 canonical work pages

  1. [6]

    PointBERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling,

    H. Yu, J. Tang, X. Liu, X. Song, and S. Li, “PointBERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling,”arXiv preprint, 2021. arXiv:2111.14819

    arXiv 2021

  2. [1]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner,et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale,” International Conference on Learning Representations (ICLR),2021.arXiv:2010.11929

    arXiv 2021

  3. [2]

    Atten- tion is All You Need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,et al., “Atten- tion is All You Need,”Advances in Neural Information Processing Systems (NeurIPS) , pp. 5998–6008, 2017. arXiv:1706.03762

    arXiv 2017

  4. [3]

    PointMAE: Masked Autoencoders for Point Cloud Self-Supervised Learning,

    J. Pang, C. Li, Z. Liu, C. Xu, C. Lin, and Y. Liu, “PointMAE: Masked Autoencoders for Point Cloud Self-Supervised Learning,”Advances in Neural Information Processing 8 Systems (NeurIPS), 2022. arXiv:2203.06604

    arXiv 2022

  5. [4]

    PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation,

    C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation,”Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 652–660, 2017. arXiv:1612.00593

    arXiv 2017

  6. [7]

    PCT: Point Cloud Trans- former,

    M. Guo, J. Cai, Z. Liu, T. Mu, R. R. Martin, and S. Hu, “PCT: Point Cloud Trans- former,” Computers & Graphics , vol. 99, pp. 92–102, 2021. arXiv:2012.09688

    arXiv 2021

  7. [8]

    PU-GAN: A Point Cloud Upsam- pling Adversarial Network,

    R. Li, X. Liu, W. Cao, C. Xu, and Y. Cheng, “PU-GAN: A Point Cloud Upsam- pling Adversarial Network,” Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pp. 7203–7212, 2019. arXiv:1904.10014

    arXiv 2019

  8. [9]

    NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis,

    B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis,”European Conference on Computer Vision (ECCV) , pp. 405–421, 2020. arXiv:2003.08934

    arXiv 2020

Show all 15 references
  1. [10]

    PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space,

    C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space,”Advances in Neural Information Processing Systems (NeurIPS), 2017. arXiv:1706.02413

    2017 arXiv

  2. [11]

    LoFTR: Detector-Free Local Feature Matching with Transformers,

    J. Sun, F. Yang, S. Tan, H. Xue, L. Huang, and K. Xu, “LoFTR: Detector-Free Local Feature Matching with Transformers,”Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR) , pp. 8922–8931, 2021. arXiv:2104.00680

    2021 arXiv

  3. [12]

    KPConv: Flexible and Deformable Convolution for Point Clouds,

    H. Thomas, C. R. Qi, J. E. Deschaud, B. Marcotegui, F. Goulette, and L. J. Guibas, “KPConv: Flexible and Deformable Convolution for Point Clouds,” Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pp. 6411–6420,

  4. [13]

    PCN: Point Completion Net- work,

    W. Yuan, T. Khot, D. Held, C. Mertz, and M. Hebert, “PCN: Point Completion Net- work,” Proceedings of the IEEE International Conference on 3D Vision (3DV) , pp. 728–737, 2018. arXiv:1808.00671

    2018 arXiv

  5. [14]

    DeepViewAgg: Learning View Aggregation for 3D Point Cloud Reconstruction,

    Y. Huang, Y. Zhou, and T. Yi, “DeepViewAgg: Learning View Aggregation for 3D Point Cloud Reconstruction,”Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022. arXiv:2205.03312

    2022 arXiv

  6. [15]

    Multi-View Convolutional Neural Networks for 3D Shape Recognition,

    H. Su, S. Maji, E. Kalogerakis, and E. Learned-Miller, “Multi-View Convolutional Neural Networks for 3D Shape Recognition,” Proceedings of the IEEE International Conference on Computer Vision (ICCV) , pp. 945–953, 2015. arXiv:1505.00880

    2015 arXiv

  7. [16]

    3D Semantic Segmentation with Submanifold Sparse Convolutional Networks,

    B. Graham, M. Engelcke, and L. van der Maaten, “3D Semantic Segmentation with Submanifold Sparse Convolutional Networks,”Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2018. arXiv:1711.10275. 9 10 6 Appendix 6.1 ObitoNet/Base Figure 6: F...

    2018 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.