pith. sign in

arxiv: 2603.10963 · v1 · submitted 2026-03-11 · 💻 cs.CV · cs.LG

Pointy - A Lightweight Transformer for Point Cloud Foundation Models

Konrad Szafer , Marek Kraft , Dominik Belter This is my paper

Pith reviewed 2026-05-15 13:24 UTC · model grok-4.3

classification 💻 cs.CV cs.LG
keywords point cloudtransformerfoundation modellightweight architecturereplication study3D data processingtokenizer-free modelefficient training
0
0 comments X

The pith

A lightweight transformer trained only on 39k point clouds outperforms larger foundation models trained on over 200k samples.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper presents Pointy, a lightweight transformer architecture for point cloud data that avoids reliance on cross-modal supervision from images or text. It demonstrates that this design, when trained on a modest set of 39,000 point clouds, can exceed the performance of several larger models trained on more than 200,000 samples and approach results from models exposed to over a million mixed samples. The authors support this through a replication study that standardizes training regimes and benchmarks across architectures to isolate the effects of design choices. This setup shows that simple, tokenizer-free backbones can compete with more complex or data-intensive strategies.

Core claim

Pointy is a lightweight transformer-based point cloud architecture trained exclusively on 39k samples that outperforms several larger foundation models trained on over 200k samples and approaches state-of-the-art performance from models trained on over a million point clouds, images, and text samples, with a replication study confirming that architectural choices and curated training contribute to these results.

What carries the argument

The lightweight transformer backbone with a tokenizer-free design that processes point clouds directly under a standardized training regime.

If this is right

  • Simple backbones can deliver competitive results to more complex or data-rich strategies in point cloud tasks.
  • Architectural choices can be isolated and compared transparently when training regimes are standardized.
  • Tokenizer-free designs reduce the need for extensive cross-modal data in foundation model training.
  • Carefully curated small-scale training can narrow the gap to models trained on much larger mixed datasets.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Resource-limited settings could adopt similar lightweight designs to reduce compute demands for 3D model development.
  • The results invite direct tests of whether scaling data always outperforms refining architecture in point cloud domains.
  • Similar standardization efforts might clarify trade-offs in other 3D vision benchmarks beyond the current study.

Load-bearing premise

The replication study standardizes training and evaluation fairly across architectures without introducing biases in data preprocessing, tokenization, or metric computation.

What would settle it

Retraining the larger comparison models on the identical 39k point cloud dataset and finding that they still outperform Pointy under the same evaluation protocol would disprove the architecture's efficiency advantage.

Figures

Figures reproduced from arXiv: 2603.10963 by Dominik Belter, Konrad Szafer, Marek Kraft.

Figure 1
Figure 1. Figure 1: Architecture Pointy – transformer backbone for po [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Training dynamics of different transformer-based models on ModelNet40, ScanObjectNN, and Objaverse-LVIS datasets. The plots show overall classification accuracy (%) versus training epochs for our proposed small model compared to existing approaches: PCT [8], PointMAE [18], and two variants of PointTransformer [31] and [5]. Our method demonstrates faster convergence across all datasets while achieving compe… view at source ↗
Figure 3
Figure 3. Figure 3: Classification accuracy on ModelNet40 as a function of input point cloud size. Models were trained for 30 epochs under identical conditions, with re￾sults showing the peak accuracy achieved. While PCT demonstrates superior performance in the 256-1024 point range, our architecture achieves competitive results and attains 89.3% for 2048 points. 4.4.2 Pre-training Next, we evaluate only transformer-based arch… view at source ↗
read the original abstract

Foundation models for point cloud data have recently grown in capability, often leveraging extensive representation learning from language or vision. In this work, we take a more controlled approach by introducing a lightweight transformer-based point cloud architecture. In contrast to the heavy reliance on cross-modal supervision, our model is trained only on 39k point clouds - yet it outperforms several larger foundation models trained on over 200k training samples. Interestingly, our method approaches state-of-the-art results from models that have seen over a million point clouds, images, and text samples, demonstrating the value of a carefully curated training setup and architecture. To ensure rigorous evaluation, we conduct a comprehensive replication study that standardizes the training regime and benchmarks across multiple point cloud architectures. This unified experimental framework isolates the impact of architectural choices, allowing for transparent comparisons and highlighting the benefits of our design and other tokenizer-free architectures. Our results show that simple backbones can deliver competitive results to more complex or data-rich strategies. The implementation, including code, pre-trained models, and training protocols, is available at https://github.com/KonradSzafer/Pointy.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces Pointy, a lightweight tokenizer-free transformer for point cloud foundation models. Trained only on 39k point clouds, it claims to outperform several larger foundation models trained on over 200k samples and approach state-of-the-art results from models pretrained on over a million multimodal (point cloud, image, text) samples. These results are supported by a replication study that standardizes training regimes and benchmarks across multiple point cloud architectures to isolate architectural effects, with code, pretrained models, and protocols released at a GitHub repository.

Significance. If the replication study fairly isolates architecture and data curation effects, the work provides evidence that simple, efficient backbones can deliver competitive point cloud performance without massive scale or cross-modal supervision. This has potential implications for resource-efficient foundation models in 3D vision. The release of full implementation details strengthens the contribution by enabling direct reproducibility.

major comments (2)
  1. [Replication study] Replication study section: The manuscript states that training and evaluation are standardized across architectures but provides no concrete protocol in the text for key steps such as point sampling density, normalization, augmentation sequences, voxelization/FPS strategies, or exact metric implementations. This detail is load-bearing for the central claim, as any inadvertent difference in preprocessing could bias comparisons and undermine attribution of performance gains to the proposed lightweight design versus data or protocol choices.
  2. [Results] Results and experimental setup: While the abstract and main claims reference outperforming models trained on 200k+ samples and approaching million-scale SOTA, the manuscript does not report the precise data splits, baseline re-implementation hyperparameters, or verification that the compared models were retrained under identical conditions within the paper itself (relying instead on the external repository). This reduces the ability to assess the claim independently from the provided code.
minor comments (2)
  1. [Abstract] The abstract mentions 'comprehensive replication study' but could more explicitly list the architectures included in the comparison for immediate clarity.
  2. [Method] Notation for model components (e.g., tokenization-free aspects) should be defined on first use to aid readers unfamiliar with point cloud transformers.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on the replication study and experimental details. We address each major comment below and will revise the manuscript to improve self-containment of the protocols while preserving the open-source repository for full reproducibility.

read point-by-point responses

  1. Referee: [Replication study] Replication study section: The manuscript states that training and evaluation are standardized across architectures but provides no concrete protocol in the text for key steps such as point sampling density, normalization, augmentation sequences, voxelization/FPS strategies, or exact metric implementations. This detail is load-bearing for the central claim, as any inadvertent difference in preprocessing could bias comparisons and undermine attribution of performance gains to the proposed lightweight design versus data or protocol choices.

    Authors: We agree that explicit protocol details belong in the manuscript text. In the revised version we will insert a concise subsection under Experimental Setup that specifies: sampling to 1024 points via FPS, unit-sphere normalization, the exact augmentation sequence (random rotation, scaling [0.8,1.2], Gaussian jitter), absence of voxelization, and standard metric implementations matching the cited benchmarks. This addition will allow readers to verify fairness directly from the paper. revision: yes

  2. Referee: [Results] Results and experimental setup: While the abstract and main claims reference outperforming models trained on 200k+ samples and approaching million-scale SOTA, the manuscript does not report the precise data splits, baseline re-implementation hyperparameters, or verification that the compared models were retrained under identical conditions within the paper itself (relying instead on the external repository). This reduces the ability to assess the claim independently from the provided code.

    Authors: We acknowledge the value of greater self-containment. The revision will add a paragraph in the Results section listing the precise data splits (39k samples drawn from ShapeNet/ModelNet with documented train/val/test partitions), the shared hyperparameters used for all re-implementations (AdamW, lr=1e-4, 300 epochs, batch size 32), and explicit confirmation that baselines were retrained from scratch under the identical regime described in the repository. These details will be summarized in the text with the full code remaining available for verification. revision: yes

Circularity Check

0 steps flagged

No significant circularity in empirical performance claims

full rationale

The paper advances empirical claims about a lightweight transformer trained on 39k point clouds outperforming larger models via a standardized replication study. No derivation chain, equations, or first-principles predictions are present that reduce to fitted inputs or self-citations by construction. The evaluation relies on experimental protocols and released code, remaining self-contained against external benchmarks without any load-bearing self-referential steps.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The central claim rests on empirical comparisons under a standardized training regime; no explicit free parameters or invented entities are described beyond standard model hyperparameters.

free parameters (1)
  • Training hyperparameters
    Standard hyperparameters for transformer training tuned on the 39k point cloud dataset.
axioms (1)
  • domain assumption Point clouds are permutation-invariant sets
    Standard assumption in point cloud processing models.

pith-pipeline@v0.9.0 · 5495 in / 1105 out tokens · 48741 ms · 2026-05-15T13:24:32.884366+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

33 extracted references · 33 canonical work pages · 1 internal anchor

  1. [1]

    La nguage models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D K aplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. La nguage models are few-shot learners. Advances in neural information processing systems , 33:1877–1901, 2020

    work page 1901

  2. [2]

    Objave rse: A universe of annotated 3d objects

    Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Osca r Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objave rse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision a nd Pattern Recognition, pages 13142–13153, 2023

    work page 2023

  3. [3]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    Jacob Devlin. Bert: Pre-training of deep bidirectional transfor mers for language understanding. arXiv preprint arXiv:1810.04805, 2018

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  4. [4]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weis senborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold , Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, IC LR 202...

    work page 2021

  5. [5]

    OpenReview.net, 2021

    work page 2021

  6. [6]

    Point transf ormer

    Nico Engel, Vasileios Belagiannis, and Klaus Dietmayer. Point transf ormer. IEEE access , 9:134826– 134840, 2021

    work page 2021

  7. [7]

    Moment: A family of open time-series foundation models

    Mononito Goswami, Konrad Szafer, Arjun Choudhry, Yifu Cai, Sh uo Li, and Artur Dubrawski. Moment: A family of open time-series foundation models. In International Conference on Machine Learning, 2024

    work page 2024

  8. [8]

    3d semantic segmentation with submanifold sparse convolutional networks

    Benjamin Graham, Martin Engelcke, and Laurens van der Maaten . 3d semantic segmentation with submanifold sparse convolutional networks. CVPR, 2018

    work page 2018

  9. [9]

    Pct: Point cloud transformer

    Meng-Hao Guo, Jun-Xiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R Martin, and Shi-Min Hu. Pct: Point cloud transformer. Computational Visual Media , 7:187–199, 2021

    work page 2021

  10. [10]

    Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following

    Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng M a, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning poin t cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint arXiv:2309.00615 , 2023

    work page arXiv 2023

  11. [11]

    Delving de ep into rectifiers: Surpassing human-level performance on imagenet classification

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving de ep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision , pages 1026–1034, 2015

    work page 2015

  12. [12]

    Deep res idual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep res idual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pa ttern recognition, pages 770–778, 2016

    work page 2016

  13. [13]

    Uni3d-llm: Unifying point cloud perception, generation and editing with large language models.arXiv preprint arXiv:2402.03327, 2024

    Dingning Liu, Xiaoshui Huang, Yuenan Hou, Zhihui Wang, Zhenfe i Yin, Yongshun Gong, Peng Gao, and Wanli Ouyang. Uni3d-llm: Unifying point cloud perception, genera tion and editing with large language models. arXiv preprint arXiv:2402.03327 , 2024

    work page arXiv 2024

  14. [14]

    Openshape: Scaling up 3d shape representation towards open-world understanding

    Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizh ong Han, Hong Cai, Fatih Porikli, and Hao Su. Openshape: Scaling up 3d shape representation towards open-world understanding. Advances in neural information processing systems , 36, 2024

    work page 2024

  15. [15]

    Swin transformer: Hierarchical vision transformer using shifted windo ws

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, St ephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windo ws. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012–10022, 2021

    work page 2021

  16. [16]

    Rethinking network design and local ge- ometry in point cloud: A simple residual MLP framework

    Xu Ma, Can Qin, Haoxuan You, Haoxi Ran, and Yun Fu. Rethinking network design and local ge- ometry in point cloud: A simple residual MLP framework. In International Conference on Learning Representations, 2022. 9

    work page 2022

  17. [17]

    Opendlign: Ope n-world point cloud understanding with depth-aligned images

    Ye Mao, Junpeng Jing, and Krystian Mikolajczyk. Opendlign: Ope n-world point cloud understanding with depth-aligned images. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

    work page 2024

  18. [18]

    Vo, and Marc Szafraniec et al

    Maxime Oquab, Timoth´ ee Darcet, Th´ eo Moutakanni, Huy V. Vo, and Marc Szafraniec et al. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024. Featured Certification

    work page 2024

  19. [19]

    Masked au- toencoders for point cloud self-supervised learning

    Yatian Pang, Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong T ian, and Li Yuan. Masked au- toencoders for point cloud self-supervised learning. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceed ings, Part II , pages 604–621. Springer, 2022

    work page 2022

  20. [20]

    Pointnet : Deep learning on point sets for 3d classification and segmentation

    Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet : Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017

    work page 2017

  21. [21]

    Poin tnet++: Deep hierarchical feature learning on point sets in a metric space

    Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Poin tnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems , 30, 2017

    work page 2017

  22. [22]

    Contrast with reconstruct: Contrastive 3d representation learning guided by g enerative pretraining

    Zekun Qi, Runpei Dong, Guofan Fan, Zheng Ge, Xiangyu Zhang, Kaisheng Ma, and Li Yi. Contrast with reconstruct: Contrastive 3d representation learning guided by g enerative pretraining. In International Conference on Machine Learning , pages 28223–28243. PMLR, 2023

    work page 2023

  23. [23]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019

    work page 2019

  24. [24]

    Revisiting point cloud classification: A new benchmark dataset and classificatio n model on real-world data

    Mikaela Angelina Uy, Quang-Hieu Pham, Binh-Son Hua, Thanh Nguy en, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classificatio n model on real-world data. In Proceedings of the IEEE/CVF international conference on co mputer vision , pages 1588–1597, 2019

    work page 2019

  25. [25]

    Attention is all you need

    A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems , 2017

    work page 2017

  26. [26]

    Dynamic graph cnn for learning on point clouds

    Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bro nstein, and Justin M Solomon. Dynamic graph cnn for learning on point clouds. ACM Transactions on Graphics (tog) , 38(5):1–12, 2019

    work page 2019

  27. [27]

    3d shapenets: A deep representation for volumetric shapes

    Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zh ang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes . In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1912–1920, 2015

    work page 1912

  28. [28]

    Pointllm: Empowering large language models to understand point clouds

    Runsen Xu, Xiaolong Wang, Tai Wang, Yilun Chen, Jiangmiao Pang, and Dahua Lin. Pointllm: Empowering large language models to understand point clouds. In European Conference on Computer Vision, pages 131–147. Springer, 2024

    work page 2024

  29. [29]

    Ulip-2: Towards scala ble multimodal pre-training for 3d understanding

    Le Xue, Ning Yu, Shu Zhang, Artemis Panagopoulou, Junnan Li, R oberto Mart ´ ın-Mart ´ ın, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, et al. Ulip-2: Towards scala ble multimodal pre-training for 3d understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision a nd Pattern Recognition, pages 27091–27101, 2024

    work page 2024

  30. [30]

    Point-bert: Pre-training 3d point cloud transformers with masked point modeling

    Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and J iwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 19313–19322, 2022

    work page 2022

  31. [31]

    Tamm: Triada pter multi-modal learning for 3d shape understanding

    Zhihao Zhang, Shengcao Cao, and Yu-Xiong Wang. Tamm: Triada pter multi-modal learning for 3d shape understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision a nd Pattern Recognition (CVPR), pages 21413–21423, June 2024

    work page 2024

  32. [32]

    Point transformer

    Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen K oltun. Point transformer. In Proceedings of the IEEE/CVF international conference on co mputer vision , pages 16259–16268, 2021

    work page 2021

  33. [33]

    Uni3d: Exploring unified 3d representation at scale

    Junsheng Zhou, Jinsheng Wang, Baorui Ma, Yu-Shen Liu, Tieju n Huang, and Xinlong Wang. Uni3d: Exploring unified 3d representation at scale. In International Conference on Learning Representations (ICLR), 2024. 10

    work page 2024