REVIEW 3 major objections 5 minor 50 references
L2G Auto-encoder: Understanding Point Clouds by Local-to-Global Reconstruction with Hierarchical Self-Attention
T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read Local-to-global reconstruction with hierarchical self-attention lets an unsupervised auto-encoder outperform prior point cloud methods.
desk verdict A promising point-cloud autoencoder architecture that is undermined by an interpolation layer whose constant makes the central local-to-global mechanism numerically inert, plus test-set tuning. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is hierarchical self-attention plus a sequential decoder. Self-attention computes attention weights between feature vectors at each same-level group, point level within each scale, scale level within each region, and region level across the cloud, and concatenates weighted outputs with inputs before MLP and max-pool aggregation. The decoder uses an interpolation layer to spread the global feature to local regions, an LSTM to emit the T=4 scale features as a sequence, skip links from the encoder, and FC layers to output both local areas and the global point cloud.
What would settle it
Retrain L2G-AE under the exact settings of the paper with Eq. (4)'s interpolation term removed (skip-link path only) and with c set to 1; if ModelNet10 accuracy remains 95.37% in both variants, the claimed spatial-interpolation mechanism is not what is carrying the local-to-global reconstruction.
Extended reading notes
Core claim
The paper's central claim is that local-to-global reconstruction is a stronger self-supervision signal for point cloud understanding than global-only reconstruction. Concretely, L2G-AE samples 256 centroids by farthest point sampling, builds four nested kNN scale areas (16, 32, 64, 128 points) around each, encodes point, scale, and region features with self-attention modules, and condenses everything into a 1024-dimensional global feature. The decoder propagates that feature through skip-linked local region features, an LSTM that treats the four scales as a sequence, and fully connected layers to produce both local area point sets and the final point cloud. Training minimizes the sum of local and global Chamfer distances. The authors claim this design is why their unsupervised features beat state-of-the-art methods on classification and retrieval and enable unsupervised upsampling.
Load-bearing premise
The load-bearing premise is that the interpolation in Eq. (4) actually injects spatial distribution information into decoding; with c=$10^{-10}$ and normalized coordinates, that term is numerically zero, so the premise may fail.
Editorial extensions
If this is right
- If the reported numbers hold, unsupervised point cloud features can match or beat supervised feature extractors on classification and retrieval, reducing the need for labeled 3D data.
- The local-to-global reconstruction loss can serve as a pretraining objective: features trained on reconstruction transfer to classification and retrieval without fine-tuning.
- The same trained auto-encoder produces dense local reconstructions that can be downsampled, making deep unsupervised point cloud upsampling possible.
- Hierarchical self-attention at point, scale, and region levels gives a reusable aggregation operator that other point cloud networks could adopt.
Reading between the lines
- If the interpolation term in Eq. (4) is negligible as written, then L2G-AE's decoder is effectively a skip-connected MLP/RNN decoder; the reported gains would then come from the hierarchical self-attention encoder and the multi-scale Chamfer loss, not from the stated spatial interpolation.
- The paper ablates self-attention only on ModelNet10; an editorially suggested extension is the same ablation on ModelNet40 to test whether the attention mechanism's contribution scales with category count.
- Because local areas are nested neighborhoods, the method may transfer to partial or noisy scans without retraining, though the paper does not test this.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes L2G-AE, an unsupervised point-cloud autoencoder that encodes multi-scale neighborhoods around sampled centroids using a hierarchical self-attention mechanism and decodes them with an interpolation layer, an RNN, and fully connected layers to reconstruct both local scale areas and the global point cloud (Eqs. (4)-(11)). The method is evaluated on shape classification on ModelNet10/ModelNet40, shape retrieval on ModelNet10, and 16x point-cloud upsampling, reporting 95.37% and 90.64% unsupervised classification accuracy and a retrieval mAP of 67.81%. The central claim is that the local-to-global reconstruction scheme enables the learned global representation to capture local structure and thereby outperform state-of-the-art unsupervised and even some supervised methods.
Significance. If the proposed local-to-global reconstruction mechanism were properly validated, this would be a solid contribution to unsupervised point-cloud representation learning, with a principled decoder design and evaluation across three tasks. The paper includes useful ablations of the hierarchical self-attention modules (Table 5) and the local/global losses (Table 6), and it provides visualizations of attention at point, scale, and region levels. However, as written the central mechanism is not established: the interpolation constant in Eq. (4) makes the global-to-local path numerically inert, and the evaluation protocol (test-set hyperparameter selection, no error bars, mixed upsampling results) does not support the paper's 'outperforming' claims. These issues are substantial but fixable, so a major revision is appropriate.
major comments (3)
- [Section 3.3, Eq. (4) and Section 4.3] The interpolation layer is the only mechanism by which the global feature g reaches the RNN decoder, yet the constant c=10^-10 makes it numerically inert. After normalizing the input into the unit ball (Section 4.2), ||pi - p0||^2 <= 1 for p0=(0,0,0), so the term c(pi-p0)^2 g has norm at most 10^-10||g||. Concatenated with the unscaled skip-linked encoder features, this term is negligible, and any reconstruction gradient reaching g through Eq. (4) is scaled by 10^-10. Thus the local-to-global reconstruction described in Sections 3.3-3.5 is not exercised as written by the losses in Eqs. (9)-(11); Tables 5 and 6 do not isolate the interpolation term, so the reported gains cannot currently be attributed to that mechanism. Please correct the constant or the equation and add an explicit ablation of the interpolation path (e.g., varying c, or removing the skip link) before claiming that L2G-AE learns local structure through global-feature decoding.
- [Section 4.2 and Tables 1-4, 7-8] Hyperparameters M, C, D, and D_global are selected by comparing test-set accuracies on ModelNet10, and all headline accuracies are single runs without error bars. Under this protocol, the 0.07 percentage-point advantage over LGAN on ModelNet10 (95.37 vs 95.30) and the 0.49-point advantage on ModelNet40 (90.64 vs 90.15) are within likely run-to-run variation, so the claim of 'significantly outperforming' all unsupervised competitors is not established. Please use a held-out validation split for model selection and report means and standard deviations over multiple runs, or clearly state the selection and evaluation protocol if a validation split was used.
- [Section 4.6, Table 9] The narrative claims that L2G-AE outperforms state-of-the-art upsampling methods, but of the ten ModelNet10 classes, PU-Net achieves lower (better) Chamfer distance than L2G-AE in nine classes, and no aggregate mean or significance test is reported. Under the stated metric, the table supports at most a mixed result for an unsupervised method, not the paper's broad 'outperforming' claim, which also appears in the abstract. Please report the mean mCD over all classes, add statistical comparison, and revise the upsampling claim accordingly.
minor comments (5)
- [Section 4.2] The sentence 'With keeping the sampled points M = 384' appears to be a typo; the preceding paragraph selects M = 256 as optimal, so the dimension experiments in Tables 2-4 should presumably keep M = 256.
- [Section 3.4, Eqs. (5)-(7)] The weight notation is inconsistent: Eq. (6) uses W_theta, Eq. (7) uses W_theta_t, and the text refers to W_d. Please unify the notation.
- [Table 8] The caption and header are incomplete: 'The comparison of retrieval in terms of under ModelNet10' does not name the metric; the table should state that the reported quantity is mean Average Precision (mAP).
- [Table 9] The unit '10^-3' appears once in the table header; please make clear that all entries are in those units and also report the mean/aggregate mCD across all classes.
- [Section 2, Related Work] The name 'Point2Seqeuce' is a typo for Point2Sequence.
Circularity Check
No significant circularity: benchmark results are independent of reconstruction training; self-citations are architectural, not load-bearing.
full rationale
The paper's central claims rest on benchmark evaluations: the encoder is trained by self-reconstruction losses (Eqs. 9-11), and the resulting global features are then evaluated by a linear SVM on ModelNet10 and ModelNet40. No parameter is fitted to the classification labels, so the reported accuracies are not forced by construction. The multi-scale establishment and self-attention design draw on prior work including the authors' Point2Sequence, but those citations justify architectural choices rather than the numerical outcomes; the comparison to MAP-VAE and other baselines is external. The interpolation term in Eq. (4) may be numerically questionable because c=10^-10 makes the global-to-local path extremely small, but that is a correctness or implementation concern, not a case where a prediction reduces to its own input by definition. No uniqueness theorem is invoked, no fitted input is renamed as a prediction, and no central derivation is equivalent to its assumptions by construction. Therefore no significant circularity is found.
Assumptions & free parameters
free parameters (7)
- M (number of centroids) =
256
- C (self-attention dimension) =
32
- D (local feature dimension) =
256
- D_global (global feature dimension) =
1024
- c (interpolation constant) =
1e-10
- gamma (loss weight) =
1
- Scale sizes (K_1..K_4) =
16, 32, 64, 128
assumptions (3)
- domain assumption Chamfer distance is a valid reconstruction objective for learning discriminative point cloud features.
- domain assumption Multi-scale kNN neighborhoods around farthest-point-sampled centroids capture the local geometry needed for representation learning.
- ad hoc to paper The T scale areas in a local region can be treated as an ordered sequence for RNN decoding.
Cite this review
Pith. "Pith review of L2G Auto-encoder: Understanding Point Clouds by Local-to-Global Reconstruction with Hierarchical Self-Attention." pith.science (2026). https://pith.science/paper/3WWRDQWC
@misc{pith2026190800720,
author = {Pith},
title = {Pith review of: L2G Auto-encoder: Understanding Point Clouds by Local-to-Global Reconstruction with Hierarchical Self-Attention},
year = {2026},
howpublished = {\url{https://pith.science/paper/3WWRDQWC}},
note = {Machine review of arXiv:1908.00720}
}
read the original abstract
Auto-encoder is an important architecture to understand point clouds in an encoding and decoding procedure of self reconstruction. Current auto-encoder mainly focuses on the learning of global structure by global shape reconstruction, while ignoring the learning of local structures. To resolve this issue, we propose Local-to-Global auto-encoder (L2G-AE) to simultaneously learn the local and global structure of point clouds by local to global reconstruction. Specifically, L2G-AE employs an encoder to encode the geometry information of multiple scales in a local region at the same time. In addition, we introduce a novel hierarchical self-attention mechanism to highlight the important points, scales and regions at different levels in the information aggregation of the encoder. Simultaneously, L2G-AE employs a recurrent neural network (RNN) as decoder to reconstruct a sequence of scales in a local region, based on which the global point cloud is incrementally reconstructed. Our outperforming results in shape classification, retrieval and upsampling show that L2G-AE can understand point clouds better than state-of-the-art methods.
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. 2018. Learning Representations and Generative Models for 3D Point Clouds. In ICML
work page 2018
-
[2]
Serge Belongie, Jitendra Malik, and Jan Puzicha. 2001. Shape Context: A New Descriptor for Shape Matching and Object Recognition. In NeurIPS. 831–837
work page 2001
-
[3]
Haowen Deng, Tolga Birdal, and Slobodan Ilic. 2018. PPF-FoldNet: Unsupervised Learning of Rotation Invariant 3D Local Descriptors. In ECCV
work page 2018
-
[4]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A Large-Scale Hierarchical Image Database. In CVPR
work page 2009
-
[5]
Aleksey Golovinskiy, Vladimir G Kim, and Thomas Funkhouser. 2009. Shape- Based Recognition of 3D Point Clouds in Urban Environments. In ICCV. 2154– 2161
work page 2009
-
[6]
Zhizhong Han, Xinhai Liu, Yu-Shen Liu, and Matthias Zwicker. 2019. Parts4Feature: Learning 3D Global Features from Generally Semantic Parts in Multiple Views. In IJCAI
work page 2019
-
[7]
Zhizhong Han, Zhenbao Liu, Junwei Han, ChiMan Vong, Shuhui Bu, and C.L.P. Chen. 2019. Unsupervised Learning of 3D Local Features from Raw Voxels Based on a Novel Permutation Voxelization Strategy. IEEE Transactions on Cybernetics 49, 2 (2019), 481–494
work page 2019
-
[8]
Zhizhong Han, Zhenbao Liu, Junwei Han, Chi-Man Vong, Shuhui Bu, and C.L.Philip Chen. 2017. Mesh Convolutional Restricted Boltzmann Machines for Unsupervised Learning of Features With Structure Preservation on 3D Meshes. IEEE Transactions on Neural Network and Learning Systems 28, 10 (2017), 2268 – 2281
work page 2017
Show all 50 references
-
[9]
Zhizhong Han, Zhenbao Liu, Junwei Han, Chi-Man Vong, Shuhui Bu, and Xuelong Li. 2016. Unsupervised 3D Local Feature Learning by Circle Convolutional Restricted Boltzmann Machine. IEEE Transactions on Image Processing 25, 11 (2016), 5331–5344
2016
-
[10]
Zhizhong Han, Zhenbao Liu, Chi-Man Vong, Yu-Shen Liu, Shuhui Bu, Junwei Han, and CL Philip Chen. 2017. BoSCC: Bag of Spatial Context Correlations for Spatially Enhanced 3D Shape Representation. IEEE Transactions on Image Processing 26, 8 (2017), 3707–3720
2017
-
[11]
Zhizhong Han, Zhenbao Liu, Chi-Man Vong, Yu-Shen Liu, Shuhui Bu, Junwei Han, and CL Philip Chen. 2018. Deep Spatiality: Unsupervised Learning of Spatially-Enhanced Global and Local 3D Features by Deep Neural Network with Coupled Softmax. IEEE Transactions on Image Processing 2...
2018
-
[12]
Zhizhong Han, Honglei Lu, Zhenbao Liu, Chi-Man Vong, Yu-Shen Liua, Matthias Zwicker, Junwei Han, and CL Philip Chen. 2019. 3D2SeqViews: Aggregating Sequential Views for 3D Global Feature Learning by CNN with Hierarchical Attention Aggregation. IEEE Transactions on Image Proces...
2019
-
[13]
Zhizhong Han, Mingyang Shang, Yu-Shen Liu, and Matthias Zwicker. 2019. View Inter-Prediction GAN: Unsupervised Representation Learning for 3D Shapes by Learning Global Shape Memories to Support Local View Predictions. In AAAI
2019
-
[14]
Zhizhong Han, Mingyang Shang, Zhenbao Liu, Chi-Man Vong, Yu-Shen Liu, Matthias Zwicker, Junwei Han, and CL Philip Chen. 2018. SeqViews2SeqLabels: Learning 3D Global Features via Aggregating Sequential Views by RNN with Attention. IEEE Transactions on Image Processing 28, 2 (20...
2018
-
[15]
Zhizhong Han, Mingyang Shang, Xiyang Wang, Yu-Shen Liu, and Matthias Zwicker. 2019. Yˆ 2Seq2Seq: Cross-Modal Representation Learning for 3D Shape and Text by Joint Reconstruction and Prediction of View and Word Sequences. In AAAI
2019
-
[16]
Zhizhong Han, Xiyang Wang, Yu-Shen Liu, and Matthias Zwicker. 2019. Multi- Angle Point Cloud-VAE: Unsupervised Feature Learning for 3D Point Clouds from Multiple Angles by Joint Self-Reconstruction and Half-to-Half Prediction. In ICCV
2019
-
[17]
Zhizhong Han, Xiyang Wang, Chi-Man Vong, Yu-Shen Liu, Matthias Zwicker, and CL Chen. 2019. 3DViewGraph: Learning Global Features for 3D Shapes from A Graph of Unordered Views with Attention. In IJCAI
2019
-
[18]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In CVPR. 770–778
2016
-
[19]
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural computation 9, 8 (1997), 1735–1780
1997
-
[20]
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger
-
[21]
Roman Klokov and Victor Lempitsky. 2017. Escape from Cells: Deep KD- Networks for the Recognition of 3D Point Cloud Models. In ICCV. 863–872
2017
-
[22]
Hema S Koppula, Abhishek Anand, Thorsten Joachims, and Ashutosh Saxena
-
[23]
Jiaxin Li, Ben M Chen, and Gim Hee Lee. 2018. SO-Net: Self-Organizing Network for Point Cloud Analysis. In CVPR. 9397–9406
2018
-
[24]
Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. 2018. PointCNN: Convolution on X-Transformed Points. In NeurIPS
2018
-
[25]
Yangyan Li, Soeren Pirk, Hao Su, Charles R Qi, and Leonidas J Guibas. 2016. FPNN: Field Probing Neural Networks for 3D Data. In NeurIPS. 307–315
2016
-
[26]
Xinhai Liu, Zhizhong Han, Yu-Shen Liu, and Matthias Zwicker. 2019. Point2Sequence: Learning the Shape Representation of 3D Point Clouds with an Attention-Based Sequence to Sequence Network. In AAAI
2019
-
[27]
Charles R Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J Guibas. 2017. Frustum PointNets for 3D Object Detection from RGB-D Data. In CVPR
2017
-
[28]
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. 2016. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation. In CVPR
2016
-
[29]
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. 2017. PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space. In NeurIPS. 5099–5108
2017
-
[30]
Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger. 2017. OctNet: Learning Deep 3D Representations at High Resolutions. In CVPR, Vol. 3
2017
-
[31]
Yiru Shen, Chen Feng, Yaoqing Yang, and Dong Tian. 2018. Mining Point Cloud Local Structures by Kernel Correlation and Graph Pooling. In CVPR, Vol. 4
2018
-
[32]
M Simon, S Milz, K Amende, and HM Gross. 2018. Complex-YOLO: Real-Time 3D Object Detection on Point Clouds. arXiv preprint arXiv:1803.06199 (2018)
2018 arXiv
-
[33]
David Stutz and Andreas Geiger. 2018. Learning 3D Shape Completion from Laser Scan Data with Weak Supervision. In CVPR
2018
-
[34]
Yuan Wang, Tianyue Shi, Peng Yun, Lei Tai, and Ming Liu. 2018. PointSeg: Real- Time Semantic Segmentation Based on 3D LiDAR Point Cloud. In arXiv preprint arXiv:1807.06288
2018 arXiv
-
[35]
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. 2018. Dynamic Graph CNN for Learning on Point Clouds. In arXiv preprint arXiv:1801.07829
2018 arXiv
-
[36]
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 2015. 3D ShapeNets: A Deep Representation for Volumetric Shapes. In CVPR. 1912–1920
2015
-
[37]
Saining Xie, Sainan Liu, Zeyu Chen, and Zhuowen Tu. 2018. Attentional ShapeContextNet for Point Cloud Recognition. In CVPR. 4606–4615
2018
-
[38]
Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, and Yu Qiao. 2018. SpiderCNN: Deep Learning on Point Sets with Parameterized Convolutional Filters. InECCV
2018
-
[39]
Bin Yang, Wenjie Luo, and Raquel Urtasun. 2018. PIXOR: Real-Time 3D Object Detection from Point Clouds. In CVPR
2018
-
[40]
Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. 2018. FoldingNet: Point Cloud Auto-Encoder via Deep Grid Deformation. In CVPR
2018
-
[41]
Wang Yifan, Shihao Wu, Hui Huang, Daniel Cohen-Or, and Olga Sorkine- Hornung. 2019. Patch-Base Progressive 3D Point Set Upsampling. In CVPR
2019
-
[42]
Haoxuan You, Yifan Feng, Rongrong Ji, and Yue Gao. 2018. PVNet: A Joint Convolutional Network of Point Cloud and Multi-View for 3D Shape Recognition. In ACM Multimedia Conference
2018
-
[44]
Lequan Yu, Xianzhi Li, Chi-Wing Fu, Daniel Cohen-Or, and Pheng-Ann Heng
-
[45]
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. 2018. Self- Attention Generative Adversarial Networks. In NeurIPS
2018
-
[46]
Yongheng Zhao, Tolga Birdal, Haowen Deng, and Federico Tombari. 2019. 3D Point-Capsule Networks. In CVPR
2019
-
[47]
Yin Zhou and Oncel Tuzel. 2017. VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection. In CVPR
2017
-
[48]
PU-Net: Point Cloud Upsampling Network. In CVPR
-
[2011]
In NeurIPS
Semantic Labeling of 3D Point Clouds for Indoor Scenes. In NeurIPS. 244–252
-
[2017]
Densely Connected Convolutional Networks. In CVPR. 4700–4708
-
[2018]
EC-Net: An Edge-Aware Point Set Consolidation Network. In ECCV
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.