{"id":"036499d3-4fe9-4837-98b5-eb37493af334","arxiv_id":"2412.11540","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SP2T adds a sparse proxy attention stream to a point transformer, improving 3D segmentation and detection accuracy on indoor and outdoor benchmarks over PTv3.","lead":"This paper introduces SP2T, a point cloud transformer that pairs local point attention with a small set of spatially sampled proxy points connected sparsely by vertex-based associations, and reports higher accuracy than prior proxy-based and mainstream transformers on ScanNet, S3DIS, nuScenes, SemanticKITTI, and Waymo benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Softmax normalization in SPA appears to sum over proxy keys rather than per-query point keys, which would invert the intended attention and weaken the stated global-fusion mechanism.","rationale":"The paper's central contribution is sparse proxy attention with a global receptive field, and its SOTA claims on ScanNet, ScanNet200, S3DIS, nuScenes, SemanticKITTI, and Waymo are all attributed to this mechanism. Equations (1)-(3) as written do not implement a softmax over each query's associations: the denominator in Eq. (2) is over associations sharing the same proxy, and Eq. (3)'s loop variable and output index are both proxy indices. If these equations reflect the implementation, then the operation is a proxy-normalized pooling rather than per-query attention, and the interpretation of the ablation gains, the attention visualizations, and the claimed global receptive field all change. The reader's verdict flagged a possible sign issue and conditional acceptance. The sign issue is real, but my reading goes further: the normalization is a load-bearing structural concern, not just a notational slip. However, I cannot rule out that Eq. (2) and Eq. (3) are merely miswritten while the code implements the correct per-query softmax; the paper's own ablations and visualizations suggest the intended operation is attention. Thus the correct verdict is not outright rejection but UNVERDICTED: the stated claims cannot be evaluated until the released code or an author clarification resolves this indexing question. A code release and a tiny normalization test would settle it immediately.","tokens_in":20340,"tokens_out":2027,"duration_ms":17215,"concrete_test":"Inspect the released SPA implementation (or, if code is not yet released, ask the authors for the exact index tensors for Eq. 2 and Eq. 3) and verify that for each query point, the normalization denominator sums over associations of that query, not over associations of the same proxy. Concretely, run a tiny unit test with N=2 points, M=1 proxy, and k=2 associations, and check that each point's two W values sum to 1. If instead the proxy's four W values sum to 1, the equations as written are implemented. Additionally, recompute the Table 5 row 10 result with a corrected per-query softmax; if the mIoU drops by more than 0.5, the reported global-fusion gain is partly an artifact of the normalization scheme.","verdict_should_be":"UNVERDICTED","load_bearing_attack":"The central claim that SP2T outperforms PTv3 via a global receptive field rests on SPA acting as a proper attention from points to proxies and back. Equation (1) defines S_i^h for each association group, and Eq. (2) defines the sparse-softmax weight as W_i^h = S_i^h / sum_{j: aspx_j = aspx_i} S_j^h. The denominator is indexed by j, aspx_j, and aspx_i, so it sums over all associations pointing to the same proxy, not over associations sharing the same query point. For a point-to-proxy pass, aspx_j is the proxy index of association j, so the normalization is over keys (proxies) rather than over queries (points). This is not a valid per-query softmax: weights are not normalized per query, and a query point's output can receive contributions whose weights sum to an arbitrary value. Furthermore, Eq. (3) has the same proxy index as the loop variable, oh_i = sum_{j: aspx_j = i} v_{aspt_j} W_j^i, which would index the output by proxy i while the text says it is the point output. The sign issue is a possible expression typo, but if the implementation follows the stated equations, the attention is not a softmax over each point's associations, and the claimed global fusion mechanism is not the one being evaluated. This is the most load-bearing concern because the headline SOTA numbers, and the ablation rows 9–10 in Table 5 that attribute gains to attention-based PPI with TRB, would be explained by a differently normalized operation; the novelty of SPA as a sparse proxy attention mechanism would be undermined. The reader's weakest_assumption concerns vertex-based association in outdoor scenes, which is a plausible geometric concern but is secondary to an indexing error that affects even the indoor ScanNet results.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SP2T, a dual-stream point transformer that adds a local-proxy stream to a point stream (PTv3-style local fusion) for indoor and outdoor 3D semantic segmentation, instance segmentation, and object detection. The main novelty is Sparse Proxy Attention (SPA), which connects each point to eight grid-vertex proxies and computes sparse attention with a map-reduce scheme, plus a Table-Based Relative Bias (TRB) term. The paper reports state-of-the-art or competitive results on ScanNet, ScanNet200, S3DIS, SemanticKITTI, nuScenes, and Waymo, with ablations on ScanNet validation supporting the proxy count, association count, sampling scheme, empty-proxy treatment, and TRB variants.","tokens_in":1726,"tokens_out":2128,"duration_ms":69599,"significance":"If correct, the core claim is important: a local-proxy dual-stream design can provide a global receptive field at O(k n d + m^2 d) cost (k=8, m << n), improving over strong PTv3 baselines by 1.2-2.0 mIoU on indoor segmentation and 1.6-1.8 mAPH on Waymo detection. The paper includes several strengths: a parameter-matched PTv3 comparison (Table 5 rows 1-2), a latency breakdown (Fig. 7), a discussion of test-time augmentation inconsistencies with a same-TTA comparison (Table 8), and a transfer study replacing PTv3 with MinkUNet (Table 7). These features make the empirical claims more credible than a pure SOTA table. However, the central methodological description in Sec. 3.3 has an indexing/normalization inconsistency that must be resolved before the architecture's mechanism can be evaluated, and the reported results are single-run numbers with hyperparameters selected on the same validation sets that are reported.","major_comments":[{"comment":"The equations defining SPA are internally inconsistent, and the described operation is not a per-query sparse softmax as the text claims. In Eq. (2), W_i^h is normalized by summing over j with aspx_j = aspx_i, i.e., over all associations sharing the same proxy (key-side normalization), not over associations sharing the same point (query-side normalization). In Eq. (3), the output o_i^h is indexed by proxy i and aggregates point values v_{aspt_j}, which describes a point-to-proxy aggregation; but the surrounding text says the example transfers features from proxy to point, and Eq. (1) puts the point feature in the query (q_{aspt_i}) and the proxy feature in the key (k_{aspx_i}). Because of these swaps, the paper does not currently specify a consistent attention direction. If the implementation follows Eqs. (1)-(3), the operation is not a valid per-query softmax attention and the gains attributed to attn-based PPI w/ TRB in Table 5 rows 7-10 would not support the stated global-fusion mechanism. Please correct the indexing, define the normalization group (per query point or per query proxy), and show the associated map-reduce pseudocode.","section":"Sec. 3.3, Eqs. (1)-(3)"},{"comment":"All structural and hyperparameter ablations - proxy count, association count, sampling method, empty-proxy handling, TRB scales, and temperature - are tuned and reported on ScanNet validation, and the main claims rest on single runs without error bars or repeated seeds. Several reported differences are small relative to typical run-to-run variance: e.g., Table 5 rows 5 and 7 differ by 0.13 mIoU, and Table 6 rows 1-4 span 0.67 mIoU across proxy counts. Because the headline SOTA claims and the ablations are single numbers, I cannot assess whether the improvements are statistically reliable. Please provide multiple seeds with mean and standard deviation for the main results and key ablations, or at minimum state the number of runs; if retraining is infeasible, report at least the held-out test numbers for the configurations compared in Table 5 and Table 10.","section":"Sec. 4.2, Tables 5, 6, and 10"},{"comment":"The TRB formula appears inconsistent with the standard relative-bias mechanism described in the text. Eq. (5) writes S_i^h = exp(q·k/√d) + TRB^h(x), which adds the bias after the exponential, so the softmax denominator sums terms of the form exp(score) + bias. This is not the usual logit-bias form exp(q·k/√d + b) that Swin-style relative bias uses, and it changes the normalization statistics. If the implementation uses logit addition, please rewrite Eq. (5) as S_i^h = exp(q·k/√d + TRB^h(x)); if the additive-after-exponential form is intentional, justify why it behaves as a relative bias and explain the effect on the softmax.","section":"Sec. 3.3, Eq. (5)"}],"minor_comments":[{"comment":"The E.P. column is not filled with explicit True/False labels, so the reader cannot tell which row corresponds to with empty proxies and which to without empty proxies despite the text in Sec. 4.2 describing both conditions.","section":"Table 6, rows 15-16"},{"comment":"The table title says Indoor instance segmentation, but the content and metrics (mIoU on ScanNet/ScanNet200 semantic segmentation) describe semantic segmentation; please retitle it.","section":"Table 8"},{"comment":"The caption of Fig. 9 refers to a point-to-point SPA attention map, but the paper only defines point-to-proxy and proxy-to-point SPA; please clarify how the point-to-point map is obtained from the proxy attention components.","section":"Sec. 4.3 and Fig. 9"},{"comment":"Algorithm 1 returns (l+r)/2 after max_iter even if the cell count is outside the target range; please state what happens in that fallback case. Also, Fig. 6(b) contains the label TRE Self-Attention with TRE, which appears to be a typo for TRB.","section":"Algorithm 1 and Fig. 6"},{"comment":"The sentence we evaluated some SOTA models ... as shown in Fig. 8 should refer to Table 8; similarly, a few places use pesade-code instead of pseudo-code.","section":"Appendix A.1"}],"recommendation":"major_revision","confidential_remarks":"The central empirical story is plausible and the paper includes useful controls, but the Eq. (1)-(3) inconsistency is load-bearing: if the implementation follows the equations, the SPA mechanism differs from what is claimed. This is fixable by correcting the equations and validating against the released code, so I do not recommend rejection. I would also ask the authors to clarify whether the public repository contains the exact version used for the reported SOTA numbers, since I did not verify the code against the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SP2T is a solid empirical architecture paper. It reports consistent gains over PTv3 across six benchmarks, ships a parameter-matched baseline, and the ablation study is thorough enough that the core claim—that sparse proxy attention plus dual-stream fusion adds value—is credible. The spatial-wise binary-search sampling and vertex-based association are simple, interpretable ideas that look genuinely useful, and the map-reduce sparse attention with table-based relative bias is a reasonable engineering contribution. The transfer study to MinkUNet and the latency breakdown are also good practice. I believe the SOTA numbers, if they reproduce, matter for the point-cloud transformer community.\n\nWhere I'd push back: the paper is light on statistical evidence—single runs, no error bars, and proxy/TRB hyperparameters tuned on the same validation splits that are reported. That is common in this literature, but it does mean the 1–2 point gains over PTv3 could partly be selection noise. The SPoTr comparison is also thin (one dataset). More serious is the normalization in Eq. (2): as written, the denominator sums over associations sharing the same proxy, not over associations sharing the same query point. That is a softmax over keys, not queries. Eq. (3) then indexes the output by proxy while the text says the output is per-point. If the implementation matches the equations, the attention is not doing what the paper claims, and the global-fusion narrative needs a different explanation. If this is just sloppy notation, fine, but the authors have to make it unambiguous, and code release becomes mandatory rather than optional. The outdoor empty-proxy concern raised by the skeptic is real but secondary: it is an efficiency and robustness issue, not something that breaks the indoor results.\n\nBottom line: this paper is worth seriously refereeing. The design is novel and the experiments are strong enough to be informative even if the exact gains don't all survive. I would send it out, with the understanding that the normalization detail and code are make-or-break for the final verdict.","headline":"Solid empirical architecture paper; the sparse-attention normalization is stated ambiguously enough that the authors must clarify before the global-fusion claim is accepted.","tokens_in":21288,"tokens_out":2168,"would_cite":true,"duration_ms":19772,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that sparse proxy attention over grid-cell corners gives point transformers a global receptive field at near-linear cost, and that the dual-stream design beats PTv3 on indoor and outdoor 3D benchmarks.","keywords":["point cloud","3D semantic segmentation","point transformer","sparse proxy attention","dual-stream network","relative position bias","LiDAR perception","receptive field"],"falsifier":"Take a held-out outdoor split (for example SemanticKITTI validation), replace the vertex-based association with exact L-infinity nearest-proxy association, and measure mIoU; if accuracy is essentially unchanged, the grid-corner assumption is not what carries the reported gains, while a large drop when grid spacing is doubled would confirm it is load-bearing.","tokens_in":20144,"feed_emoji":"🧊","tokens_out":12248,"duration_ms":97330,"temperature":0.7,"pith_summary":"Point transformers struggle to balance a large receptive field against computational cost and attention dilution. This paper proposes SP2T, a dual-stream transformer that runs a compressed proxy stream for global context alongside a point stream for local detail, exchanging features through sparse proxy attention at $O(k n d + m^2 d)$ cost with $k=8$ and $m$ much smaller than $n$. The paper claims this design achieves state-of-the-art results with acceptable latency: +3.8% mIoU over SPoTr on S3DIS, +22.9% over PointASNL on SemanticKITTI, and consistent gains over PTv3 on ScanNet, ScanNet200, S3DIS, nuScenes, SemanticKITTI, and Waymo. If correct, the work matters because it shows a path to very large receptive fields in point cloud transformers without quadratic attention cost or the attention dilution that comes from simply enlarging group windows.","feed_headline":"Sparse proxy attention lifts 3D segmentation past PTv3","feed_subtitle":"Global context at near-linear cost, with gains of +1.2 mIoU on ScanNet and +2.0 on S3DIS over PTv3.","key_machinery":"The load-bearing mechanism is sparse proxy attention (SPA). Each point is linked, by vertex-based association, to the eight corners of the containing axis-aligned grid cell, which is equivalent to L-infinity KNN with $k=8$; the attention then computes exponential similarities only within each proxy's association list, using map-reduce to obtain the sparse softmax and aggregate values. This reduces point–proxy interaction to $O(k n d)$ and leaves proxy–proxy global fusion at $O(m^2 d)$ with $m$ much smaller than $n$. A table-based relative bias (TRB) -- a trilinearly interpolated lookup table over relative positions, clamped and scaled -- injects spatial structure into the sparse similarities, and the paper's ablation attributes +1.20 mIoU on ScanNet to attention-based point–proxy interaction with TRB plus global fusion. The dual-stream layout, with PTv3 serialization attention handling local point fusion and TRB-driven self-attention among proxies handling global fusion, is what keeps local detail and global context balanced.","core_discovery":"The paper's central claim is that local proxies anchored to a spatial grid can give point transformers a global receptive field if three weaknesses of earlier proxy methods are fixed. Sampling must be spatial-wise: a binary search chooses the grid spacing so the proxy count lands in a target range, and each point is associated with the eight corners of the axis-aligned grid cell containing it, which is equivalent to L-infinity KNN with $k=8$. Interaction must be sparse: the point–proxy attention sums exponential similarities only within each proxy's association set, using a map-reduce pass, so each point touches only $k$ proxies instead of all $m$ proxies. Finally, local and global streams must run in parallel rather than being merged into one stream. With these fixes, SP2T reports state-of-the-art mIoU on ScanNet (78.7), ScanNet200 (37.0), S3DIS 6-fold (79.7), nuScenes (81.2), and SemanticKITTI (71.7), and improves over PTv3 on Waymo detection by 1.6 mAPH with a single input frame.","pith_inferences":["The vertex-based association assumes the eight corners of an axis-aligned grid cell are good proxies for a point; in sparse outdoor LiDAR with large empty cells, those corners may not be the true nearest proxies, so a direct comparison against exact L-infinity KNN association would isolate how much of the reported gain depends on this geometric shortcut.","Because TRB shows stage-level similarity across layers, the learned relative bias may act as a transferable distance kernel; a testable extension would train TRB on one dataset and freeze it on another to see whether the spatial prior transfers.","The dual-stream design is not tied to PTv3; with the Minkowski-convolution backbone it already improves both ScanNet and ScanNet200, so sparse proxy attention could be grafted onto other point, voxel, or hybrid backbones for detection and tracking.","Since empty proxies improve accuracy, occupancy of the proxy grid itself is informative; an auxiliary loss predicting whether a proxy cell contains points could make the global fusion even more explicit."],"forward_implications":["The point–proxy interaction cost drops from $O(nmd)$ to $O(knd+m^2d)$ with $k=8$, making proxy-based global receptive fields practical for large indoor and outdoor scenes; the paper reports about 20 FPS on nuScenes.","SP2T improves over PTv3 on every evaluated benchmark: +1.2 mIoU on ScanNet, +1.8 on ScanNet200, +2.0 on S3DIS 6-fold, +1.6 mAPH on Waymo single-frame detection, and +0.8/+0.9 mIoU on nuScenes and SemanticKITTI.","Proxy-based point transformers can now be competitive on indoor and outdoor tasks simultaneously; earlier proxy methods like SPoTr and Fast PT lagged on indoor datasets, while PointASNL lagged outdoors.","The sparse proxy interaction is plug-and-play: attaching it to a Minkowski-convolution backbone lifts ScanNet mIoU by +3.5 and ScanNet200 by +5.3, matching the gains seen with the PTv3 backbone.","Empty grid cells, whose proxies have no associated points, still contribute when included in global fusion with zero-initialized features, improving accuracy and indicating that the absence of points carries spatial information."],"supporting_citations":[{"why":"It provides the PTv3 baseline and local fusion backbone whose serialization attention SP2T reuses; all main gains are measured against it.","marker":"[56]"},{"why":"PointASNL is the proxy-based dual-stream method with FPS sampling that SP2T improves on SemanticKITTI by +22.9 mIoU and contrasts its sampling design against.","marker":"[57]"},{"why":"SPoTr is the proxy-based single-stream point transformer that SP2T surpasses on S3DIS by +3.8 mIoU, representing the indoor proxy baseline.","marker":"[38]"},{"why":"Fast PT is an earlier proxy-based point transformer whose indoor results are compared with SP2T.","marker":"[37]"},{"why":"PointNet++ defines the Furthest Point Sampling that SP2T replaces with spatial-wise sampling; the FPS-based ablation row depends on it.","marker":"[41]"},{"why":"Swin Transformer supplies the relative bias idea that the table-based relative bias (TRB) adapts to sparse point–proxy associations.","marker":"[31]"},{"why":"MobileFormer motivates the dual-stream architecture with parallel point and proxy streams.","marker":"[6]"},{"why":"MinkUNet is the convolutional backbone used in the transfer study, showing the sparse proxy interaction is plug-and-play.","marker":"[9]"}],"fun_headline_variants":["Sparse proxy attention: global RF for point clouds","SP2T: local proxies, global reach, beats PTv3","Sparse proxy attention: global context at near-linear cost","Local proxies, global reach: SP2T tops 3D benchmarks","SP2T: sparse proxy attention lifts 3D segmentation past PTv3"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every point is well represented by the eight corners of the axis-aligned grid cell containing it, so in sparse or uneven point clouds where those corners are empty or are not the nearest proxies, the global fusion quality depends on an association that may miss the true neighbors.","fun_headline_variants_meta":{"raw":{"variants":["Sparse proxy attention: global RF for point clouds","SP2T: local proxies, global reach, beats PTv3","Sparse proxy attention: global context at near-linear cost","Local proxies, global reach: SP2T tops 3D benchmarks","SP2T: sparse proxy attention lifts 3D segmentation past PTv3"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001149,"raw_usage":{"total_tokens":4824,"prompt_tokens":1067,"completion_tokens":3757,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":683,"completion_tokens_details":{"reasoning_tokens":3666}},"tokens_in":683,"tokens_out":3757,"duration_ms":22610,"temperature":1.0,"reasoning_tokens":3666,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:49:12.033471+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a held-out outdoor split (for example SemanticKITTI validation), replace the vertex-based association with exact L-infinity nearest-proxy association, and measure mIoU; if accuracy is essentially unchanged, the grid-corner assumption is not what carries the reported gains, while a large drop when grid spacing is doubled would confirm it is load-bearing.","supporting_citations":[{"cited_title":"Point transformer v3: Simpler faster stronger","cited_arxiv_id":null,"evidence_quote":"It provides the PTv3 baseline and local fusion backbone whose serialization attention SP2T reuses; all main gains are measured against it."},{"cited_title":"Pointasnl: Robust point clouds processing using nonlocal neural networks with adaptive sampling","cited_arxiv_id":null,"evidence_quote":"PointASNL is the proxy-based dual-stream method with FPS sampling that SP2T improves on SemanticKITTI by +22.9 mIoU and contrasts its sampling design against."},{"cited_title":"Self-positioning point-based transformer for point cloud understanding","cited_arxiv_id":null,"evidence_quote":"SPoTr is the proxy-based single-stream point transformer that SP2T surpasses on S3DIS by +3.8 mIoU, representing the indoor proxy baseline."},{"cited_title":"Fast point transformer","cited_arxiv_id":null,"evidence_quote":"Fast PT is an earlier proxy-based point transformer whose indoor results are compared with SP2T."},{"cited_title":"Swin transformer: Hierarchical vision transformer using shifted windows","cited_arxiv_id":null,"evidence_quote":"Swin Transformer supplies the relative bias idea that the table-based relative bias (TRB) adapts to sparse point–proxy associations."},{"cited_title":"Mobile-former: Bridging mobilenet and transformer","cited_arxiv_id":null,"evidence_quote":"MobileFormer motivates the dual-stream architecture with parallel point and proxy streams."}],"review_version":1}