REVIEW 3 major objections 4 minor 37 references
SeqAffordSplat: Scene-level Sequential Affordance Reasoning on 3D Gaussian Splatting
T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read The paper claims that sequential, scene-level 3D affordance reasoning can be solved end-to-end by an LLM that autoregressively emits segmentation tokens, each triggering a 3D mask over Gaussian-splat scenes.
desk verdict A genuinely new task and benchmark for sequential 3DGS affordance, with a promising architecture, but the headline SOTA numbers rest on an under-specified evaluation protocol. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the interleaved token sequence produced by an autoregressive LLM: text fragments describe each primitive step in words, and each <SEG> token acts as a trigger whose hidden-state vector becomes the query for a lightweight conditional decoder that predicts one binary mask over all Gaussians in the scene. The decoder follows query-based mask prediction, so the language model's per-step hidden state directly conditions the segmentation output. Two supporting mechanisms carry the rest of the argument: Conditional Geometric Reconstruction pretraining, which builds a geometric prior by reconstructing full affordance masks from a mask embedding plus partial geometry, and VFM Semantic Feature Injection, which renders the scene from multiple views, runs those images through a frozen 2D vision foundation model, and lifts the per-pixel features back onto each Gaussian by inverse alpha blending, then fuses them into the decoder at multiple scales.
What would settle it
Test the trained model on real 3DGS scans of furnished rooms with sequential instructions written independently by several humans and annotated without reference to the dataset; if the average overlap score in that setting falls to the 12.1 baseline level, the benchmark's ecological validity claim collapses.
Extended reading notes
Core claim
The central claim is that sequential affordance reasoning can be cast as autoregressive language generation grounded directly in 3D scenes: a large language model reads the instruction and generates an ordered primitive plan, and every time it emits a special <SEG> token a conditional decoder produces the corresponding 3D affordance mask over the Gaussian primitives. The paper argues that this unifies planning and perception in one end-to-end trainable model, supported by two additions: a pretraining scheme that makes the 3D encoder reconstruct complete affordance regions from geometric observations, and multi-scale injection of semantic features lifted from a frozen 2D vision model through inverse alpha-blended rendering. The evidence is a new large-scale benchmark, SeqAffordSplat, where the method sets the current best results in the single-step setting (37.0 mIoU vs 31.3 for the best prior method), the perception-only sequential setting (36.0 sIoU vs 30.3), and the full end-to-end sequential setting (26.2 sIoU vs 12.1 for the point-cloud SeqAfford baseline). On the existing 3DAffordSplat benchmark it also improves mIoU from 30.3 to 40.2. The paper's conclusion is that high-fidelity 3DGS representations combined with LLM-driven sequential decoding are sufficient to move affordance reasoning from single atomic interactions to long-horizon, scene-level tasks.
Load-bearing premise
The reported gains hold only if scenes composed by combining isolated object models and instructions generated by a language model then edited by humans represent real cluttered environments people and robots encounter.
Editorial extensions
If this is right
- End-to-end sequential affordance grounding on 3DGS is feasible: the method reaches 26.2 sIoU versus 12.1 for the prior sequential point-cloud method.
- The unified architecture transfers to single-step tasks, reaching 40.2 mIoU on the 3DAffordSplat benchmark, 9.9 points above the previous published result.
- Given the ground-truth step sequence, the perception module alone reaches 36.0 sIoU, indicating that geometric pretraining and semantic injection, rather than the LLM's planning, carry the localization accuracy.
- The ablation shows the geometric pretraining adds roughly 3.8 points of sIoU and DINOv2 feature injection adds about 2.1 more, so both components are needed for the reported gain.
- LLM scale is not a simple driver of success: the 0.6B backbone outperforms the 8B backbone on sIoU, suggesting that efficient task-tuned language models are sufficient for this type of reasoning.
Reading between the lines
- Because the benchmark scenes are composed by transforming single-object models rather than captured from real rooms, the strongest test of the paper's ecological claim would be a held-out set of real 3DGS room scans with independently written human instructions; the reported gains may not transfer if real clutter and occlusion patterns differ.
- The <SEG>-token interface suggests a natural bridge to robotics: the same autoregressive planning could emit grasp or placement parameters instead of masks, making the learned sequence directly executable by a manipulator.
- The observation that the 8B LLM underperforms the 0.6B one hints that the limiting factor is alignment with the curated instruction style rather than world knowledge; a smaller model tuned on a larger pool of human-curated plans may be the more economical path.
- Because all instructions come from one multimodal language model plus human curation, the diversity of language remains an open question; testing on unseen affordance verbs would separate genuine semantic generalization from memorized phrasings.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper introduces a new task, Sequential 3D Gaussian Affordance Reasoning, together with a benchmark (SeqAffordSplat) of over 1,800 composed 3D Gaussian Splatting scenes annotated with ordered affordance masks and language instructions, and a method (SeqSplatNet) that combines a fine-tuned LLM emitting interleaved text and <SEG> tokens with a conditional 3D affordance decoder, a geometric-reconstruction pre-training, and VFM semantic feature injection. The method is evaluated on the new benchmark (Table 2) and on the existing 3DAffordSplat benchmark (Table 3), reporting sIoU 26.2 vs. 12.1 for SeqAfford in the end-to-end sequential setting and mIoU 40.2 vs. 30.3 on 3DAffordSplat. Ablations (Tables 4-5) quantify the contributions of the pre-training, the VFM feature injection, and the LLM backbone.
Significance. If the empirical claims are supported, the paper makes a useful contribution by extending 3D affordance reasoning from single-object, single-step tasks to multi-object, sequential tasks on 3DGS, and by providing a benchmark that could support future research. The architecture is well motivated, and the ablations demonstrate that the proposed components contribute positively. However, the headline quantitative claims depend on baseline adaptation details and dataset split hygiene that are not currently reported, so the state-of-the-art assertions are not yet established.
major comments (3)
- [Experimental Settings (Baseline Models); Table 2] The headline end-to-end Sequential comparison against SeqAfford is not protocol-equivalent. SeqAfford operates on point clouds, and the paper never states how the 3DGS scenes were converted to its input, whether it was retrained on the SeqAffordSplat training split, or which checkpoints and hyperparameters were used. As reported, the 14.1-point sIoU gap (26.2 vs. 12.1) could reflect representation mismatch or an undertrained baseline rather than the superiority of the proposed architecture. The authors should adapt SeqAfford to 3DGS via a clearly specified conversion (e.g., Gaussian centers as point clouds), retrain it under identical data splits, and report the full adaptation and training settings.
- [Step 1: 3DGS Data Collection; Table 3] The generalization experiment on 3DAffordSplat is at risk of object-level leakage. SeqAffordSplat scenes are explicitly composed from 3D-AffordanceNet objects, the same corpus underlying 3DAffordSplat, yet the paper does not specify whether the object instances used to build training scenes are disjoint from the 3DAffordSplat test set. If overlapping instances appear, the 9.9-point mIoU improvement (40.2 vs. 30.3) may be inflated by memorization of object geometry rather than by transferable affordance reasoning. Please report the exact object-instance split and demonstrate that no training object from SeqAffordSplat appears in the 3DAffordSplat test split.
- [Step 1 and Step 2 of SeqAffordSplat Dataset Collection] The benchmark's ecological validity is asserted but not demonstrated. Scenes are generated by manually composing single-object models with translations, rotations, and scalings, and labels are transferred from single-object annotations; the paper provides no quantitative or qualitative evidence that these compositions reproduce realistic multi-object layouts, occlusion patterns, or object-part interactions, nor does it evaluate the naturalness and executability of the GPT-4o-generated instruction plans beyond reporting curation counts. Because the central claim is state-of-the-art performance on this benchmark, the authors should validate the benchmark, for example by reporting inter-annotator agreement on instruction plans, human ratings of scene realism, or zero-shot transfer to real captured 3DGS scenes.
minor comments (4)
- [Abstract / Figure 1 / Dataset section] The scene and annotation counts are inconsistent across the abstract (1800+ scenes, 14,000+ masks, 8,000+ instructions), Figure 1 (over 1,700 scenes, 12,000 instruction pairs), and the Dataset section (1800+ scenes, 14,000+ masks, 8,000+ instructions); please harmonize these numbers.
- [Throughout] Typos such as 'envolves', 'acurrate gemoetry', 'Benifiting', 'Obaining', and 'The the lifted semantic features' should be corrected.
- [Evaluation Metrics] The sequential metrics align sequences by padding with empty frames; please clarify whether the padding is applied to the predicted or ground-truth sequence and how this interacts with the stated length penalty.
- [Tables 2-5] No error bars or repeated-seed results are reported; given that several gains are in the 5-6 point range, please report the variance over at least three seeds.
Circularity Check
No significant circularity: the reported gains are empirical benchmark results, not by-construction equivalences.
full rationale
The paper's central claims are empirical benchmark outcomes (Tables 2 and 3), not derived predictions. Eq. (1) only defines the task; Eqs. (2)-(7) define losses, a mask-conditioned pretraining objective, and VFM feature lifting, none of which is equivalent to its input by construction. The pretraining stage does take the ground-truth mask M^gt as a conditioning input, but at inference the mask encoder is replaced by the LLM's <SEG> token embedding, so the pretraining is a representation-initialization step rather than a reported prediction. The dataset is assembled from the published 3DAffordSplat/3D-AffordanceNet corpus; although one current co-author (G. Li) is also affiliated with 3DAffordSplat, that corpus is external, published, and independently usable, and the paper also evaluates on the existing 3DAffordSplat benchmark (Table 3), providing an external check. The GPT-4o-generated instruction plans are human-curated, and the method's own LLM is a different model (Qwen3), so no by-construction reduction exists. Concerns about baseline retraining protocol, input conversion, and potential train/test object overlap are experimental-validity/correctness issues, not circularity.
Assumptions & free parameters
free parameters (2)
- lambda_mask (loss balancing weight) =
not specified
- Number of views m for VFM semantic feature lifting =
not specified
assumptions (5)
- domain assumption 3DGS representation is sufficient for fine-grained affordance localization.
- domain assumption Composing objects from an existing dataset via geometric transformations yields realistic complex scenes.
- domain assumption Point-wise affordance labels transfer correctly from source objects to 3DGS Gaussians in composed scenes.
- domain assumption GPT-4o generated instruction sequences, after human curation, are valid ground-truth action plans.
- domain assumption Padding predicted and ground-truth sequences to the same length with empty frames is a fair evaluation of sequence quality.
Cite this review
Pith. "Pith review of SeqAffordSplat: Scene-level Sequential Affordance Reasoning on 3D Gaussian Splatting." pith.science (2026). https://pith.science/paper/YBH7IIME
@misc{pith2026250723772,
author = {Pith},
title = {Pith review of: SeqAffordSplat: Scene-level Sequential Affordance Reasoning on 3D Gaussian Splatting},
year = {2026},
howpublished = {\url{https://pith.science/paper/YBH7IIME}},
note = {Machine review of arXiv:2507.23772}
}
read the original abstract
3D affordance reasoning, the task of associating human instructions with the functional regions of 3D objects, is a critical capability for embodied agents. Current methods based on 3D Gaussian Splatting (3DGS) are fundamentally limited to single-object, single-step interactions, a paradigm that falls short of addressing the long-horizon, multi-object tasks required for complex real-world applications. To bridge this gap, we introduce the novel task of Sequential 3D Gaussian Affordance Reasoning and establish SeqAffordSplat, a large-scale benchmark featuring 1800+ scenes to support research on long-horizon affordance understanding in complex 3DGS environments. We then propose SeqSplatNet, an end-to-end framework that directly maps an instruction to a sequence of 3D affordance masks. SeqSplatNet employs a large language model that autoregressively generates text interleaved with special segmentation tokens, guiding a conditional decoder to produce the corresponding 3D mask. To handle complex scene geometry, we introduce a pre-training strategy, Conditional Geometric Reconstruction, where the model learns to reconstruct complete affordance region masks from known geometric observations, thereby building a robust geometric prior. Furthermore, to resolve semantic ambiguities, we design a feature injection mechanism that lifts rich semantic features from 2D Vision Foundation Models (VFM) and fuses them into the 3D decoder at multiple scales. Extensive experiments demonstrate that our method sets a new state-of-the-art on our challenging benchmark, effectively advancing affordance reasoning from single-step interactions to complex, sequential tasks at the scene level.
Figures
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al
Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774
arXiv 2023
-
[4]
Chen, D.; Kong, D.; Li, J.; and Yin, B. 2025. MaskPrompt: Open-Vocabulary Affordance Segmentation with Object Shape Mask Prompts. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, 2034--2042
work page 2025
-
[5]
Cheng, B.; Schwing, A.; and Kirillov, A. 2021. Per-pixel classification is not all you need for semantic segmentation. Advances in neural information processing systems, 34: 17864--17875
work page 2021
-
[6]
Dalgarno, B.; and Lee, M. J. 2010. What are the learning affordances of 3-D virtual environments? British journal of educational technology, 41(1): 10--32
work page 2010
-
[7]
Deng, S.; Xu, X.; Wu, C.; Chen, K.; and Jia, K. 2021. 3d affordancenet: A benchmark for visual object affordance understanding. In proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 1778--1787
work page 2021
-
[8]
Do, T.-T.; Nguyen, A.; and Reid, I. 2018. Affordancenet: An end-to-end deep learning approach for object affordance detection. In 2018 IEEE international conference on robotics and automation (ICRA), 5882--5889. IEEE
work page 2018
Show all 37 references
-
[9]
J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; Chen, W.; et al
Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; Chen, W.; et al. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2): 3
2022
-
[10]
Kerbl, B.; Kopanas, G.; Leimk \"u hler, T.; and Drettakis, G. 2023. 3D Gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4): 139--1
2023
-
[11]
P.; and Ba, J
Kingma, D. P.; and Ba, J. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980
2014 arXiv
-
[12]
Li, G.; Sun, D.; Sevilla-Lara, L.; and Jampani, V. 2024 a . One-shot open affordance learning with foundation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3086--3096
2024
-
[13]
Li, Y.; Zhao, N.; Xiao, J.; Feng, C.; Wang, X.; and Chua, T.-s. 2024 b . Laso: Language-guided affordance segmentation on 3d object. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14251--14260
2024
-
[14]
Lu, D.; Kong, L.; Huang, T.; and Lee, G. H. 2025. Geal: Generalizable 3d affordance learning with cross-modal consistency. In Proceedings of the Computer Vision and Pattern Recognition Conference, 1680--1690
2025
-
[15]
Lu, G.; Zhang, S.; Wang, Z.; Liu, C.; Lu, J.; and Tang, Y. 2024. Manigaussian: Dynamic gaussian splatting for multi-task robotic manipulation. In European Conference on Computer Vision, 349--366. Springer
2024
-
[16]
Marrie, J.; M \'e n \'e gaux, R.; Arbel, M.; Larlus, D.; and Mairal, J. 2024. LUDVIG: Learning-free uplifting of 2d visual features to Gaussian splatting scenes. arXiv preprint arXiv:2410.14462
2024 arXiv
-
[17]
Mo, K.; Qin, Y.; Xiang, F.; Su, H.; and Guibas, L. 2022. O2o-afford: Annotation-free large-scale object-object affordance learning. In Conference on robot learning, 1666--1677. PMLR
2022
-
[18]
S.; Duarte, N
Mohammadi, S. S.; Duarte, N. F.; Dimou, D.; Wang, Y.; Taiana, M.; Morerio, P.; Dehban, A.; Moreno, P.; Bernardino, A.; Del Bue, A.; et al. 2023. 3dsgrasp: 3d shape-completion for robotic grasp. arXiv preprint arXiv:2301.00866
2023
-
[19]
Nagarajan, T.; and Grauman, K. 2020. Learning affordance landscapes for interaction exploration in 3d environments. Advances in Neural Information Processing Systems, 33: 2005--2015
2020
-
[20]
Oquab, M.; Darcet, T.; Moutakanni, T.; Vo, H.; Szafraniec, M.; Khalidov, V.; Fernandez, P.; Haziza, D.; Massa, F.; El-Nouby, A.; et al. 2023. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193
2023 arXiv
-
[21]
R.; Yi, L.; Su, H.; and Guibas, L
Qi, C. R.; Yi, L.; Su, H.; and Guibas, L. J. 2017. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30
2017
-
[22]
Qian, S.; Chen, W.; Bai, M.; Zhou, X.; Tu, Z.; and Li, L. E. 2024. Affordancellm: Grounding affordance from vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 7587--7597
2024
-
[23]
W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al
Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748--8763. PmLR
2021
-
[24]
Radford, A.; Wu, J.; Child, R.; Luan, D.; Amodei, D.; Sutskever, I.; et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8): 9
2019
-
[25]
Roy, A.; and Todorovic, S. 2016. A multi-scale cnn for affordance segmentation in rgb images. In European conference on computer vision, 186--201. Springer
2016
-
[26]
Shao, Y.; Zhai, W.; Yang, Y.; Luo, H.; Cao, Y.; and Zha, Z.-J. 2025. Great: Geometry-intention collaborative inference for open-vocabulary 3d object affordance grounding. In Proceedings of the Computer Vision and Pattern Recognition Conference, 17326--17336
2025
-
[27]
Shorinwa, O.; Tucker, J.; Smith, A.; Swann, A.; Chen, T.; Firoozi, R.; Kennedy III, M.; and Schwager, M. 2024. Splat-mover: Multi-stage, open-vocabulary robotic manipulation via editable gaussian splatting. arXiv preprint arXiv:2405.04378
2024 arXiv
-
[28]
H.; Gaskin, J
Steffen, J. H.; Gaskin, J. E.; Meservy, T. O.; Jenkins, J. L.; and Wolman, I. 2019. Framework of affordances for virtual reality and augmented reality. Journal of management information systems, 36(3): 683--729
2019
-
[29]
C.; Robb, A
Venkatakrishnan, R.; Venkatakrishnan, R.; Raveendranath, B.; Pagano, C. C.; Robb, A. C.; Lin, W.-C.; and Babu, S. V. 2023. How virtual hand representations affect the perceptions of dynamic affordances in virtual reality. IEEE Transactions on Visualization and Computer Graphic...
2023
-
[30]
Wei, Z.; Lin, J.; Liu, Y.; Chen, W.; Luo, J.; Li, G.; and Lin, L. 2025. 3DAffordSplat: Efficient Affordance Reasoning with 3D Gaussians. arXiv preprint arXiv:2504.11218
2025 arXiv
-
[31]
Xu, C.; Chen, Y.; Wang, H.; Zhu, S.-C.; Zhu, Y.; and Huang, S. 2022. Partafford: Part-level affordance discovery from 3d objects. arXiv preprint arXiv:2202.13519
2022 arXiv
-
[32]
G.; Petit, D.; Tsuji, T.; Akizuki, S.; Hashimoto, M.; Nagata, K.; and Harada, K
Yamanobe, N.; Wan, W.; Ramirez-Alpizar, I. G.; Petit, D.; Tsuji, T.; Akizuki, S.; Hashimoto, M.; Nagata, K.; and Harada, K. 2017. A brief review of affordance in robotic manipulation research. Advanced Robotics, 31(19-20): 1086--1101
2017
-
[33]
Yang, A.; Li, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Gao, C.; Huang, C.; Lv, C.; et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388
2025 arXiv
-
[34]
Yang, Y.; Zhai, W.; Luo, H.; Cao, Y.; Luo, J.; and Zha, Z.-J. 2023. Grounding 3d object affordance from 2d interactions in images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 10905--10915
2023
-
[35]
Yu, C.; Wang, H.; Shi, Y.; Luo, H.; Yang, S.; Yu, J.; and Wang, J. 2025. Seqafford: Sequential 3d affordance reasoning via multimodal large language model. In Proceedings of the Computer Vision and Pattern Recognition Conference, 1691--1701
2025
-
[36]
Zheng, Y.; Chen, X.; Zheng, Y.; Gu, S.; Yang, R.; Jin, B.; Li, P.; Zhong, C.; Wang, Z.; Liu, L.; et al. 2024. Gaussiangrasper: 3d language gaussian splatting for open-vocabulary robotic grasping. IEEE Robotics and Automation Letters
2024
-
[37]
Zhu, S.; Wang, G.; Kong, X.; Kong, D.; and Wang, H. 2024. 3d gaussian splatting in robotics: A survey. arXiv preprint arXiv:2410.12262
2024 arXiv
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.