pith. sign in

arxiv: 2605.16832 · v1 · pith:FWUC24DSnew · submitted 2026-05-16 · 💻 cs.CV

Coarse Semantic Injection for LLM-Conditioned Structured Indoor Prediction

Shuliang Zhu , Tomiwa Adey , Jinjia Zhou This is my paper

Pith reviewed 2026-05-19 20:49 UTC · model grok-4.3

classification 💻 cs.CV
keywords semantic injectionpoint cloudLLM decoderindoor scene understandingstructured predictionRGBB codingsparse tokenizationopening localization
0
0 comments X

The pith

Appending a coarse four-group semantic color code to raw point attributes before tokenization improves LLM-based structured indoor prediction while leaving the decoder unchanged.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that reliable coarse semantic labels for points can be reduced to four groups—furniture, walls, openings, and others—and represented as an RGBB code that is simply appended to the original point attributes. This shared path through sparse tokenization lets the language model decoder exploit the extra cues for better recovery of thin structures and individual instances. A reader would care because voxelization and pooling routinely erase doors, windows, and cluttered furniture, yet the method requires no change to the downstream LLM or output format. The approach works from RGB-derived semantics and adds only a lightweight routed shift module at training time. Ablations confirm that the color coding and injection steps each contribute to gains on Structured3D, SpatialLM, and ARKitScenes.

Core claim

The central claim is that associating semantic evidence with each point, reducing it to a four-group code, and encoding it as an RGBB point interface—red for furniture, green for walls, blue for openings, black for others—before tokenization strengthens LLM-conditioned structured decoding. Geometry and semantics therefore follow the identical sparse tokenization path, the language-model decoder and output serialization stay untouched, and a lightweight routed semantic shift module (with an auxiliary training head) further reinforces cues after pooling. Under controlled semantic-source settings the metrics rise, especially for opening localization and per-instance furniture detection in dense

What carries the argument

The RGBB semantic color code appended to raw point attributes, which lets geometry and semantics share the same sparse tokenization path without altering the LLM decoder or serialization.

If this is right

  • Opening localization accuracy rises because thin structural elements receive explicit semantic cues before pooling.
  • Per-instance furniture detection improves in cluttered scenes by distinguishing individual objects through the appended labels.
  • The same LLM decoder and output serialization can be used unchanged, preserving compatibility with existing structured-prediction pipelines.
  • Ablations show that both the color coding choice and the shift-injection module contribute measurably to the reported gains across three indoor datasets.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same coarse four-group injection could be tested on outdoor or dynamic scenes where RGB evidence is also available, to check whether the benefit generalizes beyond static indoor layouts.
  • If higher-quality semantic sources become cheap, the method offers a low-cost way to upgrade existing point-token LLM pipelines without retraining the core decoder.
  • The auxiliary ratio-regularization head used only at training time suggests a route for distilling the semantic signal into the main model for inference-time efficiency.

Load-bearing premise

Reliable coarse semantic evidence for the four groups can be obtained from RGB or similar sources and injected without errors that outweigh the benefits after sparse pooling and LLM decoding.

What would settle it

A controlled experiment on the same test sets that removes the RGBB color channel or replaces it with random labels and measures whether opening localization and per-instance furniture F-scores fall back to the baseline level would falsify the claim.

Figures

Figures reproduced from arXiv: 2605.16832 by Jinjia Zhou, Shuliang Zhu, Tomiwa Adey.

Figure 1
Figure 1. Figure 1: Structured indoor modeling with semantic-colored point condition [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Overview of the proposed semantic-colored point interface. Coarse semantic cues are encoded as RGBB point attributes and injected before sparse [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative comparison on three benchmarks. On Structured3D and the SpatialLM dataset, we visualize point-cloud-to-structure predictions (layout [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Controllability via click-based 2D segmentation. A user clicks on a [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative comparison highlighting reduced empty-space halluci [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
read the original abstract

Large language models (LLMs) have recently been used as structured decoders for indoor understanding from 3D point-token inputs. However, point cloud encoders often under-represent thin structural elements such as doors and windows after voxelization and sparse pooling, and may miss individual furniture instances in cluttered scenes. We propose an interface-preserving semantic augmentation for LLM-conditioned structured decoding. The key idea is to associate semantic evidence with the point-cloud representation, reduce it to a coarse four-group code (furniture, walls, openings, and others), and encode it as an RGBB point interface: red for furniture, green for walls, blue for openings, and black for others, where RGBB denotes four semantic color states represented in three RGB channels rather than an additional fourth channel. This semantic color code is appended to the original raw point attributes before tokenization, so geometry and semantics share the same sparse tokenization path while the downstream language model decoder and output serialization remain unchanged. We further introduce a lightweight routed semantic shift module, with an auxiliary head used only for training-time ratio/budget regularization and analysis, to strengthen semantic cues after sparse pooling. The overall pipeline can use RGB-derived semantic evidence. Under these controlled semantic-source settings, the reported metrics improve across Structured3D, the SpatialLM dataset, and ARKitScenes, especially for opening localization and per-instance furniture detection in cluttered scenes. Ablations clarify the roles of semantic source, color coding, token fusion, and shift injection, while also showing that color/entropy effects remain nontrivial.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper proposes Coarse Semantic Injection for LLM-conditioned structured indoor prediction from 3D point clouds. Coarse four-class semantics (furniture/walls/openings/others) are encoded as an RGBB color code and appended to raw point attributes before tokenization, allowing geometry and semantics to share the same sparse tokenization path while leaving the LLM decoder and output serialization unchanged. A lightweight routed semantic shift module (with auxiliary training-time head) is introduced to counteract semantic dilution after sparse pooling. The pipeline supports RGB-derived semantics; under controlled semantic-source settings, metrics improve on Structured3D, SpatialLM, and ARKitScenes (especially opening localization and per-instance furniture detection in clutter), with ablations on semantic source, color coding, token fusion, and shift injection.

Significance. If the gains hold, the interface-preserving design offers a lightweight augmentation for existing point-cloud LLM pipelines without decoder changes, directly addressing under-representation of thin structures and cluttered instances. The color-based semantic encoding and routed shift module are pragmatic responses to sparse pooling effects. Credit for the empirical ablations and explicit focus on controlled vs. realistic semantic sources.

major comments (2)
  1. [Abstract] Abstract: the central claim of metric improvements is stated without any numerical values, baselines, error bars, or dataset statistics. This renders the magnitude and reliability of the reported gains unverifiable from the provided text and is load-bearing for assessing whether the semantic injection delivers substantive benefit.
  2. [Semantic injection pipeline] Semantic injection pipeline (as described in the abstract and method outline): the load-bearing assumption that reliable coarse semantic evidence (furniture/walls/openings/others) can be obtained from RGB or similar sources and survive sparse tokenization/pooling plus LLM decoding without net degradation is not sufficiently tested. The paper reports gains only under controlled semantic-source settings and introduces the shift module to mitigate dilution, yet provides no quantitative evaluation of performance erosion under realistic RGB-derived label noise (e.g., misclassification of thin openings or cluttered furniture), which could propagate through the shared token path.
minor comments (2)
  1. [Abstract] The RGBB encoding (four states in three channels, with black for 'others') is introduced without an explicit mapping table or example; a short clarification or diagram would remove potential ambiguity in how the color vector is appended to raw point attributes.
  2. [Overall] Consider adding a figure that visualizes the point-attribute augmentation step and the routed semantic shift module to improve clarity of the interface-preserving claim.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback on our manuscript. We address each major comment point by point below, indicating revisions made to the manuscript where appropriate.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central claim of metric improvements is stated without any numerical values, baselines, error bars, or dataset statistics. This renders the magnitude and reliability of the reported gains unverifiable from the provided text and is load-bearing for assessing whether the semantic injection delivers substantive benefit.

    Authors: We agree that the abstract would be strengthened by including concrete numerical results. In the revised manuscript, we have updated the abstract to report key quantitative improvements, such as relative gains in opening localization and per-instance furniture detection on Structured3D, SpatialLM, and ARKitScenes, while referencing the specific baselines, tables, and dataset statistics provided in the experimental section. This makes the central claims more verifiable without substantially increasing length. revision: yes

  2. Referee: [Semantic injection pipeline] Semantic injection pipeline (as described in the abstract and method outline): the load-bearing assumption that reliable coarse semantic evidence (furniture/walls/openings/others) can be obtained from RGB or similar sources and survive sparse tokenization/pooling plus LLM decoding without net degradation is not sufficiently tested. The paper reports gains only under controlled semantic-source settings and introduces the shift module to mitigate dilution, yet provides no quantitative evaluation of performance erosion under realistic RGB-derived label noise (e.g., misclassification of thin openings or cluttered furniture), which could propagate through the shared token path.

    Authors: We thank the referee for this observation. The paper deliberately employs controlled semantic sources to isolate the contribution of the injection mechanism and the routed shift module from confounding factors in semantic prediction. The manuscript is explicit about this choice and includes ablations on semantic source, color coding, and shift injection to demonstrate robustness. We have added a dedicated paragraph in the revised discussion section that analyzes potential degradation under realistic RGB label noise, explains how the shift module is intended to counteract dilution effects, and identifies this as an important direction for future work. A full quantitative study of noisy RGB-derived inputs was not included in the current experiments. revision: partial

Circularity Check

0 steps flagged

Empirical augmentation with no self-referential derivations or load-bearing reductions

full rationale

The paper proposes an interface-preserving semantic augmentation by appending a coarse four-class RGBB color code to raw point attributes before tokenization, sharing the sparse tokenization path with geometry while leaving the LLM decoder unchanged, plus a lightweight routed semantic shift module with auxiliary training-time regularization. No equations, derivations, or predictions are presented that reduce by construction to fitted inputs or self-citations. Central claims rest on empirical gains reported on external datasets (Structured3D, SpatialLM, ARKitScenes) under controlled semantic-source settings, with ablations on semantic source, color coding, and shift injection. The method is self-contained against external benchmarks with no uniqueness theorems, ansatzes smuggled via citation, or renaming of known results.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The central claim rests on the assumption that coarse semantic grouping can be reliably sourced and that appending it as color channels preserves information through existing tokenization without architectural changes.

axioms (1)
  • domain assumption Coarse four-group semantic labels (furniture, walls, openings, others) are sufficient to improve downstream LLM decoding when encoded as RGBB colors.
    Abstract invokes this grouping as the basis for the color code without proving sufficiency for all indoor elements.
invented entities (1)
  • RGBB point interface no independent evidence
    purpose: Carry four semantic states through three RGB channels before tokenization
    New encoding scheme introduced to share the sparse tokenization path with geometry.

pith-pipeline@v0.9.0 · 5810 in / 1268 out tokens · 40125 ms · 2026-05-19T20:49:53.481134+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

48 extracted references · 48 canonical work pages

  1. [1]

    Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

    Structure-from-motion revisited , author=. Proceedings of the IEEE conference on computer vision and pattern recognition , pages=

    work page

  2. [2]

    Wang, Shuzhe and Leroy, Vincent and Cabon, Yohann and Chidlovskii, Boris and Revaud, Jerome , booktitle=

    work page

  3. [3]

    Grounding image matching in

    Leroy, Vincent and Cabon, Yohann and Revaud, J. Grounding image matching in. European Conference on Computer Vision , pages=. 2024 , organization=

    work page 2024

  4. [4]

    Wang, Jianyuan and Chen, Minghao and Karaev, Nikita and Vedaldi, Andrea and Rupprecht, Christian and Novotny, David , booktitle=

    work page

  5. [5]

    Nie, Yinyu and Han, Xiaoguang and Guo, Shihui and Zheng, Yujian and Chang, Jian and Zhang, Jian Jun , booktitle=

    work page

  6. [6]

    Atlas: End-to-end

    Murez, Zak and Van As, Tarrence and Bartolozzi, James and Sinha, Ayan and Badrinarayanan, Vijay and Rabinovich, Andrew , booktitle=. Atlas: End-to-end. 2020 , organization=

    work page 2020

  7. [7]

    Sun, Jiaming and Xie, Yiming and Chen, Linghao and Zhou, Xiaowei and Bao, Hujun , booktitle=

    work page

  8. [8]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Connecting the dots: Floorplan reconstruction using two-level queries , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    work page

  9. [9]

    Proceedings of the IEEE/CVF international conference on computer vision , pages=

    Segment anything , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=

    work page

  10. [10]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Masked-attention mask transformer for universal image segmentation , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    work page

  11. [11]

    Peng, Songyou and Genova, Kyle and Jiang, Chiyu and Tagliasacchi, Andrea and Pollefeys, Marc and Funkhouser, Thomas and others , booktitle=

    work page

  12. [12]

    Openmask3d: Open-vocabulary 3d instance segmenta- tion,

    Takmaz, Ay. arXiv preprint arXiv:2306.13631 , year=

    work page arXiv

  13. [13]

    Jatavallabhula, Krishna Murthy and Kuwajerwala, Alihusein and Gu, Qiao and Omama, Mohd and Chen, Tao and Maalouf, Alaa and Li, Shuang and Iyer, Ganesh and Saryazdi, Soroush and Keetha, Nikhil and others , journal=

    work page

  14. [14]

    PointNet: Deep learning on point sets for

    Qi, Charles R and Su, Hao and Mo, Kaichun and Guibas, Leonidas J , booktitle=. PointNet: Deep learning on point sets for

    work page

  15. [15]

    Qi, Charles Ruizhongtai and Yi, Li and Su, Hao and Guibas, Leonidas J , booktitle=

    work page

  16. [16]

    Proceedings of the IEEE/CVF international conference on computer vision , pages=

    Thomas, Hugues and Qi, Charles R and Deschaud, Jean-Emmanuel and Marcotegui, Beatriz and Goulette, Fran. Proceedings of the IEEE/CVF international conference on computer vision , pages=

    work page

  17. [17]

    Proceedings of the IEEE/CVF international conference on computer vision , pages=

    Point transformer , author=. Proceedings of the IEEE/CVF international conference on computer vision , pages=

    work page

  18. [18]

    Qian, Guocheng and Li, Yuchen and Peng, Houwen and Mai, Jinjie and Hammoud, Hasan and Elhoseiny, Mohamed and Ghanem, Bernard , booktitle=

    work page

  19. [19]

    Yu, Xumin and Tang, Lulu and Rao, Yongming and Huang, Tiejun and Zhou, Jie and Lu, Jiwen , booktitle=

    work page

  20. [20]

    Pang, Yatian and Wang, Wenxiao and Tay, Francis E. H. and Liu, Wei and Tian, Yonghong and Yuan, Li , booktitle=. 2022 , organization=

    work page 2022

  21. [21]

    Wu, Xiaoyang and DeTone, Daniel and Frost, Duncan and Shen, Tianwei and Xie, Chris and Yang, Nan and Engel, Jakob and Newcombe, Richard and Zhao, Hengshuang and Straub, Julian , booktitle=

    work page

  22. [22]

    2025 , doi=

    Mao, Yongsen and Zhong, Junhao and Fang, Chuan and Zheng, Jia and Tang, Rui and Zhu, Hao and Tan, Ping and Zhou, Zihan , journal=. 2025 , doi=

    work page 2025

  23. [23]

    Baruch, Gilad and Chen, Zhuoyuan and Dehghan, Afshin and Dimry, Tal and Feigin, Yuri and Fu, Peter and Gebauer, Thomas and Joffe, Brandon and Kurz, Daniel and Schwartz, Arik and others , journal=

    work page

  24. [24]

    2025 , doi=

    Wang, Yifan and Zhou, Jianjun and Zhu, Haoyi and Chang, Wenzheng and Zhou, Yang and Li, Zizun and Chen, Junyi and Pang, Jiangmiao and Shen, Chunhua and He, Tong , journal=. 2025 , doi=

    work page 2025

  25. [25]

    2020 , organization=

    Zheng, Jia and Zhang, Junfei and Li, Jing and Tang, Rui and Gao, Shenghua and Zhou, Zihan , booktitle=. 2020 , organization=

    work page 2020

  26. [26]

    2018 , doi=

    Perez, Ethan and Strub, Florian and De Vries, Harm and Dumoulin, Vincent and Courville, Aaron , booktitle=. 2018 , doi=

    work page 2018

  27. [27]

    Advances in Neural Information Processing Systems , volume=

    Modulating early visual processing by language , author=. Advances in Neural Information Processing Systems , volume=

    work page

  28. [28]

    2024 , organization=

    Avetisyan, Armen and Xie, Christopher and Howard-Jenkins, Henry and Yang, Tsun-Yi and Aroudj, Samir and Patra, Suvam and Zhang, Fuyang and Frost, Duncan and Holland, Luke and Orme, Campbell and others , booktitle=. 2024 , organization=

    work page 2024

  29. [29]

    2025 , doi=

    Carion, Nicolas and Gustafson, Laura and Hu, Yuan-Ting and Debnath, Shoubhik and Hu, Ronghang and Suris, Didac and Ryali, Chaitanya and Alwala, Kalyan Vasudev and Khedr, Haitham and Huang, Andrew and others , journal=. 2025 , doi=

    work page 2025

  30. [30]

    Wang, Peng and Bai, Shuai and Tan, Sinan and Wang, Shijie and Fan, Zhihao and Bai, Jinze and Chen, Keqin and Liu, Xuejing and Wang, Jialin and Ge, Wenbin and others , journal=

    work page

  31. [31]

    arXiv preprint arXiv:2603.03283 , year=

    Utonia: Toward One Encoder for All Point Clouds , author=. arXiv preprint arXiv:2603.03283 , year=

    work page arXiv

  32. [32]

    2009 IEEE 12th International Conference on Computer Vision , pages=

    Recovering the spatial layout of cluttered rooms , author=. 2009 IEEE 12th International Conference on Computer Vision , pages=. 2009 , organization=

    work page 2009

  33. [33]

    2009 IEEE Conference on Computer Vision and Pattern Recognition , pages=

    Geometric reasoning for single image structure recovery , author=. 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages=. 2009 , organization=

    work page 2009

  34. [34]

    2018 , doi=

    Zou, Chuhang and Colburn, Alex and Shan, Qi and Hoiem, Derek , booktitle=. 2018 , doi=

    work page 2018

  35. [35]

    Sun, Cheng and Hsiao, Chi-Wei and Sun, Min and Chen, Hwann-Tzong , booktitle=

    work page

  36. [36]

    Hong, Yining and Zhen, Haoyu and Chen, Peihao and Zheng, Shuhong and Du, Yilun and Chen, Zhenfang and Gan, Chuang , booktitle=

    work page

  37. [37]

    Qi, Charles R and Litany, Or and He, Kaiming and Guibas, Leonidas J , booktitle=

    work page

  38. [38]

    Group-free

    Liu, Ze and Zhang, Zheng and Cao, Yue and Hu, Han and Tong, Xin , booktitle=. Group-free

    work page

  39. [39]

    Choy, Christopher and Gwak, JunYoung and Savarese, Silvio , booktitle=

    work page

  40. [40]

    Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=

    Fully convolutional networks for semantic segmentation , author=. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages=. 2015 , doi=

    work page 2015

  41. [41]

    He, Kaiming and Gkioxari, Georgia and Doll. Mask. Proceedings of the IEEE International Conference on Computer Vision , pages=

    work page

  42. [42]

    Advances in Neural Information Processing Systems , volume=

    Per-pixel classification is not all you need for semantic segmentation , author=. Advances in Neural Information Processing Systems , volume=

    work page

  43. [43]

    Proceedings of the 38th International Conference on Machine Learning , pages=

    Learning transferable visual models from natural language supervision , author=. Proceedings of the 38th International Conference on Machine Learning , pages=. 2021 , volume=

    work page 2021

  44. [44]

    Kerr, Justin and Kim, Chung Min and Goldberg, Ken and Kanazawa, Angjoo and Tancik, Matthew , booktitle=

    work page

  45. [45]

    2023 , address=

    Shafiullah, Nur Muhammad Mahi and Paxton, Chris and Pinto, Lerrel and Chintala, Soumith and Szlam, Arthur , booktitle=. 2023 , address=

    work page 2023

  46. [46]

    Probabilistic Triangulation for Uncalibrated Multi-View

    Jiang, Boyuan and Hu, Lei and Xia, Shihong , booktitle=. Probabilistic Triangulation for Uncalibrated Multi-View

    work page

  47. [47]

    Adaptive Multi-View and Temporal Fusing Transformer for

    Shuai, Hui and Wu, Lele and Liu, Qingshan , journal=. Adaptive Multi-View and Temporal Fusing Transformer for. 2023 , doi=

    work page 2023

  48. [48]

    2026 , doi=

    Song, Jucheng and Yang, Xu and Wang, Yapeng and Zhang, Jie and Im, Sio Kei , journal=. 2026 , doi=

    work page 2026