Pith. sign in

REVIEW 3 major objections 5 minor 59 references

Building-MLLM turns indoor point clouds into natural-language recognition, captions, and engineering answers instead of just class labels.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Building-MLLM, with PIE, GPR, fixed prefix and multi-dimensional LoRA plus a 37k-pair synthetic dataset, reaches 88/65/68 % on recognition, captioning and multi-engineering QA for 47 indoor component categories.

T0 review reviewed 2026-07-12 challenge →

load-bearing objection Solid domain adaptation of PointLLM for indoor components with a real new dataset and clean ablations; headline scores rest on GPT-family self-evaluation and a 30-sample real transfer, so treat them as synthetic-distribution gains rather than external engineering truth. the 3 major comments →

arxiv 2607.03661 v1 pith:I3R2ULPG submitted 2026-07-04 cs.CV

From Geometric Labels to Semantic Understanding of Indoor Building Components Using Multimodal Large Language Models

Shuju Jing , Chao Yin This is my paper
classification cs.CV
keywords Indoor Building ComponentsMultimodal Large Language ModelsPoint CloudsSynthetic Point Cloud-Text DatasetEngineering InteractionGeometry-Preserving AlignmentLoRA Fine-TuningFacility Operation and Maintenance
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most point-cloud systems for building components stop at a discrete label such as "valve" or "duct bend." That is not enough for facility operation and maintenance: engineers need functional explanations, constraint reasoning, and step-by-step interaction language. This paper claims that a domain-adapted multimodal large language model can close that gap. Building-MLLM takes a component-level point cloud plus a natural-language instruction and answers three escalating tasks: simple recognition, multi-attribute captioning, and multi-engineering question answering that covers knowledge, constraints, spatial relations, and embodied procedures. The authors construct the first large synthetic point-cloud–text instruction set for indoor components (4 198 objects, 47 categories, 37 782 pairs) with a progressive engine that injects ontology priors and engineering constraints. They then add four domain-specific mechanisms—an enhancer that sharpens task-relevant geometry, a regularizer that keeps shallow layers from erasing fine shape detail, a fixed textual prefix that stabilizes domain language, and multi-dimensional LoRA that balances short-range recognition against long-range reasoning. On a held-out synthetic test set the model scores 88 %, 65 % and 68 % (GPT-4 semantic evaluation) on the three tasks, beating the general-domain PointLLM baseline, and it shows initial transfer gains on limited real-world samples.

Core claim

A point-cloud-centered multimodal LLM specialized for indoor building components can generate coherent natural-language responses across simple recognition, complex captioning and multi-engineering question answering, provided it is trained on ontology-constrained synthetic instruction data and equipped with geometry-preserving alignment plus multi-dimensional LoRA; under GPT-4 evaluation the resulting Building-MLLM reaches 88.00 %, 65.10 % and 68.14 % on the three tasks and transfers with large gains to small real-world samples.

What carries the argument

The triple-constraint geometry-language alignment (Point Information Enhancer that recursively fuses local and spatial cues, Geometry-Preserving Regularization that reconstructs shallow-layer point features, plus a fixed domain prefix) together with multi-dimensional LoRA that jointly tunes layer range, target modules, rank and progressive scaling; these keep fine geometric distinctions alive while still allowing long-range engineering reasoning.

Load-bearing premise

The progressive synthetic engine that uses ontology templates, multi-view filtering and GPT-4V produces instruction text whose engineering semantics are faithful enough that GPT-4 evaluation of the same model family can serve as a reliable external measure of correctness.

What would settle it

A larger, independently annotated real-world point-cloud–text benchmark for indoor MEP and architectural components on which Building-MLLM’s GPT-4 scores fall back to or below the general PointLLM baseline, or human expert raters systematically disagree with the GPT-4 scores on engineering logic.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Facility O&M systems can move from discrete class labels to interactive natural-language queries about function, installation constraints and maintenance steps directly from component point clouds.
  • The same progressive engine can be reused to generate instruction data for other concentrated engineering domains that currently lack multimodal corpora.
  • Geometry-preserving regularizers of the kind introduced here become a standard tool when frozen point encoders must be adapted to fine-grained professional categories.
  • Multi-dimensional LoRA schedules that trade short-range geometric grounding against long-range reasoning can be ported to other mixed point–text multimodal models.
  • Component-level language understanding becomes a modular entry point for future scene-level digital-twin agents that reason over connectivity and global constraints.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the synthetic-to-real gap can be closed with modest real annotation, the same architecture could serve as a lightweight semantic front-end for existing BIM-CMMS platforms without requiring full scene-level 3D LLMs.
  • The large performance drop on severely incomplete or atypical real scans suggests that active sensing or multi-sensor fusion will be necessary before the method can be trusted in live construction or renovation sites.
  • The seven engineering QA subtypes (especially constraint and embodied interaction) form a natural curriculum that could be used to stress-test other multimodal models claiming domain expertise.
  • Because the authors already freeze most of the 7 B backbone, further distillation to smaller edge-deployable models looks feasible for on-site mobile capture devices.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Building-MLLM, a point-cloud-centered multimodal LLM for indoor building components that unifies Simple Recognition, Complex Captioning, and Multi-Engineering QA. It introduces four domain mechanisms (Point Information Enhancer, Geometry-Preserving Regularization on shallow LLM layers, a fixed textual prefix, and multi-dimensional LoRA) to address semantic concentration and multi-task balance, together with a progressive multi-constraint instruction-generation engine that produces a synthetic dataset of 4,198 objects, 47 categories and 37,782 instruction pairs. On a held-out synthetic test set the model reports GPT-4 semantic scores of 88.00 %, 65.10 % and 68.14 % (outperforming a fine-tuned PointLLM baseline), with limited-sample transfer gains on 30 real-world point clouds from Pipework, ScanObjectNN and S3DIS.

Significance. If the reported gains hold under independent evaluation, the work supplies a concrete bridge from discrete geometric labels to language-grounded engineering understanding for facility O&M and digital twins—an underexplored niche relative to general 3D MLLMs. Strengths include a carefully ablated architecture (PIE +5.25 %, GPR +2.12 %, fixed prefix, systematic LoRA layer/module/rank/scaling tables), an open progressive data engine with ontology and cross-modal (BLIP-2/CLIP/Shap-E) filters, public code, and explicit multi-task coverage spanning knowledge, constraint and embodied interaction. These elements make the manuscript a useful methodological foundation even if absolute numbers require stronger external validation.

major comments (3)
  1. §3.1 Step 3 / Table 2 and §5.2: The headline scores (88.00 / 65.10 / 68.14) are GPT-4 judgments of model outputs against reference text generated by GPT-4V under the same ontology and task prompts. Manual review of 800 samples (κ=0.874) only flags binary “need correction” and does not supply independent human-authored ground truth. This creates a distributional loop that can inflate agreement on style and omitted constraints; the paper itself notes residual bias risk. An independent human or code-based engineering-logic audit on a non-trivial subset is required before the superiority claim can be treated as reliable.
  2. §5.3 / Table 12: Real-world transfer is demonstrated on only 30 hand-selected samples (10 per dataset) that remain GPT-4-scored. While the absolute gains over PointLLM are large, the sample size and selection procedure are insufficient to support the claim of “initial generalisability.” Either enlarge the real-world test set with stratified sampling and report confidence intervals, or clearly reframe the transfer results as qualitative feasibility only.
  3. §5.4.1–5.4.2 and Eq. (9)–(10): Ablations convincingly isolate PIE, GPR and LoRA dimensions on the synthetic validation set, yet all Stage-1/Stage-2 metrics ultimately feed into the same GPT-4 evaluator. Without a secondary metric suite (e.g., human preference rankings or external ontology consistency checks) it remains unclear how much of the multi-task balance improvement is genuine geometric-semantic gain versus better imitation of the generator’s distribution.
minor comments (5)
  1. Figure 2 and §3.2: Token-length statistics are given, but category-wise sample counts and class-balance statistics for the 47 categories would help readers assess long-tail effects.
  2. §4.2.2 / Eq. (5)–(7): The recursive-block count and Fourier parameters (α, β) are ablated, yet the precise definition of the gating function G(·) and the residual bottleneck dimensions are only sketched; a short algorithmic box or appendix would improve reproducibility.
  3. Table 9–11: Asterisked rows (independent test set) are helpful, but the distinction between validation-inference and test-inference splits should be stated once in the experimental-setup paragraph for clarity.
  4. Appendix A: Prompt-robustness numbers are reassuring; moving a one-sentence summary into the main evaluation section would strengthen the claim that results are not prompt-specific.
  5. Minor typographical inconsistencies appear (e.g., “generalisability” vs “generalizability”, occasional missing spaces around citations); a final proof-reading pass is recommended.

Circularity Check

1 steps flagged

Headline GPT-4 scores partly measure agreement with GPT-4V-generated references under shared ontology/task prompts; architecture and gains over PointLLM remain independently trained.

specific steps
  1. other [Section 3.1 Step 3 + Table 2; Section 5.2 + Tables 6–8]
    "Step 3: Complex Captioning and Multi-Engineering QA Text Instruction-Following Generation. ... we design a Task-Driven Instruction-Following Construction Prompt (Table 2) using API ... GPT-4V ... To address these limitations, we introduce GPT-4 as a high-level semantic evaluator ... we design task-specific standardized GPT-4 evaluation prompts ... Nevertheless, we conservatively acknowledge that some bias may still remain"

    Reference texts for the two complex tasks are generated by GPT-4V under ontology-constrained, task-driven prompts; the same model family (GPT-4) later scores model outputs against those references using prompts that check semantic equivalence to the generated ground truth. Agreement therefore partly reflects shared GPT-family style and prompt distribution rather than fully independent engineering correctness. Manual review and the paper’s caveat reduce but do not break the loop; the Simple Recognition task (template-based) and architectural ablations remain outside it.

full rationale

This is an empirical MLLM paper, not a first-principles derivation. No equations reduce by construction, no fitted parameters are renamed as predictions, and no uniqueness theorems or ansätze are imported via self-citation. The only load-bearing circularity risk is the closed GPT-family loop: Complex Captioning and Multi-Engineering QA references are produced by GPT-4V (Section 3.1 Step 3, Table 2) under the same ontology priors and task-driven prompts later used by the GPT-4 evaluator (Section 5.2, Tables 6–8). Manual review of 800 samples (κ=0.874, ~5 % corrected) and the paper’s own residual-bias caveat mitigate but do not eliminate distributional self-consistency. Real-world transfer (30 samples) and the architectural ablations (PIE/GPR/LoRA) supply independent content, so the central claim is not forced. Score 4 reflects partial circularity confined to the evaluation hinge, not the model itself.

Axiom & Free-Parameter Ledger

4 free parameters · 3 axioms · 2 invented entities

The central performance claims rest on a handful of hand-chosen architectural hyper-parameters, the assumption that GPT-4V under ontology constraints yields faithful engineering text, and the newly invented modules PIE and GPR whose utility is demonstrated only inside this paper.

free parameters (4)
  • PIE recursive blocks = 3
    Number of recursive Feature/Spatial Adaptation blocks set to 3 after ablation (Table 14); chosen by hand for best Stage-1 accuracy.
  • GPR unfrozen shallow layers = 2
    Number of LLM layers left unfrozen under GPR set to 2 after sensitivity sweep (Table 16).
  • LoRA rank and progressive scaling = r=16, λ=16/32/64
    Rank r=16 and layer-wise λ schedule (16/32/64) selected after multi-dimensional search (Tables 18–20) to balance recognition vs. reasoning.
  • Fourier frequency parameters α,β = α=2.5, β=0.8
    Spatial Adaptation Module frequencies set to α=2.5, β=0.8 after comparison (Table 15).
axioms (3)
  • domain assumption Ontology-constrained GPT-4V generation under BLIP-2/CLIP/Shap-E view filtering produces instruction text whose engineering semantics are sufficiently accurate for both training and GPT-4 evaluation.
    Stated as the foundation of the progressive engine (Section 3.1) and of the evaluation protocol (Section 5.2).
  • domain assumption Frozen Point-BERT encoder plus a lightweight PIE is sufficient to capture the fine-grained geometric distinctions among 47 indoor component categories.
    Inherited from PointLLM and justified by Stage-1 ablations (Section 4.2).
  • standard math Standard Transformer/LoRA mathematics and MSE reconstruction loss behave as expected under mixed point-text sequences.
    Used without modification for the Projector, GPR loss (Eq. 9) and LoRA update (Eq. 11).
invented entities (2)
  • Point Information Enhancer (PIE) no independent evidence
    purpose: Recursively fuse local geometric and spatial cues into the frozen encoder output before projection.
    Introduced in Section 4.2.2; no independent external validation outside this paper’s ablations.
  • Geometry-Preserving Regularization (GPR) no independent evidence
    purpose: MSE reconstruction constraint on shallow unfrozen LLM layers to prevent geometric erosion.
    Introduced in Section 4.2.3; utility shown only by the paper’s own Stage-1 numbers.

reviewed 2026-07-12 · how reviews work

0 comments
Cite this review

Pith. "Pith review of From Geometric Labels to Semantic Understanding of Indoor Building Components Using Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/I3R2ULPG

@misc{pith2026260703661,
  author       = {Pith},
  title        = {Pith review of: From Geometric Labels to Semantic Understanding of Indoor Building Components Using Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I3R2ULPG}},
  note         = {Machine review of arXiv:2607.03661}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Point cloud-based understanding has become an important enabler for facility operation and maintenance involving indoor building components. However, existing methods output only discrete labels without explaining component functions or natural language interactions. This paper proposes Building-MLLM, a point cloud-centered multimodal large language model (MLLM) for indoor components, which models point clouds and instructions to generate responses across Simple Recognition, Complex Captioning, and Multi-Engineering Question Answering tasks. Building-MLLM addresses semantic concentration through four domain-specific mechanisms: Point Information Enhancer for task-relevant semantics, Geometry-Preserving Regularization preventing geometric erosion, fixed textual prefix for domain stabilization, and multi-dimensional LoRA balancing recognition with reasoning. A multi-constraint progressive instruction-generation engine is developed to compile a synthetic point cloud-text dataset with 4198 objects, 37,782 instruction-following pairs, and 47 categories. Experiments show that Building-MLLM achieves 88.00%, 65.10%, and 68.14% on the three task types, respectively, demonstrating superior indoor component language understanding and providing initial generalizability in transfer inference on other real-world datasets.

Figures

Figures reproduced from arXiv: 2607.03661 by Chao Yin, Shuju Jing.

Figure 1
Figure 1. Figure 1: Progressive instruction-following generation engine for constructing point cloud-text training data with [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Object category and instruction-following statistics in the proposed dataset. [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Examples of point cloud-text instruction-following data for different tasks. [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Overview of PointLLM [18] architecture. Building on this foundation, Building-MLLM adopts a two-stage training strategy for special￾ized point cloud–language understanding of indoor building components. In stage 1, we introduce the PIE module, the GPR mechanism, and a fixed textual prefix to enhance domain-relevant se￾mantics and improve the stability of point cloud–language alignment. In stage 2, we desig… view at source ↗
Figure 5
Figure 5. Figure 5: Building-MLLM architecture for geometry-preserving cross-modal alignment in Stage 1. [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Point Information Enhancer (PIE) module for task-oriented geometric enhancement. [PITH_FULL_IMAGE:figures/full_fig_p018_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Proposed Geometry-Preserving Regularization (GPR) on shallow LLM layers. [PITH_FULL_IMAGE:figures/full_fig_p019_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Proposed Building-MLLM framework with stage-2 instruction fine-tuning training. [PITH_FULL_IMAGE:figures/full_fig_p020_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Multi-Dimensional LoRA fine-tuning Strategy for Building-MLLM. [PITH_FULL_IMAGE:figures/full_fig_p021_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Test case demonstration of Building-MLLM (1). [PITH_FULL_IMAGE:figures/full_fig_p029_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Test case demonstration of Building-MLLM (2). [PITH_FULL_IMAGE:figures/full_fig_p030_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Inference case studies with GPT-4–based evaluation. [PITH_FULL_IMAGE:figures/full_fig_p031_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Inference cases transferring Building-MLLM to real-world datasets. [PITH_FULL_IMAGE:figures/full_fig_p033_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Failure cases of Building-MLLM on real-world point cloud datasets. [PITH_FULL_IMAGE:figures/full_fig_p034_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Category-level recognition comparison [PITH_FULL_IMAGE:figures/full_fig_p036_15.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

59 extracted references · 20 canonical work pages · 1 internal anchor

  1. [1]

    M. Li, V. J. Gan, B. Wang, Integrating hierarchical segmentation and vision-language reason- ing for spatially complex and occluded MEP point clouds, Automation in Construction 179 (2025) 106455.doi:10.1016/j.autcon.2025.106455

    work page doi:10.1016/j.autcon.2025.106455 2025

  2. [3]

    C. Yin, B. Yang, J. C. Cheng, V. J. Gan, B. Wang, J. Yang, Label-efficient semantic segmen- tation of large-scale industrial point clouds using weakly supervised learning, Automation in Construction 148 (2023) 104757.doi:10.1016/j.autcon.2023.104757

    work page doi:10.1016/j.autcon.2023.104757 2023

  3. [4]

    Zhang, L

    Y. Zhang, L. Chen, Q. Lu, Y. Zou, X. Xiahou, S. Sølvsten, C. Hancock, Semi-automated localised updating for as-built BIM of piping systems using point cloud data, Automation in Construction 181 (2026) 106609.doi:10.1016/j.autcon.2025.106609

    work page doi:10.1016/j.autcon.2025.106609 2026

  4. [5]

    A. B. Ersöz, F. Bosché, Evaluating confidence in geometric matching between 3D point clouds and BIM models by integrating coverage, distance, and distribution metrics, Automation in Construction 181 (2026) 106636.doi:10.1016/j.autcon.2025.106636. 41

    doi:10.1016/j.autcon.2025.106636 2026

  5. [6]

    Z. Liu, Y. Mo, B. Hou, M. Li, W. Li, C. Xu, Digital twin modelling approaches and applica- tions in urban infrastructure operations and maintenance, Automation in Construction 179 (2025) 106445.doi:10.1016/j.autcon.2025.106445

    work page doi:10.1016/j.autcon.2025.106445 2025

  6. [7]

    H. Yue, Q. Wang, M. Zhao, Z. Yang, L. Lu, Advancements in digital twin applications for intelligent construction quality management, Journal of Construction Engineering and Man- agement 152 (1) (2026) 03125011.doi:10.1061/JCEMD4.COENG-17126

    work page doi:10.1061/jcemd4.coeng-17126 2026

  7. [8]

    C. Yin, J. C. Cheng, B. Wang, V. J. Gan, Automated classification of piping components from 3D LiDAR point clouds using SE-PseudoGrid, Automation in Construction 139 (2022) 104300.doi:10.1016/j.autcon.2022.104300

    work page doi:10.1016/j.autcon.2022.104300 2022

  8. [9]

    Teclaw, A

    W. Teclaw, A. Tomczak, M. Kasznia, M. Luczkowski, N. Labonnote, E. Hjelseth, ContextNET-Contextual classification of building components using deep neural networks, Advanced Engineering Informatics 67 (2025) 103484.doi:10.1016/j.aei.2025.103484

    work page doi:10.1016/j.aei.2025.103484 2025

  9. [10]

    H. Liu, C. Li, Q. Wu, Y. J. Lee, Visual instruction tuning, Advances in neural information processing systems 36 (2023) 34892–34916.doi:10.48550/arXiv.2304.08485

    Pith/arXiv arXiv doi:10.48550/arxiv.2304.08485 2023

  10. [11]

    Achiam, S

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al., Gpt-4 technical report, arXiv preprint arXiv:2303.08774 (2023).doi:10.48550/arXiv.2303.08774

    Pith/arXiv arXiv doi:10.48550/arxiv.2303.08774 2023

  11. [12]

    Areerob, V.-Q

    K. Areerob, V.-Q. Nguyen, X. Li, S. Inadomi, T. Shimada, H. Kanasaki, Z. Wang, M. Sug- anuma, K. Nagatani, P.-j. Chun, et al., Multimodal artificial intelligence approaches using large language models for expert-level landslide image analysis, Computer-Aided Civil and Infrastructure Engineering (2025).doi:10.1111/mice.13482

    doi:10.1111/mice.13482 2025

  12. [13]

    X. Wang, Q. Yue, X. Liu, Crack image classification and information extraction in steel bridges using multimodal large language models, Automation in Construction 171 (2025) 105995.doi:10.1016/j.autcon.2025.105995

    doi:10.1016/j.autcon.2025.105995 2025

  13. [14]

    P. K.-Y. Wong, J. C. Cheng, C.-F. Chan, P.-H. Leung, X. Tao, et al., Enhancing visual- LLM for construction site safety compliance via prompt engineering and Bi-stage retrieval- augmented generation, Automation in Construction 179 (2025) 106490.doi:10.1016/j. autcon.2025.106490

    doi:10.1016/j 2025

  14. [15]

    Armeni, O

    I. Armeni, O. Sener, A. R. Zamir, H. Jiang, I. Brilakis, M. Fischer, S. Savarese, 3d semantic parsing of large-scale indoor spaces, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 1534–1543.doi:10.1109/cvpr.2016.170

    doi:10.1109/cvpr.2016.170 2016

  15. [16]

    C. Yin, B. Wang, V. J. Gan, M. Wang, J. C. Cheng, Automated semantic segmentation of industrial point clouds using ResPointNet++, Automation in Construction 130 (2021) 103874. doi:10.1016/j.autcon.2021.103874

    work page doi:10.1016/j.autcon.2021.103874 2021

  16. [17]

    T. Luo, C. Rockwell, H. Lee, J. Johnson, Scalable 3d captioning with pretrained models, Advances in Neural Information Processing Systems 36 (2023) 75307–75337.doi:10.48550/ arXiv.2306.07279

    Pith/arXiv arXiv 2023

  17. [18]

    R. Xu, X. Wang, T. Wang, Y. Chen, J. Pang, D. Lin, Pointllm: Empowering large language models to understand point clouds, in: European Conference on Computer Vision, Springer, 2024, pp. 131–147.doi:10.1007/978-3-031-72698-9_8. 42

    work page doi:10.1007/978-3-031-72698-9_8 2024

  18. [19]

    Y. Tang, X. Han, X. Li, Q. Yu, Y. Hao, L. Hu, M. Chen, Minigpt-3d: Efficiently aligning 3d point clouds with large language models using 2d priors, in: Proceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 6617–6626.doi:10.1145/3664647. 3681257

    doi:10.1145/3664647 2024

  19. [20]

    Z. Qi, Y. Fang, Z. Sun, X. Wu, T. Wu, J. Wang, D. Lin, H. Zhao, Gpt4point: A unified framework for point-language understanding and generation, in: Proceedings of the ieee/cvf conference on computer vision and pattern recognition, 2024, pp. 26417–26427.doi:10.1109/ cvpr52733.2024.02495

    arXiv 2024

  20. [21]

    Z. Qi, R. Dong, S. Zhang, H. Geng, C. Han, Z. Ge, L. Yi, K. Ma, Shapellm: Universal 3d object understanding for embodied interaction, in: European Conference on Computer Vision, Springer, 2024, pp. 214–238.doi:10.1007/978-3-031-72775-7_13

    work page doi:10.1007/978-3-031-72775-7_13 2024

  21. [22]

    E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al., Lora: Low-rank adaptation of large language models., ICLR 1 (2) (2022) 3.doi:10.48550/arXiv. 2106.09685

    Pith/arXiv arXiv doi:10.48550/arxiv 2022

  22. [23]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., Learning transferable visual models from natural language su- pervision, in: International conference on machine learning, PmLR, 2021, pp. 8748–8763. doi:10.48550/arXiv.2103.00020

    Pith/arXiv arXiv doi:10.48550/arxiv.2103.00020 2021

  23. [24]

    J. Li, D. Li, S. Savarese, S. Hoi, Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, in: International conference on machine learning, PMLR, 2023, pp. 19730–19742.doi:10.48550/arXiv.2301.12597

    Pith/arXiv arXiv doi:10.48550/arxiv.2301.12597 2023

  24. [25]

    H. Jun, A. Nichol, Shap-e: Generating conditional 3d implicit functions, arXiv preprint arXiv:2305.02463 (2023).doi:10.48550/arXiv.2305.02463

    Pith/arXiv arXiv doi:10.48550/arxiv.2305.02463 2023

  25. [26]

    Mirzaei, M

    K. Mirzaei, M. Arashpour, E. Asadi, H. Masoumi, Y. Bai, A. Behnood, 3D point cloud data processing with machine learning for construction and infrastructure applications: A comprehensive review, Advanced Engineering Informatics 51 (2022) 101501.doi:10.1016/ j.aei.2021.101501

    arXiv 2022

  26. [27]

    H. Yue, Q. Wang, X. Nie, H. Fang, J. C. Cheng, S. Jing, B. Wang, Point cloud instance segmentation for building indoor scenes using deep learning and BIM-Generated synthetic point clouds, Advanced Engineering Informatics 72 (2026) 104453.doi:https://doi.org/ 10.1016/j.aei.2026.104453

    work page doi:10.1016/j.aei.2026.104453 2026

  27. [28]

    M. Li, V. J. Gan, K. Li, Vision foundation model with connectivity-based inferencing for zero- shot bridge element segmentation and geometric reconstruction, Automation in Construction 188 (2026) 106995.doi:10.1016/j.autcon.2026.106995

    work page doi:10.1016/j.autcon.2026.106995 2026

  28. [29]

    H. Yue, Q. Wang, Y. Huang, H. Fang, J. C. Cheng, L. Li, Enhancing 3D Object Detection for Building Indoor Scenes Using BIM-Generated Synthetic Point Clouds, Journal of Comput- ing in Civil Engineering 40 (4) (2026) 05026003.doi:https://doi.org/10.1061/JCCEE5. CPENG-7008. 43

    doi:10.1061/jccee5 2026

  29. [30]

    H. Yue, Q. Wang, L. Cui, C. Li, H. Fang, J. C. Cheng, Transfer Learning for Deep Learning-Based Point Cloud Tasks in Construction Scenes, Journal of Construction Engi- neering and Management 152 (6) (2026) 04026058.doi:https://doi.org/10.1061/JCEMD4. COENG-18262

    work page doi:10.1061/jcemd4 2026

  30. [31]

    C. R. Qi, H. Su, K. Mo, L. J. Guibas, Pointnet: Deep learning on point sets for 3d classification and segmentation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 652–660.doi:10.1109/cvpr.2017.16

    doi:10.1109/cvpr.2017.16 2017

  31. [32]

    C. R. Qi, L. Yi, H. Su, L. J. Guibas, Pointnet++: Deep hierarchical feature learning on point sets in a metric space, Advances in neural information processing systems 30 (2017). doi:10.48550/arXiv.1706.02413

    Pith/arXiv arXiv doi:10.48550/arxiv.1706.02413 2017

  32. [33]

    H. Zhao, L. Jiang, J. Jia, P. H. Torr, V. Koltun, Point transformer, in: Proceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 16259–16268.doi: 10.1109/iccv48922.2021.01595

    doi:10.1109/iccv48922.2021.01595 2021

  33. [34]

    B. Koo, R. Jung, Y. Yu, Automatic classification of wall and door BIM element subtypes using 3D geometric deep neural networks, Advanced Engineering Informatics 47 (2021) 101200. doi:10.1016/j.aei.2020.101200

    work page doi:10.1016/j.aei.2020.101200 2021

  34. [35]

    Emunds, N

    C. Emunds, N. Pauen, V. Richter, J. Frisch, C. van Treeck, SpaRSE-BIM: Classification of IFC-based geometry via sparse convolutional neural networks, Advanced Engineering Infor- matics 53 (2022) 101641.doi:10.1016/j.aei.2022.101641

    work page doi:10.1016/j.aei.2022.101641 2022

  35. [36]

    M. A. Uy, Q.-H. Pham, B.-S. Hua, T. Nguyen, S.-K. Yeung, Revisiting point cloud classifi- cation: A new benchmark dataset and classification model on real-world data, in: Proceed- ings of the IEEE/CVF international conference on computer vision, 2019, pp. 1588–1597. doi:10.1109/iccv.2019.00167

    doi:10.1109/iccv.2019.00167 2019

  36. [37]

    X. Wu, L. Jiang, P.-S. Wang, Z. Liu, X. Liu, Y. Qiao, W. Ouyang, T. He, H. Zhao, Point transformer v3: Simpler faster stronger, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, pp. 4840–4851.doi:10.48550/arXiv.2312. 10035

    doi:10.48550/arxiv.2312 2024

  37. [38]

    R. Zhai, J. Zou, Y. He, L. Meng, BIM-driven data augmentation method for semantic seg- mentation in superpoint-based deep learning network, Automation in construction 140 (2022) 104373.doi:10.1016/j.autcon.2022.104373

    doi:10.1016/j.autcon.2022.104373 2022

  38. [39]

    H. Yue, S. Jing, G. Huang, P. Peng, L. Li, Q. Wang, Weakly supervised framework for semantic segmentation of MEP point clouds, Automation in Construction 187 (2026) 106960. doi:10.1016/j.autcon.2026.106960

    work page doi:10.1016/j.autcon.2026.106960 2026

  39. [40]

    Y. Hong, H. Zhen, P. Chen, S. Zheng, Y. Du, Z. Chen, C. Gan, 3D-LLM: Injecting 3D world into large language models, Advances in Neural Information Processing Systems 36 (2023) 20482–20494.doi:10.48550/arXiv.2307.12981

    Pith/arXiv arXiv doi:10.48550/arxiv.2307.12981 2023

  40. [41]

    X. Yu, L. Tang, Y. Rao, T. Huang, J. Zhou, J. Lu, Point-bert: Pre-training 3d point cloud transformers with masked point modeling, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 19313–19322.doi:10.1109/cvpr52688. 2022.01871. 44

    doi:10.1109/cvpr52688 2022

  41. [42]

    W. Dai, J. Li, D. Li, A. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, S. Hoi, Instructblip: Towards general-purpose vision-language models with instruction tuning, Advances in neural information processing systems 36 (2023) 49250–49267.doi:10.48550/arXiv.2305.06500

    Pith/arXiv arXiv doi:10.48550/arxiv.2305.06500 2023

  42. [43]

    Deitke, D

    M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, A. Farhadi, Objaverse: A universe of annotated 3d objects, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 13142–13153.doi:10.1109/cvpr52729.2023.01263

    doi:10.1109/cvpr52729.2023.01263 2023

  43. [44]

    Utkucu, H

    D. Utkucu, H. Ying, Z. Wang, R. Sacks, Classification of architectural and MEP BIM objects for building performance evaluation, Advanced Engineering Informatics 61 (2024) 102503. doi:10.1016/j.aei.2024.102503

    doi:10.1016/j.aei.2024.102503 2024

  44. [45]

    1179–1189.doi:10.1109/cvpr52729.2023.00120

    L.Xue, M.Gao, C.Xing, R.Martín-Martín, J.Wu, C.Xiong, R.Xu, J.C.Niebles, S.Savarese, Ulip: Learning a unified representation of language, images, and point clouds for 3d un- derstanding, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 1179–1189.doi:10.1109/cvpr52729.2023.00120

    doi:10.1109/cvpr52729.2023.00120 2023

  45. [46]

    S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, E. Chen, A survey on multimodal large language models, National Science Review 11 (12) (2024) nwae403.doi:10.1093/nsr/nwae403

    doi:10.1093/nsr/nwae403 2024

  46. [47]

    A. B. Sai, A. K. Mohankumar, M. M. Khapra, A survey of evaluation metrics used for nlg systems, ACM Computing Surveys (CSUR) 55 (2) (2022) 1–39.doi:10.1145/3485766

    doi:10.1145/3485766 2022

  47. [48]

    Papineni, S

    K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, Bleu: a method for automatic evaluation of machine translation, in: Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311–318.doi:10.3115/1073083.1073135

    doi:10.3115/1073083.1073135 2002

  48. [49]

    Lin, Rouge: A package for automatic evaluation of summaries, in: Text Summarization Branches Out: Proceedings of the ACL Workshop, 2004, pp

    C.-Y. Lin, Rouge: A package for automatic evaluation of summaries, in: Text Summarization Branches Out: Proceedings of the ACL Workshop, 2004, pp. 74–81.doi:10.3115/1220355. 1220369

    work page doi:10.3115/1220355 2004

  49. [50]

    Banerjee, A

    S. Banerjee, A. Lavie, METEOR: An automatic metric for MT evaluation with improved correlation with human judgments, in: Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, 2005, pp. 65– 72.doi:10.3115/1220575.1220599

    doi:10.3115/1220575.1220599 2005

  50. [51]

    Reimers, I

    N. Reimers, I. Gurevych, Sentence-bert: Sentence embeddings using siamese bert-networks, arXiv preprint arXiv:1908.10084 (2019).doi:10.18653/v1/d19-1410

    Pith/arXiv arXiv doi:10.18653/v1/d19-1410 1908

  51. [52]

    T. Gao, X. Yao, D. Chen, Simcse: Simple contrastive learning of sentence embeddings, arXiv preprint arXiv:2104.08821 (2021).doi:10.18653/v1/2021.emnlp-main.552

    Pith/arXiv arXiv doi:10.18653/v1/2021.emnlp-main.552 2021

  52. [53]

    IFCNet: A Benchmark Dataset for IFC Entity Classification

    C. Emunds, N. Pauen, V. Richter, J. Frisch, C. Van Treeck, IFCNet: A benchmark dataset for IFC entity classification, in: Proceedings of the 28th International Workshop on Intelligent Computing in Engineering, EG-ICE, Vol. 166, 2021.doi:10.48550/arXiv.2106.09712

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2106.09712 2021

  53. [54]

    BIMobject,https://www.bimobject.com/en, accessed: 2025-11-13 (2023)

    2025

  54. [55]

    BIMstore,https://www.bimstore.co/, accessed: 2025-11-13 (2023). 45

    2025

  55. [56]

    NBS Enterprises Ltd., NBS Source,https://source.thenbs.com/, accessed: 2025-11-13 (2023)

    2025

  56. [57]

    Construction Industry Council, CIC BIM Object Library,https://www.bim.cic.hk/en/ resources/bim_objects_list, accessed: 2025-11-13 (2023)

    2025

  57. [58]

    Collins, BIMGEOM (2021).doi:10.7910/DVN/YK86XK

    F. Collins, BIMGEOM (2021).doi:10.7910/DVN/YK86XK

    work page doi:10.7910/dvn/yk86xk 2021

  58. [59]

    C. Yeo, S. Kim, H. Kim, S. Kim, D. Mun, Deep learning applications in an industrial process plant: repository of segmented point clouds for pipework components, JMST Advances 2 (1) (2020) 15–24.doi:10.1007/s42791-019-00027-y

    work page doi:10.1007/s42791-019-00027-y 2020

  59. [60]

    What is it?

    R. Zhang, Z. Guo, W. Zhang, K. Li, X. Miao, B. Cui, Y. Qiao, P. Gao, H. Li, Pointclip: Point cloud understanding by clip, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 8552–8562.doi:10.48550/arXiv.2112.02413. Appendix A. Prompt Robustness Analysis During the training stage, we introduced a diverse set of ...

    Pith/arXiv arXiv doi:10.48550/arxiv.2112.02413 2022

This paper was first reviewed by grok-4.5 on July 12, 2026.