REVIEW 3 major objections 3 minor 48 references
What makes a good BIM design: quantitative linking between design behavior and quality
T0 review · 3 major / 3 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Design behavior in BIM software predicts final model quality with an R² of 0.88.
desk verdict A useful problem and real data collection are undermined by a load-bearing evaluation flaw: the window-cropped augmentation with random split lets the model memorize session- and student-specific labels, so the claimed R2 of 0.88 is not evidence of generalization. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the window-cropping data augmentation combined with density features over the cropped sequence. Each full Revit session is sliced into subsequences of length $N=30000$ rows with step $s=2000$; every cropped window inherits the full session's quality score, expanding 136 models into thousands of training samples. From each window the authors compute 29 statistical densities (data density and time density) that compress behavior into feature vectors, then train tree ensembles on these vectors and interpret them with SHAP values. The ExtraTrees algorithm's random split-point selection is what lets the model capture the feature-score relation; SHAP attribution is what turns the model into the claim that proficiency and intention stability matter.
What would settle it
A concrete check is to split by student or by session instead of by window: if the model's test $R^2$ drops substantially (for example, below 0.5) when no window from a given session appears in both training and test, then the reported 0.88 reflects window overlap rather than a transferable behavior-quality relationship. A second check is to score the same 136 models with a second independent pair of raters and see whether the learned features still predict the new scores.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that a quantitative, learnable relationship exists between BIM design behavior and the quality of the resulting model. The relationship is measured by 29 hand-crafted features—data densities such as the share of successful transactions, added/deleted/modified components, command execution modes (ribbon, shortcut, pushbutton), pause lengths, and effective work time—and target scores assigned by expert raters. Across 42 regression algorithms, only the ensemble family learns the mapping well; ExtraTrees with a cropping window length of 30,000 rows and step of 2,000 achieves the reported test-set $R^2=0.88$ with RMSE 2.29 against a score standard deviation of 7.61. SHAP interpretation identifies command-execution features and component-manipulation densities as the main drivers, leading the authors to conclude that designer proficiency and stability of design intentions are the two behavior-level levers on final quality.
Load-bearing premise
The load-bearing premise is that a 30,000-row window clipped from the middle of a designer's session shares the same quality score as the whole finished model, and that randomly splitting these overlapping windows into training and test sets gives an honest measure of generalization.
Editorial extensions
If this is right
- If the relationship holds beyond this student dataset, BIM software could score a model in progress from behavior alone, before formal quality checks.
- Feature importances give testable guidance: training designers to use shortcuts and reduce ribbon-button hunting should improve both efficiency and quality.
- High deletion density, interpreted as large design-intention changes, becomes a measurable early-warning signal for final quality.
- The same pipeline—log capture, window cropping, density features, ensemble regression—could be applied to other software-mediated tasks where outcome quality is scored.
Reading between the lines
- Beyond the paper's claims: the reported $R^2$ likely overstates how well behavior predicts quality for a new designer, because each student contributes many overlapping windows and the random 80/20 split puts windows from the same session on both sides of the train/test boundary; a session-level or student-level split would be a harder test.
- Beyond the paper's claims: an untested extension is to replace density features with sequence models (LSTM or Transformer) that respect command order; the paper explicitly notes its statistical features may miss sequential structure, so this is a natural next experiment.
- Beyond the paper's claims: the quality score is manually assigned by two raters with a benchmark student; if ratings were gathered from more raters, inter-rater agreement could separate true behavioral signal from rater subjectivity.
- Beyond the paper's claims: a practical deployment would need to distinguish deliberate pauses from idle time; the paper treats pauses over five minutes as non-work, but a designer thinking while away from the keyboard is still productive.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper aims to establish a quantitative relationship between BIM design behaviors and design quality. The authors collected behavioral data from 68 students' Revit modeling sessions using log files and a custom plugin, scored the resulting BIM models with a hand-crafted evaluation rubric, augmented the sequence data by window cropping, extracted 29 statistical features, and trained 42 regression models. They report that an Extremely Random Trees model achieves an R2 of 0.88 on the test set and use SHAP values to identify features related to command execution and design-intention changes as influential.
Significance. If the reported result were valid, the paper would provide a useful data-driven bridge between process-level design behavior and final design quality in the AEC domain, complementing existing qualitative studies. The data collection pipeline, combining log mining with a real-time plugin, is a genuine practical contribution, and the authors are transparent about many experimental details. However, the central quantitative claim is undermined by a procedural flaw in the evaluation setup: the training and test sets are not independent, so the reported R2 does not measure generalization to unseen modeling sessions. The SHAP-based interpretations are also built on this leaky model. The underlying research question is valuable and the dataset is interesting, but the paper's main conclusion is not currently supported.
major comments (3)
- [Section 3.3.1 and Section 4.2] The window-cropping augmentation with N=30,000 and s=2,000 creates overlapping windows that all inherit the same quality score as the full session, and the random 80/20 split places near-duplicate windows from the same session in both training and test sets. Consequently, the model can effectively memorize session- or student-specific patterns, and the reported test R2 of 0.88 is not evidence of generalization to unseen modeling sessions. The evaluation must be repeated with a split that respects the cluster structure, e.g., holding out entire sessions or entire students.
- [Section 4.3] The paper acknowledges that very small step sizes can cause information leakage due to high similarity between samples, but then selects s=2,000, where adjacent windows still overlap by 28,000 rows. The claim that this setting maintains 'a reasonable level of reliability' is not supported by any analysis. Even with non-overlapping windows, all windows from the same student share the same label, so the effective number of independent samples is at most 68 students (or 136 models), not the reported 493–2,326 augmented samples.
- [Section 4.4] The SHAP interpretation is performed on a model trained under the leaky data-splitting protocol. The feature importance findings, such as the positive association of pushbutton% with quality, may reflect memorization of session-specific or student-specific patterns rather than a generalizable behavior-quality relationship. A clean model trained under a proper grouped split is required before these domain conclusions can be considered reliable.
minor comments (3)
- [Section 3.2] The authors use the normality of the score distribution to affirm data reliability, but this is not a strong check against scoring subjectivity; a normal distribution of scores can arise from many rating processes and does not validate the rubric's accuracy.
- [Section 3.3.2] The table of statistical indicators is labeled 'Table 11' even though it appears to be the first numbered table in the manuscript; the numbering should be corrected throughout.
- [Figures 7 and 8] In the hyperparameter optimization figures, the RMSE and R2 values are presented together without clear bar-chart legends for the two algorithms; the captions should explicitly state which color corresponds to Bagging and which to ExtraTrees, and whether the printed numbers refer to training or test sets.
Circularity Check
The reported test R2 of 0.88 is not evidence of generalization: window-cropped test samples are generated from the same sessions as training samples and share identical labels.
-
fitted input called prediction
[Section 3.3.1 (window-crop augmentation) with Section 4.2 (80/20 random split); hyperparameters N=30,000, s=2,000 in Section 4.3]
"Each data file is sliced into a number of sub-sequences with a length of N, and the cropping window moves forward with a step size s... The corresponding quality score for each newly generated data sample is consistent with the original data. ... The data set was randomly split in an 80-20 ratio, with 80% of the data allocated to the training set and the remaining 20% allocated to the test set."
By construction, all augmented windows from one Revit session carry the same quality score as the original file. With the final hyperparameters N=30,000 and s=2,000, adjacent windows overlap by 28,000 rows, so training and test windows can be near-duplicates of the same session. The random 80/20 split therefore places windows from the same session (and even overlapping windows) on both sides, letting the model memorize session- or student-specific patterns instead of learning a general behavior-to-quality relationship. The reported test R2=0.88 is thus not an unbiased estimate for unseen modeling sessions; the central predictive claim reduces to recall of labels already seen during training.
full rationale
The paper's central claim rests on the test-set R2, but the test set is constructed by randomly splitting overlapping windows from the same Revit sessions, so training and test labels are not independent. The paper itself acknowledges that small step sizes cause information leakage, yet the chosen s=2,000 still yields 93% overlap, and even non-overlapping windows from the same student share one score. No definitional equivalence or self-citation chain is present: the features and scores are not defined in terms of one another, and the only self-citation ([33]) describes the data-collection plugin rather than the quantitative result. The circularity is therefore confined to the validation protocol, but it is load-bearing because the paper's core empirical claim is the R2 value.
Assumptions & free parameters
free parameters (3)
- Augmentation window length N =
30000
- Augmentation step size s =
2000
- Quality score weights =
20/20/20/10 (completeness, accuracy, complexity, structural completeness)
assumptions (4)
- domain assumption Design behavior and final design quality are inherently interlinked.
- domain assumption Manual expert scores are a valid quantitative index of BIM design quality.
- ad hoc to paper Each cropped subsequence inherits the quality score of the original session.
- domain assumption SHAP feature importance can be interpreted as causal guidance for design practice.
Cite this review
Pith. "Pith review of What makes a good BIM design: quantitative linking between design behavior and quality." pith.science (2026). https://pith.science/paper/OUIOWMSM
@misc{pith2026241109481,
author = {Pith},
title = {Pith review of: What makes a good BIM design: quantitative linking between design behavior and quality},
year = {2026},
howpublished = {\url{https://pith.science/paper/OUIOWMSM}},
note = {Machine review of arXiv:2411.09481}
}
read the original abstract
In the Architecture Engineering & Construction (AEC) industry, how design behaviors impact design quality remains unclear. This study proposes a novel approach, which, for the first time, identifies and quantitatively describes the relationship between design behaviors and quality of design based on Building Information Modeling (BIM). Real-time collection and log mining are integrated to collect raw data of design behaviors. Feature engineering and various machine learning models are then utilized for quantitative modeling and interpretation. Results confirm an existing quantifiable relationship which can be learned by various models. The best-performing model using Extremely Random Trees achieved an R2 value of 0.88 on the test set. Behavioral features related to designer's skill level and changes of design intentions are identified to have significant impacts on design quality. These findings deepen our understanding of the design process and help forming BIM designs with better quality.
Reference graph
Works this paper leans on
-
[1]
No.1, Records with “Transaction Successful” : Records with the phrase "Transaction Successful" come from log files, which usually represent a group of modifications of the BIM model have been successfully executed and recorded through Revit's own transaction mechanism. Therefore, the density of such records can reflect how many substantial modifications t...
-
[2]
the total number of modified elements
No. 2-6, Records related to component manipulation : Records of adding, deleting and modifying elements come from data files collected via the Design Tracker, which can also reflect the efficiency of the designer for different types of specific modification behaviors. Among them, the total number of modified elements is not considered as a valuable statis...
-
[3]
No.7-11, Records related to commands: In the BIM-based model design process, the most important behavior is executing commands. Many statistical indicators related to the execution of commands are considered important , including the total number of commands executed, the number of commands canceled or undone, the number of commands executed through the r...
-
[4]
No.12-14, Records related to pauses: Designers may turn to deal with other affairs or take a rest in the middle of the design process, resulting in a pause in the data sample, that is, an interval of time between the two adjacent data records. In general, considering the time of the operation itself and the response time of the computer, the time interval...
-
[5]
No.15, Effective work time: The last time-related feature is calculated from the statistical indicator “effective work time”. In this study, effective work time is defined as the sum of the total time span of the sample excluding the sum of all time intervals over 5 minutes. A pause of more than 5 minutes usually means the designer is no longer working on...
work page 1917
-
[6]
C. Nelson, W. Ronco, J. Beveridge, J. Reigle, J. Cramer, Managing Quality in Architecture: Integrating BIM, Risk and Design Process, 2nd ed., Routledge, London, 2017. https://doi.org/10.4324/9781315272382
-
[7]
The data acquired is insufficient in volume, making it challenging to support the training of more complex deep learning algorithms with higher data requirements
-
[8]
More diverse data samples are needed to improve the accuracy of the models in real-world scenarios
The designers studied were students, whose modeling capabilities may not be representative of the industry standard. More diverse data samples are needed to improve the accuracy of the models in real-world scenarios
Show all 48 references
-
[9]
Exploring sequential analysis methods, such as Recurrent Neural Network and Long Short-Term Memory Network, could be a valuable direction for further research
Design behavior data has a strong sequential characteristic, which the statistical features may have difficulty capturing. Exploring sequential analysis methods, such as Recurrent Neural Network and Long Short-Term Memory Network, could be a valuable direction for further rese...
-
[10]
Moayeri, O
V . Moayeri, O. Moselhi, Z. Zhu, BIM-based model for quantifying the design change time ripple effect, Canadian Journal of Civil Engineering 44 (2017) 626 –
2017
-
[11]
Y . Pan, L. Zhang, BIM log mining: Learning and predicting design commands, Automation in Construction 112 (2020) 103107. https://doi.org/10.1016/j.autcon.2020.103107
2020
-
[12]
Lin, Y .-C
J.-R. Lin, Y .-C. Zhou, Semantic classification and hash code accelerated detection of design changes in BIM models, Automation in Construction 115 (2020) 103212. https://doi.org/10.1016/j.autcon.2020.103212
2020
-
[13]
Lee Hansen, J
K. Lee Hansen, J. Vanegas, Improving design quality through briefing automation, Building Research & Information 31 (2003) 379 –386. https://doi.org/10.1080/0961321032000105395
2003 doi
-
[14]
Sampaio, E
A.Z. Sampaio, E. Berdeja, Collaborative BIM environment as a support to conflict analysis in building design, in: 2017 4th Experiment@International Conference (Exp.at’17), IEEE, Faro, Portugal, 2017: pp. 77 –82. https://doi.org/10.1109/EXPAT.2017.7984348
2017
-
[15]
Knotten, F
V . Knotten, F. Svalestuen, G.K. Hansen, O. Lædre, Design management in the building process - A review of current literature, Procedia Economics and Finance 21 (2015) 120–127. https://doi.org/10.1016/S2212-5671(15)00158-6
2015 doi
-
[16]
Bloch, R
T. Bloch, R. Sacks, Comparing machine learning and rule -based inferencing for semantic enrichment of BIM models, Automation in Construction 91 (2018) 256–
2018
-
[17]
Kasim, N.A.Z
N. Kasim, N.A.Z. Abidin, R. Zainal, N. Sarpin, M.H.I.A. Rahim, M. Saikah, Best practices of Building Information Modelling (BIM) implementation in design phase for construction project, IOP Conf. Ser.: Mater. Sci. Eng. 271 (2017) 012038. https://doi.org/10.1088/1757-899X/271/1/012038
2017 doi
-
[18]
Saka, D.W.M
A.B. Saka, D.W.M. Chan, Knowledge, skills and functionalities requirements for quantity surveyors in building information modelling (BIM) work environment: an international Delphi study, Architectural Engineering and Design Management 16 (2020) 227–246. https://doi.org/10.1080...
2020
-
[19]
Alsaadani, C
S. Alsaadani, C. Bleil De Souza, Performer, consumer or expert? A critical review of building performance simulation training paradigms for building design decision-making, Journal of Building Performance Simulation 12 (2019) 289–307. https://doi.org/10.1080/19401493.2018.1447602
2019
-
[20]
Brown, Design performance and designer preference in an interactive, data- driven conceptual building design scenario, Design Studies 68 (2020) 1 –33
N.C. Brown, Design performance and designer preference in an interactive, data- driven conceptual building design scenario, Design Studies 68 (2020) 1 –33. https://doi.org/10.1016/j.destud.2020.01.001
2020 doi
-
[21]
Vegard Knotten, G.K
O.L. Vegard Knotten, G.K. Hansen, Building design management – key success factors, Architectural Engineering and Design Management 13 (2017) 479 –493. https://doi.org/10.1080/17452007.2017.1345718
2017
-
[22]
S. Han, S. Lee, F. Peña -Mora, Identification and quantification of Non -Value- Adding Effort from errors and changes in design and construction projects, Journal of Construction Engineering and Management 138 (2012) 98 –109. https://doi.org/10.1061/(ASCE)CO.1943-7862.0000406
2012
-
[23]
Koo, J.T
H.J. Koo, J.T. O’Connor, Building information modeling as a tool for prevention of design defects, Construction Innovation 22 (2021) 870 –890. https://doi.org/10.1108/CI-02-2021-0033
2021 doi
-
[24]
Leygonie, A
R. Leygonie, A. Motamedi, I. Iordanova, Development of quality improvement procedures and tools for facility management BIM, Developments in the Built Environment 11 (2022) 100075. https://doi.org/10.1016/j.dibe.2022.100075
2022
-
[25]
black box
utilized the integration of the Kohonen Clustering Network (KCN) and Fuzzy C-Means (FCM) to analyze the design efficiency of designers across different time periods. By clustering designers based on this analysis, they aimed to optimize work schedules and enhance team producti...
2017
-
[26]
Akponeware, Z.A
A.O. Akponeware, Z.A. Adamu, Clash detection or clash avoidance? An investigation into coordination problems in 3D BIM, Buildings 7 (2017) 75. https://doi.org/10.3390/buildings7030075
2017 doi
-
[27]
Zhang, B
L. Zhang, B. Ashuri, BIM log mining: Discovering social networks, Automation in Construction 91 (2018) 31–43. https://doi.org/10.1016/j.autcon.2018.03.009
2018 doi
-
[28]
Jiang, X
S. Jiang, X. Feng, B. Zhang, J. Shi, Semantic enrichment for BIM: Enabling technologies and applications, Advanced Engineering Informatics 56 (2023) 101961. https://doi.org/10.1016/j.aei.2023.101961
2023
-
[29]
Fortineau, T
V . Fortineau, T. Paviot, L. Louis-Sidney, S. Lamouri, SWRL as a rule language for ontology-based models in power plant design, in: L. Rivest, A. Bouras, B. Louhichi (Eds.), Product Lifecycle Management. Towards Knowledge -Rich Enterprises, Springer, Berlin, Heidelberg, 2012: ...
2012 doi
-
[30]
Tomasevic, N
N. Tomasevic, N. Gvozdenovic, S. Vranes, An overview and comparison of supervised data mining techniques for student exam performance prediction, Computers & Education 143 (2020) 103676. https://doi.org/10.1016/j.compedu.2019.103676
2020
-
[31]
Falkenberg, S
S.F. Falkenberg, S. Spinler, Integrating operational and human factors to predict daily productivity of warehouse employees using extreme gradient boosting, International Journal of Production Research 61 (2023) 8654 –8673. https://doi.org/10.1080/00207543.2022.2159563
2023
-
[32]
S. Jang, G. Lee, S. Shin, H. Roh, Lexicon -based content analysis of BIM logs for diverse BIM log mining use cases, Advanced Engineering Informatics 57 (2023) 102079. https://doi.org/10.1016/j.aei.2023.102079
2023
-
[33]
Y . Pan, L. Zhang, Integrating BIM and AI for Smart Construction Management: Current Status and Future Directions, Arch Computat Methods Eng 30 (2023) 1081–1110. https://doi.org/10.1007/s11831-022-09830-8
2023 doi
-
[34]
W. Gao, C. Wu, W. Huang, B. Lin, X. Su, A data structure for studying 3D modeling design behavior based on event logs, Automation in Construction 132 (2021) 103967. https://doi.org/10.1016/j.autcon.2021.103967
2021
-
[35]
Yarmohammadi, R
S. Yarmohammadi, R. Pourabolghasem, D. Castro -Lacouture, Mining implicit 3D modeling patterns from unstructured temporal BIM log text data, Automation in Construction 81 (2017) 17–24. https://doi.org/10.1016/j.autcon.2017.04.012
2017 doi
-
[36]
Y . Pan, L. Zhang, Z. Li, Mining event logs for knowledge discovery based on adaptive efficient fuzzy Kohonen clustering network, Knowledge-Based Systems 209 (2020) 106482. https://doi.org/10.1016/j.knosys.2020.106482
2020
-
[37]
Zhang, M
L. Zhang, M. Wen, B. Ashuri, BIM log mining: Measuring design productivity, J. Comput. Civ. Eng. 32 (2018) 04017071. https://doi.org/10.1061/(ASCE)CP .1943- 5487.0000721
2018
-
[39]
Adadi, M
A. Adadi, M. Berrada, Peeking inside the black -box: A survey on Explainable Artificial Intelligence (XAI), IEEE Access 6 (2018) 52138 –52160. https://doi.org/10.1109/ACCESS.2018.2870052
2018
-
[40]
Linardatos, V
P . Linardatos, V . Papastefanopoulos, S. Kotsiantis, Explainable AI: A review of machine learning interpretability methods, Entropy 23 (2021) 18. https://doi.org/10.3390/e23010018
2021 doi
-
[41]
Roscher, B
R. Roscher, B. Bohn, M.F. Duarte, J. Garcke, Explainable machine learning for scientific insights and discoveries, IEEE Access 8 (2020) 42200 –42216. https://doi.org/10.1109/ACCESS.2020.2976199
2020
-
[42]
Lundberg, S.-I
S.M. Lundberg, S.-I. Lee, A unified approach to interpreting model predictions, in: Advances in Neural Information Processing Systems, Curran Associates, Inc., 2017. https://proceedings.neurips.cc/paper/2017/hash/8a20a8621978632d76c43dfd 28b67767-Abstract.html (accessed April 1, 2024)
2017
-
[44]
X.-R. Ni, Z. Zheng, J. -R. Lin, Z.-Z. Hu, X. Zhang, DesignTracking: Track and replay BIM-based design process, in: Proceedings of the Creative Construction Conference 2023, 2023: pp. 42–47. https://doi.org/10.3311/CCC2023-006
2023 doi
-
[45]
Zadeh, S
P .A. Zadeh, S. Staub-French, R. Pottinger, Review of BIM quality assessment approaches for facility management, (2015). https://doi.org/10.14288/1.0076364
2015 doi
-
[46]
Q. Wen, L. Sun, F. Yang, X. Song, J. Gao, X. Wang, H. Xu, Time series data augmentation for deep learning: A survey, in: Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, 2021: pp. 4653 –4660. https://doi.org/10.24963/ijcai.2021/631
2021 doi
- [47]
-
[48]
Dietterich, An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, Boosting, and Randomization, Machine Learning 40 (2000) 139–157
T.G. Dietterich, An experimental comparison of three methods for constructing ensembles of decision trees: Bagging, Boosting, and Randomization, Machine Learning 40 (2000) 139–157. https://doi.org/10.1023/A:1007607513941
2000 doi
-
[272]
https://doi.org/10.1016/j.autcon.2018.03.018
2018 doi
-
[642]
https://doi.org/10.1139/cjce-2016-0413
2016 doi
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.