{"id":"2b4d67c1-6a11-40d0-b680-cf55b613478d","arxiv_id":"2411.09481","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper claims a machine learning model can predict BIM design quality scores from logged design behavior features, with an ExtraTrees model reaching R2=0.88.","lead":"This paper tries to link how designers behave in BIM software (actions like commands, pauses, edits) to the final quality of their models, using machine learning and a manual scoring system. The headline result, an R2 of 0.88, is weakened by a validation setup that allows overlapping data from the same designer to appear in both training and test sets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported test R2 of 0.88 is not evidence of generalization: window-cropped samples with identical labels were randomly split, so training and test sets contain overlapping windows from the same session and same student.","rationale":"The reader's weakest assumption correctly identifies the load-bearing flaw: window-cropped data augmentation followed by a random 80/20 split creates train/test overlap and shared labels, so the reported R2 does not measure generalization. My own reading of Sections 3.3.1, 4.2, and 4.3 confirms this mechanism concretely: with N=30000 and s=2000, adjacent windows share 28000 rows, and all windows from a student are labeled with the same quality score. This is not a minor methodological quibble but a direct threat to the paper's central quantitative claim. The paper itself acknowledges information leakage for very small step sizes in Section 4.3, but the chosen hyperparameters still produce massive overlap, and the label-sharing across sessions from the same student would persist even with non-overlapping windows. The effective independent sample size is 68 students, not the hundreds or thousands of augmented samples, so the variance of the R2 estimate under a valid split could be very high. I also considered whether the subjectivity of the expert quality scores or the use of a total score for both architectural and structural sessions could be an equally serious issue; however, those concern label quality and construct validity, whereas the leakage problem alone is sufficient to invalidate the headline result. The proposed concrete test, a student-level or at least session-level split with no overlapping windows, would settle whether any genuine predictive signal exists. If the leakage-free R2 remains high, the authors' claim could be revived; if it collapses, the rejection stands. I therefore see no reason to change the reader's REJECT verdict, and I agree with the identified weakest assumption.","tokens_in":17555,"tokens_out":2763,"duration_ms":26949,"concrete_test":"Re-run the model-selection and hyperparameter experiments with a student-level split: assign all windows from each student entirely to either the training or test set (e.g., train on 80% of students, test on the remaining 20%), and ensure no test window overlaps any training window by using non-overlapping crops or setting the step size s at least as large as the window length N. Report test R2 and RMSE under this leakage-free protocol, ideally with repeated random student-level splits. If the R2 drops substantially (for example, below 0.5 or near zero), the claimed generalizable relationship and the headline 0.88 result are not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that design-behavior features can predict final design quality for unseen modeling sessions rests on the test-set R2 reported in Section 4.2, but that evaluation is invalidated by leakage in the data-generation and splitting procedure. In Section 3.3.1, each Revit session is sliced into overlapping windows of length N=30000 with step s=2000 (chosen in Section 4.3), so adjacent windows overlap by 28000 rows. Every window is assigned the same quality score as the full session (and, in effect, the same score as all windows from that student). The data set is then randomly split 80/20 in Section 4.2, meaning training and test sets can contain near-duplicate windows from the same session, and will in general contain different sessions from the same student with identical labels. Under this protocol the model can memorize session- or student-specific patterns rather than learn a general behavior-to-quality relationship, so the R2 of 0.88 overstates performance on genuinely unseen modeling sessions. The paper acknowledges in Section 4.3 that smaller step sizes cause information leakage, but the chosen hyperparameters still produce heavy overlap, and the more fundamental issue remains: even non-overlapping windows from the same student share a label, so the effective number of independent samples is at most 68 students (or 136 models), not the 493-2326 augmented samples used. Without a split that respects this cluster structure, the quantitative link claimed in the abstract is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper aims to establish a quantitative relationship between BIM design behaviors and design quality. The authors collected behavioral data from 68 students' Revit modeling sessions using log files and a custom plugin, scored the resulting BIM models with a hand-crafted evaluation rubric, augmented the sequence data by window cropping, extracted 29 statistical features, and trained 42 regression models. They report that an Extremely Random Trees model achieves an R2 of 0.88 on the test set and use SHAP values to identify features related to command execution and design-intention changes as influential.","tokens_in":17884,"tokens_out":2331,"duration_ms":23378,"significance":"If the reported result were valid, the paper would provide a useful data-driven bridge between process-level design behavior and final design quality in the AEC domain, complementing existing qualitative studies. The data collection pipeline, combining log mining with a real-time plugin, is a genuine practical contribution, and the authors are transparent about many experimental details. However, the central quantitative claim is undermined by a procedural flaw in the evaluation setup: the training and test sets are not independent, so the reported R2 does not measure generalization to unseen modeling sessions. The SHAP-based interpretations are also built on this leaky model. The underlying research question is valuable and the dataset is interesting, but the paper's main conclusion is not currently supported.","major_comments":[{"comment":"The window-cropping augmentation with N=30,000 and s=2,000 creates overlapping windows that all inherit the same quality score as the full session, and the random 80/20 split places near-duplicate windows from the same session in both training and test sets. Consequently, the model can effectively memorize session- or student-specific patterns, and the reported test R2 of 0.88 is not evidence of generalization to unseen modeling sessions. The evaluation must be repeated with a split that respects the cluster structure, e.g., holding out entire sessions or entire students.","section":"Section 3.3.1 and Section 4.2"},{"comment":"The paper acknowledges that very small step sizes can cause information leakage due to high similarity between samples, but then selects s=2,000, where adjacent windows still overlap by 28,000 rows. The claim that this setting maintains 'a reasonable level of reliability' is not supported by any analysis. Even with non-overlapping windows, all windows from the same student share the same label, so the effective number of independent samples is at most 68 students (or 136 models), not the reported 493–2,326 augmented samples.","section":"Section 4.3"},{"comment":"The SHAP interpretation is performed on a model trained under the leaky data-splitting protocol. The feature importance findings, such as the positive association of pushbutton% with quality, may reflect memorization of session-specific or student-specific patterns rather than a generalizable behavior-quality relationship. A clean model trained under a proper grouped split is required before these domain conclusions can be considered reliable.","section":"Section 4.4"}],"minor_comments":[{"comment":"The authors use the normality of the score distribution to affirm data reliability, but this is not a strong check against scoring subjectivity; a normal distribution of scores can arise from many rating processes and does not validate the rubric's accuracy.","section":"Section 3.2"},{"comment":"The table of statistical indicators is labeled 'Table 11' even though it appears to be the first numbered table in the manuscript; the numbering should be corrected throughout.","section":"Section 3.3.2"},{"comment":"In the hyperparameter optimization figures, the RMSE and R2 values are presented together without clear bar-chart legends for the two algorithms; the captions should explicitly state which color corresponds to Bagging and which to ExtraTrees, and whether the printed numbers refer to training or test sets.","section":"Figures 7 and 8"}],"recommendation":"reject","confidential_remarks":"The data-leakage problem is not a stylistic or presentational issue; it invalidates the paper's headline result. The authors would need to re-run the entire experimental pipeline with a grouped train/test split, and the result could plausibly change. If the authors can perform such a re-analysis and the R2 remains high, the paper might be reconsidered, but as submitted the central claim is unsupported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper tries something genuinely useful—predicting final BIM design quality from process behavior captured via Revit logs plus a custom plugin—but the headline R2 of 0.88 does not mean what they claim. The underlying idea is worth saving; the evaluation isn't.\n\nWhat's new and good: they built a real data collection pipeline (Design Tracker plugin plus log mining), scored 136 student models with a rubric, extracted 29 sensible density features, and benchmarked many ML models with SHAP interpretation. That combination—linking overall design quality, not just efficiency or command prediction, to behavioral features—is a legitimate gap in the BIM literature. The feature set is reasonable, and the SHAP findings (e.g., command-execution proficiency and deletion density as proxies for design intention stability) are plausible and practically interesting.\n\nThe soft spot is critical and in the middle of the method. Section 3.3.1 slices each session into overlapping windows of length N=30,000 with step s=2,000, assigning every window the same quality score as the full session. Then Section 4.2 randomly splits these augmented samples 80/20. That means training and test sets share near-duplicate windows from the same session and, even for non-overlapping windows, different sessions from the same student share a label. The effective number of independent units is at most 68 students (or 136 models), not the 493–2,326 augmented samples. The R2 of 0.88 therefore mostly reflects the model's ability to memorize session- or student-specific patterns, not a general behavior-to-quality relationship. The paper even acknowledges in Section 4.3 that small step sizes cause information leakage, but then chooses s=2,000, which still overlaps heavily; the deeper issue—labels shared across all windows of one student—is never addressed.\n\nWhat would fix it: a split that respects cluster structure—hold out entire students or entire models—and report performance on those truly unseen sessions. The SHAP analysis, while interesting, is built on a model whose input distribution was corrupted by the same leakage, so the feature importance rankings may also be unreliable.\n\nFor peer review: I would send this to a serious referee rather than desk-reject. The data collection effort and problem framing are valuable, and the leakage is fixable with a reanalysis. But as it stands, the central quantitative claim is not established. The paper needs major revision with leakage-aware evaluation before publication.","headline":"A useful problem and real data collection are undermined by a load-bearing evaluation flaw: the window-cropped augmentation with random split lets the model memorize session- and student-specific labels, so the claimed R2 of 0.88 is not evidence of generalization.","tokens_in":18346,"tokens_out":1614,"would_cite":false,"duration_ms":16306,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Design behavior in BIM software predicts final model quality with an R² of 0.88.","keywords":["BIM","design quality","design behavior","machine learning","feature engineering","SHAP","Extremely Random Trees","data augmentation"],"falsifier":"A concrete check is to split by student or by session instead of by window: if the model's test $R^2$ drops substantially (for example, below 0.5) when no window from a given session appears in both training and test, then the reported 0.88 reflects window overlap rather than a transferable behavior-quality relationship. A second check is to score the same 136 models with a second independent pair of raters and see whether the learned features still predict the new scores.","tokens_in":17383,"feed_emoji":"🏗️","tokens_out":4283,"duration_ms":38120,"temperature":0.7,"pith_summary":"This paper tries to establish that how a designer behaves inside BIM modeling software—what commands they run, how they execute them, and how often they delete or redo work—carries enough signal to predict the quality of the finished model. Using Revit log files supplemented by a plugin that records keystrokes and element operations, the authors turn 68 students' modeling sessions into 29 density features and score each submitted model on completeness, accuracy, and complexity. They then train regression models on window-cropped session subsequences and report that the best model, Extremely Random Trees, reaches an $R^2$ of 0.88 on a held-out test set. The wider point is that design quality is not just a property of the final drawings but is inscribed in the process, so behavior data could be used to guide designers in real time.","feed_headline":"BIM design behavior predicts final quality: R² = 0.88","feed_subtitle":"Command, rework and pause patterns in Revit sessions predict final model scores, pointing to skill and stable intentions.","key_machinery":"The load-bearing mechanism is the window-cropping data augmentation combined with density features over the cropped sequence. Each full Revit session is sliced into subsequences of length $N=30000$ rows with step $s=2000$; every cropped window inherits the full session's quality score, expanding 136 models into thousands of training samples. From each window the authors compute 29 statistical densities (data density and time density) that compress behavior into feature vectors, then train tree ensembles on these vectors and interpret them with SHAP values. The ExtraTrees algorithm's random split-point selection is what lets the model capture the feature-score relation; SHAP attribution is what turns the model into the claim that proficiency and intention stability matter.","core_discovery":"On the paper's own terms, the central discovery is that a quantitative, learnable relationship exists between BIM design behavior and the quality of the resulting model. The relationship is measured by 29 hand-crafted features—data densities such as the share of successful transactions, added/deleted/modified components, command execution modes (ribbon, shortcut, pushbutton), pause lengths, and effective work time—and target scores assigned by expert raters. Across 42 regression algorithms, only the ensemble family learns the mapping well; ExtraTrees with a cropping window length of 30,000 rows and step of 2,000 achieves the reported test-set $R^2=0.88$ with RMSE 2.29 against a score standard deviation of 7.61. SHAP interpretation identifies command-execution features and component-manipulation densities as the main drivers, leading the authors to conclude that designer proficiency and stability of design intentions are the two behavior-level levers on final quality.","pith_inferences":["Beyond the paper's claims: the reported $R^2$ likely overstates how well behavior predicts quality for a new designer, because each student contributes many overlapping windows and the random 80/20 split puts windows from the same session on both sides of the train/test boundary; a session-level or student-level split would be a harder test.","Beyond the paper's claims: an untested extension is to replace density features with sequence models (LSTM or Transformer) that respect command order; the paper explicitly notes its statistical features may miss sequential structure, so this is a natural next experiment.","Beyond the paper's claims: the quality score is manually assigned by two raters with a benchmark student; if ratings were gathered from more raters, inter-rater agreement could separate true behavioral signal from rater subjectivity.","Beyond the paper's claims: a practical deployment would need to distinguish deliberate pauses from idle time; the paper treats pauses over five minutes as non-work, but a designer thinking while away from the keyboard is still productive."],"forward_implications":["If the relationship holds beyond this student dataset, BIM software could score a model in progress from behavior alone, before formal quality checks.","Feature importances give testable guidance: training designers to use shortcuts and reduce ribbon-button hunting should improve both efficiency and quality.","High deletion density, interpreted as large design-intention changes, becomes a measurable early-warning signal for final quality.","The same pipeline—log capture, window cropping, density features, ensemble regression—could be applied to other software-mediated tasks where outcome quality is scored."],"supporting_citations":[{"why":"Supplies the real-time behavior capture plugin that fills gaps in Revit log data.","marker":"[33]"},{"why":"Provides the window-cropping augmentation technique used to generate training samples.","marker":"[35]"},{"why":"Describes the Extremely Randomized Trees algorithm that yields the best-performing model.","marker":"[36]"},{"why":"Provides the baseline ensemble method (Bagging) compared against ExtraTrees.","marker":"[37]"},{"why":"Supplies the SHAP interpretation method used to rank feature influence.","marker":"[31]"},{"why":"Shows prior use of BIM log data for efficiency analysis, positioning the paper's quality-focused target.","marker":"[26]"},{"why":"Defines BIM quality categories that inform the scoring rubric.","marker":"[14]"},{"why":"Shows the state of the art in behavior-sequence prediction, which this paper extends to quality prediction.","marker":"[11]"}],"fun_headline_variants":["BIM actions predict design quality: R²=0.88","Skill and stable intent drive BIM design quality","ExtraTrees links BIM behavior to quality: R²=0.88","29 BIM behavior features forecast model score"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a 30,000-row window clipped from the middle of a designer's session shares the same quality score as the whole finished model, and that randomly splitting these overlapping windows into training and test sets gives an honest measure of generalization.","fun_headline_variants_meta":{"raw":{"variants":["BIM actions predict design quality: R²=0.88","Skill and stable intent drive BIM design quality","ExtraTrees links BIM behavior to quality: R²=0.88","29 BIM behavior features forecast model score"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000563,"raw_usage":{"total_tokens":2640,"prompt_tokens":884,"completion_tokens":1756,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":1691}},"tokens_in":500,"tokens_out":1756,"duration_ms":14449,"temperature":1.0,"reasoning_tokens":1691,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:35:31.205369+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check is to split by student or by session instead of by window: if the model's test $R^2$ drops substantially (for example, below 0.5) when no window from a given session appears in both training and test, then the reported 0.88 reflects window overlap rather than a transferable behavior-quality relationship. A second check is to score the same 136 models with a second independent pair of raters and see whether the learned features still predict the new scores.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the real-time behavior capture plugin that fills gaps in Revit log data."},{"cited_title":"Yarmohammadi, R","cited_arxiv_id":null,"evidence_quote":"Provides the window-cropping augmentation technique used to generate training samples."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes the Extremely Randomized Trees algorithm that yields the best-performing model."},{"cited_title":"Akponeware, Z.A","cited_arxiv_id":null,"evidence_quote":"Shows prior use of BIM log data for efficiency analysis, positioning the paper's quality-focused target."},{"cited_title":"Sampaio, E","cited_arxiv_id":null,"evidence_quote":"Defines BIM quality categories that inform the scoring rubric."}],"review_version":1}