{"id":"cde5f476-b47a-4c72-955c-76d43f679b40","arxiv_id":"2412.02041","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Standard regression models predict simulated scope-change cost and schedule impacts on a small construction dataset, but no real-world validation is provided.","lead":"This paper applies six machine learning models to a 221-row dataset from a single construction project to predict how scope changes affect cost and schedule. The key impact variables were simulated, not measured, so the reported accuracy describes performance on synthetic data rather than real projects.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on simulated impact targets with no external validation; an R2 of 0.985 on synthetic data does not establish real-world predictive utility.","rationale":"The paper's stated goal is to provide actionable prediction of scope-change impacts on real projects. For that claim to hold, the target variables must either be real measurements or come from a simulation independently validated against real outcomes. Neither condition is met: the targets are explicitly simulated, no generation details are given, no external dataset is used, and the paper's own limitation section asks for future validation. The reader's weakest assumption correctly identifies this as load-bearing. I also note internal inconsistencies, such as XGBoost schedule R2=0.1804 in Section III.C versus cost R2=0.985 in Section IV.B, and the use of classification-style Decision Tree parameters with regression metrics, but these are secondary: even a perfectly consistent fit to a synthetic target would not establish the practical claim. Therefore I agree with the REJECT verdict and see no reason to change it.","tokens_in":8705,"tokens_out":5909,"duration_ms":60729,"concrete_test":"Obtain real project data with documented scope changes and measured cost/schedule impacts (e.g., the original data source [18] or another empirical construction dataset), rerun the authors' preprocessing and XGBoost/Decision Tree pipelines on that data, and compare test-set R2/MSE with the reported values (0.985 for cost; roughly 0.18-0.21 for schedule). If real-data performance drops materially, or if no such data can be supplied, the central claim is unsupported. As a secondary check, reproduce the reported R2=0.985 from released code and data; if code or data are unavailable, the number is not independently confirmable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section II.A states that productivity rate, impact on cost, scope change magnitude, and scope change timing are 'simulated values ... generated to represent realistic project scenarios,' but the paper provides no generation mechanism, no distributional comparison to empirical project outcomes, and no external dataset. Section V concedes the models 'could be further validated with additional data sets when they become available.' The central claim that XGBoost predicts cost impact with R2=0.985 (Section IV.B) is therefore a statement about fit to a synthetic target. If the simulated relationship between scope changes and cost/schedule does not match real projects, the high R2 is an artifact and the practical conclusion fails. The descriptive statistics in Table I also suggest Impact_on_Schedule is binary (min 0, max 1) while regression MSE/R2 are used, complicating interpretation of the schedule claim; however, the load-bearing issue is the unvalidated synthetic target itself.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper trains six machine-learning models on a 221-row dataset of construction project tasks to predict the impact of scope changes on cost and schedule. The features include WBS, task type, estimated cost, actual cost, duration, task dependencies, and simulated values of productivity rate, scope change magnitude, impact on cost, impact on schedule, and scope change timing. The authors report that XGBoost performs best for cost impact (R2=0.985) and that Decision Tree performs best for schedule impact, with feature importance highlighting productivity, scope change magnitude, task dependencies, and WBS elements. The paper concludes that these models can help project managers forecast and mitigate the consequences of scope changes.","tokens_in":8915,"tokens_out":5111,"duration_ms":44701,"significance":"If the reported accuracy held on real project data, the paper would offer a practically useful early-warning tool for scope-change cost and schedule overruns. The paper uses a reasonable preprocessing pipeline, applies multiple standard models, includes cross-validation and residual plots, and explicitly acknowledges the need for additional validation data. However, the significance is currently not established: the target variables are simulated rather than observed, the generation mechanism is undisclosed, the evaluation is entirely internal to the synthetic dataset, and several reported performance figures are mutually inconsistent. As a result, the paper is better read as an illustrative ML workflow on synthetic data than as evidence for predictive utility in real projects.","major_comments":[{"comment":"The central claim of accurate prediction rests on target values that were simulated by the authors ('simulated values for ... impact on cost, scope change magnitude, scope change timing, which were generated to represent realistic project scenarios'), not on observed outcomes from real scope-change events. The paper does not disclose the data-generation mechanism, compare the simulated variables to empirical project data, or evaluate on any external dataset; Section V concedes that 'the models could be further validated with additional data sets when they become available.' Therefore the headline R2 values, such as XGBoost's 0.985 in Section IV.B, demonstrate fit to a synthetic target rather than predictive skill on real projects, and the practical conclusions in Sections IV.C and V are unsupported.","section":"Section II.A, Section V"},{"comment":"The reported model rankings are internally inconsistent. Section IV.B states that XGBoost achieved R2=0.985 for cost impact, while Section III.C reports an XGBoost MSE of 0.2094 and R2 of 0.1804; the manuscript does not explain whether these refer to different targets or different experimental setups. Similarly, Ridge Regression is reported with R2=0.991 in Section IV.B but with R2=0.0281 in Section III.B for schedule impact. Without a consistent set of metrics for the same prediction tasks, the claim that XGBoost best predicts cost impact and Decision Tree best predicts schedule impact is not reproducible.","section":"Section IV.B vs Section III.C"},{"comment":"The Decision Tree is configured as a classifier ('method' set to 'class', evaluation via confusionMatrix), but the stated target, Impact_on_Schedule, is analyzed as a numeric variable with regression MSE and R2 elsewhere. Table I shows this variable has min=0, max=1, and mean=0.57, strongly suggesting a binary outcome. Mixing classification and regression treatments of the same target makes the schedule-impact results ambiguous and prevents meaningful comparison with the cost-impact results.","section":"Section III.A and Table I"},{"comment":"The feature importance findings are contradictory. Section III.F reports that, for an XGBoost model trained without actual costs, the most influential feature is wbs.86, with scope change magnitude and task dependencies also prominent. Section IV.A and Section IV.B instead state that productivity rate is the most influential feature and that productivity rate, scope change magnitude, and task dependencies are the most critical factors. Because the manuscript does not identify which experimental configuration produced which importance ranking, the feature importance claims are not reproducible and cannot be used as practical guidance.","section":"Section III.F, Section IV.A, Section IV.B"}],"minor_comments":[{"comment":"Reference [3] appears to be mis-copied from another bibliography ('Synthetic structure of industrial plastics (Book style with paper title and editor)') and is not a usable citation for scope creep in project management.","section":"References"},{"comment":"The text states that Fig. 4 shows 'different cp values for the XGBoost model,' but cp is a complexity parameter for rpart-style trees, not an XGBoost hyperparameter; the figure and text should be aligned with the XGBoost parameters (eta, max_depth, etc.) actually selected.","section":"Section III.D"},{"comment":"Several sentences in this subsection are incomplete, for example 'The RMSE values across different cp values, indicating the optimal cp value for minimizing RMSE in the model,' and should be rewritten as full sentences.","section":"Section III.D"},{"comment":"The column names contain inconsistent spacing and underscores (e.g., 'Productivity_ Rate'), and the table would benefit from consistent naming to match the feature names used in the text.","section":"Table I"},{"comment":"The paper does not provide access to the code, the processed dataset, or the simulation script; for reproducibility, these should be made available or the data citation [18] should be extended with a persistent link.","section":"Reproducibility"}],"recommendation":"reject","confidential_remarks":"The main barrier is not presentation but the evidential base: the target variables are synthetic, the generation process is undisclosed, and the reported metrics conflict across sections. Even a careful revision would not turn this paper into a validation of scope-change impact prediction on real projects within the current scope; at best it could be reframed as a synthetic-data benchmark. I would suggest the editor weigh whether the journal is appropriate for a paper whose core empirical claim cannot currently be verified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a clearly described benchmark of six standard ML models on 221 rows of construction task data. That is the main thing to credit: the preprocessing steps, model configurations, and feature-importance analysis are concrete enough to reproduce on the same data. The feature-importance findings (productivity rate, scope change magnitude, task dependencies, WBS elements) are unsurprising but consistent with project management domain knowledge.\n\nThe central problem is that the target variables—impact on cost and impact on schedule—are simulated. Section II.A states these values were 'generated to represent realistic project scenarios,' but no generation mechanism, no distributional comparison to empirical outcomes, and no external dataset are provided. So the headline result, XGBoost R2 = 0.985 for cost impact, is a measure of fit to an invented ground truth. The paper itself concedes in the conclusion that the models need validation on additional datasets when they become available. That is honest, but it also means the practical conclusion is unsupported.\n\nThe soft spots are serious. First, the schedule-impact analysis is muddled: Impact_on_Schedule is binary (min 0, max 1, mean 0.57), yet the authors run regression and even configure the decision tree with method='class', then evaluate it with a confusion matrix. The reported MSE/R2 for schedule impact are not clearly interpretable. Second, the numbers are internally inconsistent across sections: the XGBoost model gets R2 = 0.985 for cost (Section IV.B), while Section III.C reports an XGBoost R2 of 0.18 for schedule, and the paper mixes the two without making the distinction obvious. Third, there is a clear citation error (reference [3] is a random book entry).\n\nThese are not minor quibbles. The simulated-target issue is load-bearing, and the schedule analysis is hard to trust. That said, the paper is not a crank document; it is straightforward and honest about its limitations, and it could serve as a starting point for a real empirical study.\n\nWho is the audience? Someone looking for a cautionary example of why simulated targets need external validation, or a project-management researcher planning a data-collection effort. I would not cite it as evidence of predictive performance, and I would not send it for full peer review as-is; a desk reject with an encouraging note about proper validation would be the right call.","headline":"Clear ML benchmark on a small construction dataset, but the simulated target variables mean the reported R2 of 0.985 says little about real projects.","tokens_in":9369,"tokens_out":3849,"would_cite":false,"duration_ms":38808,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that machine-learning models, especially XGBoost, can predict the cost impact of project scope changes almost perfectly on simulated construction data, with a Decision Tree model best for schedule impact.","keywords":["cost impact","schedule impact","scope changes","predictive modeling","machine learning","XGBoost","decision tree","construction projects"],"falsifier":"Hold out a set of real construction projects with recorded scope changes and observed cost and schedule impacts, run the published preprocessing and model pipeline on them, and compare predicted versus actual impacts; if the cost-impact $R^2$ falls far below the reported 0.985, or the Decision Tree's schedule predictions do not beat a simple majority-class baseline, the simulated-data result is an artifact of the data generation process.","tokens_in":8517,"feed_emoji":"📈","tokens_out":7201,"duration_ms":60848,"temperature":0.7,"pith_summary":"Scope changes in construction projects routinely cause cost overruns and schedule delays, but their impact is usually assessed with subjective judgment. This paper tries to replace that judgment with machine learning: it trains six models on a 221-row dataset of construction project tasks, including simulated scope-change variables, to predict the monetary and schedule impact of a change. The author claims that XGBoost predicts cost impact with an $R^2$ of 0.985, and that a Decision Tree model predicts schedule impact better than the alternatives. If those results hold on real projects, project managers could estimate the consequences of a proposed scope change before approving it. The paper itself cautions that the models still need validation on additional datasets.","feed_headline":"XGBoost predicts scope-change cost impact with R-squared 0.985","feed_subtitle":"Decision Tree models lead on schedule impact; productivity and scope-change size are the strongest levers.","key_machinery":"The central mechanism is the supervised learning pipeline built around the XGBoost algorithm, an optimized gradient-boosting implementation that combines many shallow decision trees with regularization, and the Decision Tree classifier used for schedule impact. The input features are project task attributes—WBS codes, task type, productivity rate, estimated and actual cost, duration, and task dependencies—plus two scope-change variables (magnitude and timing). The targets are the generated 'impact on cost' and 'impact on schedule' values. The pipeline scales numeric features, splits data 80/20, applies cross-validation and hyperparameter tuning, and uses residual plots and feature-importance rankings to evaluate and interpret the models.","core_discovery":"The paper's central claim is that a supervised machine-learning model, XGBoost, predicts the monetary impact of a scope change on a construction project with an $R^2$ of 0.985 on the test set, and that a Decision Tree model gives the best predictions of schedule impact among the six models compared. Feature-importance analysis identifies productivity rate, scope change magnitude, task dependencies, estimated cost, actual cost, duration, and specific WBS elements as the dominant predictors of both cost and schedule impact. The author presents these results as evidence that data-driven models can give project managers objective, early estimates of scope-change consequences, replacing subjective judgment. The work is framed as an extension of earlier machine-learning forecasting to the specific problem of scope-change impact.","pith_inferences":["The near-perfect $R^2$ values, combined with the high correlation (0.94) between estimated and actual cost, suggest the cost model may be leaning heavily on near-duplicate cost information; the model trained without actual costs is the more realistic deployment test.","The schedule target appears to be binary (0/1) from the descriptive statistics, yet the paper evaluates schedule models with regression metrics; a fair comparison would use classification accuracy, precision, and recall.","A cheap test of the whole approach is to benchmark against a trivial baseline that always predicts the mean cost impact; if tree ensembles only beat that baseline on simulated data, their practical value is unproven.","Because the data are simulated, the feature-importance findings are really hypotheses about which project attributes matter, and they could be checked against published empirical studies of scope-creep cost drivers."],"forward_implications":["Project managers could estimate the cost and schedule impact of a proposed scope change before approving it, turning scope-change review from a reactive into a proactive process.","The feature-importance rankings suggest that monitoring productivity rate, scope change magnitude, and task dependencies would catch the largest cost and schedule risks early.","The same pipeline, retrained on richer data, could be extended to predict the probability and magnitude of cost overruns at project completion, as the paper suggests.","Organizations that already track WBS-level task data could use these models to run scenario analyses for stakeholder negotiations and resource allocation."],"supporting_citations":[{"why":"Supplies the 221-row construction project task dataset, including WBS, costs, durations, dependencies, and the simulated scope-change variables.","marker":"[18]"},{"why":"Provides the precedent of using ANNs and regression on preproject planning data, which the study extends to scope-change impact prediction.","marker":"[15]"},{"why":"The author's earlier machine-learning forecasting work that this study builds on for dynamic project behavior.","marker":"[8]"},{"why":"Establishes scope creep in construction as the problem that motivates predicting cost overruns and delays.","marker":"[1]"},{"why":"Documents the gap in handling scope and change in agile project management, motivating a data-driven approach.","marker":"[4]"},{"why":"Shows KDD techniques extracting useful knowledge from construction labor data, supporting the use of ML on project records.","marker":"[5]"}],"fun_headline_variants":["XGBoost predicts scope-change cost impact with R² 0.985","Scope-change impact: productivity and size are top predictors","Decision trees best for schedule impact, XGBoost for cost","Machine learning quantifies scope-change costs and delays","Predicting scope-change fallout: ML models from project data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the simulated values for productivity rate, scope change magnitude, timing, and their impacts on cost and schedule are realistic enough to stand in for real project outcomes; if those synthetic relationships differ from reality, the reported accuracy will not transfer to actual projects.","fun_headline_variants_meta":{"raw":{"variants":["XGBoost predicts scope-change cost impact with R² 0.985","Scope-change impact: productivity and size are top predictors","Decision trees best for schedule impact, XGBoost for cost","Machine learning quantifies scope-change costs and delays","Predicting scope-change fallout: ML models from project data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000338,"raw_usage":{"total_tokens":1893,"prompt_tokens":997,"completion_tokens":896,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":811}},"tokens_in":613,"tokens_out":896,"duration_ms":8660,"temperature":1.0,"reasoning_tokens":811,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:52:50.154387+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out a set of real construction projects with recorded scope changes and observed cost and schedule impacts, run the published preprocessing and model pipeline on them, and compare predicted versus actual impacts; if the cost-impact $R^2$ falls far below the reported 0.985, or the Decision Tree's schedule predictions do not beat a simple majority-class baseline, the simulated-data result is an artifact of the data generation process.","supporting_citations":[{"cited_title":"Developing a dataset of real projects for portfolio, program and project control management research,","cited_arxiv_id":null,"evidence_quote":"Supplies the 221-row construction project task dataset, including WBS, costs, durations, dependencies, and the simulated scope-change variables."},{"cited_title":"A study of preproject planning and project success using ANNs and regression models,","cited_arxiv_id":null,"evidence_quote":"Provides the precedent of using ANNs and regression on preproject planning data, which the study extends to scope-change impact prediction."},{"cited_title":"Enhancing Project Performance Forecasting using Machine Learning Techniques","cited_arxiv_id":"2411.17914","evidence_quote":"The author's earlier machine-learning forecasting work that this study builds on for dynamic project behavior."},{"cited_title":"Managing project scope creep in construction industry,","cited_arxiv_id":null,"evidence_quote":"Establishes scope creep in construction as the problem that motivates predicting cost overruns and delays."},{"cited_title":"Agile project management challenge in handling scope and change: A systematic literature review,","cited_arxiv_id":null,"evidence_quote":"Documents the gap in handling scope and change in agile project management, motivating a data-driven approach."},{"cited_title":"Application of KDD techniques to extract useful knowledge from labor resources data in industrial construction projects,","cited_arxiv_id":null,"evidence_quote":"Shows KDD techniques extracting useful knowledge from construction labor data, supporting the use of ML on project records."}],"review_version":1}