{"id":"0ecf67ca-c1fd-4e14-b958-95f0e49f52cd","arxiv_id":"2411.17914","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A machine learning case study claims LSTM outperforms ARIMA and EVM for forecasting project cost variance, but the paper reports no numerical results to support that claim.","lead":"This paper applies ARIMA and LSTM machine learning models to forecast cost variance and earned value for an urban road reconstruction project, and claims LSTM beats traditional project management methods. The finding could help project managers detect cost and schedule problems earlier, but the paper's supporting numerical evidence is absent.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim that LSTM outperforms ARIMA/EVM is unverifiable: Section IV cites MAE/MSE/RMSE and Figs. 1–2, but the manuscript contains no numeric error values and the referenced figures are absent, so no quantitative evidence supports the conclusion.","rationale":"The reader's core critique identifies two serious weaknesses: (a) the central comparative claim is asserted without numeric error metrics, and (b) the simulated external factors make the feature-importance and external-factor conclusions circular. I agree with the verdict of REJECT but focus on (a) as the single most load-bearing concern, because it invalidates the primary comparison (LSTM > ARIMA > EVM) regardless of whether external factors are real or simulated. The absence of Figs. 1–2 and any reported MAE/MSE/RMSE values means there is no falsifiable quantitative claim to examine. Even if the simulated external factors were perfectly realistic, the reader still could not verify the 'superior performance' claim. The simulation issue is important but secondary: it undermines the external-factor contribution specifically, whereas the missing metrics undermine the entire results section. The proposed concrete test—requesting the missing figures/tables and re-running with a chronological split—would settle whether the concern lands. If the figures and tables are simply absent, the paper does not meet the evidentiary standard for its central claim. If they are supplied, the comparison might be checkable, but the random k-fold issue would remain a methodological risk. Therefore the verdict stays REJECT rather than CONDITIONAL, because the evidence required for the central claim is absent and the test cannot be performed without additional unreported artifacts.","tokens_in":5870,"tokens_out":2296,"duration_ms":20820,"concrete_test":"Obtain the complete manuscript or supplementary material containing Fig. 1, Fig. 2, and the underlying numeric MAE/MSE/RMSE tables (at minimum per model and per WBS category). If these figures/tables cannot be supplied, the comparative claim remains unsubstantiated. Additionally, re-run the LSTM/ARIMA comparison with a chronological train/validation split (e.g., first 18 months train, last 4 months test) and report the error table; if the random k-fold results differ materially from the chronological split, the reported superiority is an artifact of leakage.","verdict_should_be":"REJECT","load_bearing_attack":"The paper's headline result—that the LSTM model 'showcased superior performance' and that ML models 'significantly improved' forecasting—depends entirely on the evaluation metrics claimed in Section IV. However, no MAE, MSE, or RMSE values appear anywhere in the text; the reader is referred to Figs. 1 and 2, which are not present in the provided manuscript. Table II reports only raw correlations, not model errors. Consequently, the comparative claim cannot be checked, reproduced, or even assessed for statistical significance. Moreover, Section III.C states k-fold cross-validation was used on a 22-month time series; standard random k-fold splits leak future information into training windows, which would bias the error estimates optimistically, but since the estimates themselves are not reported, this cannot be verified either. The central argument therefore rests on unreported quantitative evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a machine-learning-based forecasting approach for project performance metrics (cost variance, earned value) in an urban road reconstruction project, using ARIMA and LSTM time-series models and incorporating external factors (weather patterns, resource availability) as features. The authors report summary statistics and a correlation table, claim that both ARIMA and LSTM outperform traditional earned value management, and state that LSTM gives the most accurate forecasts. The case-study period, model-development details, and feature-importance analysis are described, but the central quantitative evidence—MAE/MSE/RMSE values and the figures that display them—does not appear in the manuscript.","tokens_in":6051,"tokens_out":2859,"duration_ms":26840,"significance":"If substantiated, the finding that LSTM-based forecasting outperforms both ARIMA and traditional EVM in a road reconstruction project would be a practically useful contribution to construction project management, especially with the inclusion of external-factor features and SHAP-based interpretability. However, the current manuscript does not provide the evidence needed to assess that claim: no numeric error metrics are reported, the referenced figures are absent, the external factors are simulated, and the cross-validation design is not described with enough specificity to rule out temporal leakage. The paper also provides no reproducible code or data release. The potential significance is therefore real but entirely unverified; the contribution at present is an architecture proposal and a set of claims rather than a validated empirical study.","major_comments":[{"comment":"The central claim that “the LSTM model, in particular, showcased superior performance” and that both models “outperformed the traditional EVM approach” is not supported by any quantitative evidence in the text. Section IV refers to MAE, MSE, and RMSE and to Figs. 1 and 2, but the figures are not present in the manuscript and no numeric values of these metrics are given in any table or sentence. Table II reports only Pearson correlations between features, not forecast errors. Without the actual MAE/MSE/RMSE values, per-metric breakdowns, and ideally confidence intervals or significance tests, the comparative claim cannot be checked or reproduced.","section":"Section IV, Figs. 1–2"},{"comment":"The external factors are simulated: the text states that weather and resource data were “simulated based on reasonable assumptions to demonstrate the potential impact of external factors.” The paper then treats the SHAP feature-importance results and the correlation analysis as evidence that external factors matter for forecasting. This is circular: the simulated variables can only influence the models through the assumptions baked into their generation, so the findings about weather and resource availability are artifacts of the simulation unless the simulation is validated against real measurements or explicitly presented only as an illustrative sensitivity exercise. The manuscript does neither, and Section V repeats the claim that “the integration of simulated external factors… contributed to a more comprehensive and accurate assessment.”","section":"Section III.A, Section III.D"},{"comment":"The description of k-fold cross-validation is insufficient and potentially invalid for time-series data. The dataset spans only 22 months, yet the text does not state the value of k, the split ratio, or whether the folds are contiguous time blocks. Standard random k-fold splitting of a time series leaks future information into the training set, producing optimistically biased error estimates. Since the error estimates themselves are not reported, the claimed “confidence in the models’ predictive capabilities” is unsubstantiated. The authors should use walk-forward validation or an explicit temporal split and report fold-wise results.","section":"Section III.C"},{"comment":"The LSTM architecture is described only as “multiple layers” with “the number of neurons and layers determined through experimentation and hyperparameter tuning.” No layer counts, neuron counts, learning rate, batch size, sequence length, loss function, optimizer, or training epochs are given. Likewise, the ARIMA model is mentioned without its (p,d,q) orders. These details are load-bearing because the claimed superiority of LSTM may depend on specific hyperparameter choices, and the absence of such details makes the experiments impossible to reproduce or to assess for overfitting. At minimum, a table of hyperparameters and a description of the tuning procedure should be provided.","section":"Section III.B"},{"comment":"There is an inconsistency in the time periods: data collection is described as January 2011 to October 2012 in Section III.A, while the case study application is described as October 2011 to November 2012 in Section III.D. It is also unclear how the data are split into training, validation, and test sets, and what forecast horizon the models were asked to predict. Clarifying the exact temporal structure of the training and evaluation is necessary for the results to be meaningful.","section":"Section III.D, Section III.A"}],"minor_comments":[{"comment":"There are duplicated subsection headings (two “A.” subsections in Section III), inconsistent references to “Table 2” vs. “Table II,” and a repeated title header in the body text; these editorial issues should be fixed.","section":"Throughout"},{"comment":"The captions for Figs. 1–3 are present but the figures themselves are missing from the manuscript; either the figures must be included or the text should report the numerical values they would show.","section":"Section IV"},{"comment":"Reference [8] is marked “submitted for publication” and reference [3] is marked “unpublished”; these are not citable sources in their current form, and several other references appear incomplete or in preprint form, which weakens the literature-review grounding.","section":"Section II / References"},{"comment":"The text states that “data preprocessing steps, including handling missing values, outliers, and inconsistencies” were applied, but no details are given about the amount of missing data, the outlier criteria, or the normalization method; such details are needed for reproducibility.","section":"Section III.A"},{"comment":"The phrase “significant improvements” and “significantly improved” in the conclusion is not supported by any statistical significance testing; the authors should either provide tests with p-values or use language such as “substantially lower error” backed by the reported metrics.","section":"Section IV"}],"recommendation":"reject","confidential_remarks":"The manuscript is not yet at the level of a publishable research paper: the key empirical evidence is missing (no error metrics, no figures), the external-factor analysis is based on simulated data, and the experimental protocol is underspecified. These problems are load-bearing, not cosmetic. Rejection seems appropriate, though a future submission could be considered if the authors include the actual quantitative results, validate or explicitly reframe the simulated external factors, use a proper temporal cross-validation scheme, and report full model-configuration details."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi,\n\nQuick take: this is a small case study applying ARIMA and LSTM to a single road-construction project. The structure is clear and the literature review touches the right works. But the paper's main claim—LSTM beats ARIMA and EVM—is asserted, not shown. No MAE/MSE/RMSE numbers appear anywhere in the text. The two figures that supposedly carry the comparison are missing from the copy I have. Without those, the central result is unverifiable.\n\nThe bigger problem is the simulated external factors. Section III.A states weather and resource availability were unavailable and therefore 'simulated based on reasonable assumptions to demonstrate the potential impact.' Later, the feature-importance analysis is presented as evidence that external factors matter. That is circular: the inputs were constructed to show an effect, and then the model's preference for those inputs is read as a real-world finding. This is not a minor issue—it means the paper cannot support its claim that incorporating weather and resource data improves forecasts, regardless of how the model comparison turns out.\n\nThere is also a technical concern: k-fold cross-validation on a 22-month time series with random splits will leak future information into training windows, biasing error estimates optimistically. Since no numbers are reported, we can't know how much this matters, but it should be addressed.\n\nTo be fair, the paper does several things right. The motivation is sensible, the choice of ARIMA and LSTM is standard for this kind of comparison, and the author is upfront in Section IV that the study relies on a specific project dataset and assumptions about external factors. The writing is readable. But a case study with no shipped data or code and no quantitative results is not a complete scientific contribution.\n\nWho is this for? Someone wanting a quick example of how ML could be applied to earned-value forecasting might skim it, but they'd be better off reading the cited hybrid-model papers. As submitted, it does not deserve serious referee time: the load-bearing evidence is absent, and the simulation circularity would require new real data to fix.\n\nMy recommendation: desk reject, with encouragement to resubmit after collecting real weather/resource data and reporting the actual error metrics. If the numbers and figures were added, and the external factors came from measurements, this could be a modest practical case study worth publishing in a venue that accepts applied project-management work.","headline":"Standard ARIMA/LSTM case study whose central LSTM-vs-ARIMA claim is unsupported by any reported numbers and whose external-factor findings are artifacts of simulated inputs.","tokens_in":6560,"tokens_out":3344,"would_cite":false,"duration_ms":30290,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Machine-learning time-series models, especially LSTM, can forecast project cost variance and earned value more accurately than traditional earned value management in a road reconstruction case study.","keywords":["project performance forecasting","machine learning","time series forecasting","LSTM","ARIMA","earned value management","cost variance","construction projects"],"falsifier":"Retrain ARIMA and LSTM on the same project with recorded, not simulated, weather and resource data, and compute MAE, MSE, and RMSE on a held-out period; if LSTM's errors are not below ARIMA's and EVM's, the paper's central claim fails.","tokens_in":5616,"feed_emoji":"📈","tokens_out":7355,"duration_ms":62493,"temperature":0.7,"pith_summary":"The paper sets out to prove that machine-learning time series models can forecast project performance indicators better than the traditional earned value management (EVM) baseline. It trains ARIMA and LSTM models on historical cost and earned-value data from an urban road reconstruction project, adding simulated weather patterns and resource availability as external features. The case study claims LSTM produces the most accurate forecasts, with lower mean absolute error, mean squared error, and root mean squared error than ARIMA and EVM, and that feature-importance analysis shows cost history and planned value are the strongest drivers. If this holds, project managers gain an early-warning tool that flags deviations from baseline before they grow.","feed_headline":"LSTM forecasts beat ARIMA and EVM on project cost variance","feed_subtitle":"A road-reconstruction case study says deep learning catches deviations traditional tracking misses.","key_machinery":"The central mechanism is a per-Work Breakdown Structure (WBS) time series forecast of cost variance and earned value, built from engineered rolling averages and simulated external factors. ARIMA captures the linear trend component of the series, while LSTM, a recurrent neural network, captures long-term dependencies and nonlinear patterns; both are compared against the EVM baseline. SHAP (SHapley Additive exPlanations) values are used to attribute each feature's contribution to the predictions, and k-fold cross-validation provides the robustness check.","core_discovery":"The paper argues that the weakness of EVM is its reliance on static baseline plans, which ignore the dynamic, nonlinear patterns in project progress and external influences. By treating cost variance and earned value as time series and modeling them with ARIMA and LSTM, the forecast can adapt as new progress data arrives. In the road reconstruction case study, the LSTM model is reported to have the best performance on MAE, MSE, and RMSE, and cross-validation confirms the models generalize across data subsets; SHAP analysis identifies rolling average actual cost and rolling average planned value as the most influential features. The conclusion drawn is that machine learning, with external factors included, significantly improves project performance prediction.","pith_inferences":["A natural extension the paper does not pursue is an online forecasting loop that retrains or updates the model as each progress payment and actual cost record arrives.","Because the external factors were simulated, the strongest test of the paper's message is to rerun the same models with recorded weather and resource data and compare the SHAP rankings.","The per-WBS granularity suggests the method would transfer to other infrastructure projects, but projects with short histories or irregular reporting may not have enough data to train an LSTM reliably.","If the findings hold, the practical bottleneck shifts from model choice to data quality: the value of the forecast depends on how faithfully cost and progress histories are recorded."],"forward_implications":["Project managers could identify WBS categories drifting from baseline early and take corrective action before cost variance widens.","Collecting real weather and resource-availability data would make the same models directly usable for ongoing project monitoring.","The reported superiority of LSTM over ARIMA implies that nonlinear, long-memory patterns carry forecasting signal that linear models miss.","Feature-importance rankings can tell project teams which metrics deserve the most disciplined data collection."],"supporting_citations":[{"why":"Defines earned value management and its performance indicators, the traditional baseline the paper compares against.","marker":"[6]"},{"why":"Documents EVM's limitations under dynamic conditions, motivating the machine-learning approach.","marker":"[7]"},{"why":"Provides the ARIMA-versus-LSTM comparison that the paper's model selection builds on.","marker":"[11]"},{"why":"Establishes LSTM's ability to model long- and short-term temporal patterns, the reason LSTM is chosen.","marker":"[12]"},{"why":"Shows weather-related time series forecasting with LSTM, supporting weather as an external feature.","marker":"[13]"},{"why":"Supports the claim that external factors such as weather and resources affect construction project performance.","marker":"[14]"},{"why":"Supplies the dataset of real project records used for the case study.","marker":"[16]"}],"fun_headline_variants":["LSTM beats ARIMA and EVM on project cost forecasting","ML project forecasting outperforms static EVM baselines","Deep learning sharpens road project performance predictions","Time-series AI predicts cost variance better than traditional EVM","Machine learning boosts construction project forecasting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that simulated weather and resource-availability values can stand in for real measurements of those factors; if the simulation is not representative of the actual project, the findings about external factors and the comparison with EVM do not necessarily hold for real conditions.","fun_headline_variants_meta":{"raw":{"variants":["LSTM beats ARIMA and EVM on project cost forecasting","ML project forecasting outperforms static EVM baselines","Deep learning sharpens road project performance predictions","Time-series AI predicts cost variance better than traditional EVM","Machine learning boosts construction project forecasting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000643,"raw_usage":{"total_tokens":2927,"prompt_tokens":884,"completion_tokens":2043,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":1970}},"tokens_in":500,"tokens_out":2043,"duration_ms":12073,"temperature":1.0,"reasoning_tokens":1970,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:41:48.061809+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain ARIMA and LSTM on the same project with recorded, not simulated, weather and resource data, and compute MAE, MSE, and RMSE on a held-out period; if LSTM's errors are not below ARIMA's and EVM's, the paper's central claim fails.","supporting_citations":[{"cited_title":"Vanhoucke, Measuring time: Improving project performance using earned value management (Vol","cited_arxiv_id":null,"evidence_quote":"Defines earned value management and its performance indicators, the traditional baseline the paper compares against."},{"cited_title":"Measuring the impact of scope changes on project plan using EVM,","cited_arxiv_id":null,"evidence_quote":"Documents EVM's limitations under dynamic conditions, motivating the machine-learning approach."},{"cited_title":"A comparison of ARIMA and LSTM in forecasting time series,","cited_arxiv_id":null,"evidence_quote":"Provides the ARIMA-versus-LSTM comparison that the paper's model selection builds on."},{"cited_title":"Modeling long-and short-term temporal patterns with deep neural networks,","cited_arxiv_id":null,"evidence_quote":"Establishes LSTM's ability to model long- and short-term temporal patterns, the reason LSTM is chosen."},{"cited_title":"Day -ahead solar irradiance forecasting for microgrids using a long short -term memory recurrent neural network: A deep learning approach,","cited_arxiv_id":null,"evidence_quote":"Shows weather-related time series forecasting with LSTM, supporting weather as an external feature."},{"cited_title":"System dynamics approach for forecasting performance of construction projects,","cited_arxiv_id":null,"evidence_quote":"Supports the claim that external factors such as weather and resources affect construction project performance."},{"cited_title":"Developing a dataset of real projects for portfolio, program and project control management research,","cited_arxiv_id":null,"evidence_quote":"Supplies the dataset of real project records used for the case study."}],"review_version":1}