{"id":"237010e6-e365-4dc4-bc5e-31379cde1d47","arxiv_id":"1909.11637","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"XGBoost outperformed 19 other AI techniques in predicting conceptual costs of Egyptian field canal improvement projects, achieving 9.091% MAPE and 0.929 adjusted R-squared on a 144-case dataset.","lead":"This study compares 20 machine learning and artificial intelligence methods for predicting the early-stage cost of field canal improvement projects in Egypt, using 144 historical cases. It reports that XGBoost gives the most accurate predictions, with a mean absolute percentage error of 9.091%.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-split evaluation leaves the XGBoost-vs-quadratic-regression ranking within noise; no significance testing or cross-validation supports the headline claim.","rationale":"The reader's weakest assumption correctly identifies the single random split and absent significance testing as the main fragility of the paper's central claim. My stress-test agrees and adds two concrete technical details: the difference between the top two models is far smaller than the expected sampling variability for a 33-case test set, and the feature-selection step performed on the full dataset in the companion paper leaks test information into all models. Both points reinforce the conditional verdict rather than changing it. The paper is a plausible applied comparison, but its headline ranking cannot be accepted as established without a resampling-based uncertainty analysis. The proposed test would settle whether XGBoost's advantage is real or an artifact of one split. I do not see an internal inconsistency that would justify rejection; the concern is purely about statistical support for the comparison.","tokens_in":14671,"tokens_out":2724,"duration_ms":27495,"concrete_test":"Re-run the full comparison using repeated stratified 5-fold cross-validation (or 100 bootstrap resamples) on the 144 FCIP cases, retraining every model with the same hyperparameters described in the paper (or, if those are unavailable, with inner cross-validation for hyperparameter choice). Record the MAPE distribution and the rank of XGBoost across folds/resamples. If XGBoost is not the top-ranked model in a majority of resamples, or if the 95% confidence interval for the MAPE difference (M1 − M2) includes zero, the reported superiority of XGBoost is not established. As a second check, redo feature selection within each training fold only before training; if the gap between M1 and M2 changes materially or the absolute MAPE values shift, the original comparison leaks selection information from the test set.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that XGBoost is the most accurate method (MAPE 9.091%, adjusted R2 0.929; Table 1, M1) rests entirely on a single random split of 144 projects into 111 training and 33 test cases. The margin over the second-ranked model, quadratic regression (M2), is 0.029 percentage points in MAPE, and the top five models span only 9.091–9.270%. With 33 test points, the standard error of MAPE is on the order of 1–2 percentage points, so the observed differences are within noise. No cross-validation, bootstrap, or significance test (e.g., paired t-test or Wilcoxon) is reported, despite the paper's own statement that 'the most accurate and suitable method is XGBoost.' Additionally, the four cost drivers were selected by Pearson correlation and stepwise regression on the full 144-case dataset in Elmousalami et al. (2018b); the same dataset is then split for training and testing, so feature selection leaks test information into every model. This leakage inflates absolute accuracy and can distort the ranking because the preselected features were chosen partly based on their relationship with cost in the test cases as well. The paper itself notes that the previous study (Elmousalami et al. 2018b) found the quadratic regression 'the most accurate' on the same data, underscoring that the winner label is not stable across studies with similar data. Since the top two models differ by less than the expected split-to-split variation, the headline claim is not statistically supported as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares twenty AI and machine-learning techniques for conceptual cost prediction of field canals improvement projects (FCIPs) in Egypt. Using a dataset of 144 projects split into 111 training and 33 test cases, the authors evaluate models from multiple families (regression, ANNs/DNNs, SVM, fuzzy and genetic-fuzzy systems, CBR, decision trees, and ensemble methods) with MAPE and adjusted R^2. The headline claim is that XGBoost is the most accurate and suitable model, with 9.091% MAPE and 0.929 adjusted R^2 (Table 1, M1), narrowly ahead of quadratic regression at 9.120% MAPE. The paper also discusses qualitative properties such as interpretability, uncertainty handling, and robustness to noisy data, and provides a strengths/weaknesses table for all models.","tokens_in":15085,"tokens_out":4417,"duration_ms":46217,"significance":"If the reported ranking were robust, the paper would provide useful practical guidance for cost-estimation practitioners and a structured benchmark across many model families. The study is creditable for assembling twenty models on a real dataset, for giving a transparent table of model characteristics, and for explicitly discussing the uncertainty/interpretability trade-off. However, the significance is limited by the evaluation methodology: the ranking rests on a single random split with no significance testing, and several reported metrics appear internally inconsistent. These issues are fixable in revision, but as it stands the central claim is not statistically supported.","major_comments":[{"comment":"The headline claim that XGBoost is the most accurate model is not statistically supported. The evaluation uses a single random split (111/33) with no cross-validation, bootstrap, or paired significance test. The top two models differ by only 0.029 percentage points in MAPE (9.091% vs. 9.120%), while the top five span 9.091–9.270%. With 33 test points the standard error of MAPE is plausibly on the order of 1–2 percentage points, so the observed differences are within noise. Please report repeated cross-validation results and a paired test (e.g., Wilcoxon or paired t-test) of the XGBoost-vs-quadratic-regression difference, or explicitly temper the conclusion to 'performs well on this split'.","section":"Comparison and analysis, Table 1"},{"comment":"Feature selection leaks test information into every model. The four cost drivers were selected by Pearson correlation and stepwise regression on the full 144-case dataset in Elmousalami et al. (2018b), and the same dataset is then split for training and testing. This can inflate absolute accuracy and distort the ranking because the preselected features were chosen partly from test-case information. The prior study's finding that quadratic regression was 'the most accurate' on the same data further indicates that the winner label is unstable across studies. Please use a nested procedure (e.g., feature selection inside each cross-validation fold) or at least discuss this limitation explicitly.","section":"Data collection and feature selection"},{"comment":"Several reported adjusted R^2 values exceed the corresponding R^2 values (e.g., M8: R2=0.905, R*2=0.910; M5: R2=0.913, R*2=0.912). By the definition in Eq. (10), adjusted R^2 cannot exceed R^2 when K>0, so these entries are internally inconsistent and indicate computational or rounding errors in the metric calculations. Please recheck the reported adjusted R^2 values and the underlying formulas; the ranking by R*2 may change.","section":"Table 1, Eqs. (9)–(10)"},{"comment":"Equation (4), y_i(W·X_i+b) ≥ 0 − ξ, is a mis-specified classification constraint, not the support vector regression formulation the paper claims to apply. The subsequent objective in Eq. (5) also corresponds to classification-style soft-margin SVM, not to ε-insensitive SVR. Please correct the mathematical description of the SVM model, or remove the equations if the model was fitted with a standard library using default settings.","section":"Support Vector Machines, Eq. (4)"},{"comment":"Equation (8) is garbled: it begins with the binomial expansion identity (x+a)^n = ..., which has no connection to the XGBoost objective, and then mixes a loss term and a regularization term without a clear definition of the components. The intended XGBoost loss with regularization is standard; please rewrite Eq. (8) properly so the description is technically accurate.","section":"Extreme Gradient Boosting, Eq. (8)"},{"comment":"There is a numerical inconsistency in the fuzzy rule count. The text says 'the number of possible rules equals 74 rules' while later stating '2401 rules represent the whole possible search space for GA.' With four inputs and seven membership functions per input, the correct number is 7^4 = 2401. The '74' appears to be a typo for '7^4,' but as written it is confusing; please correct it.","section":"Genetic-Fuzzy model"}],"minor_comments":[{"comment":"The abstract says 'twenty artificial intelligence techniques' but the conclusion says 'Twenty one machine learning models are developed'; please make the count consistent (Table 1 lists 20).","section":"Abstract and Conclusion"},{"comment":"The MAPE categorization equation is hard to parse because of missing percentage signs and ambiguous inequalities; for clarity, rewrite as explicit categories (e.g., MAPE < 10% excellent, 10% ≤ MAPE < 20% good, MAPE ≥ 20% unacceptable).","section":"Evaluation techniques"},{"comment":"The text refers to 'Haytham et al. (2018 b)' but the reference list uses 'Elmousalami et al. 2018b'; please standardize the author name throughout.","section":"MRA and transformed regressions"},{"comment":"Several references are duplicated (Fan et al. 2008, Schapire et al. 1998, Stoy et al. 2012); please remove duplicates.","section":"References"},{"comment":"Hyperparameter values are only reported for the genetic algorithm (crossover 0.7, mutation 0.01). Please report the hyperparameters used for SVM (C, gamma/kernel parameters), XGBoost (learning rate, max depth, number of trees, etc.), and the DNN (optimizer, epochs, batch size) to make the comparisons reproducible.","section":"Model development sections"},{"comment":"Figure 7 plots MAPE and R^2 with different scales on separate subfigures; consider adding clear axis labels and possibly a combined figure with normalized scales for easier visual comparison across models.","section":"Fig. 7"}],"recommendation":"major_revision","confidential_remarks":"The single-split evaluation and the metric inconsistencies (adjusted R2 > R2) are serious enough that the central claim is currently unsupported. The authors should be asked to rerun the comparison with repeated cross-validation and significance testing and to recompute all metrics; if they cannot, the paper should probably be rejected. The reliance on prior work for seven of the twenty models and for feature selection also needs to be addressed, but that is fixable with a nested procedure or explicit discussion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"XGBoost is almost certainly not the winner here. The paper compares 20 AI methods on 144 Egyptian field canal projects and finds XGBoost best by a hair (9.091% vs 9.120% MAPE for quadratic regression), but a single random split of 111/33 with no cross-validation or significance testing means the top five finishers are statistically indistinguishable. That's the main thing to know.\n\nCredit where it's due: the paper is a genuine benchmark, not a toy. The author applies a broad set of techniques—regression, ANNs, DNNs, CBR, fuzzy, genetic-fuzzy, SVM, and several ensemble methods—to a real cost-estimation problem, and the comparison on this specific FCIP dataset hasn't been published before. The text is organized around a practical question, and there's an honest discussion of model interpretability, uncertainty, and small-sample issues. The author also correctly notes that his earlier work had quadratic regression as the best model; he's not hiding the fact that the winner label is unstable.\n\nThe soft spots, in order of severity: (1) The evaluation protocol. One random split gives no error bars. With 33 test points, the standard error of MAPE is on the order of 1–2 percentage points, so a 0.029-point gap between M1 and M2 is meaningless. No paired test, no bootstrap, no repeated CV. (2) Feature selection leakage. The four cost drivers were selected by Pearson correlation and stepwise regression on the full 144-project dataset in Elmousalami et al. 2018b, so those features already encode test-set information. That inflates absolute accuracy and can distort the ranking. (3) Missing reproducibility. No code, no data, no hyperparameter settings for the 20 models. The fuzzy and GA portions are described only in outline. (4) Minor typos: Eq. 4 has an inequality that doesn't make sense, Eq. 8 writes the XGBoost objective as $(x+a)^n$ (a binomial expansion, not the actual loss), and the conclusion says 21 models while Table 1 lists 20.\n\nThese are not fatal to the paper's existence, but they are fatal to the headline claim as stated. The comparison itself is not circular—the XGBoost result is empirical, not derived from its inputs—and the author's engagement with the prior literature is honest. But the practical recommendation to cost engineers ('use XGBoost') is not supported by this evidence.\n\nWho gets value: a cost engineer wanting a quick map of what AI methods exist and roughly how they do on this kind of data, or a graduate student looking for a case study in why single-split benchmarks are dangerous. The paper is not a research contribution that will change methodology.\n\nIf I were the editor, I'd send it to peer review only with a strong directive: rerun with cross-validation or bootstrap, add significance tests, release the data and hyperparameters, and fix the leakage issue. If the author can do that, a revised version could be useful. As it stands, I'd treat the XGBoost recommendation as unverified.","headline":"The XGBoost win is almost certainly within noise due to a single-split evaluation, but the benchmark is a useful practical survey if properly re-validated.","tokens_in":15541,"tokens_out":3616,"would_cite":false,"duration_ms":36601,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"XGBoost beats 19 AI models at predicting early-stage project costs, with 9.091% MAPE on a 144-project irrigation dataset.","keywords":["conceptual cost prediction","XGBoost","ensemble methods","machine learning","construction cost estimation","fuzzy logic","genetic fuzzy","parametric cost model"],"falsifier":"Run the same models on the same FCIP data with repeated stratified k-fold cross-validation and record the variance of MAPE. If the cross-validated mean gap between XGBoost and quadratic regression is not reliably positive, or if feature selection is done inside each fold and the ranking flips, the paper's central claim is not supported.","tokens_in":14410,"feed_emoji":"🏗️","tokens_out":3270,"duration_ms":29053,"temperature":0.7,"pith_summary":"The paper tries to establish that, among twenty supervised regression techniques applied to a real-world dataset of 144 Egyptian field canal improvement projects, XGBoost gives the most accurate conceptual cost estimates, with 9.091% MAPE and adjusted R-squared 0.929. This matters because conceptual cost estimates are made early in a project with little information, and practitioners need guidance on which method to trust. The paper also argues that ensemble methods outperform single models, that data transformations help regression and neural models, and that fuzzy models trade accuracy for uncertainty handling. The comparison itself is the contribution: a single benchmark ranking on one dataset, not a proof of general superiority.","feed_headline":"XGBoost wins the 20-model cost prediction contest","feed_subtitle":"On 144 Egyptian irrigation projects, boosted trees hit 9.091% MAPE, edging out quadratic regression and deep nets.","key_machinery":"The central object is the XGBoost ensemble: an additive model $\\hat{y}_i = \\sum_k f_k(x_i)$ of regression trees trained with a regularized objective $L = \\sum_k \\ell(\\hat{y}_k, y_k) + \\sum_k \\Omega(f_k)$, where $\\Omega(f) = \\gamma T + \\frac{1}{2}\\lambda \\|w\\|^2$. The regularization term and default missing-value handling are what the paper credits for XGBoost's top ranking; the comparison table and the MAPE and adjusted R-squared metrics carry the argument.","core_discovery":"The central claim is that XGBoost is the most accurate and suitable AI technique for conceptual cost prediction on the FCIP dataset, edging out quadratic regression at 9.120% MAPE and plain regression at 9.130%, and clearly beating DNNs at 12.059%, decision trees at 12.488%, genetic-fuzzy at 14.700%, case-based reasoning at 17.300%, SVM at 21.217%, and expert fuzzy logic at 26.300%. The paper presents this as evidence that scalable tree boosting, with its built-in regularization and missing-value handling, is the appropriate default model for small, noisy project-cost data.","pith_inferences":["The margin between first and second place is 0.029 percentage points, far smaller than the uncertainty of a single random split; repeated k-fold cross-validation could easily reorder the top models.","The feature selection was performed on the full dataset before splitting, so information from the test cases may have leaked into model construction and the reported test MAPEs could be optimistic.","A testable extension is to run the same twenty-model comparison on other construction project types and report cross-validated means and variances for each model rather than single-split scores.","The paper's uncertainty-handling argument points toward a natural follow-up: a hybrid that combines XGBoost accuracy with fuzzy interval outputs, giving both accuracy and uncertainty bounds."],"forward_implications":["If the ranking holds, project managers estimating early-stage irrigation improvement costs should prefer XGBoost over the previously recommended quadratic regression, at least for similar FCIP-type data.","Ensemble methods (Extra Trees, bagging, random forest, AdaBoost, stochastic gradient boosting) all land below 11% MAPE, supporting ensemble approaches over single learners when data are small and noisy.","The gap between genetic-fuzzy at 14.7% and expert fuzzy at 26.3% implies that automated fuzzy rule generation is the route to making uncertainty-handling fuzzy models usable in cost estimation.","DNNs at 12.059% MAPE were worse than every ensemble method, indicating that deep networks are not suitable for datasets of this size.","The paper supplies a benchmark ranking that future conceptual cost prediction studies can compare against when choosing or proposing new models."],"supporting_citations":[{"why":"Provides the XGBoost algorithm, its regularized objective, scalability, and missing-value handling that the paper credits for the top result.","marker":"(Chen and Guestrin 2016)"},{"why":"Supplies the FCIP dataset, the four cost drivers, and the quadratic regression and ANN baselines that XGBoost must beat.","marker":"(Elmousalami et al. 2018 b)"},{"why":"Supplies the qualitative cost-driver ranking used to define the features for all twenty models.","marker":"(Elmousalami et al. 2018 a)"},{"why":"Justifies the 111-instance training sample as acceptable for four predictors.","marker":"(Green 1991)"},{"why":"Sets the 20% MAPE acceptability threshold used to classify models as acceptable or unacceptable.","marker":"(Peurifoy and Oberlender 2002)"},{"why":"Provides the MAPE quality bands (below 10%, 10–20%, above 20%) used in the evaluation categorization.","marker":"(Lewis 1982)"},{"why":"Defines random forests, the bagging-based ensemble baseline in the comparison.","marker":"(Breiman 2001)"},{"why":"Defines AdaBoost, one of the ensemble baselines in the twenty-model comparison.","marker":"(Freund and Schapire 1997)"}],"fun_headline_variants":["XGBoost beats 19 AI models for conceptual cost prediction","Boosted trees win cost forecast contest among 20 AI methods","XGBoost nails 9.091% MAPE in cost prediction shootout","XGBoost wins 20-model cost prediction face-off","XGBoost outperforms 19 AI rivals for project cost estimates"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking depends on one random split of 144 projects into 111 training and 33 test cases, and the top two models are within 0.03 percentage points, so a different split could change the winner.","fun_headline_variants_meta":{"raw":{"variants":["XGBoost beats 19 AI models for conceptual cost prediction","Boosted trees win cost forecast contest among 20 AI methods","XGBoost nails 9.091% MAPE in cost prediction shootout","XGBoost wins 20-model cost prediction face-off","XGBoost outperforms 19 AI rivals for project cost estimates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001091,"raw_usage":{"total_tokens":4556,"prompt_tokens":943,"completion_tokens":3613,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":3519}},"tokens_in":559,"tokens_out":3613,"duration_ms":22748,"temperature":1.0,"reasoning_tokens":3519,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:19:30.802795+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same models on the same FCIP data with repeated stratified k-fold cross-validation and record the variance of MAPE. If the cross-validated mean gap between XGBoost and quadratic regression is not reliably positive, or if feature selection is done inside each fold and the ranking flips, the paper's central claim is not supported.","supporting_citations":[],"review_version":1}