{"id":"6318610a-920b-425c-9917-01e90f7d8ee4","arxiv_id":"2506.14593","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Leave-one-out cross-validation is often a useful model selection tool for small designed experiments, though ordinary k-fold cross-validation is uneven in performance.","lead":"This paper uses extensive simulations to test whether cross-validation, the standard machine learning tool for picking models, can be safely used on small designed experiments. It finds that leave-one-out cross-validation is often competitive, and challenges the long-standing caution against using cross-validation in this setting.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Response-surface simulations evaluate CV under a random-X prediction metric, not the fixed-X setting that motivated the paper's caution; the LOOCV 'success' may be an artifact of the evaluation criterion.","rationale":"The reader's weakest_assumption is external generalizability: whether the simulated response surfaces and screening scenarios represent real experiments. My concern is internal: the response-surface evaluation metric does not match the fixed-X prediction setting that the paper itself identifies as the basis for the prior caution and for including the little bootstrap. This is a more fundamental threat to the central claim because it affects the validity of the headline prediction result even within the simulations. The screening results are not subject to this critique, so the overall conclusion that LOOCV works for screening remains plausible. The correct response is to keep the CONDITIONAL verdict but add a specific condition: the prediction claim must be re-verified under a fixed-X prediction-error criterion, or the paper's scope should be narrowed to random-X prediction over the experimental region. Since the reader already issued CONDITIONAL, I recommend no change to the verdict, though the condition I identify is distinct from the reader's.","tokens_in":16678,"tokens_out":6932,"duration_ms":76642,"concrete_test":"Re-run the Section 3.1 response-surface simulations with a fixed-X evaluation: for each simulated surface and selected model, draw new independent responses at the original design's predictor matrix X (same design points, fresh error draws) and compute RMSPE at those design points, for all methods (full regression, regression-CV/LOOCV, LB, random forest). If regression-LOOCV is no longer competitive with the full model and LB (or if CV becomes more variable), the paper's claim that the literature's fixed-X caution is unsupported would not survive. Supplementary code at github.com/weeseml/CVandDOE should make this a direct modification of the existing evaluation step.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing weakness is an evaluation-metric mismatch in the response-surface study. Section 3.1 step 2 evaluates every method by generating 1000 fresh out-of-sample points uniformly on U(-1,1) and computing RMSPE there. That is a random-X prediction-error criterion. But the paper's motivation is Breiman's (1992) fixed-X critique: Section 2 states that CV 'assumes that the predictors are randomly sampled' and 'in the fixed predictor case, as in designed experiments, [Breiman] argues that CV will overestimate the prediction error.' The little bootstrap was included specifically to address fixed-X prediction error. By scoring all methods at uniform random new points, the simulations test CV in exactly the regime where it is known to work and test LB in a regime it was not designed for. Consequently, the headline finding that 'regression using LOOCV performed particularly well' and that Breiman's concern is 'not supported' does not follow for fixed-X prediction at design points, which is the case the prior literature cautioned about. The screening experiments (Section 4) are not affected by this critique, so the screening conclusion retains support, but the response-surface prediction claim is not established for the setting that motivated the paper.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper empirically studies whether cross-validation (CV) is useful for model selection in small, structured designed experiments, a setting where prior literature has cautioned against CV. For response-surface prediction, the authors compare full second-order regression, regression with 5-fold CV, regression with LOOCV, regression with the little bootstrap (LB), and random forests tuned by CV or LOOCV, across five design types and two sample sizes. For screening, they compare Gauss-Lasso, Lasso with CV or LOOCV, regression with CV or LOOCV, and regression with LB, using both nonregular fractional factorial designs and supersaturated designs. The headline claim, stated in Section 5, is that the earlier caution is not supported: CV-based methods are often competitive, and regression with LOOCV performs particularly well in both prediction and screening.","tokens_in":16900,"tokens_out":7077,"duration_ms":73364,"significance":"If fully established, the paper would provide practical, evidence-based guidance for the growing DOE+ML community. The simulation design is a real strength: it covers multiple design families, two sample sizes, 250-500 replications, independent out-of-sample evaluation, and a priori fixed tuning constants for LB and Gauss-Lasso. The code and designs are available on GitHub, which supports reproducibility. However, the response-surface evaluation appears to measure random-X prediction error, not the fixed-X prediction error that motivates the paper's central question; as a result, the strongest claim in Section 5 is only partially established. The screening results are less affected by this concern and retain value.","major_comments":[{"comment":"The response-surface simulations are scored on 1000 new points drawn uniformly from U(-1,1), which is a random-X prediction-error criterion. The paper's stated motivation, however, is Breiman's fixed-X critique, which Section 2 describes as the case where prediction error is computed at the same design points or at known fixed points, and where CV overestimates prediction error. The little bootstrap was introduced specifically for X-fixed prediction error. Evaluating all methods at freshly sampled uniform X therefore tests CV in a regime where it is known to behave well and tests LB outside the regime for which it was designed. Consequently, the statement in Section 5 that the findings 'do not support' the prior caution about CV in designed experiments is not supported for fixed-X prediction at design points. The authors should either evaluate RMSPE at the n design points or on a fixed prediction grid that includes the design points, and report both fixed-X and random-X metrics separately.","section":"Sec. 3.1 (simulation step 2), Sec. 2, and Sec. 5"},{"comment":"The central comparative claims, including the headline that regression with LOOCV 'performed particularly well' and that Lasso CV performed poorly, are based entirely on visual inspection of boxplots. Because the simulations involve 250 or 500 replications, the authors should provide numerical summaries such as means, medians, standard errors, or confidence intervals, and ideally formal paired comparisons. Without these, it is difficult to tell whether visible differences are systematic effects or sampling variation, especially in the small designs where boxplots overlap substantially.","section":"Sec. 3.2 and Sec. 4.1-4.2 (results figures)"},{"comment":"Regression with LOOCV and regression with LB are reported as failing for the MEPI design, with the cause left unexplained. Since the paper later recommends regression with LOOCV across screening designs, a complete failure of that method on one of the five screening designs should be quantified and discussed: how often the failure occurred, which simulation scenarios were affected, and whether it is an implementation artifact of the leaps package or a property of the design. The current wording, 'it is unclear as to why this occurred', is insufficient for a recommendation that is supposed to hold across designs.","section":"Sec. 4.1 (Figures 9-10) and Sec. 5"}],"minor_comments":[{"comment":"The phrase 'small, structured is often useful' is missing a noun; it should read 'small, structured designs is often useful'.","section":"Abstract"},{"comment":"The acronym 'RSMPE' should be 'RMSPE' for consistency with the rest of the paper.","section":"Sec. 3.1, simulation step 4"},{"comment":"The caption says 'n = 16, k = 3' but should refer to m = 3, the number of factors.","section":"Figure 8 caption"},{"comment":"The text says 'RMSPE vales' and should read 'RMSPE values'.","section":"Sec. 3.2"},{"comment":"The random forest hyperparameter grid is only described as being in the Supplementary Materials; because random forests are part of the comparative claims, at least the key grid values should be summarized in the main text.","section":"Sec. 3.1, notes on methods"}],"recommendation":"major_revision","confidential_remarks":"The random-X versus fixed-X evaluation-metric issue is the main reason for major revision. The screening portion of the paper is valuable and could be published with modest changes; the response-surface portion needs to be either supplemented with fixed-X prediction-error evaluation or explicitly reframed as a study of random-X prediction. The paper is not internally inconsistent, but the current framing overstates what the simulations show."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know up front. First, this is the first systematic simulation study of cross-validation in designed experiments, and the screening half is solid enough to be useful on its own. Second, the response-surface half does not actually test the Breiman caution the paper claims to address: all methods are scored on 1000 freshly drawn uniform points, a random-X criterion, whereas Breiman's fixed-X critique and the little bootstrap are about prediction at the fixed design points. So the headline claim that LOOCV 'performed particularly well' and that prior cautions are unsupported is, for the response-surface setting, not established.\n\nWhat the paper does well: it runs a broad, careful simulation campaign. Response-surface designs include CCD, BBD, I-optimal, MaxPro; screening includes nonregular factorials and supersaturated designs. Tuning constants for LB and Gauss-Lasso are fixed a priori from prior literature, not fit to the simulation outcomes. The screening analysis measures power and type I error directly, which is the right criterion for selection, and there LOOCV regression comes out as a sensible choice—lower type I error than lasso-based methods with comparable power. Code and designs are on GitHub. That is real, reproducible work.\n\nSoft spots, in order of seriousness. (1) The metric mismatch above. If the question is 'does CV overestimate prediction error in fixed-X designs,' you have to evaluate at the design points or at specified fixed locations. Scoring at uniform random points puts every method in the random-X regime, where CV is expected to do well. The authors should either add a fixed-X evaluation or explicitly reframe the response-surface claim as being about prediction over the whole factor region—a legitimate but different question. (2) Conclusions rest on boxplot inspection. With 500 replications, a paired comparison or error bars would firm up claims like 'competitive' and 'uneven.' (3) The LOOCV/LB failures on the MEPI design are unexplained; the authors' aliasing hypothesis is plausible but unverified. (4) Minor editorial: the abstract has an incomplete sentence.\n\nWho should read this: applied statisticians and engineers using ML on small experiments, and people who cite Breiman's caution as received wisdom. They will get practical guidance for screening, and a clear example of how evaluation metrics shape conclusions.\n\nMy recommendation: send it to peer review. The screening results and the general question are worth referee time. Ask for a fixed-X evaluation of the response-surface methods (or a careful reframing) and some formal comparisons before publication.","headline":"A first systematic simulation study of CV in designed experiments; the screening results are solid and useful, but the response-surface claim is undercut by a random-X evaluation metric that never tests Breiman's fixed-X caution.","tokens_in":17461,"tokens_out":3920,"would_cite":true,"duration_ms":36339,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62K05","62K20","62J05","62F40"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that cross-validation, especially leave-one-out, is often a reliable model selection tool for small structured designed experiments, contrary to longstanding cautions.","keywords":["cross-validation","leave-one-out cross-validation","design of experiments","response surface design","screening design","supersaturated design","little bootstrap","model selection"],"falsifier":"Re-run the same comparison on data from a real designed experiment whose active factors are confirmed by independent follow-up runs, or generate surfaces from empirical response-surface case studies instead of the testbed; if regression with LOOCV no longer matches the full model on out-of-sample RMSPE or loses its power/type-I balance in screening, the central claim would be refuted.","tokens_in":16453,"feed_emoji":"📊","tokens_out":7985,"duration_ms":72590,"temperature":0.7,"pith_summary":"Cross-validation has been treated with suspicion in the analysis of small, structured experiments because the design matrix is fixed and the designs are too small to give CV room to work. This paper presents a systematic simulation comparison of CV against alternatives, including full regression, the little bootstrap, and machine-learning baselines, in two settings: response-surface prediction and screening. Its central claim is that the caution is overstated: leave-one-out cross-validation (LOOCV) is frequently competitive, and in screening it offers the best balance of detecting active factors and avoiding false ones. More general k-fold CV is more uneven, and the little bootstrap offers no consistent advantage. If the claim holds, experimenters using machine-learning workflows on designed experiments have a defensible way to select models without abandoning CV.","feed_headline":"Use leave-one-out cross-validation for small designed experiments","feed_subtitle":"Simulation shows LOOCV is competitive for prediction and screening, contradicting earlier cautions.","key_machinery":"The argument rests on a fixed-design resampling protocol. For each cross-validation split, best subsets regression selects the best model of each candidate size from the training folds, the held-out fold supplies a root mean squared prediction error, and the model size with the lowest average RMSPE is refit on the full data; LOOCV repeats this n times with n-1 training runs, which the paper argues retains the design geometry almost intact. The little bootstrap is the same model-size selection driven by bootstrap-perturbed responses with a bias correction rather than by held-out folds. This comparison machinery is what lets the paper separate the effect of the selection rule from the effect of the design itself.","core_discovery":"The paper's central discovery, stated on its own terms, is that prior warnings against combining CV with designed experiments find no empirical support in the settings studied. Across five response surface designs (central composite, axial central composite, I-optimal, Box-Behnken, and MaxPro) at two sizes, and across true models that are full second order, reduced second order, and sixth order, regression with LOOCV is competitive with the full second-order model on out-of-sample root mean squared prediction error and often more accurate when the true model is a reduced surface. In screening with nonregular fractional factorial designs and in supersaturated designs where the number of runs is smaller than the number of factors, regression with LOOCV gives the most favorable combination of power and type I error among the tested methods. K-fold CV performs unevenly, lasso tuned by CV is comparatively weak, and the little bootstrap, included as a theoretically motivated alternative, does not outperform CV. The paper interprets LOOCV's strength as preserving the structure of the small design better than k-fold.","pith_inferences":["A natural next test is whether LOOCV's advantage extends to other machine-learning models used in DOE+ML practice, such as neural networks, support vector regression, or gradient boosting, since the paper's strongest evidence is regression-based.","Because LOOCV preserves design structure, one would expect its advantage over k-fold to grow as designs become more highly structured or more correlated; this could be tested by varying orthogonality and run-to-factor ratios across a wider class of designs.","The paper's results imply a practical decision rule: use LOOCV when model uncertainty is real, but expect it to add little when the full second-order model is known to be correct, since the full model is already a robust baseline.","A hybrid not tested in the paper, using lasso's tuning path with LOOCV plus coefficient thresholding as in Gauss-Lasso, might combine the power of lasso LOOCV with better type I error control."],"forward_implications":["Response-surface practitioners can use regression with LOOCV as a competitive alternative to the full second-order model, particularly when the true surface is believed to be a subset of second-order terms.","In screening studies, including supersaturated ones, regression with LOOCV is a defensible default: it detects active effects well without the inflated type I error of lasso-based CV.","K-fold CV and random forests tuned by 5-fold CV should be treated cautiously, since they overfit when the true model is simple.","The little bootstrap, despite its theoretical motivation, does not currently justify replacing CV in either prediction or screening tasks."],"supporting_citations":[{"why":"Supplies the theoretical argument that CV overestimates prediction error with fixed predictors and proposes the little bootstrap, which the paper tests as an alternative.","marker":"Breiman (1992)"},{"why":"Provides an earlier LOOCV application to a small design and the caution that CV is less effective than viewing coefficient paths, a claim the paper revisits.","marker":"Yuan et al. (2007)"},{"why":"Reports poor CV performance for shrinkage models in supersaturated scenarios, defining the screening baseline the paper contests.","marker":"Draguljić et al. (2014)"},{"why":"Supplies the best-subsets-plus-CV regression protocol that the paper implements for response-surface model selection.","marker":"James et al. (2013)"},{"why":"Provides the nonregular fractional factorial screening designs, simulation scenarios, and power/type I error protocol for the n=20, m=7 study.","marker":"Mee et al. (2017)"},{"why":"Supplies the supersaturated design sizes, construction criteria, and scenario structure used in the screening comparisons.","marker":"Marley and Woods (2010)"},{"why":"Provides the response-surface testbed used to generate realistic true models with control over flatness and steepness.","marker":"McDaniel and Ankenman (2000)"},{"why":"Adds the implementation details and calibration approach for generating the response surfaces from the testbed.","marker":"Smucker et al. (2021)"},{"why":"Provides the ridge-regression estimate of error variance that lets the little bootstrap run in screening settings where the full model cannot be fit.","marker":"Liu et al. (2020)"}],"fun_headline_variants":["LOOCV defies CV warnings for small experiments","Cross-validation works for small designed experiments","Leave-one-out CV rivals k-fold in design analysis","Small designs benefit from leave-one-out cross-validation","LOOCV rescues designed experiments from CV warnings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The simulations' response surfaces and screening scenarios stand in for real experiments: if actual response surfaces are not represented by the calibrated flat and steep testbed surfaces, or actual screening effects do not look like the fixed magnitudes and sparsity levels used, the ranking of methods, including LOOCV's apparent success, could change.","fun_headline_variants_meta":{"raw":{"variants":["LOOCV defies CV warnings for small experiments","Cross-validation works for small designed experiments","Leave-one-out CV rivals k-fold in design analysis","Small designs benefit from leave-one-out cross-validation","LOOCV rescues designed experiments from CV warnings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000723,"raw_usage":{"total_tokens":3212,"prompt_tokens":886,"completion_tokens":2326,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":2254}},"tokens_in":502,"tokens_out":2326,"duration_ms":16753,"temperature":1.0,"reasoning_tokens":2254,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:50:55.211121+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same comparison on data from a real designed experiment whose active factors are confirmed by independent follow-up runs, or generate surfaces from empirical response-surface case studies instead of the testbed; if regression with LOOCV no longer matches the full model on out-of-sample RMSPE or loses its power/type-I balance in screening, the central claim would be refuted.","supporting_citations":[{"cited_title":"(1992), The little bootstrap and other methods for dimensionality selection in regression: X-fixed prediction error, Journal of the American Statistical Association, 87, 738--754","cited_arxiv_id":null,"evidence_quote":"Supplies the theoretical argument that CV overestimates prediction error with fixed predictors and proposes the little bootstrap, which the paper tests as an alternative."},{"cited_title":"R., and Lin, Y","cited_arxiv_id":null,"evidence_quote":"Provides an earlier LOOCV application to a small design and the caution that CV is less effective than viewing coefficient paths, a claim the paper revisits."},{"cited_title":"(2013), An introduction to statistical learning, vol","cited_arxiv_id":null,"evidence_quote":"Supplies the best-subsets-plus-CV regression protocol that the paper implements for response-surface model selection."},{"cited_title":"W., Schoen, E","cited_arxiv_id":null,"evidence_quote":"Provides the nonregular fractional factorial screening designs, simulation scenarios, and power/type I error protocol for the n=20, m=7 study."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the supersaturated design sizes, construction criteria, and scenario structure used in the screening comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the response-surface testbed used to generate realistic true models with control over flatness and steepness."},{"cited_title":"J., Edwards, D","cited_arxiv_id":null,"evidence_quote":"Adds the implementation details and calibration approach for generating the response surfaces from the testbed."},{"cited_title":"(2020), Estimation of error variance via ridge regression, Biometrika, 107, 481--488","cited_arxiv_id":null,"evidence_quote":"Provides the ridge-regression estimate of error variance that lets the little bootstrap run in screening settings where the full model cannot be fit."}],"review_version":2}