Pith. sign in

REVIEW 3 major objections 5 minor 79 references

The use of cross validation in the analysis of designed experiments

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that cross-validation, especially leave-one-out, is often a reliable model selection tool for small structured designed experiments, contrary to longstanding cautions.

desk verdict A first systematic simulation study of CV in designed experiments; the screening results are solid and useful, but the response-surface claim is undercut by a random-X evaluation metric that never tests Breiman's fixed-X caution. read the letter →

arxiv 2506.14593 v1 pith:J2TO3UZA submitted 2025-06-17 stat.AP stat.ML

Maria L. Weese , Byran J. Smucker , David J. Edwards This is my paper · ORCID
classification stat.APstat.ML MSC 62K0562K2062J0562F40
keywords cross-validationleave-one-outdesignofexperimentsresponsesurfacescreeningsupersaturatedlittlebootstrapmodelselection
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Cross-validation has been treated with suspicion in the analysis of small, structured experiments because the design matrix is fixed and the designs are too small to give CV room to work. This paper presents a systematic simulation comparison of CV against alternatives, including full regression, the little bootstrap, and machine-learning baselines, in two settings: response-surface prediction and screening. Its central claim is that the caution is overstated: leave-one-out cross-validation (LOOCV) is frequently competitive, and in screening it offers the best balance of detecting active factors and avoiding false ones. More general k-fold CV is more uneven, and the little bootstrap offers no consistent advantage. If the claim holds, experimenters using machine-learning workflows on designed experiments have a defensible way to select models without abandoning CV.

What carries the argument

The argument rests on a fixed-design resampling protocol. For each cross-validation split, best subsets regression selects the best model of each candidate size from the training folds, the held-out fold supplies a root mean squared prediction error, and the model size with the lowest average RMSPE is refit on the full data; LOOCV repeats this n times with n-1 training runs, which the paper argues retains the design geometry almost intact. The little bootstrap is the same model-size selection driven by bootstrap-perturbed responses with a bias correction rather than by held-out folds. This comparison machinery is what lets the paper separate the effect of the selection rule from the effect of the design itself.

What would settle it

Re-run the same comparison on data from a real designed experiment whose active factors are confirmed by independent follow-up runs, or generate surfaces from empirical response-surface case studies instead of the testbed; if regression with LOOCV no longer matches the full model on out-of-sample RMSPE or loses its power/type-I balance in screening, the central claim would be refuted.

Watch

Extended reading notes

Core claim

The paper's central discovery, stated on its own terms, is that prior warnings against combining CV with designed experiments find no empirical support in the settings studied. Across five response surface designs (central composite, axial central composite, I-optimal, Box-Behnken, and MaxPro) at two sizes, and across true models that are full second order, reduced second order, and sixth order, regression with LOOCV is competitive with the full second-order model on out-of-sample root mean squared prediction error and often more accurate when the true model is a reduced surface. In screening with nonregular fractional factorial designs and in supersaturated designs where the number of runs is smaller than the number of factors, regression with LOOCV gives the most favorable combination of power and type I error among the tested methods. K-fold CV performs unevenly, lasso tuned by CV is comparatively weak, and the little bootstrap, included as a theoretically motivated alternative, does not outperform CV. The paper interprets LOOCV's strength as preserving the structure of the small design better than k-fold.

Load-bearing premise

The simulations' response surfaces and screening scenarios stand in for real experiments: if actual response surfaces are not represented by the calibrated flat and steep testbed surfaces, or actual screening effects do not look like the fixed magnitudes and sparsity levels used, the ranking of methods, including LOOCV's apparent success, could change.

Editorial extensions

If this is right

  • Response-surface practitioners can use regression with LOOCV as a competitive alternative to the full second-order model, particularly when the true surface is believed to be a subset of second-order terms.
  • In screening studies, including supersaturated ones, regression with LOOCV is a defensible default: it detects active effects well without the inflated type I error of lasso-based CV.
  • K-fold CV and random forests tuned by 5-fold CV should be treated cautiously, since they overfit when the true model is simple.
  • The little bootstrap, despite its theoretical motivation, does not currently justify replacing CV in either prediction or screening tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next test is whether LOOCV's advantage extends to other machine-learning models used in DOE+ML practice, such as neural networks, support vector regression, or gradient boosting, since the paper's strongest evidence is regression-based.
  • Because LOOCV preserves design structure, one would expect its advantage over k-fold to grow as designs become more highly structured or more correlated; this could be tested by varying orthogonality and run-to-factor ratios across a wider class of designs.
  • The paper's results imply a practical decision rule: use LOOCV when model uncertainty is real, but expect it to add little when the full second-order model is known to be correct, since the full model is already a robust baseline.
  • A hybrid not tested in the paper, using lasso's tuning path with LOOCV plus coefficient thresholding as in Gauss-Lasso, might combine the power of lasso LOOCV with better type I error control.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper empirically studies whether cross-validation (CV) is useful for model selection in small, structured designed experiments, a setting where prior literature has cautioned against CV. For response-surface prediction, the authors compare full second-order regression, regression with 5-fold CV, regression with LOOCV, regression with the little bootstrap (LB), and random forests tuned by CV or LOOCV, across five design types and two sample sizes. For screening, they compare Gauss-Lasso, Lasso with CV or LOOCV, regression with CV or LOOCV, and regression with LB, using both nonregular fractional factorial designs and supersaturated designs. The headline claim, stated in Section 5, is that the earlier caution is not supported: CV-based methods are often competitive, and regression with LOOCV performs particularly well in both prediction and screening.

Significance. If fully established, the paper would provide practical, evidence-based guidance for the growing DOE+ML community. The simulation design is a real strength: it covers multiple design families, two sample sizes, 250-500 replications, independent out-of-sample evaluation, and a priori fixed tuning constants for LB and Gauss-Lasso. The code and designs are available on GitHub, which supports reproducibility. However, the response-surface evaluation appears to measure random-X prediction error, not the fixed-X prediction error that motivates the paper's central question; as a result, the strongest claim in Section 5 is only partially established. The screening results are less affected by this concern and retain value.

major comments (3)
  1. [Sec. 3.1 (simulation step 2), Sec. 2, and Sec. 5] The response-surface simulations are scored on 1000 new points drawn uniformly from U(-1,1), which is a random-X prediction-error criterion. The paper's stated motivation, however, is Breiman's fixed-X critique, which Section 2 describes as the case where prediction error is computed at the same design points or at known fixed points, and where CV overestimates prediction error. The little bootstrap was introduced specifically for X-fixed prediction error. Evaluating all methods at freshly sampled uniform X therefore tests CV in a regime where it is known to behave well and tests LB outside the regime for which it was designed. Consequently, the statement in Section 5 that the findings 'do not support' the prior caution about CV in designed experiments is not supported for fixed-X prediction at design points. The authors should either evaluate RMSPE at the n design points or on a fixed prediction grid that includes the design points, and report both fixed-X and random-X metrics separately.
  2. [Sec. 3.2 and Sec. 4.1-4.2 (results figures)] The central comparative claims, including the headline that regression with LOOCV 'performed particularly well' and that Lasso CV performed poorly, are based entirely on visual inspection of boxplots. Because the simulations involve 250 or 500 replications, the authors should provide numerical summaries such as means, medians, standard errors, or confidence intervals, and ideally formal paired comparisons. Without these, it is difficult to tell whether visible differences are systematic effects or sampling variation, especially in the small designs where boxplots overlap substantially.
  3. [Sec. 4.1 (Figures 9-10) and Sec. 5] Regression with LOOCV and regression with LB are reported as failing for the MEPI design, with the cause left unexplained. Since the paper later recommends regression with LOOCV across screening designs, a complete failure of that method on one of the five screening designs should be quantified and discussed: how often the failure occurred, which simulation scenarios were affected, and whether it is an implementation artifact of the leaps package or a property of the design. The current wording, 'it is unclear as to why this occurred', is insufficient for a recommendation that is supposed to hold across designs.
minor comments (5)
  1. [Abstract] The phrase 'small, structured is often useful' is missing a noun; it should read 'small, structured designs is often useful'.
  2. [Sec. 3.1, simulation step 4] The acronym 'RSMPE' should be 'RMSPE' for consistency with the rest of the paper.
  3. [Figure 8 caption] The caption says 'n = 16, k = 3' but should refer to m = 3, the number of factors.
  4. [Sec. 3.2] The text says 'RMSPE vales' and should read 'RMSPE values'.
  5. [Sec. 3.1, notes on methods] The random forest hyperparameter grid is only described as being in the Supplementary Materials; because random forests are part of the comparative claims, at least the key grid values should be summarized in the main text.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity; the paper is an empirical simulation comparison with independent test points and fixed a priori tuning constants.

full rationale

The paper is an empirical simulation study, not a derivation. The central comparisons evaluate each model selection method by generating true response surfaces from the McDaniel-Ankenman testbed (Section 3.1), selecting a model on simulated training data, and scoring the selected model on 1000 independent U(-1,1) points via RMSPE (Section 3.1, steps 1-5); screening is scored by power and type I error against known active effects (Sections 4.1 and 4.2). No tuning constant is fit to the outcomes: the little bootstrap uses t=0.6 and nbootstrap=25 fixed from Breiman (1992), and Gauss-Lasso uses gamma=0.1*max|beta| with BIC, both stated a priori. The authors' self-citations (Smucker et al. 2021 testbed implementation, Ockuly et al. 2017 effect probabilities, Weese et al. 2021 Gauss-Lasso implementation, Mee et al. 2017 screening protocol) are inputs to the simulation or comparator definitions, not results that are then 'predicted'; none of the cited works is invoked as a uniqueness theorem that forces the LOOCV conclusion. The skeptic's concern about the Section 3 evaluation metric (random-X U(-1,1) points rather than fixed design points) is a substantive external-validity criticism of whether the response-surface experiment addresses Breiman's fixed-X claim, but it is not circularity: the reported RMSPE is not equivalent to any fitted input or to the conclusion by construction. The screening conclusions are independent of that critique. Therefore no circular step is present, and the appropriate score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim is empirical and rests on simulation testbeds, method tuning constants taken from prior literature, and standard software implementations. No new entities are introduced. The main assumptions are representativeness of the simulated scenarios and correctness of the R packages used.

free parameters (4)
  • t (little bootstrap perturbation scale) = 0.6
    Chosen following Breiman (1992) rather than fitted to data; affects the perturbation variance in LB and thus the selected model size.
  • nbootstrap (little bootstrap iterations) = 25
    Set to 25 following Breiman (1992); the authors themselves note LB performance may suffer from too few iterations (Section 5).
  • gamma (Gauss-Lasso threshold) = 0.1 * max(|beta_hat|)
    Data-dependent threshold used in the non-CV Gauss-Lasso comparator; from prior work (Weese et al. 2021), not fitted to the simulation outcomes.
  • flatness/steepness calibration for response surface testbed
    Second-order surfaces were 'adjusted to be relatively flat' and the sixth-order surface 'steeper' (Section 3.1); exact values are not stated and are hand-tuned inputs to the testbed.
assumptions (5)
  • domain assumption The McDaniel-Ankenman testbed, with the Ockuly et al. (2017) effect probabilities and hand-chosen flatness/steepness, generates response surfaces representative of real experiments.
    Used in Section 3.1 to generate the true surfaces that all methods are evaluated against; if unrepresentative, the ranking of methods may not transfer to practice.
  • standard math In the linear model y = X beta + epsilon, errors are independent and normally distributed with variance sigma^2.
    Stated in Section 2 (LB implementation) and used in the simulation protocols in Sections 3.1 and 4; standard linear model assumptions.
  • domain assumption Breiman's little bootstrap estimator PE_s = RSS_s + 2 Bbar_s correctly estimates fixed-X prediction error.
    The LB is adopted from Breiman (1992, 1996) as a comparator, with no re-derivation; its validity is assumed.
  • domain assumption The fixed-predictor (X-fixed) interpretation is the correct framework for evaluating prediction error in designed experiments.
    Section 2 adopts Breiman's argument that CV overestimates prediction error when predictors are fixed; the entire comparison assumes this framing.
  • standard math R packages leaps, glmnet, ranger, and RidgeVar correctly implement best subsets, lasso, random forest, and ridge-based variance estimation.
    All results depend on these implementations; the MEPI missing values suggest at least one package has limitations, so assumption is not fully secure.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The use of cross validation in the analysis of designed experiments." pith.science (2026). https://pith.science/paper/J2TO3UZA

@misc{pith2026250614593,
  author       = {Pith},
  title        = {Pith review of: The use of cross validation in the analysis of designed experiments},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/J2TO3UZA}},
  note         = {Machine review of arXiv:2506.14593}
}
abstract

Cross-validation (CV) is a common method to tune machine learning methods and can be used for model selection in regression as well. Because of the structured nature of small, traditional experimental designs, the literature has warned against using CV in their analysis. The striking increase in the use of machine learning, and thus CV, in the analysis of experimental designs, has led us to empirically study the effectiveness of CV compared to other methods of selecting models in designed experiments, including the little bootstrap. We consider both response surface settings where prediction is of primary interest, as well as screening where factor selection is most important. Overall, we provide evidence that the use of leave-one-out cross-validation (LOOCV) in the analysis of small, structured is often useful. More general $k$-fold CV may also be competitive but its performance is uneven.

Figures

Figures reproduced from arXiv: 2506.14593 by the authors.

Figure 1
Figure 1. Count of Web of Science citations 1990-2023 for the query: TS=(“Design of Experi [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An illustration of k-fold CV where k = 3. Reproduced from Kuhn et al. (2013). of out-of-sample prediction error, then, is RMSPE(k) = 1 k Pk i=1 RMSPEi . When k = n, the result is leave-one-out cross-validation (LOOCV). Cross-validation provides a reasonable estimate of how the model that produced the predictions ˆyf will fare in predicting new observations, and provides an objective method to compare competing predi… view at source ↗
Figure 3
Figure 3. RMSPE for all design types and all analysis methods for a true response surface that is [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: RMSPE for all design types and all analysis methods for a true response surface that is [PITH_FULL_IMAGE:figures/full_fig_p013_4.png]
Figure 5
Figure 5. Figure 5: Y-axis zoom of the RMSPE for all design types and all analysis methods for a true [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 3
Figure 3. Figure 3: There are potentially minor differences, however. For example, the LB method shows [PITH_FULL_IMAGE:figures/full_fig_p014_3.png]
Figure 6
Figure 6. Figure 6: RMSPE for all design types and all analysis methods for a true response surface that is [PITH_FULL_IMAGE:figures/full_fig_p014_6.png]
Figure 7
Figure 7. Figure 7: RMSPE for all design types and all analysis methods for a true response surface that is [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Y-axis zoom of the RMSPE for all design types and all analysis methods for a true [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Power across all n = 20 and m = 7 design types across the combinations of main effects (ME) and two-factor interactions (Int) in the true model.The blue line indicates the average value. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Type 1 error rate across all n = 20 and m = 7 design types across the combinations of main effects (ME) and two-factor interactions (Int) in the true model. The blue line indicates the average value. 4.2 Supersaturated Screening In this Section we explore a more extre…
Figure 11
Figure 11. Figure 11: Power for each of the simulation scenarios (1-4) for each analysis method and each SSD [PITH_FULL_IMAGE:figures/full_fig_p023_11.png]
Figure 12
Figure 12. Figure 12: Type 1 error rate for each of the simulation scenarios (1-4) for each analysis method [PITH_FULL_IMAGE:figures/full_fig_p023_12.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

79 extracted references · 72 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.all := #1 'mid.sentence := #2 '...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in " " * FUNCTION format....

  3. [3]

    Ramezan, C., A

    A. Ramezan, C., A. Warner, T., and E. Maxwell, A. (2019), Evaluation of sampling and cross-validation tuning strategies for regional-scale machine learning classification, Remote Sensing, 11, 185

    work page 2019

  4. [4]

    Abedpour, H., Moghaddas, J., Sori, A., and Alizadeh, R. (2023), Experimental study and machine learning simulation of Pb (II) separation from aqueous solutions via a nanocomposite adsorbent, Journal of the Taiwan Institute of Chemical Engineers, 147, 104923

    work page 2023

  5. [5]

    (1974), A new look at the statistical model identification, IEEE transactions on automatic control, 19, 716--723

    Akaike, H. (1974), A new look at the statistical model identification, IEEE transactions on automatic control, 19, 716--723

    work page 1974

  6. [6]

    (2022 a ), Design choice and machine learning model performances, Quality and Reliability Engineering International, 38, 3357--3378

    Arboretti, R., Ceccato, R., Pegoraro, L., and Salmaso, L. (2022 a ), Design choice and machine learning model performances, Quality and Reliability Engineering International, 38, 3357--3378

    work page 2022

  7. [7]

    --- (2022 b ), Design of Experiments and machine learning for product innovation: A systematic literature review, Quality and Reliability Engineering International, 38, 1131--1156

    work page 2022

  8. [8]

    (2021), Machine learning and design of experiments with an application to product innovation in the chemical industry, Journal of Applied Statistics, 1--26

    Arboretti, R., Ceccato, R., Pegoraro, L., Salmaso, L., Housmekerides, C., Spadoni, L., Pierangelo, E., Quaggia, S., Tveit, C., and Vianello, S. (2021), Machine learning and design of experiments with an application to product innovation in the chemical industry, Journal of Applied Statistics, 1--26

    work page 2021

Show all 79 references
  1. [9]

    --- (2022 c ), Machine learning and design of experiments with an application to product innovation in the chemical industry, Journal of Applied Statistics, 49, 2674--2699

    2022

  2. [10]

    and Celisse, A

    Arlot, S. and Celisse, A. (2010), A survey of cross-validation procedures for model selection, Statistics Surveys, 4, 40--79

    2010

  3. [11]

    (2024), Cross-validation: what does it estimate and how well does it do it? Journal of the American Statistical Association, 119, 1434--1445

    Bates, S., Hastie, T., and Tibshirani, R. (2024), Cross-validation: what does it estimate and how well does it do it? Journal of the American Statistical Association, 119, 1434--1445

    2024

  4. [12]

    (1999), Beating the hold-out: Bounds for k-fold and progressive cross-validation, in Proceedings of the twelfth annual conference on Computational learning theory, pp

    Blum, A., Kalai, A., and Langford, J. (1999), Beating the hold-out: Bounds for k-fold and progressive cross-validation, in Proceedings of the twelfth annual conference on Computational learning theory, pp. 203--208

    1999

  5. [13]

    Booth, K. H. and Cox, D. R. (1962), Some systematic supersaturated designs, Technometrics, 4, 489--495

    1962

  6. [14]

    Box, G. E. and Draper, N. R. (2007), Response Surfaces, Mixtures, and Ridge Analyses, Springer

    2007

  7. [15]

    Box, G. E. P. and Wilson, K. B. (1951), On the Experimental Attainment of Optimum Conditions, Journal of the Royal Statistical Society. Series B, Methodological, 13, 1--45

    1951

  8. [16]

    (1992), The little bootstrap and other methods for dimensionality selection in regression: X-fixed prediction error, Journal of the American Statistical Association, 87, 738--754

    Breiman, L. (1992), The little bootstrap and other methods for dimensionality selection in regression: X-fixed prediction error, Journal of the American Statistical Association, 87, 738--754

    1992

  9. [17]

    --- (1996), Heuristics of instability and stabilization in model selection, The annals of statistics, 24, 2350--2383

    1996

  10. [18]

    Dietterich, T. G. (1998), Approximate statistical tests for comparing supervised classification learning algorithms, Neural computation, 10, 1895--1923

    1998

  11. [19]

    C., Dean, A

    Dragulji \'c , D., Woods, D. C., Dean, A. M., Lewis, S. M., and Vine, A.-J. E. (2014), Screening strategies in the presence of interactions, Technometrics, 56, 1--1

    2014

  12. [20]

    K., and Holena, M

    Dropka, N., Tang, X., Chappa, G. K., and Holena, M. (2022), Smart Design of Cz-Ge Crystal Growth Furnace and Process, Crystals, 12, 1764

    2022

  13. [21]

    and Wainer, J

    Duarte, E. and Wainer, J. (2017), Empirical comparison of cross-validation and internal metrics for tuning SVM hyperparameters, Pattern Recognition Letters, 88, 6--11

    2017

  14. [22]

    and Jones, B

    DuMouchel, W. and Jones, B. (1994), A simple Bayesian modification of D-optimal designs to reduce dependence on an assumed model, Technometrics, 36, 37--47

    1994

  15. [23]

    F., Alateyah, A

    Elkatatny, S., Alsharekh, M. F., Alateyah, A. I., El-Sanabary, S., Nassef, A., Kamel, M., Alawad, M. O., BaQais, A., El-Garaihy, W. H., and Kouta, H. (2023), Optimizing the Powder Metallurgy Parameters to Enhance the Mechanical Properties of Al-4Cu/xAl2O3 Composites Using Mach...

    2023

  16. [24]

    Escribano-Garc \' a, R., Lostado-Lorza, R., Fern \'a ndez-Mart \' nez, R., Villanueva-Rold \'a n, P., and Mac Donald, B. J. (2014), Improvement in manufacturing welded products through multiple response surface methodology and data mining techniques, in International Joint Con...

    2014

  17. [25]

    (2021), Package ‘glmnet’, CRAN R Repositary, 595

    Friedman, J., Hastie, T., Tibshirani, R., Narasimhan, B., Tay, K., Simon, N., and Qian, J. (2021), Package ‘glmnet’, CRAN R Repositary, 595

    2021

  18. [26]

    (1975), The predictive sample reuse method with applications, Journal of the American statistical Association, 70, 320--328

    Geisser, S. (1975), The predictive sample reuse method with applications, Journal of the American statistical Association, 70, 320--328

    1975

  19. [27]

    Ghalandari, M., Ziamolki, A., Mosavi, A., Shamshirband, S., Chau, K.-W., and Bornassi, S. (2019), Aeromechanical optimization of first row compressor test stand blades using a hybrid machine learning model of genetic algorithm, artificial neural networks and design of experime...

    2019

  20. [28]

    C., Luber, E

    Ginige, G., Song, Y., Olsen, B. C., Luber, E. J., Yavuz, C. T., and Buriak, J. M. (2021), Solvent vapor annealing, defect analysis, and optimization of self-assembly of block copolymers using machine learning approaches, ACS Applied Materials & Interfaces, 13, 28639--28649

    2021

  21. [29]

    M., Clemente, R

    Gorriz, J. M., Clemente, R. M., Segovia, F., Ramirez, J., Ortiz, A., and Suckling, J. (2024), Is K-fold cross validation the best model selection method for Machine Learning?

    2024

  22. [30]

    (2013), An introduction to statistical learning, vol

    James, G., Witten, D., Hastie, T., Tibshirani, R., et al. (2013), An introduction to statistical learning, vol. 112, Springer

    2013

  23. [31]

    K., and Nachtsheim, C

    Jones, B., Lin, D. K., and Nachtsheim, C. J. (2008), Bayesian D-optimal supersaturated designs, Journal of Statistical Planning and Inference, 138, 86--92

    2008

  24. [32]

    R., Gul, E., and Ba, S

    Joseph, V. R., Gul, E., and Ba, S. (2015), Maximum projection designs for computer experiments, Biometrika, 102, 371--380

    2015

  25. [33]

    --- (2020), Designing computer experiments with multiple types of factors: The MaxPro approach, Journal of Quality Technology, 52, 343--354

    2020

  26. [34]

    (2022), Multi-object optimization of Navy-blue anodic oxidation via response surface models assisted with statistical and machine learning techniques, Chemosphere, 291, 132818

    Khan, H., Wahab, F., Hussain, S., Khan, S., and Rashid, M. (2022), Multi-object optimization of Navy-blue anodic oxidation via response surface models assisted with statistical and machine learning techniques, Chemosphere, 291, 132818

    2022

  27. [35]

    Kohavi, R. et al. (1995), A study of cross-validation and bootstrap for accuracy estimation and model selection, in Ijcai, Montreal, Canada, vol. 14, pp. 1137--1145

    1995

  28. [36]

    J., Leahy, D

    Krstajic, D., Buturovic, L. J., Leahy, D. E., and Thomas, S. (2014), Cross-validation pitfalls when selecting and assessing regression and classification models, Journal of cheminformatics, 6, 1--15

    2014

  29. [37]

    (2013), Applied predictive modeling, vol

    Kuhn, M., Johnson, K., et al. (2013), Applied predictive modeling, vol. 26, Springer

    2013

  30. [38]

    Lemkus, T., Gotwalt, C., Ramsey, P., and Weese, M. L. (2021), Self-validated ensemble models for design of experiments, Chemometrics and Intelligent Laboratory Systems, 219, 104439

    2021

  31. [39]

    and Nachtsheim, C

    Li, W. and Nachtsheim, C. J. (2000), Model-robust factorial designs, Technometrics, 42, 345--352

    2000

  32. [40]

    (2021), Forward stepwise random forest analysis for experimental designs, Journal of Quality Technology, 53, 488--504

    Lin, C.-Y. (2021), Forward stepwise random forest analysis for experimental designs, Journal of Quality Technology, 53, 488--504

    2021

  33. [41]

    (2020), Estimation of error variance via ridge regression, Biometrika, 107, 481--488

    Liu, X., Zheng, S., and Feng, X. (2020), Estimation of error variance via ridge regression, Biometrika, 107, 481--488

    2020

  34. [42]

    L., Sitter, R

    Loeppky, J. L., Sitter, R. R., and Tang, B. (2007), Nonregular designs with desirable projection properties, Technometrics, 49, 454--467

    2007

  35. [43]

    and Lumley, M

    Lumley, T. and Lumley, M. T. (2013), Package ‘leaps’, Regression subset selection. Thomas Lumley Based on Fortran Code by Alan Miller. Available online: http://CRAN. R-project. org/package= leaps (Accessed on 18 March 2018)

    2013

  36. [44]

    Marley, C. J. and Woods, D. C. (2010), A comparison of design and model selection methods for supersaturated experiments, Computational Statistics & Data Analysis, 54, 3158--3167

    2010

  37. [45]

    B., and Kulkarni, N

    Mathew, S., Karandikar, P. B., and Kulkarni, N. R. (2020), Modeling and Optimization of a Jackfruit Seed-Based Supercapacitor Electrode Using Machine Learning, Chemical Engineering & Technology, 43, 1765--1773

    2020

  38. [46]

    McDaniel, W. R. and Ankenman, B. E. (2000), A response surface test bed, Quality and Reliability Engineering International, 16, 363--372

    2000

  39. [47]

    W., Schoen, E

    Mee, R. W., Schoen, E. D., and Edwards, D. J. (2017), Selecting an orthogonal or nonorthogonal two-level design for screening, Technometrics, 59, 305--318

    2017

  40. [48]

    H., Montgomery, D

    Myers, R. H., Montgomery, D. C., and Anderson-Cook, C. M. (2016), Response surface methodology: process and product optimization using designed experiments, John Wiley & Sons

    2016

  41. [49]

    Nikita, S., Sharma, R., Fahmi, J., and Rathore, A. S. (2023), Process optimization using machine learning enhanced design of experiments (DOE): ranibizumab refolding as a case study, Reaction Chemistry & Engineering, 8, 592--603

    2023

  42. [50]

    A., Weese, M

    Ockuly, R. A., Weese, M. L., Smucker, B. J., Edwards, D. J., and Chang, L. (2017), Response surface experiments: A meta-analysis, Chemometrics and Intelligent Laboratory Systems, 164, 64--75

    2017

  43. [51]

    R., Oliveira, R., and von Stosch, M

    Pinto, J., de Azevedo, C. R., Oliveira, R., and von Stosch, M. (2019), A bootstrap-aggregated hybrid semi-parametric modeling framework for bioprocess development, Bioprocess and biosystems engineering, 42, 1853--1865

    2019

  44. [52]

    (2019), Tunability: Importance of hyperparameters of machine learning algorithms, Journal of Machine Learning Research, 20, 1--32

    Probst, P., Boulesteix, A.-L., and Bischl, B. (2019), Tunability: Importance of hyperparameters of machine learning algorithms, Journal of Machine Learning Research, 20, 1--32

    2019

  45. [53]

    H., Tahmasbi, V., and Qasemi, M

    Rabiee, A. H., Tahmasbi, V., and Qasemi, M. (2023), Experimental evaluation, modeling and sensitivity analysis of temperature and cutting force in bone micro-milling using support vector regression and EFAST methods, Engineering Applications of Artificial Intelligence, 120, 105874

    2023

  46. [54]

    R., Sakuma, K., Doll, R., Biesheuvel, K., Van Borkulo, J., and Boughorbel, F

    Raghavan, S., Jain, A., Chowdhury, P. R., Sakuma, K., Doll, R., Biesheuvel, K., Van Borkulo, J., and Boughorbel, F. (2023), A Methodology to Optimize Laser Dicing Parameters to Maximize Dicing Quality Through Machine Learning, in 2023 IEEE 73rd Electronic Components and Techno...

    2023

  47. [55]

    (2020), Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning,

    Raschka, S. (2020), Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning,

    2020

  48. [56]

    K., Prahasam, V., and Siddharth, S

    Ratnavel, R., Viswanath, S., Subramanian, J., Selvaraj, V. K., Prahasam, V., and Siddharth, S. (2022), Predicting the Optimal Input Parameters for the Desired Print Quality Using Machine Learning, Micromachines, 13, 2231

    2022

  49. [57]

    K., Wood, P., Mockus, L., DeLaurentis, D

    Raz, A. K., Wood, P., Mockus, L., DeLaurentis, D. A., and Llinas, J. (2018), Identifying interactions for information fusion system design using machine learning techniques, in 2018 21st International Conference on Information Fusion (FUSION), IEEE, pp. 226--233

    2018

  50. [58]

    M., Saubamea, B., Andrieux, K., Mignet, N., and Alhareth, K

    Rebollo, R., Oyoun, F., Corvis, Y., El-Hammadi, M. M., Saubamea, B., Andrieux, K., Mignet, N., and Alhareth, K. (2022), Microfluidic manufacturing of liposomes: development and optimization by design of experiment and machine learning, ACS Applied Materials & Interfaces, 14, 3...

    2022

  51. [59]

    (1997), Robust linear model selection by cross-validation, Journal of the American Statistical Association, 92, 1017--1023

    Ronchetti, E., Field, C., and Blanchard, W. (1997), Robust linear model selection by cross-validation, Journal of the American Statistical Association, 92, 1017--1023

    1997

  52. [60]

    (2021), Battery stress factor ranking for accelerated degradation test planning using machine learning, energies, 14, 723

    Saxena, S., Roman, D., Robu, V., Flynn, D., and Pecht, M. (2021), Battery stress factor ranking for accelerated degradation test planning using machine learning, energies, 14, 723

    2021

  53. [61]

    (2019), Hyperparameter tuning and performance assessment of statistical and machine-learning algorithms using spatial data, Ecological Modelling, 406, 109--120

    Schratz, P., Muenchow, J., Iturritxa, E., Richter, J., and Brenning, A. (2019), Hyperparameter tuning and performance assessment of statistical and machine-learning algorithms using spatial data, Ecological Modelling, 406, 109--120

    2019

  54. [62]

    (1978), Estimating the dimension of a model, The annals of statistics, 461--464

    Schwarz, G. (1978), Estimating the dimension of a model, The annals of statistics, 461--464

    1978

  55. [63]

    (1993), Linear model selection by cross-validation, Journal of the American statistical Association, 88, 486--494

    Shao, J. (1993), Linear model selection by cross-validation, Journal of the American statistical Association, 88, 486--494

    1993

  56. [64]

    J., Edwards, D

    Smucker, B. J., Edwards, D. J., and Weese, M. L. (2021), Response surface models: To reduce or not to reduce? Journal of Quality Technology, 53, 197--216

    2021

  57. [65]

    (1974), Cross-validatory choice and assessment of statistical predictions, Journal of the royal statistical society: Series B (Methodological), 36, 111--133

    Stone, M. (1974), Cross-validatory choice and assessment of statistical predictions, Journal of the royal statistical society: Series B (Methodological), 36, 111--133

    1974

  58. [66]

    and Simon, R

    Varma, S. and Simon, R. (2006), Bias in error estimation when using cross-validation for model selection, BMC bioinformatics, 7, 1--8

    2006

  59. [67]

    R., Engemann, D

    Varoquaux, G., Raamana, P. R., Engemann, D. A., Hoyos-Idrobo, A., Schwartz, Y., and Thirion, B. (2017), Assessing and tuning brain decoders: cross-validation, caveats, and guidelines, NeuroImage, 145, 166--179

    2017

  60. [68]

    L., Edwards, D

    Weese, M. L., Edwards, D. J., and Smucker, B. J. (2017), A criterion for constructing powerful supersaturated designs when effect directions are known, Journal of Quality Technology, 49, 265--277

    2017

  61. [69]

    L., Stallrich, J

    Weese, M. L., Stallrich, J. W., Smucker, B. J., and Edwards, D. J. (2021), Strategies for supersaturated screening: Group orthogonal and constrained var (s) designs, Technometrics, 63, 443--455

    2021

  62. [70]

    (2019), Data mining methodology for engineering applications (DMME)—A holistic extension to the CRISP-DM model, Applied Sciences, 9, 2407

    Wiemer, H., Drowatzky, L., and Ihlenfeldt, S. (2019), Data mining methodology for engineering applications (DMME)—A holistic extension to the CRISP-DM model, Applied Sciences, 9, 2407

    2019

  63. [71]

    Wright, M. N. and Ziegler, A. (2015), ranger: A fast implementation of random forests for high dimensional data in C++ and R, arXiv preprint arXiv:1508.04409

    2015 arXiv

  64. [72]

    and Liang, Y.-Z

    Xu, Q.-S. and Liang, Y.-Z. (2001), Monte Carlo cross validation, Chemometrics and Intelligent Laboratory Systems, 56, 1--11

    2001

  65. [73]

    (2007), Consistency of cross validation for comparing regression procedures, Annals of Statistics, 35, 2450--2473

    Yang, Y. (2007), Consistency of cross validation for comparing regression procedures, Annals of Statistics, 35, 2450--2473

    2007

  66. [74]

    A., Aandahl, Z., Richards, S

    Yates, L. A., Aandahl, Z., Richards, S. A., and Brook, B. W. (2023), Cross validation for model selection: a review with examples from ecology, Ecological Monographs, 93, e1557

    2023

  67. [75]

    R., and Lin, Y

    Yuan, M., Joseph, V. R., and Lin, Y. (2007), An efficient variable selection approach for analyzing designed experiments, Technometrics, 49, 430--439

    2007

  68. [76]

    (2022), Predictive Analysis of Fluid-Hammer Effect on LNG Regasification System Pipeline Network, in 2022 Annual Reliability and Maintainability Symposium (RAMS), IEEE, pp

    Zalkikar, A., Nepal, B., Husin, H., Yadav, O., and Banerjee, A. (2022), Predictive Analysis of Fluid-Hammer Effect on LNG Regasification System Pipeline Network, in 2022 Annual Reliability and Maintainability Symposium (RAMS), IEEE, pp. 1--6

    2022

  69. [77]

    (1993), Model selection via multifold cross validation, The annals of statistics, 299--313

    Zhang, P. (1993), Model selection via multifold cross validation, The annals of statistics, 299--313

    1993

  70. [78]

    and Yang, Y

    Zhang, Y. and Yang, Y. (2015), Cross-validation for selecting a model selection procedure, Journal of Econometrics, 187, 95--112

    2015

  71. [79]

    Zhong, E., Fan, W., Yang, Q., Verscheure, O., and Ren, J. (2010), Cross validation framework to choose amongst models and datasets for transfer learning, in Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2010, Barcelona, Spain, September ...

    2010

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.