Pith. sign in

REVIEW 2 major objections 7 minor 60 references

Sample size calculations for the experimental comparison of multiple algorithms on multiple problem instances

T0 review · 2 major / 7 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper shows how to choose the number of instances and the number of runs per instance for comparing any number of algorithms, with controlled error rates and prespecified power to detect a minimally relevant difference.

desk verdict A genuinely useful extension of the two-algorithm sample-size machinery to arbitrary A, with real software; the main soft spot is the unvalidated mean-power approximation in Algorithm 2. read the letter →

arxiv 1908.01720 v1 pith:XMNKNIQZ submitted 2019-08-05 stat.ME cs.LGcs.NE

Felipe Campelo , Elizabeth F. Wanner This is my paper · ORCID
classification stat.MEcs.LGcs.NE MSC 62F0362K10
keywords samplesizecalculationexperimentalcomparisonofalgorithmsHolmstep-downprocedurefamilywiseerrorratestatisticalpowerminimallyrelevanteffectpaireddifferencesbenchmarkinstances
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Experimental comparisons of stochastic algorithms need two sample sizes: the number of problem instances and the number of repeated runs per algorithm per instance. This paper claims that both can be planned in advance, for any number of algorithms, from two user-chosen targets: a minimally relevant standardised difference $d^*$ and a desired statistical power for detecting it, together with a controlled familywise error rate. The instance count comes from a noncentral $t$ power calculation with Holm-corrected significance levels, so an experiment can be sized for best-, worst-, mean-, or median-case power across the planned pairwise tests. The run count comes from a greedy per-instance allocation that drives every pairwise standard error below a chosen threshold $se^*$, using optimal sample-size ratios derived for simple and percent differences. A sympathetic reader would care because this replaces the common practices of using as many instances as budget allows and an arbitrary 30 or 50 runs with quantities that have a stated statistical meaning.

What carries the argument

Minimally relevant effect size (MRES), $d^*=|\delta^*|/\sigma$, the smallest standardised mean paired difference judged practically relevant, anchors both the power calculation and the interpretation of detected effects. Holm's step-down procedure assigns significance level $\alpha'_r=\alpha_f/(K-r+1)$ to the $r$-th ordered hypothesis, controlling the familywise error rate while being less conservative than Bonferroni; the paper uses these levels in the instance-count calculation. The instance-count engine is Equation (38), a noncentral $t$ power formula with noncentrality $|d^*|\sqrt{N}$, together with Algorithm 2, which finds $N$ by averaging per-comparison powers. The run-count engine is Algorithm 1, a greedy allocation that reduces the worst pairwise standard error on each instance subject to $se^*$, using the optimal ratios $n_i/n_j=\sigma_i/\sigma_j$ and $n_1/n_j=(\sigma_1/\mu_1)/(\sigma_j/\mu_j)$ to decide which algorithm receives the next run.

What would settle it

Run a Monte Carlo simulation of the designed experiment: generate $A$ algorithms on $N$ instances with all pairwise standardised differences exactly $d^*$ (and also with only some pairs differing and the rest null), apply Holm's step-down procedure, and compare the empirical mean, best, and worst rejection power across many repetitions to the values predicted by Equation (38) and Algorithm 2. A systematic shortfall would show that the recommended instance count is miscalibrated in mean-power mode.

Watch

Extended reading notes

Core claim

The paper establishes that the comparison of $A$ algorithms over a problem class can be planned as a set of paired comparisons on instance-blocked differences, and that the required number of instances $N$ is the smallest integer satisfying the noncentral $t$ condition $t_{N-1}^{1-\alpha/2}\le t_{N-1}^{\beta^*;|ncp^*|}$ with $|ncp^*|=|d^*|\sqrt{N}$, where the significance level is corrected by Holm's step-down procedure. For the least-favourable comparison this correction is $\alpha_f/K$; for mean-power design, the paper's Algorithm 2 iterates $N$ until the average of per-comparison powers $\mathrm{Power}(\alpha/i, N, d^*, H_1)$ reaches $\pi^*$. Within each instance, the number of runs is the solution of a constrained minimization: the smallest total runs such that every pairwise standard error $se_{(ij)|\ell}^2\le(se^*)^2$. The paper derives optimal allocation ratios $n_i/n_j=\sigma_i/\sigma_j$ for simple differences and for all-vs-all percent differences, and $n_1/n_j=(\sigma_1/\mu_1)/(\sigma_j/\mu_j)$ for all-vs-one percent differences, and embeds them in a greedy algorithm that allocates the next run to the algorithm most responsible for the current worst standard error. The case study with 21 variants of a simulated-annealing scheduler demonstrates that the instance count (57 in that experiment) and per-instance run counts determined this way deliver the planned power and controlled standard errors.

Load-bearing premise

The load-bearing premise is that the planned mean power of a Holm-corrected experiment is well approximated by the average of per-comparison powers computed at fixed thresholds $\alpha/i$ ($i=1,\dots,K$) with every comparison at effect size $d^*$; actual Holm thresholds are assigned to ordered, dependent $p$-values, so the realised mean power can differ from the planned $\pi^*$.

Editorial extensions

If this is right

  • An experimenter who fixes $d^*$, $\pi^*$, and $\alpha_f$ can compute the minimum number of instances directly, instead of using the entire benchmark set or an arbitrary default.
  • Because Holm's procedure is less conservative than Bonferroni, designing for mean power yields a smaller $N$ than the Bonferroni-equivalent design, while the worst-case power stays close to $\pi^*$ in the paper's examples.
  • For a fixed available set of instances, the same formulas give a power-versus-effect-size curve, so researchers can report the sensitivity their chosen benchmark set actually provides.
  • The per-instance run allocation guarantees, up to the computational budget, that all pairwise estimates meet the chosen accuracy $se^*$, which keeps estimation noise from inflating the residual variance used in the paired tests.
  • All-vs-one and one-sided designs are systematically cheaper than all-vs-all and two-sided designs, since they create fewer and less heavily corrected hypotheses.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension, not explored in the paper, would replace the single $d^*$ with a distribution of effect sizes across pairs or instance subclasses; the same noncentral-$t$ machinery could then be integrated over that distribution to size experiments under heterogeneity.
  • The mean-power shortcut in Algorithm 2 ignores the dependence among Holm-ordered $p$-values and the possibility that some null hypotheses are true; a Monte Carlo calibration across those scenarios would state how much the realised power can deviate from $\pi^*$.
  • The optimal-run ratios are derived per pair and ignore correlations among comparisons that share the reference algorithm or the grand mean; a joint allocation rule could in principle beat the greedy rule in total runs when such correlations are strong.
  • Because the method can reserve unused benchmark instances, it offers a practical guard against tuning a heuristic to a fixed benchmark set; this suggests a testable protocol where held-out instances, not just held-out runs, are used to validate algorithm-design choices.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 7 minor

Summary. The paper generalizes the authors' earlier sample-size methodology from two algorithms to an arbitrary number. The number of repeated runs per instance is framed as a constrained minimization problem on the standard errors of estimated paired differences; the authors derive optimal sample-size ratios for simple differences and for percent differences under all-vs-one and all-vs-all comparisons, and give a greedy sampling algorithm (Algorithm 1). The number of instances is derived from a paired-t-test power calculation (Eq. 38), with significance levels corrected by either Bonferroni or Holm. For Holm's procedure they propose worst-case, mean-power, and median-power planning modes (Algorithm 2), and they illustrate the whole protocol on a 22-algorithm comparison of simulated-annealing variants on a scheduling benchmark. The main advertised advances are the multi-algorithm generalization, the Holm-based instance-number formulas, and the per-instance run-allocation ratios.

Significance. If the proposed formulas are valid, the paper supplies a practical and much-needed alternative to arbitrary choices of instance counts and repetition numbers in metaheuristic comparisons. The worst-case (Bonferroni) instance-size formulas are standard and correctly applied, and the derivations of the per-instance optimal ratios are carefully executed for the three difference types. The accompanying R package and the realistic case study are valuable. The principal weakness is that the recommended mean-power mode for Holm's procedure rests on an unvalidated approximation of the achieved rejection rate, so the central claim about this design mode is not yet established.

major comments (2)
  1. [Section 4, Algorithm 2; Eq. (39)] The mean-power planning in Algorithm 2 averages Power(α/i, N, d*, H1) over i = 1,...,K, treating the Holm thresholds as fixed quantities assigned to comparisons. In the actual Holm procedure, the threshold available to a comparison depends on the rank of its p-value among all K tests, and that rank is random and correlated with the p-value itself; for example, with K = 2 independent tests under the global null, the average of the fixed thresholds is 3α/4, whereas the expected per-test rejection rate under Holm is α/2 + 3α^2/8. The paper provides no proof or simulation that the achieved mean rejection probability equals the averaged quantity when some alternatives are true or effect sizes vary, and Figure 2 is generated from the same approximation, so it cannot serve as evidence. The case study in Section 5.4 reports realized p-values and confidence intervals but not achieved power, so it also does not validate the approximation. Because Section 4 recommends mean power as the preferred design target, this gap is load-bearing; please provide either an analytic justification under stated conditions or a simulation study covering equal and unequal effect sizes, and revise the corresponding claims if the approximation is optimistic.
  2. [Section 3, Algorithm 1 and text following Eq. (13)] The text states that if Algorithm 1 is interrupted by the computational budget, 'the resulting standard errors of estimation will be the smallest ones achievable.' This global optimality claim is not established: the greedy rule of augmenting the algorithm that contributes most to the current maximal standard error does not in general imply that the realized standard-error vector is minimal for the total number of runs used, because early choices constrain later possibilities. The procedure is a reasonable heuristic for reducing the worst-case standard error, but the claim should either be proved or weakened to say that the procedure is designed to reduce the worst-case standard error. This secondary issue does not affect the derivation of the pairwise sample-size ratios, but it is directly tied to the optimality language used to motivate Algorithm 1.
minor comments (7)
  1. [Section 4, paragraph after Eq. (39)] Holm's thresholds become less strict as r increases, since α'_r = α/(K - r + 1) grows with r; the phrase 'increasingly stricter significance level' is therefore inverted and should be corrected.
  2. [Section 5.2] The sentence 'an all vs. one design was be the most appropriate' contains a grammatical error and should read 'was the most appropriate.'
  3. [Section 3.1, Eq. (13)] The lower-bound constraints n_k|ℓ ≥ 2 are mentioned only in the prose; include them in the displayed formulation or explicitly state that they are implicit.
  4. [Table 1] The inserted row 'Stop rejecting H0' is informal; use a conventional separation, such as a rule or boldfaced boundary, and state clearly that all comparisons below the line are not rejected.
  5. [Section 5.3 vs. reference [14]] The text says CAISEr version 1.0.5 was used, while reference [14] lists version 1.0.13; please make these consistent.
  6. [Section 4, Algorithm 2] The pseudocode does not state explicitly that all K comparisons are assumed to be under the alternative with effect size d*; this assumption should be written into the algorithm's contract.
  7. [Figure 3] The figure is extremely dense and the instance labels are difficult to read; consider faceting by instance size or using a rotated, smaller set of labels.

Circularity Check

1 steps flagged · score 6.0 of 10

Algorithm 2's 'mean power' design is self-definitional: it sizes N until an average of fixed-threshold per-comparison powers reaches π*, so the mean-power outcome is true by construction rather than a predicted property of Holm's step-down procedure.

  1. self definitional [Section 4, Algorithm 2 (pseudocode preceding Fig. 2)]
    "5: for i∈{ 1,...,K } do 6: pi← Power (α/i,N,d∗,H 1) ⊿ See [15] for details. 7: end for 8: ¯p =∑Ki=1pi/K ⊿ Calculate mean power"

    Holm's thresholds (Eq. 39) are α_f/(K−r+1) applied to ordered p-values, so the significance level available to a comparison depends on its p-value rank and on the realized ordering of all p-values. The pseudocode instead evaluates Power(α/i, N, d*, H1) at K fixed levels and defines the experiment's 'mean power' as their arithmetic mean; the while-loop stops when this defined average reaches π*.

full rationale

The derivation chain of Section 3 is self-contained: the optimal run-allocation ratios follow from KKT conditions on minimizing total runs subject to se^2 ≤ (se*)^2, and Algorithm 1 is a greedy sampling procedure whose behavior is not asserted as a statistical prediction. The instance-number formula (38) is the standard paired-t power calculation; its use for Bonferroni and worst-case Holm designs is an ordinary alpha-adjustment and is not circular. The self-citations to Campelo and Takahashi [15] supply derivations and nonparametric extensions, but the invoked formulas are standard, parameter-free, and externally checkable, so they do not make the paper circular. The one genuinely circular element is Algorithm 2. Holm's procedure assigns thresholds by p-value rank (Eq. 39), so actual mean power is a property of the joint distribution of the ordered p-values. Algorithm 2 instead defines the experiment's mean power as the arithmetic average of Power(α/i, N, d*, H1) over fixed i, and stops when that defined average reaches π*. The returned N is therefore the solution of an equation involving the algorithm's own definition; the paper supplies no proof or simulation that this quantity equals the mean rejection probability of the Holm procedure, and the case study does not verify it. Because mean-power planning is one of the four advertised design modes and is explicitly recommended in Section 4, this is partial circularity rather than a harmless approximation.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method introduces no hidden fitted constants; its inputs are user-specified design parameters (d*, alpha, power, se*) and unknown population quantities (sigma_d, per-instance sigma_i) that must be supplied. The main hidden assumptions are the i.i.d. normal paired-difference model, the additive block model, positivity of performance for percent differences, and two approximations in the derivations (uncoupled ratios in the all-vs-all percent case, fixed-threshold mean power under Holm).

free parameters (3)
  • MRES d* = 0.5 in case study
    User-specified minimum standardized effect; the method does not provide a way to choose it, but the instance count N depends strongly on it.
  • Per-instance standard error threshold se* = 0.05 in case study
    User-specified accuracy bound for estimated pairwise differences; drives total runs per instance in Algorithm 1.
  • Standard deviation of paired differences across instances (sigma_d)
    Needed to turn d* into sample size; the paper gives no procedure for estimating it from pilot data and it is not estimated in the case study.
assumptions (5)
  • domain assumption Paired differences across instances are independent and approximately normal, supporting the paired t-test and noncentral t power formula (Eq. 38).
    Underlies the number-of-instances calculation; not tested in the paper.
  • domain assumption Performance follows the additive block model Y_k|l = mu_k + theta_l + epsilon_kl (Eq. 1), so instance effects can be blocked out and each algorithm has a constant mean effect over the problem class.
    Justifies summarizing per-instance runs and using instances as blocks; fails if algorithms interact with instance characteristics.
  • domain assumption All performance observations are strictly positive when percent differences are used (Section 3.2.2).
    Required for the Fieller-based standard error of ratios; not valid for performance measures that can be negative.
  • ad hoc to paper In the all-vs-all percent-difference standard error, the term C_ij^2 is treated as constant when differentiating with respect to n_i and n_j.
    This approximation produces the clean optimal ratio sigma_i/sigma_j but ignores coupling through other algorithms' sample sizes.
  • ad hoc to paper Algorithm 2 treats Holm thresholds as fixed at alpha/i and assumes every comparison has effect size d* when computing mean power.
    This makes the planned mean power calculable, but the actual mean power of Holm's procedure depends on the random ordering and dependence of p-values.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Sample size calculations for the experimental comparison of multiple algorithms on multiple problem instances." pith.science (2026). https://pith.science/paper/XMNKNIQZ

@misc{pith2026190801720,
  author       = {Pith},
  title        = {Pith review of: Sample size calculations for the experimental comparison of multiple algorithms on multiple problem instances},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XMNKNIQZ}},
  note         = {Machine review of arXiv:1908.01720}
}
read the original abstract

This work presents a statistically principled method for estimating the required number of instances in the experimental comparison of multiple algorithms on a given problem class of interest. This approach generalises earlier results by allowing researchers to design experiments based on the desired best, worst, mean or median-case statistical power to detect differences between algorithms larger than a certain threshold. Holm's step-down procedure is used to maintain the overall significance level controlled at desired levels, without resulting in overly conservative experiments. This paper also presents an approach for sampling each algorithm on each instance, based on optimal sample size ratios that minimise the total required number of runs subject to a desired accuracy in the estimation of paired differences. A case study investigating the effect of 21 variants of a custom-tailored Simulated Annealing for a class of scheduling problems is used to illustrate the application of the proposed methods for sample size calculations in the experimental comparison of algorithms.

Figures

Figures reproduced from arXiv: 1908.01720 by the authors.

Figure 1
Figure 1. Example of required sample sizes for Bonferroni-corrected tests in a common case with [PITH_FULL_IMAGE:figures/full_fig_p016_1.png] view at source ↗
Figure 2
Figure 2. Example of the resulting power of each comparison when designing experiments focused [PITH_FULL_IMAGE:figures/full_fig_p017_2.png] view at source ↗
Figure 4
Figure 4. Distribution of nk|` , i.e., of the number of runs for each algorithm variant on each instance. The majority of runs were allocated either to the full algorithm - which was involved in all comparisons - or to variants that had the Task Move (TSK) suppressed, suggesting a strong effect of this perturbation function [PITH_FULL_IMAGE:figures/full_fig_p022_4.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Point estimates and confidence intervals (95% joint confidence level) for the mean of [PITH_FULL_IMAGE:figures/full_fig_p024_5.png]
Figure 6
Figure 6. Figure 6: Power curve for the follow-up experiment, obtained by setting [PITH_FULL_IMAGE:figures/full_fig_p025_6.png]
Figure 7
Figure 7. Figure 7: Joint 95% confidence intervals and sample sizes in the follow-up experiment. Notice that no comparison yielded statistically significant results, and that the narrow confidence intervals suggest that even if the neighbourhoods tested have some effect on the algorithm p…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 60 canonical work pages

  1. [15]

    Journal of Heuristics25(2), 305–338 (2019)

    Campelo, F., Takahashi, F.: Sample size estimation for power and accuracy in the experimental comparison of algorithms. Journal of Heuristics25(2), 305–338 (2019)

    work page 2019

  2. [1]

    Applied Soft Computing12(10), 3176–3192 (2012)

    del Amo, I.G., Pelta, D.A., González, J.R., Masegosa, A.D.: An algorithm comparison for dynamic optimization problems. Applied Soft Computing12(10), 3176–3192 (2012)

    work page 2012

  3. [2]

    Journal of Heuristics1(1), 9–32 (1995)

    Barr, R.S., Golden, B.L., Kelly, J.P., Resende, M.G.C., Stewart, W.R.: Designing and report- ing on computational experiments with heuristic methods. Journal of Heuristics1(1), 9–32 (1995)

    work page 1995

  4. [3]

    Springer (2013)

    Bartroff, J., Lai, T., Shih, M.C.: Sequential Experimentation in Clinical Trials: Design and Analysis. Springer (2013)

    work page 2013

  5. [4]

    Bartz-Beielstein, T.: New Experimentalism Applied to Evolutionary Computation. Ph.D. thesis, Universität Dortmund, Germany (2005)

    work page 2005

  6. [5]

    Springer (2006) 27

    Bartz-Beielstein, T.: Experimental Research in Evolutionary Computation. Springer (2006) 27

    work page 2006

  7. [6]

    Bartz-Beielstein, T.: How to create generalizable results. In: J. Kacprzyk, W. Pedrycz (eds.) Handbook of Computational Intelligence. Springer (2015)

    work page 2015

  8. [7]

    Springer (2010)

    Bartz-Beielstein, T., Chiarandini, M., Paquete, L., Preuss, M.: Experimental Methods for the Analysis of Optimization Algorithms. Springer (2010)

    work page 2010

Show all 60 references
  1. [8]

    In: 30th International conference on machine learning, pp

    Benavoli, A., Corani, G., Mangili, F., Zaffalon, M., Ruggeri, F.: A bayesian wilcoxon signed- rank test based on the dirichlet process. In: 30th International conference on machine learning, pp. 1026–1034 (2014)

    2014

  2. [9]

    Birattari, M.: On the estimation of the expected performance of a metaheuristic on a class of instances: how many instances, how many runs? Tech. Rep. IRIDIA/2004-001, Université Libre de Bruxelles, Belgium (2004)

    2004

  3. [10]

    Springer Berlin Heidelberg (2009)

    Birattari, M.: Tuning Metaheuristics – A Machine Learning Perspective. Springer Berlin Heidelberg (2009)

    2009

  4. [11]

    Birattari, M., Dorigo, M.: How to assess and report the performance of a stochastic algorithm on a benchmark problem: Mean or best result on a number of runs? Optimization Letters1, 309–311 (2007)

    2007

  5. [12]

    Behavior Research Methods38(1), 65–76 (2006)

    Botella, J., Ximénez, C., Revuelta, J., Suero, M.: Optimization of sample size in controlled experiments: The CLAST rule. Behavior Research Methods38(1), 65–76 (2006)

    2006

  6. [13]

    In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO ’19, pp

    Calvo, B., Shir, O.M., Ceberio, J., Doerr, C., Wang, H., Bäck, T., Lozano, J.A.: Bayesian performance analysis for black-box optimization benchmarking. In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO ’19, pp. 1789–1797. ACM (2019)

    2019

  7. [14]

    URL https://CRAN.R-project.org/package=CAISEr

    Campelo, F.: CAISEr: Comparison of Algorithms with Iterative Sample Size Estimation (2019). URL https://CRAN.R-project.org/package=CAISEr. Package version 1.0.13

    2019

  8. [16]

    IEEE Transactions on Evolutionary Computation 15(6), 848–870 (2011)

    Carrano, E.G., Wanner, E.F., Takahashi, R.H.C.: A multicriteria statistical based compari- son methodology for evaluating evolutionary algorithms. IEEE Transactions on Evolutionary Computation 15(6), 848–870 (2011)

    2011

  9. [17]

    In: Experimental Methods for the Analysis of Optimization Algorithms, pp

    Chimani, M., Klein, K.: Algorithm engineering: Concepts and practice. In: Experimental Methods for the Analysis of Optimization Algorithms, pp. 131–158. Springer Berlin Heidelberg (2010)

    2010

  10. [18]

    INFORMS Journal on Computing12(1), 24–44 (2000)

    Coffin, M., Saltzman, M.J.: Statistical analysis of computational tests of algorithms and heuris- tics. INFORMS Journal on Computing12(1), 24–44 (2000)

    2000

  11. [19]

    In: Proceedings of the 2004 IEEE Congress on Evolutionary Computation

    Czarn, A., MacNish, C., Vijayan, K., Turlach, B.: Statistical exploratory analysis of genetic algorithms: the importance of interaction. In: Proceedings of the 2004 IEEE Congress on Evolutionary Computation. Institute of Electrical & Electronics Engineers (IEEE) (2004)

    2004

  12. [20]

    Journal of Machine Learning Research7, 1–30 (2006) 28

    Demšar, J.: Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research7, 1–30 (2006) 28

    2006

  13. [21]

    Information Sciences289, 41–58 (2014)

    Derrac, J., García, S., Hui, S., Suganthan, P.N., Herrera, F.: Analyzing convergence perfor- mance of evolutionary algorithms: A statistical approach. Information Sciences289, 41–58 (2014)

    2014

  14. [22]

    Swarm and Evolutionary Computation1(1), 3–18 (2011)

    Derrac, J., García, S., Molina, D., Herrera, F.: A practical tutorial on the use of nonpara- metric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms. Swarm and Evolutionary Computation1(1), 3–18 (2011)

    2011

  15. [23]

    Journal of the American Statistical Associ- ation 56(293), 52–64 (1961)

    Dunn, O.J.: Multiple comparisons among means. Journal of the American Statistical Associ- ation 56(293), 52–64 (1961)

    1961

  16. [24]

    In: Proceedings of the 2002 IEEECongress on Evolutionary Computation

    Eiben, A., Jelasity, M.: A critical note on experimental research methodology in EC. In: Proceedings of the 2002 IEEECongress on Evolutionary Computation. Institute of Electrical & Electronics Engineers (IEEE) (2002)

    2002

  17. [25]

    Cambridge University Press (2010)

    Ellis, P.D.: The Essential Guide to Effect Sizes, 1st edn. Cambridge University Press (2010)

    2010

  18. [26]

    Journal of the Royal Statistical Society

    Fieller, E.C.: Some problems in interval estimation. Journal of the Royal Statistical Society. Series B (Methodological)16(2), 175–185 (1954)

    1954

  19. [27]

    org/pdf/0710.2024v1.pdf

    Franz, V.: Ratios: A short guide to confidence limits and proper use (2007).https://arxiv. org/pdf/0710.2024v1.pdf

    2007 arXiv

  20. [28]

    Soft Computing 13(10), 959–977 (2009)

    García, S., Fernández, A., Luengo, J., Herrera, F.: A study of statistical techniques and performance measures for genetics-based machine learning: accuracy and interpretability. Soft Computing 13(10), 959–977 (2009)

    2009

  21. [29]

    Information Sciences180(10), 2044–2064 (2010)

    García, S., Fernández, A., Luengo, J., Herrera, F.: Advanced nonparametric tests for multi- ple comparisons in the design of experiments in computational intelligence and data mining: Experimental analysis of power. Information Sciences180(10), 2044–2064 (2010)

    2010

  22. [30]

    Journal of Heuristics15(6), 617–644 (2008)

    García, S., Molina, D., Lozano, M., Herrera, F.: A study on the use of non-parametric tests for analyzing the evolutionary algorithms’ behaviour: a case study on the CEC’2005 Special session on real parameter optimization. Journal of Heuristics15(6), 617–644 (2008)

    2008

  23. [31]

    Cam- bridge university press (2006)

    Gelman, A., Hill, J.: Data analysis using regression and multilevel/hierarchical models. Cam- bridge university press (2006)

    2006

  24. [32]

    Annals of Discrete Mathematics 5, 287–326 (1979)

    Graham, R.L., Lawler, E.L., Lenstra, J.K., Rinnooy Kan, A.H.G.: Optimization and approxi- mation in deterministic sequencing and scheduling: a survey. Annals of Discrete Mathematics 5, 287–326 (1979)

    1979

  25. [33]

    URL https://arxiv.org/abs/1603.08776

    Hansen, N., Tˇ usar, T., Mersmann, O., Auger, A., Brockoff, D.: COCO: The experimental procedure (2016). URL https://arxiv.org/abs/1603.08776

    2016 arXiv

  26. [34]

    Scandinavian journal of statistics 6(2), 65–70 (1979)

    Holm, S.: A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics 6(2), 65–70 (1979)

    1979

  27. [35]

    Operations Research42(2), 201–212 (1994)

    Hooker, J.N.: Needed: An empirical science of algorithms. Operations Research42(2), 201–212 (1994)

    1994

  28. [36]

    Journal of Heuristics1(1), 33–42 (1996) 29

    Hooker, J.N.: Testing heuristics: We have it all wrong. Journal of Heuristics1(1), 33–42 (1996) 29

    1996

  29. [37]

    Ecological Monographs 54(2), 187–211 (1984)

    Hurlbert, S.H.: Pseudoreplication and the design of ecological field experiments. Ecological Monographs 54(2), 187–211 (1984)

    1984

  30. [38]

    John Wiley and Sons Ltd (1991)

    Jain, R.K.: The Art of Computer Systems Performance Analysis. John Wiley and Sons Ltd (1991)

    1991

  31. [39]

    Johnson, D.: A theoretician’s guide to the experimental analysis of algorithms. In: M. Gold- wasser, D. Johnson, C. McGeoch (eds.) Data Structures, Near Neighbor Searches, and Method- ology: Fifth and Sixth DIMACS Implementation Challenges, DIMACS Series in Discrete Mathematic...

    2002

  32. [40]

    Applied Soft Computing37, 217–226 (2015)

    Krohling, R.A., Lourenzutti, R., Campos, M.: Ranking and comparing evolutionary algorithms with hellinger-TOPSIS. Applied Soft Computing37, 217–226 (2015)

    2015

  33. [41]

    Aca- demic Press, Inc

    Kruschke, J.K.: Doing Bayesian Data Analysis: A Tutorial with R and BUGS, 1st edn. Aca- demic Press, Inc. (2010)

    2010

  34. [42]

    In: Handbooks in Operations Research and Management Science, vol

    Lawler, E.L., Lenstra, J.K., Rinnooy Kan, A.H., Shmoys, D.B.: Sequencing and scheduling: Algorithms and complexity. In: Handbooks in Operations Research and Management Science, vol. 4, chap. 9, pp. 445–522. Elsevier (1993)

    1993

  35. [43]

    Lazic, S.E.: The problem of pseudoreplication in neuroscientific studies: is it affecting your analysis? BMC Neuroscience11(5), 397–407 (2010)

    2010

  36. [44]

    The American Statistician 55(3), 187–193 (2001)

    Lenth, R.V.: Some practical guidelines for effective sample size determination. The American Statistician 55(3), 187–193 (2001)

    2001

  37. [45]

    In preparation

    Maravilha, A.L., Pereira, L.M., Campelo, F.: Statistical characterization of neighborhood structures for the unrelated parallel machine problem with sequence-dependent setup times. In preparation

  38. [46]

    Matthews Malnar & Bailey Inc

    Mathews, P.: Sample Size Calculations: Practical Methods for Engineers and Scientists, 1st edn. Matthews Malnar & Bailey Inc. (2010)

    2010

  39. [47]

    INFORMS Journal on Computing8(1), 1–15 (1996)

    McGeoch, C.C.: Feature article—toward an experimental method for algorithm simulation. INFORMS Journal on Computing8(1), 1–15 (1996)

    1996

  40. [48]

    Fisheries Research 70, 397–407 (2004)

    Millar, R., Anderson, M.: Remedies for pseudoreplication. Fisheries Research 70, 397–407 (2004)

    2004

  41. [49]

    John Wiley & Sons (2013)

    Montgomery, D.C.: Design and Analysis of Experiments, 8th edn. John Wiley & Sons (2013)

    2013

  42. [50]

    Wiley (2013)

    Montgomery, D.C., Runger, G.C.: Applied Statistics and Probability for Engineers, 6th edn. Wiley (2013)

    2013

  43. [51]

    Master’s thesis, Universi- dade Federal de Minas Gerais, Belo Horizonte, Brazil (2019)

    Pereira, L.M.: Análise de Estruturas de Vizinhança para o Problema de Sequenciamento de Máquinas Paralelas Não Relacionadas com Tempos de Preparação . Master’s thesis, Universi- dade Federal de Minas Gerais, Belo Horizonte, Brazil (2019). URLhttps://ppgee.ufmg.br/ defesas/1615...

    2019

  44. [52]

    R Foundation for Statistical Computing, Vienna, Austria (2017)

    R Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria (2017). URLhttps://www.R-project.org/

    2017

  45. [53]

    Ridge, E.: Design of Experiments for the Tuning of Optimisation Algorithms. Ph.D. thesis, The University of York, UK (2007)

    2007

  46. [54]

    International Transactions in Operational Research00 (2016)

    Santos, H.G., Toffolo, T.A., Silva, C.L., Berghe, G.V.: Analysis of stochastic local search methods for the unrelated parallel machine scheduling problem. International Transactions in Operational Research00 (2016)

    2016

  47. [55]

    Annual review of psychology46(1), 561–584 (1995)

    Shaffer, J.P.: Multiple hypothesis testing. Annual review of psychology46(1), 561–584 (1995)

    1995

  48. [56]

    Taylor & Francis (2011)

    Sheskin, D.J.: Handbook of Parametric and Nonparametric Statistical Procedures. Taylor & Francis (2011)

    2011

  49. [57]

    In: Handbook of Heuristics, pp

    Sörensen, K., Sevaux, M., Glover, F.: A history of metaheuristics. In: Handbook of Heuristics, pp. 1–18. Springer International Publishing (2018)

    2018

  50. [58]

    European Journal of Operational Research211(3), 612–622 (2011)

    Vallada, E., Ruiz, R.: A genetic algorithm for the unrelated parallel machine scheduling prob- lem with sequence dependent setup times. European Journal of Operational Research211(3), 612–622 (2011)

    2011

  51. [59]

    Parallel Problem Solving From Nature - PPSN VIII3242, 172–181 (2004)

    Yuan, B., Gallagher, M.: Statistical racing techniques for improved empirical evaluation of evolutionary algorithms. Parallel Problem Solving From Nature - PPSN VIII3242, 172–181 (2004)

    2004

  52. [60]

    In: Proceedings of the 11th Annual conference on Genetic and evolutionary computation - GECCO09

    Yuan, B., Gallagher, M.: An improved small-sample statistical test for comparing the success rates of evolutionary algorithms. In: Proceedings of the 11th Annual conference on Genetic and evolutionary computation - GECCO09. Association for Computing Machinery (ACM) (2009) 31

    2009

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.