Pith. sign in

REVIEW 3 major objections 5 minor 23 references

The general Nature of Saturated Designs

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper proves that in a saturated $2^k$-factorial design, a deletion set is admissible exactly when its negligible-parameter block $C$ is nonsingular, and the d-optimal design is the complement of the deletion set maximizing $|\det C|$.

desk verdict A correct, modest reformulation of saturated design selection via complementary Hadamard blocks, with a useful worked example, held back by a sloppy statement of the zero-effect assumption. read the letter →

arxiv 1908.03317 v2 pith:YF4TFXQG submitted 2019-08-09 math.ST stat.TH

Francois Domagni , Samad Hedayat , Sinha Bikas This is my paper · ORCID
classification math.STstat.TH MSC 62K1562K0505B20
keywords saturateddesignstwo-levelfactorialexperimentsHadamardmatricesnegligibleeffectsadmissibledeletionsetsd-optimalitydeterminantspectra
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper considers a $2^k$-factorial screening experiment under an acute resource limit: only $n$ runs can be made, precisely enough to estimate the $n$ effects and interactions the experimenter regards as non-negligible, with no degrees of freedom left to estimate noise. It establishes that choosing such a saturated design is equivalent to choosing which runs to delete. When the full Hadamard matrix of order $N=2^k$ is partitioned into a kept block $D$ for the $n$ non-negligible parameters and a deleted block $C$ for the $d=N-n$ negligible parameters, $D$ is invertible if and only if $C$ is invertible, and $|\det D| = N^{(n-d)/2}|\det C|$. Therefore the complement of any admissible deletion set (one with nonsingular $C$) is automatically a saturated design, and the best d-optimal design is obtained by maximizing $|\det C|$. This matters because when only a few effects are negligible, searching over deletion sets is far cheaper than searching over all $n$-run saturated designs.

What carries the argument

The load-bearing object is the block partition $H_N = \begin{pmatrix} D & E \\ V & C \end{pmatrix}$ of the Hadamard matrix of order $N=2^k$, with $D$ and $C$ square of orders $n$ and $d$. Theorem 1's identity $|\det D| = N^{(n-d)/2}|\det C|$ and the equivalence of invertibility turn the design-selection problem around: instead of searching over $n$-run saturated designs, one searches over $d$-run deletion sets with nonsingular $C$, and d-optimality of the kept design becomes maximization of $|\det C|$ on the smaller block. The Section 2 deletion algorithm completes the mechanism by expressing the best linear unbiased predictor of the unobserved deleted runs and then the best linear unbiased estimator of the non-negligible parameters through the block $D - E C^{-1} V$.

What would settle it

Simulate the paper's $2^4$ example on its d-optimal deletion set with the three-factor interaction $F_{123}$ set to 0.1 and all other 'negligible' effects zero, then estimate the eleven non-negligible parameters by least squares; the estimates of the main effects will differ from their true values by nonzero multiples of 0.1, directly contradicting the claimed unbiasedness when neglected effects are merely small.

Watch

Extended reading notes

Core claim

Under the paper's model that negligible effects are exactly zero, the discovery is a block-partition identity for the $2^k$-factorial model. In the full Hadamard matrix $H_N$, with rows indexed by runs and columns by factorial effects, fix the $n$ kept runs and the $n$ non-negligible parameters; the resulting block $D$ is the saturated design matrix, while the $d$ deleted runs and negligible parameters form the square block $C$. Theorem 1 states that $|\det D| = N^{(n-d)/2}|\det C|$, that $D$ is invertible if and only if $C$ is invertible, and that in that case $D^{-1} = \frac{1}{N}(D - E C^{-1} V)^T$. Defining a deletion set as admissible when $C$ is nonsingular, the paper proves its complement is a saturated design for $\theta^{(1)}$ and gives the best linear unbiased estimator $\hat\theta^{(1)} = \frac{1}{N}(D - E C^{-1} V)^T Y^{(1)}$. Since the generalized variance of this estimator is governed by $|\det D|$, maximizing it is equivalent to maximizing $|\det C|$; in the paper's $2^4$ example with eleven non-negligible and five negligible effects, the determinant spectra of the $5\times 5$ matrices $C$ place all saturated designs into three classes, the best having $|\det C| = 48$.

Load-bearing premise

The load-bearing premise is that every effect or interaction labeled negligible is exactly zero, not merely small or likely negligible; if any omitted effect is nonzero, the saturated-design estimates the paper calls unbiased pick up contamination from that effect and are no longer unbiased.

Editorial extensions

If this is right

  • An experimenter who knows which $d$ effects are negligible can find all valid saturated designs by enumerating $d$-run deletion sets and keeping only those with nonsingular $C$; the complement of each admissible set is a saturated design for the non-negligible parameters.
  • A d-optimal saturated design is obtained by maximizing $|\det C|$ over admissible deletion sets, so the optimal design problem over $n\times n$ matrices reduces to a determinant-maximization problem over $d\times d$ $\{-1,1\}$-matrices.
  • The determinant of any saturated design matrix is restricted to the values $N^{(n-d)/2}$ times the determinant spectrum of the corresponding $C$ matrices; in the $2^4$ example, this gives exactly three determinant classes of designs.
  • Every nonsingular $\{-1,1\}$ saturated design matrix with first column $\mathbf{1}_n$ can be completed to a full Hadamard matrix, and its inverse has one row summing to 1 and all other rows summing to 0, so every non-mean effect estimate is a linear contrast in the runs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The block-partition argument relies only on the Hadamard orthogonality structure of the full model matrix, so a similar kept-versus-deleted complementarity could be carried over to other orthogonal-array designs, a step the paper does not take.
  • If 'negligible' means small rather than exactly zero, the unbiasedness claim fails; a natural extension would rank deletion sets by their worst-case bias under bounded nonzero omitted effects, rather than by determinant alone.
  • The determinant-spectrum classification works because spectra of small $\{-1,1\}$-matrices are known; applying the complement trick to larger $k$ would require computing or approximating those spectra, which is a concrete computational extension of the paper's cataloguing idea.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies saturated designs for 2^k factorials when resources allow only n runs to estimate n non-negligible parameters. It partitions the full Hadamard matrix H_N into blocks D, E, V, C according to kept/deleted runs and non-negligible/negligible effects. Theorem 1 establishes that |det D| = N^{(n-d)/2}|det C| and that D is invertible if and only if C is invertible, with a formula for D^{-1}. The authors use this to propose a deletion rule: select a set of runs R2 making C non-singular, and for d-optimality maximize |det(C)|, since the D-matrix's determinant is proportional to that of C. The approach is illustrated with 2^3 and 2^4 examples, including a classification of saturated designs for an 11-parameter 2^4 model.

Significance. If the claims hold, the paper provides a clean and computationally useful reduction: when the number of negligible parameters d is small, finding a good saturated design for n non-negligible parameters reduces to searching over the d by d matrix C rather than the n by n matrix D. Theorem 1 is proved by a standard and correct eigenvalue argument, and the determinant identity is a genuine, non-circular structural result. The concrete examples in Section 3.2 are valuable and the claimed d-optimal design for the 2^4 case is verified through the determinant of C. The main limitation, which the paper itself states in Section 2 Step 4 but not in the abstract, is that all unbiasedness results are conditional on the negligible effects being exactly zero.

major comments (3)
  1. [Section 2, Deletion Algorithm Step 4; Definition 1(2); Section 3.1] The abstract and introduction describe the omitted effects as 'likely to be negligible', but the derivation in Section 2 Step 4 sets theta(2) to zero exactly. If a negligible effect is nonzero, then E[Y(1)] = D theta(1) + E theta(2), and the estimator in Step 7 has bias D^{-1} E theta(2), which is generically nonzero. Consequently, the deletion set maximizing |det(C)| minimizes variance but not mean squared error, and the claim that the complement of an admissible deletion set is a saturated design for unbiased estimation of the non-negligible parameters is valid only under the exact-zero assumption. The paper should state this assumption prominently and add a remark quantifying the bias, e.g. bias = D^{-1} E theta(2), and discussing its implications for the optimality criterion.
  2. [Section 2, Steps 8-9] The dispersion matrix in Step 8 is written as [D - V C^{-1} E]^T [D - V C^{-1} E], but the matrix product V C^{-1} E is not defined because V is d by n, C^{-1} is d by d, and E is n by d. The correct expression, following from D^{-1} = (1/N)(D - E C^{-1} V)^T, is [D - E C^{-1} V]^T [D - E C^{-1} V]. The same typo appears in Step 9 in the determinant to be minimized. Since this expression underlies the d-optimality equivalence, it should be corrected.
  3. [Section 2, Step 6] The predictor of Y(2)* is called a 'best linear unbiased predictor [BLUP]' without specifying the error model. In the fixed-effects model of Steps 4-5, Y(2)* is a fixed unobserved response, and the BLUP terminology requires explicit assumptions on the error covariance and on whether theta(1) is estimated or known. Please clarify the error model or replace 'BLUP' with a more precise description, such as 'the least squares predictor under the stated model'.
minor comments (5)
  1. [Theorem 1 proof, after Eq. (3)] The sentence 'The remaining r eigenvalues of DTD are N-gamma_1, ..., N-gamma_r' should refer to CC^T rather than D^T D, and the index range 'i = 1, ..., n-r' should be 'i = 1, ..., d-r'.
  2. [Corollary 1 proof] The symbol C is used both for the block matrix in the Hadamard partition and for the set of columns {m_1, ..., m_n} of the matrix M_n; this overloaded notation is confusing and should be changed.
  3. [Lemma 1 proof] The intermediate line '= -V^T 1_d + V^T 1_d = 0' omits the [N; 0_{n-1}] term that is carried through the calculation; as written the proof appears to conclude the vector is zero before giving the correct final value.
  4. [Section 3.2, spectrum discussion] The displayed list for S11 uses semicolons and gaps in a way that is not defined; either explain the notation precisely (e.g., all integers in the indicated intervals) or omit the S11 list, since the classification in the example only needs S5.
  5. [Abstract] The abstract says the neglected effects are 'likely to be negligible', which is weaker than the exact-zero assumption used in Section 2 Step 4; please reword the abstract to state that the effects are assumed to be zero so that the unbiasedness claims are not overstated.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the determinant identity and the D-optimality reduction are derived from the Hadamard property and checked against external determinant spectra.

full rationale

The paper's central chain is self-contained linear algebra. Theorem 1 starts from the Hadamard property H_N H_N^T = N I_N, derives the eigenvalue relation between D^T D and V^T V, and concludes |det(D)| = N^{(n-d)/2} |det(C)| together with the equivalence of invertibility of D and C. The deletion algorithm then uses this theorem to justify that a non-singular C-matrix gives a valid saturated design for the non-negligible parameters, and the BLUE formula in Section 2 step 7 is algebraically derived from the partitioned Yates representation, not assumed. The step 'maximize |det(C)| for D-optimality' is also derived from the determinant identity and the dispersion matrix of the BLUE, rather than being imposed as an input. The determinant spectra used to identify the maximal |det(C)| = 48 for the 2^4 example are cited from external literature (Orrick, Zivkovic, Metropolis, Stein and Weil), not from the authors' own fitted values or prior claims. Self-citations (Hedayat and Pesotan; Hedayat and Zhu) appear only as contextual references for existing construction methods and algorithms, and none of the paper's conclusions depends on the correctness of those papers. The only substantive limitation is the modeling premise in Section 2 step 4, where θ(2) is declared to be 'zero' effects while the abstract says 'likely to be negligible'; this is an assumption-strength issue about unbiasedness under nonzero omitted effects, not a circularity, because the paper's derivations are conditional on the exact-zero model and do not use the conclusion to justify the premise. No prediction is fitted to data, no parameter is renamed as a derived result, and no load-bearing uniqueness theorem is imported from the authors' own work.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted; the only input is the experimenter's a priori choice of non-negligible effects. The central derivation is a self-contained linear-algebra consequence of the Hadamard property. The main domain assumption is that negligible effects are exactly zero, which is standard in screening but should be stated explicitly.

assumptions (3)
  • domain assumption All effects not in the non-negligible set are exactly zero
    Invoked in the deletion algorithm step 4 and in the unbiasedness and best linear unbiased prediction derivation; the abstract's wording likely negligible is weaker than the model actually used.
  • standard math The full 2^k factorial model matrix is a Hadamard matrix with orthogonal columns
    Used throughout Section 2 via the Yates representation E[Y]=H_N theta; true for two-level factorial designs with plus or minus one coding.
  • domain assumption Gauss-Markov best linear unbiased estimation theory for the saturated linear model with independent homoscedastic errors
    Invoked to call the proposed estimator the BLUE and to link dispersion to the design matrix in Section 2, steps 7 to 9.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The general Nature of Saturated Designs." pith.science (2026). https://pith.science/paper/YF4TFXQG

@misc{pith2026190803317,
  author       = {Pith},
  title        = {Pith review of: The general Nature of Saturated Designs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YF4TFXQG}},
  note         = {Machine review of arXiv:1908.03317}
}
abstract

We contemplate an experimental situation in a $2^k$-factorial experiment with acute resource crunch so that we need to conduct just a saturated design [SD] - with the understanding that precision of the estimates cannot be estimated from the data. It is known beforehand which effect(s)/interaction(s) are likely to be negligible. We examine the flexibility to the extent that an experimenter can make a choice of an SD in order to retain information on all the remaining [non-negligible] effects/interactions.

Figures

Figures reproduced from arXiv: 1908.03317 by the authors.

Figure 1
Figure 1. Hadamard matrix of order 24 10 [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 2
Figure 2. Classification of saturated designs by determinant [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 23 canonical work pages

  1. [1]

    Chadjipantelis, S

    T. Chadjipantelis, S. Kounias and C. Moyssiadis (1987), The maximum determinant of 21 21 (+1, −1)-matrices and D-optimal designs, J. Statist. Plann. Inference, 16 167-178

    work page 1987

  2. [2]

    Hedayat , H

    A.S. Hedayat , H. Pesotan (1992), Two-level factorial designs for main-effects and selected two-factor interactions Statistica Sinica 2 , 453-464

    work page 1992

  3. [3]

    Hedayat , H

    A.S. Hedayat , H. Pesotan (2007), Tools for constructing optimal two-level factorial designs for a linear model containing main effects and one two-factor interaction J. Statist. Plann. Inference, 137(4):1452--1463

    work page 2007

  4. [4]

    A. S. Hedayat and Haiyuan Zhu (2011), An effective algorithm for searching for D-optimal saturated two-level factorial designs. J. Stat. Theory Appl., 10(2):209--227

    work page 2011

  5. [5]

    Metropolis (1971), Spectra of determinant values in (0,1) matrices

    N. Metropolis (1971), Spectra of determinant values in (0,1) matrices. In A. O. L. Atkin and B. J. Birch, editors, Computers in Number Theory:\\ Proceedings of the Science Research Atlas Symposium No. 2 held at Oxford, 18-23 August, 1969, Academic Press, London, 1971, 271-276

    work page 1971

  6. [6]

    D.C Montgomery (1996), Design and Analysis of Experiments, 3rd Edition, Wiley, New York

    work page 1996

  7. [7]

    W. P. Orrick (2005), The maximal -1,1 -determinant of order 15. Metrika 62, 195-219

    work page 2005

  8. [8]

    Z ivkovi\'c (2006), Classification of small (0 , 1) matrices, Linear Algebra Appl

    M. Z ivkovi\'c (2006), Classification of small (0 , 1) matrices, Linear Algebra Appl. 414 , 1, 310-346

    work page 2006

Show all 23 references
  1. [9]

    W. P. Orrick (2005). The maximal -1,1 -determinant of order 15. Metrika 62, 195-219

    2005

  2. [10]

    G.Barba, Intorno al teorema di Hadamard sui determinanti a valore massimo\\ Giorn. Mat. Battaglini 71 (1933), 70-86

    1933

  3. [11]

    Ehlich, Determinantenabsch\..atzungen f\..ur bin\..are Matrizen , Math

    H. Ehlich, Determinantenabsch\..atzungen f\..ur bin\..are Matrizen , Math. Z.\\ 83 (1964), 123-132

    1964

  4. [12]

    Ehlich, Determinantenabsch\..atzungen f\..ur bin\..are Matrizen \\ mit n 3 4 Math

    H. Ehlich, Determinantenabsch\..atzungen f\..ur bin\..are Matrizen \\ mit n 3 4 Math. Z.84 (1964), 438-447

    1964

  5. [13]

    (1964), On Hadamard's inequality for the determinants of order non-divisible by 4 \\ Colloq

    Wojtas, M. (1964), On Hadamard's inequality for the determinants of order non-divisible by 4 \\ Colloq. Math., 12: 73-83

    1964

  6. [14]

    (2000), Almost D-optimal designs\\ Utilitas Math., 57: 121-128

    Cohn, J.H.E. (2000), Almost D-optimal designs\\ Utilitas Math., 57: 121-128

    2000

  7. [15]

    J. H. Osborn, The Hadamard Maximal Determinant Problem, Honours thesis, University of Melbourne, 2002. verbatim http://wwwmaths.anu.edu.au/ osborn/publications/pubsall.html. verbatim

    2002

  8. [16]

    Metropolis, Spectra of determinant values in (0,1) matrices

    N. Metropolis, Spectra of determinant values in (0,1) matrices. In A. O. L. Atkin and B. J. Birch, editors, Computers in Number Theory:\\ Proceedings of the Science Research Atlas Symposium No. 2 held at Oxford, 18-23 August, 1969, Academic Press, London, 1971, 271-276

    1969

  9. [17]

    Craigen, The range of the determinant function on the set of n n \\ (0,1)-matrices, J

    R. Craigen, The range of the determinant function on the set of n n \\ (0,1)-matrices, J. Combin. Math. Combin. Comput. 8 (1990), 161-171

    1990

  10. [18]

    http://www.indiana.edu/ maxdet/spectrum.html

  11. [19]

    , Classification of small (0 , 1) matrices, Linear Algebra Appl

    M. , Classification of small (0 , 1) matrices, Linear Algebra Appl. 414 (2006), 1, 310–346

    2006

  12. [20]

    D.C Montgomery, Design and Analysis of Experiments, 3rd Edition, Wiley, New York, 1996

    1996

  13. [21]

    A. S. Hedayat; H. Pesotan. Two-Level Factorial Designs For Main Effects and Selected Two-Factor Interactions. (1992), 2 , 453-464

    1992

  14. [22]

    A. S. Hedayat; H. Pesotan. Tools for constructing optimal two-level factorial designs for a linear model containing main effects and one two-factor interaction. J. Statist. Plann. Inference, 137( 2007) 4 : 1452--1463

    2007

  15. [23]

    An effective algorithm for searching for D-optimal saturated two-level factorial designs. J. Stat. Theory Apply., 10, 2 209-227. Singh, Rakhi; Chai, Feng-Shun; Das, Ashish. Optimal two-level choice designs for any number of choice sets. Biometrika 102(2015), 4 967-973. Chai, F...

    2015

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.