{"id":"a3ec507f-89f6-47b7-90f2-f19825b3230e","arxiv_id":"1908.03317","paper_version":2,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"For a saturated 2^k factorial design, deleting a set of runs preserves unbiased estimation of all non-negligible effects exactly when the corresponding complement matrix is nonsingular, and the D-optimal choice maximizes the determinant of that complement.","lead":"This paper gives a rule for choosing which runs to drop from a full two-level factorial experiment when the budget allows only one run per effect that matters. The rule looks at the matrix for the effects believed to be zero on the dropped runs, which makes it easier to find valid and statistically efficient saturated designs.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Exact-zero assumption on 'negligible' effects is the load-bearing premise; outside it, unbiasedness and D-optimality of Section 3 fail.","rationale":"The reader's weakest_assumption already names the exact-zero premise, and I agree it is the load-bearing condition. The mathematical statements of Theorem 1 and the determinant relation are correct under the partitioned-Hadamard setup; the proof has typographical slips but no substantive gap. The d-optimality equivalence is also valid once θ(2) is exactly zero, because the variance of the BLUE is (D^T D)^{-1}σ² and |det(D)| is proportional to |det(C)|. However, the paper's framing using 'likely negligible' in the abstract and introduction overstates the unbiasedness claim. A reader following the deletion algorithm with merely small nonzero omitted effects will obtain biased estimates, and the recommended determinant-optimal design need not minimize MSE. This does not invalidate the paper's linear algebra, but it should change the verdict from a clean accept to an accept conditioned on making the exact-zero assumption explicit wherever 'negligible' is used. No other load-bearing error was found: the worked k=4 example is internally consistent, and the determinant spectra cited are used only as upper bounds with explicit attainment examples.","tokens_in":11394,"tokens_out":16846,"duration_ms":180065,"concrete_test":"Set θ(2)=ε·1 in the k=4 example of §3.2 with ε≠0 and compute E[θ̂(1)]=θ(1)+D^{-1}Eθ(2) for the reported |det(C)|=48 design and for one admissible design with |det(C)|=16. If the bias term is nonzero and the MSE ordering of the two designs changes with ε, the exact-zero premise is necessary for the claimed unbiasedness and D-optimality.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The algebraic core of the paper is sound: Theorem 1's determinant identity and invertibility equivalence follow from the spectrum argument, and the D-optimality reduction to maximizing |det(C)| is correct under the stated model. The soft spot is the modeling premise. Section 2, Step 4 defines θ(2) as a vector of 'zero' effects, and this exact-zero condition is what makes the derived estimator unbiased. The abstract and introduction, however, repeatedly describe the same effects as only 'likely negligible' or 'unimportant'. If θ(2) is nonzero but small, the full model gives E[Y(1)] = Dθ(1) + Eθ(2), and the estimator of Section 2, Step 7 has bias D^{-1}Eθ(2), which is nonzero generically. The design then is not unbiased for θ(1), and the deletion set maximizing |det(C)| from Definition 1(3) and Section 3.1 optimizes variance only, not mean squared error. Thus the central claim 'complement of any admissible deletion set is a saturated design for unbiased estimation of non-negligible parameters' is conditional on exact zero, not on mere likelihood of negligibility.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies saturated designs for 2^k factorials when resources allow only n runs to estimate n non-negligible parameters. It partitions the full Hadamard matrix H_N into blocks D, E, V, C according to kept/deleted runs and non-negligible/negligible effects. Theorem 1 establishes that |det D| = N^{(n-d)/2}|det C| and that D is invertible if and only if C is invertible, with a formula for D^{-1}. The authors use this to propose a deletion rule: select a set of runs R2 making C non-singular, and for d-optimality maximize |det(C)|, since the D-matrix's determinant is proportional to that of C. The approach is illustrated with 2^3 and 2^4 examples, including a classification of saturated designs for an 11-parameter 2^4 model.","tokens_in":11599,"tokens_out":15502,"duration_ms":148136,"significance":"If the claims hold, the paper provides a clean and computationally useful reduction: when the number of negligible parameters d is small, finding a good saturated design for n non-negligible parameters reduces to searching over the d by d matrix C rather than the n by n matrix D. Theorem 1 is proved by a standard and correct eigenvalue argument, and the determinant identity is a genuine, non-circular structural result. The concrete examples in Section 3.2 are valuable and the claimed d-optimal design for the 2^4 case is verified through the determinant of C. The main limitation, which the paper itself states in Section 2 Step 4 but not in the abstract, is that all unbiasedness results are conditional on the negligible effects being exactly zero.","major_comments":[{"comment":"The abstract and introduction describe the omitted effects as 'likely to be negligible', but the derivation in Section 2 Step 4 sets theta(2) to zero exactly. If a negligible effect is nonzero, then E[Y(1)] = D theta(1) + E theta(2), and the estimator in Step 7 has bias D^{-1} E theta(2), which is generically nonzero. Consequently, the deletion set maximizing |det(C)| minimizes variance but not mean squared error, and the claim that the complement of an admissible deletion set is a saturated design for unbiased estimation of the non-negligible parameters is valid only under the exact-zero assumption. The paper should state this assumption prominently and add a remark quantifying the bias, e.g. bias = D^{-1} E theta(2), and discussing its implications for the optimality criterion.","section":"Section 2, Deletion Algorithm Step 4; Definition 1(2); Section 3.1"},{"comment":"The dispersion matrix in Step 8 is written as [D - V C^{-1} E]^T [D - V C^{-1} E], but the matrix product V C^{-1} E is not defined because V is d by n, C^{-1} is d by d, and E is n by d. The correct expression, following from D^{-1} = (1/N)(D - E C^{-1} V)^T, is [D - E C^{-1} V]^T [D - E C^{-1} V]. The same typo appears in Step 9 in the determinant to be minimized. Since this expression underlies the d-optimality equivalence, it should be corrected.","section":"Section 2, Steps 8-9"},{"comment":"The predictor of Y(2)* is called a 'best linear unbiased predictor [BLUP]' without specifying the error model. In the fixed-effects model of Steps 4-5, Y(2)* is a fixed unobserved response, and the BLUP terminology requires explicit assumptions on the error covariance and on whether theta(1) is estimated or known. Please clarify the error model or replace 'BLUP' with a more precise description, such as 'the least squares predictor under the stated model'.","section":"Section 2, Step 6"}],"minor_comments":[{"comment":"The sentence 'The remaining r eigenvalues of DTD are N-gamma_1, ..., N-gamma_r' should refer to CC^T rather than D^T D, and the index range 'i = 1, ..., n-r' should be 'i = 1, ..., d-r'.","section":"Theorem 1 proof, after Eq. (3)"},{"comment":"The symbol C is used both for the block matrix in the Hadamard partition and for the set of columns {m_1, ..., m_n} of the matrix M_n; this overloaded notation is confusing and should be changed.","section":"Corollary 1 proof"},{"comment":"The intermediate line '= -V^T 1_d + V^T 1_d = 0' omits the [N; 0_{n-1}] term that is carried through the calculation; as written the proof appears to conclude the vector is zero before giving the correct final value.","section":"Lemma 1 proof"},{"comment":"The displayed list for S11 uses semicolons and gaps in a way that is not defined; either explain the notation precisely (e.g., all integers in the indicated intervals) or omit the S11 list, since the classification in the example only needs S5.","section":"Section 3.2, spectrum discussion"},{"comment":"The abstract says the neglected effects are 'likely to be negligible', which is weaker than the exact-zero assumption used in Section 2 Step 4; please reword the abstract to state that the effects are assumed to be zero so that the unbiasedness claims are not overstated.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The algebraic core of the paper is sound, and Theorem 1 is a useful, correctly proved identity. The main reason for major revision is not the mathematics but the framing: the unbiasedness and d-optimality claims are conditional on an exact-zero assumption that is buried in Step 4 and contradicted by the abstract's 'likely negligible' language. The dispersion-matrix typo in Steps 8-9 also needs correction. These are fixable within the manuscript's scope, so I do not recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a modest but sound note on saturated 2^k factorial designs. The main result is a clean reformulation: if you know which effects are negligible and treat them as exactly zero, choosing a saturated design for the remaining parameters is equivalent to choosing a non-singular matrix C for the negligible effects; the determinant identity |det D| = N^{(n-d)/2} |det C| means D-optimality reduces to maximizing |det C|. That identity is classical, but the deletion-set terminology and the worked 11-run D-optimal design for the 2^4 model with all main effects and two-factor interactions are genuinely useful. The 2^3 examples and the determinant-based classification are nice.\n\nThe algebra in Theorem 1 is essentially correct. The typos are minor: the dispersion matrix in Section 2, step 8 should be [D - E C^{-1} V]^T [D - E C^{-1} V], not [D - V C^{-1} E]^T, and the eigenvalue count for CC^T has an index slip (n-r should be d-r). These don't change the conclusions.\n\nThe real soft spot is the modeling premise. Step 4 defines the negligible parameters as 'zero effects', but the abstract and introduction say 'likely negligible'. If those effects are nonzero, even small, the estimator D^{-1}Y is biased by D^{-1}E θ(2), and the D-optimality criterion based on |det C| optimizes variance, not mean squared error. The paper should be explicit that unbiased estimation requires the negligible effects to be exactly zero. That is a standard screening assumption, but the paper shouldn't blur it. The examples and the d-optimal design are still valid under that assumption.\n\nWho is this for? Practitioners of small two-level factorial experiments and researchers working on determinant spectra of ±1 matrices. It deserves a serious referee; a careful revision should fix the typos and sharpen the zero-effect wording. I would take it after those changes.","headline":"A correct, modest reformulation of saturated design selection via complementary Hadamard blocks, with a useful worked example, held back by a sloppy statement of the zero-effect assumption.","tokens_in":12102,"tokens_out":5555,"would_cite":true,"duration_ms":53007,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62K15","62K05","05B20"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proves that in a saturated $2^k$-factorial design, a deletion set is admissible exactly when its negligible-parameter block $C$ is nonsingular, and the d-optimal design is the complement of the deletion set maximizing $|\\det C|$.","keywords":["saturated designs","two-level factorial experiments","Hadamard matrices","negligible effects","admissible deletion sets","d-optimality","determinant spectra"],"falsifier":"Simulate the paper's $2^4$ example on its d-optimal deletion set with the three-factor interaction $F_{123}$ set to 0.1 and all other 'negligible' effects zero, then estimate the eleven non-negligible parameters by least squares; the estimates of the main effects will differ from their true values by nonzero multiples of 0.1, directly contradicting the claimed unbiasedness when neglected effects are merely small.","tokens_in":11203,"feed_emoji":"📐","tokens_out":15689,"duration_ms":146256,"temperature":0.7,"pith_summary":"The paper considers a $2^k$-factorial screening experiment under an acute resource limit: only $n$ runs can be made, precisely enough to estimate the $n$ effects and interactions the experimenter regards as non-negligible, with no degrees of freedom left to estimate noise. It establishes that choosing such a saturated design is equivalent to choosing which runs to delete. When the full Hadamard matrix of order $N=2^k$ is partitioned into a kept block $D$ for the $n$ non-negligible parameters and a deleted block $C$ for the $d=N-n$ negligible parameters, $D$ is invertible if and only if $C$ is invertible, and $|\\det D| = N^{(n-d)/2}|\\det C|$. Therefore the complement of any admissible deletion set (one with nonsingular $C$) is automatically a saturated design, and the best d-optimal design is obtained by maximizing $|\\det C|$. This matters because when only a few effects are negligible, searching over deletion sets is far cheaper than searching over all $n$-run saturated designs.","feed_headline":"The best saturated design is found by deleting the right runs","feed_subtitle":"A determinant identity ties the kept runs to the deleted runs, so a hard design search becomes a small complement search.","key_machinery":"The load-bearing object is the block partition $H_N = \\begin{pmatrix} D & E \\\\ V & C \\end{pmatrix}$ of the Hadamard matrix of order $N=2^k$, with $D$ and $C$ square of orders $n$ and $d$. Theorem 1's identity $|\\det D| = N^{(n-d)/2}|\\det C|$ and the equivalence of invertibility turn the design-selection problem around: instead of searching over $n$-run saturated designs, one searches over $d$-run deletion sets with nonsingular $C$, and d-optimality of the kept design becomes maximization of $|\\det C|$ on the smaller block. The Section 2 deletion algorithm completes the mechanism by expressing the best linear unbiased predictor of the unobserved deleted runs and then the best linear unbiased estimator of the non-negligible parameters through the block $D - E C^{-1} V$.","core_discovery":"Under the paper's model that negligible effects are exactly zero, the discovery is a block-partition identity for the $2^k$-factorial model. In the full Hadamard matrix $H_N$, with rows indexed by runs and columns by factorial effects, fix the $n$ kept runs and the $n$ non-negligible parameters; the resulting block $D$ is the saturated design matrix, while the $d$ deleted runs and negligible parameters form the square block $C$. Theorem 1 states that $|\\det D| = N^{(n-d)/2}|\\det C|$, that $D$ is invertible if and only if $C$ is invertible, and that in that case $D^{-1} = \\frac{1}{N}(D - E C^{-1} V)^T$. Defining a deletion set as admissible when $C$ is nonsingular, the paper proves its complement is a saturated design for $\\theta^{(1)}$ and gives the best linear unbiased estimator $\\hat\\theta^{(1)} = \\frac{1}{N}(D - E C^{-1} V)^T Y^{(1)}$. Since the generalized variance of this estimator is governed by $|\\det D|$, maximizing it is equivalent to maximizing $|\\det C|$; in the paper's $2^4$ example with eleven non-negligible and five negligible effects, the determinant spectra of the $5\\times 5$ matrices $C$ place all saturated designs into three classes, the best having $|\\det C| = 48$.","pith_inferences":["The block-partition argument relies only on the Hadamard orthogonality structure of the full model matrix, so a similar kept-versus-deleted complementarity could be carried over to other orthogonal-array designs, a step the paper does not take.","If 'negligible' means small rather than exactly zero, the unbiasedness claim fails; a natural extension would rank deletion sets by their worst-case bias under bounded nonzero omitted effects, rather than by determinant alone.","The determinant-spectrum classification works because spectra of small $\\{-1,1\\}$-matrices are known; applying the complement trick to larger $k$ would require computing or approximating those spectra, which is a concrete computational extension of the paper's cataloguing idea."],"forward_implications":["An experimenter who knows which $d$ effects are negligible can find all valid saturated designs by enumerating $d$-run deletion sets and keeping only those with nonsingular $C$; the complement of each admissible set is a saturated design for the non-negligible parameters.","A d-optimal saturated design is obtained by maximizing $|\\det C|$ over admissible deletion sets, so the optimal design problem over $n\\times n$ matrices reduces to a determinant-maximization problem over $d\\times d$ $\\{-1,1\\}$-matrices.","The determinant of any saturated design matrix is restricted to the values $N^{(n-d)/2}$ times the determinant spectrum of the corresponding $C$ matrices; in the $2^4$ example, this gives exactly three determinant classes of designs.","Every nonsingular $\\{-1,1\\}$ saturated design matrix with first column $\\mathbf{1}_n$ can be completed to a full Hadamard matrix, and its inverse has one row summing to 1 and all other rows summing to 0, so every non-mean effect estimate is a linear contrast in the runs."],"supporting_citations":[{"why":"It constructs two-level factorial designs for main effects and selected two-factor interactions, the problem class the deletion approach generalizes.","marker":"[2]"},{"why":"It presents an algorithm that searches for D-optimal saturated two-level factorial designs, the search the paper proposes to shortcut through deletion sets.","marker":"[4]"},{"why":"It provides determinant spectra and maximal determinants for {-1,1}-matrices, used in the paper to identify d-optimal deletion sets.","marker":"[7]"},{"why":"It classifies small matrix determinants, supplying spectral data behind the determinant classification of saturated designs.","marker":"[8]"}],"fun_headline_variants":["Find the best saturated design by a small determinant check","The best saturated design is a complement of a tiny matrix","Saturated design search: check the deleted runs instead","A determinant identity reveals the optimal saturated design"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every effect or interaction labeled negligible is exactly zero, not merely small or likely negligible; if any omitted effect is nonzero, the saturated-design estimates the paper calls unbiased pick up contamination from that effect and are no longer unbiased.","fun_headline_variants_meta":{"raw":{"variants":["Find the best saturated design by a small determinant check","The best saturated design is a complement of a tiny matrix","Saturated design search: check the deleted runs instead","A determinant identity reveals the optimal saturated design"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000432,"raw_usage":{"total_tokens":2195,"prompt_tokens":927,"completion_tokens":1268,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":1206}},"tokens_in":543,"tokens_out":1268,"duration_ms":11598,"temperature":1.0,"reasoning_tokens":1206,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:19:54.687019+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate the paper's $2^4$ example on its d-optimal deletion set with the three-factor interaction $F_{123}$ set to 0.1 and all other 'negligible' effects zero, then estimate the eleven non-negligible parameters by least squares; the estimates of the main effects will differ from their true values by nonzero multiples of 0.1, directly contradicting the claimed unbiasedness when neglected effects are merely small.","supporting_citations":[{"cited_title":"Hedayat , H","cited_arxiv_id":null,"evidence_quote":"It constructs two-level factorial designs for main effects and selected two-factor interactions, the problem class the deletion approach generalizes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It presents an algorithm that searches for D-optimal saturated two-level factorial designs, the search the paper proposes to shortcut through deletion sets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides determinant spectra and maximal determinants for {-1,1}-matrices, used in the paper to identify d-optimal deletion sets."},{"cited_title":"Z ivkovi\\'c (2006), Classification of small (0 , 1) matrices, Linear Algebra Appl","cited_arxiv_id":null,"evidence_quote":"It classifies small matrix determinants, supplying spectral data behind the determinant classification of saturated designs."}],"review_version":1}