Pith. sign in

REVIEW 3 major objections 4 minor 21 references

Optimal $2^K$ Paired Comparison Designs for Third-Order Interactions

T0 review · 3 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read For paired-comparison experiments with binary attributes and up to four-attribute interactions, this paper proves that D-optimal designs reduce to a single comparison depth for the interaction block and to at most four depths for the full…

desk verdict The h4 formula and full-profile D-optimal designs are solid, but the partial-profile claims fail for K>S—the Table 2 designs violate Kiefer-Wolfowitz in a concrete counterexample. read the letter →

arxiv 1908.06092 v1 pith:MPRP2YCS submitted 2019-08-16 stat.ME

Eric Nyarko This is my paper · ORCID
classification stat.ME MSC 62K0562J1562K15
keywords pairedcomparisondesignsthird-orderinteractionsD-optimalitydepthprofilestrengthpartialprofilesbinaryattributesinvariant
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Paired-comparison experiments ask people to choose between alternatives built from binary attributes; when the model includes all interactions up to the four-attribute (“third-order”) level, the space of possible pairs is enormous, so experimenters need a recipe for which pairs to show. This paper gives that recipe in terms of comparison depth $d$, the number of attributes on which the two alternatives differ. It proves that within the symmetric class of designs that are uniform on the fixed-depth orbits, the D-optimal design for the third-order interaction block uses a single depth $d^*$, and the D-optimal design for the whole parameter vector is supported on at most four depths. For profile strengths 5 through 12 it identifies explicit two-depth designs with weights $w_{d^*}=d_1/(d^*+d_1)$ and verifies their optimality numerically. The practical payoff is that an optimal design can be built by counting differing attributes rather than by exhaustive search.

What carries the argument

The load-bearing object is the comparison-depth orbit $X_d^{(S)}$: all ordered pairs of alternatives that have the same profile strength $S$ and differ in exactly $d$ attributes. The paper restricts attention to invariant designs, which are uniform on each orbit, so every candidate design reduces to a weight vector over depths $d=1,\ldots,S$. Within this class the information matrix is block-diagonal with scalar entries $h_1,\ldots,h_4$ for the main-effect, two-attribute, three-attribute and four-attribute interaction blocks; the fourth block entry $h_4(d)$ is what determines $d^*$, and the variance formula of Theorem 2 is what lets the equivalence theorem certify optimality.

What would settle it

A direct check is available from the paper's own formulas: take the Table 2 candidate for a given $S$, compute $h_1(\bar\xi),\ldots,h_4(\bar\xi)$, then evaluate $V(d,\bar\xi)$ from Theorem 2 at every integer depth $1\le d\le S$. If any calculated $V(d,\bar\xi)$ exceeds the number of parameters $p$, the candidate is not D-optimal; if all are at most $p$, the equivalence theorem makes it D-optimal. Reproducing Table 3's maximum-normalized-variance entries for $S=5,\ldots,12$ is therefore a complete test of the paper's main numerical claim.

Watch

Extended reading notes

Core claim

The central claim is that D-optimality in this model is governed by a small set of scalar functions of the comparison depth $d$. For the third-order interaction effects, the relevant scalar is $h_4(d)=\frac{16d(S-d)(2d^2-2Sd+S^2-3S+4)}{K(K-1)(K-2)(K-3)}$; maximizing $h_4(d)$ over integer depths yields the depths listed in Table 1. The paper further shows that the variance function of any invariant design is a degree-four polynomial in $d$, so the D-optimal design for the full parameter vector can be supported on at most four depths; for the cases $S=5,\ldots,12$ it finds that two depths, $d^*$ and $d^*_1=S+1-d^*$, with weights $d^*_1/(d^*+d^*_1)$, are D-optimal, as confirmed by the equivalence theorem. In the special case $S=K=4$, the design puts weights $4/15$, $2/5$, $4/15$ and $1/15$ on depths 1, 2, 3 and 4 and is D-optimal.

Load-bearing premise

The paper's whole construction rests on the assumption that no D-optimal design is lost by restricting to invariant designs that are uniform on the fixed-comparison-depth orbits; the paper cites this group-invariance reduction but does not prove it in detail. If that reduction failed, every uniform depth design in the tables could be suboptimal.

Editorial extensions

If this is right

  • For the third-order interaction block alone, an experimenter should present all pairs at the single optimal depth $d^*$ from Table 1; no depth mixing improves the determinant.
  • For the full parameter vector, two carefully weighted depths are D-optimal in every numerically checked case ($S=5,\ldots,12$), and the weights follow the rational relation $w_{d^*}=d^*_1/(d^*+d^*_1)$ with $d^*+d^*_1=S+1$.
  • The same optimal depths apply whether the alternatives are full or partial profiles, as long as the profile strength $S$ is at least 4; attributes not shown are simply coded as zero.
  • The design for $S=K=4$ is fully explicit: uniform over all non-zero comparison depths with weights $4/15$, $2/5$, $4/15$ and $1/15$.
  • Because Theorem 3 caps the support at four depths, checking optimality of a candidate design reduces to evaluating a one-variable polynomial at $S+1$ points.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The tabulated two-depth designs suggest a sharper conjecture the paper does not state: for every $S$, the full-parameter D-optimal design may be supported on exactly two depths $d$ and $S+1-d$ with weights $(S+1-d)/(S+1)$, rather than four; verifying or disproving this for $S\ge 13$ would settle the gap between Theorem 3 and the tables.
  • For fixed $S$ and varying $K$, the third-order-only depth $d^*$ is unchanged, but the full-parameter optimum may shift because the relative sizes of the parameter blocks change; the paper's invariance machinery could be used to test this numerically.
  • The exact finite-sample implementation is not addressed: approximate weights such as $2/3$ and $1/3$ require rounding to integers for a given number of respondents, and the efficiency loss from rounding can be bounded using the variance formula.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The manuscript studies D-optimal approximate designs for paired comparison experiments with K two-level attributes, allowing both full and partial profiles of strength S, under a linear model containing main effects, first-, second-, and third-order interactions. It proves that for the third-order interaction subvector the D-optimal invariant design is uniform on a single comparison depth (Theorem 1), derives a closed-form variance function for invariant designs (Theorem 2), bounds the number of support depths of the full-parameter D-optimal design (Theorem 3), gives an explicit four-depth D-optimal design for S=K=4 (Theorem 4), and tabulates two-depth designs for S=5,...,12 that are verified numerically for full profiles by the Kiefer-Wolfowitz equivalence theorem.

Significance. The paper contributes closed-form expressions for the information matrix of uniform comparison-depth designs and for the variance function of invariant designs, and the h4 formula and Theorem 2 pass direct checks. The full-profile numerical designs in Table 3 appear to satisfy the Kiefer-Wolfowitz condition, so, if restricted to full profiles, the results usefully extend existing paired-comparison design theory from first- and second-order interactions to third-order interactions. The invariance reduction and the direct derivations of h4 and the variance formula are strengths of the paper. However, the claimed extension to partial profiles is not supported and is in fact contradicted by a concrete counterexample, which undermines a central part of the paper's stated scope.

major comments (3)
  1. [Section 4, Tables 2 and 3] The two-depth designs in Table 2 are claimed for profile strength S, but their optimality is verified numerically only for full profiles S=K in Table 3. The variance function in Theorem 2 depends on K through h_r and through p=C(K,1)+...+C(K,4), so the S=K check does not carry over to partial profiles. Concretely, for S=5 and K=6, the Table 2 design xi*=(2/3)xi_2+(1/3)xi_4 has h1=16/9, h2=64/45, h3=16/15, h4=32/45. At comparison depth d=4, Theorem 2 gives V(4)=4*4*((9/16)+(45/64)+P/(6*16/15)+Q/(6*32/45)) with P=Q=6, which equals 57.75. Since p=56, the Kiefer-Wolfowitz inequality V<=p is violated, and the design is not D-optimal for K=6. Thus the partial-profile optimality claim is false or at least unsupported; a K-dependent optimization is required, and the paper should either restrict all claims to full profiles or provide correct designs for K>S.
  2. [Appendix, Proof of Theorem 4] The proof states h1(xi*)=8/15, h2(xi*)=2/15, h3(xi*)=1/30, h4(xi*)=1/120 for the design xi*=4/15 xi_1+2/5 xi_2+4/15 xi_3+1/15 xi_4 with S=K=4. Direct evaluation from Lemma 1 gives h1=h2=h3=h4=32/15. With the stated values, Theorem 2 yields V(1)=937.5 rather than 15, so the proof as written is invalid. With the correct values, one obtains V(d)=15 for d=1,2,3,4, so the design is indeed D-optimal; the proof must be corrected.
  3. [Appendix, Proof of Theorem 3] The step 'by the shape of the variance function' is not justified. A quartic with negative leading coefficient can satisfy V(d)<=p at all integer depths and have equality at isolated interior points: for example q(d)=-(d-1)^2(d-3)^2 is nonpositive everywhere and vanishes at d=1 and d=3. The proof needs a careful argument based on the number and multiplicity of local maxima of a quartic to derive the claimed support structure, and as written it is incomplete.
minor comments (4)
  1. [Proof of Lemma 1] The sentence 'i2 = 1 and ij = −1' is ambiguous; it should read 'ij = −1 when i differs from j'.
  2. [Table 1] The table lists a single d* for each S, but h4(d) is symmetric under d -> S-d, so for S=4,5,6,7 several depths attain the maximum (for S=4, d=1 and d=3 both maximize). The table should state that d* is one of possibly multiple maximizing depths.
  3. [Corollary 1] The quantities p1,...,p4 are used in the statement of Corollary 1 but are not defined there; they should be defined in the statement or the notation should be introduced before the corollary.
  4. [Section 4, after Theorem 1] The phrase 'differ in a portion of the attributes presented' is vague; it could be made precise as 'differ in a fraction d*/S of the attributes presented'.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity; the new h4 derivation and determinant maximizations are self-contained, with only routine reliance on published lower-order results.

full rationale

The paper's central new result (the h4 contribution and the optimal comparison depths d*) is derived directly from combinatorial counting in the proof of Lemma 1 and by maximizing the resulting function h4(d); the table entries are the outcome of that optimization, not fitted inputs renamed as predictions. The variance function in Theorem 2 is obtained by direct summation, with only the first three sums (main effects, first-order, and second-order interactions) taken from the published Nyarko and Schwabe (2019); these are not the target third-order result and are externally verified published formulas, so citing them does not make the derivation circular. The invariance reduction is attributed to the standard external textbook Schwabe (1996), not to a self-cited uniqueness theorem. The numerical designs of Table 2 are obtained by direct maximization of ln det, and their optimality is checked for full profiles via the Kiefer-Wolfowitz equivalence theorem in Table 3, an external mathematical criterion. No equation in the paper reduces by construction to its own input; the only caveat is that the partial-profile case K>S is not covered by the Table 3 verification, which is a possible correctness gap rather than a circularity.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper introduces no fitted constants or new physical entities. It relies on standard design-theory axioms: the linear paired comparison model, the invariance reduction, the Kiefer-Wolfowitz equivalence theorem, and the partial-profile convention of setting unshown attributes to zero. The only assumptions are domain-level modeling choices, not ad hoc expedients.

assumptions (5)
  • domain assumption The preference response is the difference of utilities Y_n(i,j) = Y_n1(i) - Y_n2(j), and the errors are uncorrelated with constant variance.
    Section 2, model (2); this defines the linear paired comparison model.
  • domain assumption The Bradley-Terry binary choice model can be linearized at indifference beta=0, so the information matrix of the linear paired comparison model coincides with the logit model at that point.
    Section 2, paragraph 'It is worthwhile mentioning...'; standard in the literature.
  • standard math The D-criterion is invariant under permutations of attributes and levels, so there exists an optimal invariant design uniform on orbits of fixed comparison depth.
    Section 4, after equation (10); relies on group invariance theory for linear optimal design (Schwabe 1996).
  • domain assumption For partial profiles, attributes not shown are set to level 0 and the same S attributes are active in both alternatives.
    Section 3, equations (7)-(9).
  • standard math The regression functions for interactions are products of two-level codes and are orthogonal over the full factorial.
    Used in Lemma 1 proof to show off-diagonal entries vanish.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimal $2^K$ Paired Comparison Designs for Third-Order Interactions." pith.science (2026). https://pith.science/paper/MPRP2YCS

@misc{pith2026190806092,
  author       = {Pith},
  title        = {Pith review of: Optimal $2^K$ Paired Comparison Designs for Third-Order Interactions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MPRP2YCS}},
  note         = {Machine review of arXiv:1908.06092}
}
read the original abstract

In psychological research often paired comparisons are used in which either full or partial profiles of the alternatives described by a common set of two-level attributes are presented. For this situation the problem of finding optimal designs is considered in the presence of third-order interactions.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 10 canonical work pages

  1. [1]

    Berkum, E. E. M. (1987a). Optimal paired comparison designs for factorial and quadratic models. Journal of Statistical Planning and Inference , 15:265--276

    work page 1987

  2. [2]

    Berkum, E. E. M. (1987b). Optimal P aired C omparison D esigns for F actorial E xperiments . CWI Tract 31 , Amsterdam

    work page 1987

  3. [3]

    Bradley, R. A. and Terry, M. E. (1952). Rank analysis of incomplete block designs: I. T he method of paired comparisons. Biometrika , 39:324--345

    1952

  4. [4]

    Chrzan, K. (2010). Using partial profile choice experiments to handle large numbers of attributes. International Journal of Market Research , 52:827--840

    2010

  5. [5]

    J., and Davey, K

    Elrod, T., Louviere, J. J., and Davey, K. S. (1992). An empirical comparison of ratings-based and choice-based conjoint models. Journal of Marketing Research , 29:368--377

    work page 1992

  6. [6]

    Gra hoff, U., Gro mann, H., Holling, H., and Schwabe, R. (2003). Optimal paired comparison designs for first-order interactions. Statistics , 37:373--386

    2003

  7. [7]

    Gra hoff, U., Gro mann, H., Holling, H., and Schwabe, R. (2004). Optimal designs for main effects in linear paired comparison models. Journal of Statistical Planning and Inference , 126:361--376

    2004

  8. [8]

    Gro mann, H. (2018). A practical approach to designing partial-profile choice experiments with two alternatives for estimating main effects and interactions of many two-level attributes. Journal of Choice Modelling

    work page 2018

Show all 21 references
  1. [9]

    Gro mann, H., Holling, H., and Schwabe, R. (2002). Advances in optimum experimental design for conjoint analysis and discrete choice models. In F ranses, P . H . and M ontgomery, A . L . (eds.), Advances in Econometrics, Vol. 16. Econometric Models in Marketing , pages 93--117...

    2002

  2. [10]

    and Schwabe, R

    Gro mann, H. and Schwabe, R. (2015). Design for discrete choice experiments. In Dean, A ., M orris, M ., S tufken, J . and B ingham, D . (eds.), H andbook of D esign and A nalysis of E xperiments , pages 787--831. Chapman and Hall/CRC, Boca Raton

    2015

  3. [11]

    Kessels, R., Jones, B., and Goos, P. (2011). Bayesian optimal designs for discrete choice experiments with partial profiles. Journal of Choice Modelling , 4:52--74

    2011

  4. [12]

    Kiefer, J. (1959). Optimum experimental designs. Journal of the Royal Statistical Society. Series B , 21:272--304

    1959

  5. [13]

    and Wolfowitz, J

    Kiefer, J. and Wolfowitz, J. (1960). The equivalence of two extremum problems. Canadian Journal of Mathematics , 12:363--366

    1960

  6. [14]

    J., Hensher, D

    Louviere, J. J., Hensher, D. A., and Swait, J. D. (2000). Stated C hoice M ethods: A nalysis and A pplication . Cambridge University Press, Cambridge

    2000

  7. [15]

    Nyarko, E. (2019). Optimal paired comparison experiments for second-order interactions. Statistics (submitted)

    2019

  8. [16]

    and Schwabe, R

    Nyarko, E. and Schwabe, R. (2019). Optimal designs for second-order interactions in paired comparison experiments with binary attributes. Journal of Statistical Theory and Practice , 13

    2019

  9. [17]

    Schwabe, R. (1996). Optimum Designs for Multi-Factor Models . Springer, New York

    1996

  10. [18]

    Schwabe, R., Gra hoff, U., Gro mann, H., and Holling, H. (2003). Optimal 2^ K paired comparison designs for partial profiles. In PROBA\-STAT\ 2002, Proceedings of the Fourth International Conference on Mathematical Statistics, Smolenice 2002, Tatra Mountains Mathematical Publi...

    2003

  11. [19]

    Shiao, S.-Y. P. K., Ahn, C. W., and Akazawa, K. (2007). Second-order interactions with the treatment groups in controlled clinical trials. Computer Methods and Programs in Biomedicine , 86:10--16

    2007

  12. [20]

    J., Bunch, D

    Street, D. J., Bunch, D. S., and Moore, B. J. (2001). Optimal designs for 2^ k paired comparison experiments. Communications in Statistics-Theory and Methods , 30:2149--2171

    2001

  13. [21]

    Street, D. J. and Burgess, L. (2007). The C onstruction of O ptimal S tated C hoice E xperiments: T heory and M ethods . Wiley, Hoboken

    2007

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.