Pith. sign in

REVIEW 3 major objections 5 minor 20 references

Optimal Paired Comparison Experiments for Second-Order Interactions

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proves that paired comparison experiments with second-order interactions have D-optimal designs supported on at most three comparison depths: S, d*, and d*+1.

desk verdict Useful extension of paired-comparison optimal designs to second-order interactions, with a real gap in Theorem 1 that is repairable and a tie-failure that should be acknowledged. read the letter →

arxiv 1908.06090 v2 pith:IMGCMJN5 submitted 2019-08-16 stat.ME

Eric Nyarko This is my paper · ORCID
classification stat.ME MSC 62K0562J1562K15
keywords pairedcomparisonsD-optimalitysecond-orderinteractionscomparisondepthpartialprofilesprofilestrengthinvariantdesignsconjointanalysis
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks how to design paired comparison experiments, in which respondents rate or choose between two product profiles, when the statistical model includes three-attribute (second-order) interactions and the profiles may be partial instead of full. It proves that the D-optimal design never needs more than three distinct comparison depths (the number of attributes in which the two profiles differ): the full profile strength S and at most two adjacent intermediate depths $d^*$ and $d^*+1$. By restricting attention to invariant designs that are uniform on each depth, the information matrix becomes diagonal with explicitly known entries, so the optimal design is a simple weighted mixture of designs on these few depths. This reduces a combinatorial search over all possible choice sets to a small explicit calculation, yielding benchmark designs for real experiments.

What carries the argument

The central object is the comparison depth $d$, the number of attributes in which the two alternatives differ, and the uniform invariant design $\bar{\xi}_d$ that assigns equal weight to every pair of that depth. The load-bearing identity is the diagonal information matrix of Lemma 1: the matrix is block-diagonal with blocks proportional to $h_1(d)$, $h_2(d)$, and $h_3(d)$, where $h_3(d)$ is an explicit cubic in $d$ capturing the three-attribute interaction information. The Kiefer-Wolfowitz equivalence theorem converts D-optimality into a bound on the variance function $V(d,\bar{\xi})$, which is also a cubic in $d$; since a cubic can equal the parameter count $p$ at most three times, the support of an optimal design is limited to three depths, and the structure of the cubic identifies them as $S$, $d^*$, and $d^*+1$.

What would settle it

Compute $h_3(d)$ from the explicit formula in Lemma 1 for a parameter triple not covered by Table 1, such as $K=11$, $S=10$, $v=9$ or $K=12$, $S=6$, $v=5$, and check whether the maximum over $d=1,\ldots,S$ is attained at two distinct depths; if it is, Theorem 1 is false and the claimed three-depth support may fail. Alternatively, run a numerical search over invariant designs for the full parameter vector and look for a D-optimal design whose support contains four different comparison depths, which would directly contradict Theorem 3.

Watch

Extended reading notes

Core claim

In the second-order interactions model with $K$ attributes, $v$ common levels, and profile strength $S$, the paper proves that the uniform design on a single comparison depth $d^*$ is D-optimal for the second-order interaction block alone, where $d^*$ maximizes the explicit cubic function $h_3(d)$ (Theorem 1). For the full parameter vector, the D-optimal design is supported on at most three comparison depths, namely $S$, $d^*$, and $d^*+1$ (Theorem 3). The proof shows the variance function of any invariant design is a cubic polynomial in the depth $d$, so the equivalence theorem allows at most three depths to achieve the parameter-count bound; the shape of the cubic forces these depths to be exactly $S$, $d^*$, and $d^*+1$ when three are needed. Numerical tables list the optimal depths and weights for $K=4,\ldots,10$ and $v=2,\ldots,8$, with one, two, or three supporting depths depending on the parameters.

Load-bearing premise

The construction depends on the assumption that $h_3(d)$ has a unique maximizer $d^*$ among the depths $1,\ldots,S$ for every combination of attribute count $K$, level count $v$, and profile strength $S$; the paper verifies this only numerically for the parameter values in its tables, offering no proof, so a parameter combination with two equal maxima would break the single-depth optimality of the interaction block and alter the support claimed in Theorem 3.

Editorial extensions

If this is right

  • For every combination of attribute count $K$, level count $v$, and profile strength $S$, a D-optimal paired comparison design can be built by mixing uniform designs on at most the comparison depths $S$, $d^*$, and $d^*+1$.
  • The optimal depth $d^*$ for second-order interactions is the maximizer of the explicit cubic $h_3(d)$, so no numerical search over full designs is needed.
  • No single comparison depth can be D-optimal for main effects, first-order interactions, and second-order interactions simultaneously, since the three blocks have different preferred depths ($S$, an intermediate depth, and $d^*$).
  • For large numbers of levels $v$, the tables show the optimal intermediate depth $d^*$ drops to $S-2$ in the partial-profile case $S=K-1$, so optimal designs become less demanding of full profile differences.
  • These benchmark designs are starting points for constructing exact designs or fractions with a reasonable number of comparisons.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • This suggests a general pattern: in paired comparison models with up to $r$-factor interactions, a D-optimal design for the full parameter vector may be supported on at most $r+1$ comparison depths; the paper's cubic argument proves the case $r=3$ but does not state the generalisation.
  • An experimenter could turn the benchmark design into a concrete exact design by rounding the optimal weights on the few supporting depths; the paper notes the benchmark use but does not detail the rounding step.
  • A proof that $h_3(d)$ is strictly unimodal, perhaps via its derivative, would close the only numerical gap and turn Table 1 into a theorem; that is a natural next step.
  • The invariance argument assumes all attributes have the same number of levels; extending to unequal level counts would require a new argument, since the diagonal information structure would no longer hold.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper studies approximate D-optimal designs for linear paired comparison experiments in which the alternatives are described by K attributes with a common number v of levels, under full or partial profiles of strength S, and the model includes main effects, first-order interactions, and second-order (three-attribute) interactions. The design region is partitioned into orbits of fixed comparison depth d, and the paper derives the information matrix of uniform designs on these orbits (Lemma 1), the variance function of invariant designs (Theorem 2), and a formula for the variance of single-depth designs (Corollary 1). The main structural claims are that a single depth d* optimizes the second-order interaction block (Theorem 1) and that the D-optimal design for the full parameter vector is supported on at most three depths, S, d*, and d*+1 (Theorem 3). Numerical tables list optimal depths and weights for selected values of K, S, and v, with Kiefer-Wolfowitz checks reported for full profiles in Table 3.

Significance. If the main characterization is correct, the paper provides benchmark D-optimal designs for paired comparison experiments with second-order interactions for general level numbers, extending earlier results for binary attributes and for first-order interaction models. The derivations are largely self-contained: Lemma 1 is proved by explicit combinatorial counts, Theorem 2 gives a closed-form variance formula, and the D-optimality of the tabulated full-profile designs is checked numerically by the Kiefer-Wolfowitz equivalence theorem. The formulas for h3(d) and lambda(d) are concrete and directly usable. However, the proof of the central support theorem is only sketched, and the statement of Theorem 1 is not correct as written; these issues are repairable but currently leave the main structural claims unsubstantiated.

major comments (3)
  1. [Section 4, Theorem 1] The theorem asserts the existence of a single comparison depth d* that maximizes h3(d), but no proof is given and the uniqueness assertion is false as stated. For K=4, S=3, v=2, Lemma 1 gives h3(d) proportional to d(4d^2 - 18d + 20), so h3(1)=h3(3) > h3(2)=0; both depths 1 and 3 are D-optimal for the second-order interaction block, yet Table 1 reports only d*=1. Since the later construction in Table 2 and the phrase 'the optimal comparison depth' depend on a selected maximizer, the paper needs either a proof of uniqueness for all (K,S,v), which the example shows is impossible, or a consistent tie-handling rule and an explicit statement of which maximizer is used. The absence of such a statement also leaves the behavior for untabulated parameter values unsupported.
  2. [Section 4, Theorem 3] The proof of Theorem 3 is only a sketch and does not establish the claimed support pattern. From Theorem 2, V(d, xi) is a cubic polynomial in d with positive leading coefficient, but to conclude that equality V(d, xi*)=p can occur only at depths forming a set {S, d*, d*+1} one must analyze the sign of V-p on the integer depths; the manuscript merely says 'by the shape of the variance function' and omits the required sign-change argument. In addition, the symbol d* in Theorem 3 is not defined and does not coincide with the h3-maximizer of Theorem 1 in the paper's own examples: for K=5, S=4, v=2, Table 1 gives d*=4 for the interaction block, while the full-model design in Table 2 is supported on depths 2 and 4. A complete proof and a clear definition of d* are needed before the central characterization can be accepted.
  3. [Section 4, Tables 2 and 3] The paper states that the D-optimality of the designs in Table 2 has been checked numerically via the Kiefer-Wolfowitz equivalence theorem, but Table 3 displays normalized variance values only for the full-profile case S=K. For the partial-profile rows (S<K), which form a substantial part of Table 2, no normalized variance values are shown, so the numerical verification is not reproducible from the manuscript and the optimality of those entries is not documented. Please provide the corresponding values for partial profiles or state explicitly where the verification can be found.
minor comments (5)
  1. [Section 3, Eq. (12)] In the sentence following equation (12), 'f1(i1) ⊗ f1(i2) ⊗ f3(i3)' should read 'f1(i3)' instead of 'f3(i3)'.
  2. [Appendix, proof of Lemma 1, third displayed equation] In the third displayed equation of the proof of Lemma 1, the term 'f1(jk) ⊗ f1(jl) ⊗ g(jm)' should be 'f1(jk) ⊗ f1(jl) ⊗ f1(jm)'.
  3. [Section 4, after Theorem 3] The sentence 'For S >= 4 numerical computations indicate that at most two different comparison depths S and d* may be required' should be labeled as a conjecture or supported by a proof, since the paper currently gives no formal result covering this case.
  4. [Section 4, Table 3 caption] The caption 'boldface 1 corresponds to the optimal comparison depths d*' is confusing because the table entries are not visibly bold in the text; please clarify that entries equal to 1 are the normalized variance at support depths and that boldface marks the optimal depth.
  5. [Abstract] The first sentence of the abstract is grammatically awkward and should be rewritten for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the optimal-design derivation is self-contained, with self-citations only to a non-load-bearing v=2 special case.

full rationale

The derivation chain is self-contained. Lemma 1 computes h3(d) from the effects-coded regression and combinatorial orbit counts, and Lemma 2 builds invariant-design information matrices by convex combination. Theorem 2 and Corollary 1 derive the variance function algebraically from the model's Kronecker structure, and Theorem 3 applies the Kiefer-Wolfowitz equivalence theorem plus the cubic shape of V(d,xi). The numerical tables are computations from these formulas and are verified by the Kiefer-Wolfowitz sufficient condition (maximal normalized variance <= 1), so the reported designs are not fitted inputs renamed as predictions. The citations to Nyarko and Schwabe (2019) cover only the binary-attribute special case (v=2) and are used for context or for results obtained elsewhere; they are not load-bearing for the general v>=2 derivation, and the h1/h2 formulas are attributed to Graßhoff et al. (2003). The concern about uniqueness of d* in Theorem 1, such as possible ties in h3, is a correctness or rigor issue rather than circularity: no equation is defined in terms of the target result and no parameter is fitted before being predicted.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The paper introduces no free parameters and no new entities. Its results rest on the standard linear paired-comparison model, the D-optimality criterion with the Kiefer-Wolfowitz equivalence theorem, the invariance reduction to uniform designs on comparison-depth orbits, and the domain restriction to common levels v and profile strength S.

assumptions (5)
  • domain assumption Linear paired comparison model (Eq. 2) with uncorrelated, constant-variance, zero-mean errors; only differences of latent utilities are observed.
    The model defines the information matrix and all downstream results.
  • standard math D-optimality as the design criterion, with the Kiefer-Wolfowitz equivalence theorem applied.
    The equivalence theorem is used to verify optimality and to prove Theorem 3.
  • standard math Invariance reduction: the D-criterion is invariant under permutations of levels and attributes, so attention can be restricted to uniform designs on comparison-depth orbits.
    Invoked in Section 4 to restrict to invariant designs ξ.
  • domain assumption For partial profiles, profile strength S >= 3 is required for identifiability of second-order interactions, and the same S attributes are presented in both alternatives.
    Defines the design region X^(S) in Eq. (15).
  • domain assumption All attributes share a common number of levels v, and effects coding with the last level recoverable from the others.
    The regression functions f1 and the dimension p rely on this.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimal Paired Comparison Experiments for Second-Order Interactions." pith.science (2026). https://pith.science/paper/IMGCMJN5

@misc{pith2026190806090,
  author       = {Pith},
  title        = {Pith review of: Optimal Paired Comparison Experiments for Second-Order Interactions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IMGCMJN5}},
  note         = {Machine review of arXiv:1908.06090}
}
read the original abstract

In real life situations often paired comparisons involving alternatives of either full or partial profiles to mitigate cognitive burden are presented. For this situation the problem of finding optimal designs is considered in the presence of second-order interactions when all attributes have general common number of levels.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 20 canonical work pages

  1. [1]

    Bradley, R. A. and El-Helbawy, A. T. (1976). Treatment contrasts in paired comparisons: Basic procedures with application to factorials. Biometrika , 63:255--262

    work page 1976

  2. [2]

    Bradley, R. A. and Terry, M. E. (1952). Rank analysis of incomplete block designs: I. T he method of paired comparisons. Biometrika , 39:324--345

    work page 1952

  3. [3]

    Chrzan, K. (2010). Using partial profile choice experiments to handle large numbers of attributes. International Journal of Market Research , 52:827--840

    work page 2010

  4. [4]

    El-Helbawy, A. T. and Bradley, R. A. (1978). Treatment contrasts in paired comparisons: Large-sample results, applications, and some optimal designs. Journal of the American Statistical Association , 73:831--839

    work page 1978

  5. [5]

    Gra hoff, U., Gro mann, H., Holling, H., and Schwabe, R. (2003). Optimal paired comparison designs for first-order interactions. Statistics , 37:373--386

    work page 2003

  6. [6]

    Gra hoff, U., Gro mann, H., Holling, H., and Schwabe, R. (2004). Optimal designs for main effects in linear paired comparison models. Journal of Statistical Planning and Inference , 126:361--376

    work page 2004

  7. [7]

    Green, P. E. (1974). On the design of choice experiments involving multifactor alternatives. Journal of Consumer Research , 1:61--68

    work page 1974

  8. [8]

    Green, P. E. and Srinivasan, V. (1990). Conjoint analysis in marketing: N ew developments with implications for research and practice. Journal of Marketing , 54:3--19

    work page 1990

Show all 20 references
  1. [9]

    Gro mann, H., Holling, H., and Schwabe, R. (2002). Advances in optimum experimental design for conjoint analysis and discrete choice models. In In: F ranses, P . H . and M ontgomery, A . L . (Eds.) , Advances in Econometrics, Vol. 16. Econometric Models in Marketing , pages 93...

    2002

  2. [10]

    and Schwabe, R

    Gro mann, H. and Schwabe, R. (2015). Design for discrete choice experiments. In In: Dean, A ., M orris, M ., S tufken, J . and B ingham, D . (Eds.) , H andbook of D esign and A nalysis of E xperiments , pages 787--831. Chapman and Hall/CRC, Boca Raton

    2015

  3. [11]

    Kessels, R., Jones, B., and Goos, P. (2011). Bayesian optimal designs for discrete choice experiments with partial profiles. Journal of Choice Modelling , 4:52--74

    2011

  4. [12]

    Kiefer, J. (1959). Optimum experimental designs. Journal of the Royal Statistical Society. Series B , 21:272--304

    1959

  5. [13]

    and Wolfowitz, J

    Kiefer, J. and Wolfowitz, J. (1960). The equivalence of two extremum problems. Canadian Journal of Mathematics , 12:363--366

    1960

  6. [14]

    and Schwabe, R

    Nyarko, E. and Schwabe, R. (2019). Optimal designs for second-order interactions in paired comparison experiments with binary attributes. Journal of Statistical Theory and Practice , 13

    2019

  7. [15]

    Scheff \'e , H. (1952). An analysis of variance for paired comparisons. Journal of the American Statistical Association , 47:381--400

    1952

  8. [16]

    Schwabe, R. (1996). Optimum Designs for Multi-Factor Models . Springer, New York

    1996

  9. [17]

    Schwabe, R., Gra hoff, U., Gro mann, H., and Holling, H. (2003). Optimal 2^ K paired comparison designs for partial profiles. In PROBA\-STAT\ 2002, Proceedings of the Fourth International Conference on Mathematical Statistics, Smolenice 2002, Tatra Mountains Mathematical Publi...

    2003

  10. [18]

    Singh, R., Chai, F.-S., and Das, A. (2015). Optimal two-level choice designs for any number of choice sets. Biometrika , 102:967--973

    2015

  11. [19]

    J., Bunch, D

    Street, D. J., Bunch, D. S., and Moore, B. J. (2001). Optimal designs for 2^ k paired comparison experiments. Communications in Statistics-Theory and Methods , 30:2149--2171

    2001

  12. [20]

    van Berkum, E. E. M. (1987). Optimal P aired C omparison D esigns for F actorial E xperiments . CWI Tract 31 , Amsterdam

    1987

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.