{"id":"c96c1eca-abf9-4d3a-b7d9-d7f8cce125ca","arxiv_id":"1908.06092","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"For the third-order interaction subvector a single comparison depth is optimal, while D-optimal designs for the full parameter vector use two depths, explicitly proven for S=4 and numerically verified for full profiles with S=5 to 12.","lead":"This paper derives D-optimal paired comparison designs for two-level attribute experiments that include third-order (four-attribute) interactions. It gives the information matrix for uniform designs at each comparison depth and tabulates optimal depths and weights for profile strengths 4 through 12.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Partial-profile claim fails: Table 2 designs are checked only for S=K; for K>S they can violate Kiefer-Wolfowitz (S=5,K=6 gives V(4)=57.75 > p=56).","rationale":"The reader's weakest assumption was the invariance reduction. I do not see that as the main risk: the group action induces orthogonal reparameterizations, and Kiefer's averaging argument justifies restricting to invariant designs, so that step is standard and likely sound. The load-bearing gap is the extension from full profiles to partial profiles. Table 2 gives no K values, and the paper says the designs apply under profile strength S, but the only verification (Table 3) is for S=K. The S=5,K=6 computation above shows the proposed design is not D-optimal for the full parameter vector, so the partial-profile claims are incorrect unless K-specific designs are provided. This is a concrete, fixable issue: either add K-dependent tables and verify the equivalence theorem for all K>=S, or restrict the optimality claims to full profiles. The internal h-value misprints in the proof of Theorem 4 and the incomplete argument in Theorem 3 reinforce the need for revision, but the partial-profile counterexample is the decisive concern. Since the full-profile results and the third-order subvector results appear sound, the appropriate outcome remains a conditional acceptance subject to correction, matching the reader's original verdict.","tokens_in":12054,"tokens_out":23294,"duration_ms":208448,"concrete_test":"Perform a direct D-optimal search over all mixtures of depths 1..5 for S=5, K=6 and compute V(d)/p for the Table 2 candidate. If the candidate has max V/p > 1 or is dominated by a design with higher log-determinant, the partial-profile claim is falsified. More generally, re-run the Table 2 optimization for K > S (e.g., K = S+1) and check whether the optimal support or weights change; if they do, the tables need a K column and partial-profile claims must be restricted or re-derived.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4 presents Table 2 two-depth designs for the full parameter vector under profile strength S, but the numerical Kiefer-Wolfowitz check in Table 3 is only for full profiles S=K. For partial profiles K>S the same designs are not generally D-optimal, because the variance function depends on K through p_r = C(K,r), so K does not cancel. Example: S=5, K=6. The Table 2 design xi* = (2/3) xi_2 + (1/3) xi_4 has h1=16/9, h2=64/45, h3=16/15, h4=32/45. At comparison depth d=4, Theorem 2 gives V(4) = 4d/h1 + 4d(S-d)/h2 + 4dP/(6h3) + 4d(S-d)Q/(6h4) = 9 + 11.25 + 15 + 22.5 = 57.75, where P=3S^2-6dS+4d^2-3S+2=6 and Q=2d^2-2Sd+S^2-3S+4=6. Since V(4)>p=56, the candidate violates the Kiefer-Wolfowitz condition and is not D-optimal for K=6. Thus the claimed partial-profile optimality is unsupported or incorrect; a separate K-dependent optimization is needed. The invariance reduction itself is standard and is not the main risk.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript studies D-optimal approximate designs for paired comparison experiments with K two-level attributes, allowing both full and partial profiles of strength S, under a linear model containing main effects, first-, second-, and third-order interactions. It proves that for the third-order interaction subvector the D-optimal invariant design is uniform on a single comparison depth (Theorem 1), derives a closed-form variance function for invariant designs (Theorem 2), bounds the number of support depths of the full-parameter D-optimal design (Theorem 3), gives an explicit four-depth D-optimal design for S=K=4 (Theorem 4), and tabulates two-depth designs for S=5,...,12 that are verified numerically for full profiles by the Kiefer-Wolfowitz equivalence theorem.","tokens_in":12243,"tokens_out":20796,"duration_ms":186511,"significance":"The paper contributes closed-form expressions for the information matrix of uniform comparison-depth designs and for the variance function of invariant designs, and the h4 formula and Theorem 2 pass direct checks. The full-profile numerical designs in Table 3 appear to satisfy the Kiefer-Wolfowitz condition, so, if restricted to full profiles, the results usefully extend existing paired-comparison design theory from first- and second-order interactions to third-order interactions. The invariance reduction and the direct derivations of h4 and the variance formula are strengths of the paper. However, the claimed extension to partial profiles is not supported and is in fact contradicted by a concrete counterexample, which undermines a central part of the paper's stated scope.","major_comments":[{"comment":"The two-depth designs in Table 2 are claimed for profile strength S, but their optimality is verified numerically only for full profiles S=K in Table 3. The variance function in Theorem 2 depends on K through h_r and through p=C(K,1)+...+C(K,4), so the S=K check does not carry over to partial profiles. Concretely, for S=5 and K=6, the Table 2 design xi*=(2/3)xi_2+(1/3)xi_4 has h1=16/9, h2=64/45, h3=16/15, h4=32/45. At comparison depth d=4, Theorem 2 gives V(4)=4*4*((9/16)+(45/64)+P/(6*16/15)+Q/(6*32/45)) with P=Q=6, which equals 57.75. Since p=56, the Kiefer-Wolfowitz inequality V<=p is violated, and the design is not D-optimal for K=6. Thus the partial-profile optimality claim is false or at least unsupported; a K-dependent optimization is required, and the paper should either restrict all claims to full profiles or provide correct designs for K>S.","section":"Section 4, Tables 2 and 3"},{"comment":"The proof states h1(xi*)=8/15, h2(xi*)=2/15, h3(xi*)=1/30, h4(xi*)=1/120 for the design xi*=4/15 xi_1+2/5 xi_2+4/15 xi_3+1/15 xi_4 with S=K=4. Direct evaluation from Lemma 1 gives h1=h2=h3=h4=32/15. With the stated values, Theorem 2 yields V(1)=937.5 rather than 15, so the proof as written is invalid. With the correct values, one obtains V(d)=15 for d=1,2,3,4, so the design is indeed D-optimal; the proof must be corrected.","section":"Appendix, Proof of Theorem 4"},{"comment":"The step 'by the shape of the variance function' is not justified. A quartic with negative leading coefficient can satisfy V(d)<=p at all integer depths and have equality at isolated interior points: for example q(d)=-(d-1)^2(d-3)^2 is nonpositive everywhere and vanishes at d=1 and d=3. The proof needs a careful argument based on the number and multiplicity of local maxima of a quartic to derive the claimed support structure, and as written it is incomplete.","section":"Appendix, Proof of Theorem 3"}],"minor_comments":[{"comment":"The sentence 'i2 = 1 and ij = −1' is ambiguous; it should read 'ij = −1 when i differs from j'.","section":"Proof of Lemma 1"},{"comment":"The table lists a single d* for each S, but h4(d) is symmetric under d -> S-d, so for S=4,5,6,7 several depths attain the maximum (for S=4, d=1 and d=3 both maximize). The table should state that d* is one of possibly multiple maximizing depths.","section":"Table 1"},{"comment":"The quantities p1,...,p4 are used in the statement of Corollary 1 but are not defined there; they should be defined in the statement or the notation should be introduced before the corollary.","section":"Corollary 1"},{"comment":"The phrase 'differ in a portion of the attributes presented' is vague; it could be made precise as 'differ in a fraction d*/S of the attributes presented'.","section":"Section 4, after Theorem 1"}],"recommendation":"major_revision","confidential_remarks":"The decisive issue is the partial-profile claim. The authors should be asked either to provide K-dependent optimal designs for all K>S or to restrict the paper explicitly to full profiles and adjust the title and abstract accordingly. The full-profile results appear sound, including Theorem 4 once the h values in its proof are corrected, so the paper is salvageable with substantial revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the h4 formula is a real extension, and the full-profile designs check out, but the paper's partial-profile claims do not survive for K>S. A concrete S=5, K=6 example violates the Kiefer-Wolfowitz condition, so the Table 2 designs are not D-optimal for partial profiles as claimed.\n\nWhat is genuinely new: Lemma 1's h4(d) formula for third-order interactions is correct and not in the cited prior work on first- and second-order interactions. Theorem 1, the optimal single depth for the third-order block, is a valid consequence. The variance formula in Theorem 2 also holds. For full profiles with S=K, the two-depth designs in Table 2 and their weights pass the Kiefer-Wolfowitz check in Table 3, as far as I can see. Theorem 4's explicit S=K=4 design is D-optimal, but the proof lists h1,...,h4 as 8/15, 2/15, 1/30, 1/120 when the actual values are all 32/15. The conclusion survives once corrected.\n\nNow the soft spot. The abstract and introduction claim full and partial profiles are treated, and Table 2 is indexed only by S. But the h_r(d) functions depend on K, and so do the determinants being maximized; the numerical verification in Table 3 is only for S=K. Try S=5, K=6. The Table 2 design is (2/3)ξ2+(1/3)ξ4, giving h1=16/9, h2=64/45, h3=16/15, h4=32/45. Theorem 2 yields V(4)=57.75, while p=56. That violates Kiefer-Wolfowitz, so the design is not D-optimal for K=6. The paper would need a separate K-dependent optimization for partial profiles, or a restriction of the claims to full profiles. A minor issue: the proof of Theorem 3 assumes the support can consist of at most two adjacent pairs, but a quartic variance function can equal p at up to four non-adjacent integer points without this being ruled out; that argument is underdeveloped.\n\nBottom line: the h4 derivation and the full-profile optimal-design results are a legitimate contribution to the optimal paired comparison design literature. The partial-profile overclaim is a serious flaw, but it is correctable by restricting the scope or redoing the optimization with K in the objective. This deserves peer review with major revision expected.\n\nRecommendation: send to a serious referee; the core is salvageable.","headline":"The h4 formula and full-profile D-optimal designs are solid, but the partial-profile claims fail for K>S—the Table 2 designs violate Kiefer-Wolfowitz in a concrete counterexample.","tokens_in":12890,"tokens_out":9868,"would_cite":false,"duration_ms":79457,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62K05","62J15","62K15"],"pacs":[],"model":"deepseek-v4-flash","headline":"For paired-comparison experiments with binary attributes and up to four-attribute interactions, this paper proves that D-optimal designs reduce to a single comparison depth for the interaction block and to at most four depths for the full…","keywords":["paired comparison designs","third-order interactions","D-optimality","comparison depth","profile strength","partial profiles","binary attributes","invariant designs"],"falsifier":"A direct check is available from the paper's own formulas: take the Table 2 candidate for a given $S$, compute $h_1(\\bar\\xi),\\ldots,h_4(\\bar\\xi)$, then evaluate $V(d,\\bar\\xi)$ from Theorem 2 at every integer depth $1\\le d\\le S$. If any calculated $V(d,\\bar\\xi)$ exceeds the number of parameters $p$, the candidate is not D-optimal; if all are at most $p$, the equivalence theorem makes it D-optimal. Reproducing Table 3's maximum-normalized-variance entries for $S=5,\\ldots,12$ is therefore a complete test of the paper's main numerical claim.","tokens_in":11705,"feed_emoji":"📊","tokens_out":12796,"duration_ms":113031,"temperature":0.7,"pith_summary":"Paired-comparison experiments ask people to choose between alternatives built from binary attributes; when the model includes all interactions up to the four-attribute (“third-order”) level, the space of possible pairs is enormous, so experimenters need a recipe for which pairs to show. This paper gives that recipe in terms of comparison depth $d$, the number of attributes on which the two alternatives differ. It proves that within the symmetric class of designs that are uniform on the fixed-depth orbits, the D-optimal design for the third-order interaction block uses a single depth $d^*$, and the D-optimal design for the whole parameter vector is supported on at most four depths. For profile strengths 5 through 12 it identifies explicit two-depth designs with weights $w_{d^*}=d_1/(d^*+d_1)$ and verifies their optimality numerically. The practical payoff is that an optimal design can be built by counting differing attributes rather than by exhaustive search.","feed_headline":"One comparison depth rules third-order choice designs","feed_subtitle":"For binary-attribute paired comparisons, show pairs that differ in exactly d* attributes; tables give d* and weights for S=5–12.","key_machinery":"The load-bearing object is the comparison-depth orbit $X_d^{(S)}$: all ordered pairs of alternatives that have the same profile strength $S$ and differ in exactly $d$ attributes. The paper restricts attention to invariant designs, which are uniform on each orbit, so every candidate design reduces to a weight vector over depths $d=1,\\ldots,S$. Within this class the information matrix is block-diagonal with scalar entries $h_1,\\ldots,h_4$ for the main-effect, two-attribute, three-attribute and four-attribute interaction blocks; the fourth block entry $h_4(d)$ is what determines $d^*$, and the variance formula of Theorem 2 is what lets the equivalence theorem certify optimality.","core_discovery":"The central claim is that D-optimality in this model is governed by a small set of scalar functions of the comparison depth $d$. For the third-order interaction effects, the relevant scalar is $h_4(d)=\\frac{16d(S-d)(2d^2-2Sd+S^2-3S+4)}{K(K-1)(K-2)(K-3)}$; maximizing $h_4(d)$ over integer depths yields the depths listed in Table 1. The paper further shows that the variance function of any invariant design is a degree-four polynomial in $d$, so the D-optimal design for the full parameter vector can be supported on at most four depths; for the cases $S=5,\\ldots,12$ it finds that two depths, $d^*$ and $d^*_1=S+1-d^*$, with weights $d^*_1/(d^*+d^*_1)$, are D-optimal, as confirmed by the equivalence theorem. In the special case $S=K=4$, the design puts weights $4/15$, $2/5$, $4/15$ and $1/15$ on depths 1, 2, 3 and 4 and is D-optimal.","pith_inferences":["The tabulated two-depth designs suggest a sharper conjecture the paper does not state: for every $S$, the full-parameter D-optimal design may be supported on exactly two depths $d$ and $S+1-d$ with weights $(S+1-d)/(S+1)$, rather than four; verifying or disproving this for $S\\ge 13$ would settle the gap between Theorem 3 and the tables.","For fixed $S$ and varying $K$, the third-order-only depth $d^*$ is unchanged, but the full-parameter optimum may shift because the relative sizes of the parameter blocks change; the paper's invariance machinery could be used to test this numerically.","The exact finite-sample implementation is not addressed: approximate weights such as $2/3$ and $1/3$ require rounding to integers for a given number of respondents, and the efficiency loss from rounding can be bounded using the variance formula."],"forward_implications":["For the third-order interaction block alone, an experimenter should present all pairs at the single optimal depth $d^*$ from Table 1; no depth mixing improves the determinant.","For the full parameter vector, two carefully weighted depths are D-optimal in every numerically checked case ($S=5,\\ldots,12$), and the weights follow the rational relation $w_{d^*}=d^*_1/(d^*+d^*_1)$ with $d^*+d^*_1=S+1$.","The same optimal depths apply whether the alternatives are full or partial profiles, as long as the profile strength $S$ is at least 4; attributes not shown are simply coded as zero.","The design for $S=K=4$ is fully explicit: uniform over all non-zero comparison depths with weights $4/15$, $2/5$, $4/15$ and $1/15$.","Because Theorem 3 caps the support at four depths, checking optimality of a candidate design reduces to evaluating a one-variable polynomial at $S+1$ points."],"supporting_citations":[{"why":"Provides the orbit/invariance framework and the first-order interaction results that this paper extends to third-order interactions.","marker":"Graßhoff et al. (2003)"},{"why":"Gives the second-order interaction information-matrix entries and the variance-function proof technique used here.","marker":"Nyarko and Schwabe (2019)"},{"why":"Supplies the equivalence theorem that certifies D-optimality of the proposed depth designs.","marker":"Kiefer and Wolfowitz (1960)"},{"why":"Justifies the invariance and reparameterization argument that restricts attention to uniform designs on the orbits.","marker":"Schwabe (1996)"},{"why":"Establishes the paired-comparison design problem and earlier full-profile optimal designs that motivate the setting.","marker":"Berkum (1987b)"},{"why":"Frames the linear paired-comparison model and design theory used throughout the paper.","marker":"Großmann and Schwabe (2015)"}],"fun_headline_variants":["Two depths suffice for D-optimal paired comparisons","Depth governs D-optimal design for third-order choices","Optimal depth found for third-order paired comparisons","D-optimal designs hinge on comparison depth","For third-order effects, two depths optimize"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's whole construction rests on the assumption that no D-optimal design is lost by restricting to invariant designs that are uniform on the fixed-comparison-depth orbits; the paper cites this group-invariance reduction but does not prove it in detail. If that reduction failed, every uniform depth design in the tables could be suboptimal.","fun_headline_variants_meta":{"raw":{"variants":["Two depths suffice for D-optimal paired comparisons","Depth governs D-optimal design for third-order choices","Optimal depth found for third-order paired comparisons","D-optimal designs hinge on comparison depth","For third-order effects, two depths optimize"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000466,"raw_usage":{"total_tokens":2268,"prompt_tokens":828,"completion_tokens":1440,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":444,"completion_tokens_details":{"reasoning_tokens":1370}},"tokens_in":444,"tokens_out":1440,"duration_ms":11032,"temperature":1.0,"reasoning_tokens":1370,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:58:37.196273+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct check is available from the paper's own formulas: take the Table 2 candidate for a given $S$, compute $h_1(\\bar\\xi),\\ldots,h_4(\\bar\\xi)$, then evaluate $V(d,\\bar\\xi)$ from Theorem 2 at every integer depth $1\\le d\\le S$. If any calculated $V(d,\\bar\\xi)$ exceeds the number of parameters $p$, the candidate is not D-optimal; if all are at most $p$, the equivalence theorem makes it D-optimal. Reproducing Table 3's maximum-normalized-variance entries for $S=5,\\ldots,12$ is therefore a complete test of the paper's main numerical claim.","supporting_citations":[],"review_version":1}