{"id":"1694b63e-eb6d-4a21-b97e-32b6d3ebdd60","arxiv_id":"2412.10213","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"For multiple experiments sharing subjects, the authors derive the treatment-effect precision matrix under a mixed-effects model and propose greedy and single-SDP randomized algorithms that balance covariates and orthogonalize allocations across experiments.","lead":"This paper proposes methods for jointly designing several controlled experiments when the same subjects take part in all of them and subject characteristics are known. The authors show that designing experiments together, rather than separately, can give more precise treatment-effect estimates, and they provide two randomized algorithms to construct such designs.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SDR algorithm's claimed cross-experiment orthogonality is not delivered by its rounding scheme; sign(W^{1/2}v_j) vectors are not orthogonal when v_j are, so the scalable algorithm may not solve the stated D-optimality problem.","rationale":"The reader identified full participation and known variance components as the weakest assumptions. I agree those are important, but full participation is explicitly stated and acknowledged as future work, and known variance components are standard in locally optimal design. The more load-bearing concern is the SDR algorithm's orthogonality mechanism, which the reader flagged as heuristic. This is concrete: the rounding step does not preserve orthogonality of the continuous random vectors, so the algorithm is unlikely to deliver the cross-experiment orthogonality that the D-optimality criterion requires for the performance advantage. The central theoretical result (Eq. 8 vs. independent analysis) remains mathematically correct under ideal designs, and the greedy algorithm provides a valid, if computationally heavier, route. Therefore the paper should be accepted with conditions: either prove or correct the SDR orthogonality mechanism, or weaken the claims made for SDR. The reader's conditional verdict is appropriate, but the specific reason should emphasize the SDR flaw rather than the full-participation assumption alone.","tokens_in":20155,"tokens_out":14998,"duration_ms":138778,"concrete_test":"Run a small simulation with N=96, K=8, W*=I (or the SDP solution from Section 3.2). Generate 1000 random orthonormal vector sets via QR, form allocations x_j = sign((W*)^{1/2} v_j), and compute the empirical distribution of |x_j^T x_{j'}|/N for j≠j'. If the mean absolute inner product is not close to 0, and is no better than using independent random vectors, the orthogonality claim in Section 3.2 fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 3.2 claims that SDR's randomization scheme 'addresses orthogonality across experiments' by drawing K orthogonal vectors v_j and rounding them to allocations. However, the rounding x_j = sign((W*)^{1/2} v_j) does not preserve orthogonality: even if v_j and v_k are orthogonal, the sign vectors of their projections are not generally orthogonal. For example, with N=3, v1=(1,1,1)/sqrt(3) and v2=(1,-2,1)/sqrt(6) are orthogonal, but x1=(1,1,1) and x2=(1,-1,1) have inner product 1, not 0. Thus the mechanism proposed to achieve the orthogonality that is essential for the D-efficiency advantage in Eq. (8) is unsupported and likely ineffective. The greedy algorithm does enforce orthogonality through its sequential quadratic form, so the central theoretical comparison is not invalidated, but the second algorithmic contribution lacks a valid justification. The D-optimality objective (3) rewards exact orthogonality, and the simulation does not directly compare SDR against the independent-analysis baseline, so the claimed practical advantage in the large-covariate regime rests on an unverified mechanism.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper considers the D-optimal allocation of subjects to treatment and control when the same subjects participate in K separate controlled experiments and subject covariates are observed. A linear mixed-effects model with a subject-specific random effect is used, and the GLS precision matrix of the treatment-effect estimates is derived in Proposition 1. The authors show analytically that, under ideal covariate-balanced and mutually orthogonal allocations, collaborative design and analysis yields D-efficiency N/σ² · (1+b(K−1))/(1+bK) and per-experiment variance σ²/N · [1 + b/(1+b(K−1))], both preferable to the corresponding independent-analysis values. They then propose two randomized algorithms: a greedy method that sequentially solves quadratic binary subproblems via SDP relaxation with a 2/π guarantee, and a single-SDP method (SDR) intended to produce orthogonal allocations. A simulation study compares RAND, Plackett-Burman, SDR, and two greedy variants across N=96, K=4 or 8, τ=0.25 or 2, and p=10 or 70 covariates.","tokens_in":20409,"tokens_out":19336,"duration_ms":194101,"significance":"The central precision-matrix calculation is standard GLS algebra and is presented with enough detail to be checked; the closed-form comparison in Eqs. (5), (8), and (9) is a clean, internally consistent result that makes a convincing analytical case for collaborative design when exact covariate balance and cross-experiment orthogonality are achievable. The paper is also honest about its scope: it assumes complete subject participation and known variance components, and it lists relaxation of these assumptions as future work. If the algorithmic claims are repaired, this would be a useful contribution to the design of experiments on shared subject pools, with clear practical motivation from online experimentation platforms. The analytical upper-bound comparison in the simulation is a sensible way to benchmark without knowing the global optimum, and the paper explicitly reports that covariate-agnostic methods degrade as p grows.","major_comments":[{"comment":"The SDR rounding scheme does not deliver the orthogonality that the text claims it delivers. After solving the SDP, the allocation is x_j = sign((W*)^{1/2} v_j) using Eq. (18), where the v_j are mutually orthogonal unit vectors. The sign map is nonlinear and does not preserve orthogonality: for example, with N=3, v1=(1,1,1)/√3 and v2=(1,-2,1)/√6 are orthogonal, but x1=(1,1,1) and x2=(1,-1,1) have inner product 1, not 0. Since the D-efficiency advantage in Eq. (8) requires exact orthogonality and Section 3.2 states that the randomization scheme 'addresses orthogonality across experiments,' this mechanism is unsupported. The simulation does not report the achieved orthogonality of SDR designs or compare SDR against the independent-analysis baseline, so the claimed large-covariate advantage of SDR is not established. A correct rounding scheme or an explicit probabilistic analysis of near-orthogonality is needed.","section":"Section 3.2"},{"comment":"The proof that the matrix M_j = (1/c)Q_j P⊥ − B(P*)^{-1} B^T is positive semidefinite is not valid as written. The text argues from Sylvester's criterion and from nonnegativity of the scalar Schur complement in Eq. (14), but Eq. (14) only gives x_j^T M_j x_j ≥ 0 for x_j ∈ {−1,1}^N, not for all real vectors, and Sylvester's criterion concerns principal minors of a fixed matrix. The PSD property is nevertheless true: writing the precision matrix as a Hadamard product of the PSD matrix S and the PSD matrix X^T P⊥ X, and replacing x_j by an arbitrary real vector in the Schur-complement argument, proves the required semidefiniteness. The proof should be rewritten because the 2/π performance guarantee in Eq. (17) depends on this property.","section":"Section 3.1, Eqs. (15)-(17)"},{"comment":"The statement that 'the gap in precision attained by designing separately and designing collaboratively tends to zero as the number of experiments increases' is contradicted by the paper's own formulas. From Eq. (8), the best-case collaborative D-efficiency tends to N/σ² as K→∞, while the independent-analysis D-efficiency is N/((1+b)σ²), a positive gap. Likewise, the variance in Eq. (9) tends to σ²/N, while the independent-analysis variance is σ²/N(1+b). If the intended comparison is instead between the SDR and greedy algorithms, that claim should be stated precisely and proved. As written, this sentence gives an incorrect motivation for SDR.","section":"Section 1 and Section 3.2"}],"minor_comments":[{"comment":"The second equality in Eq. (9) is a typo: σ²/N · (1+bK)/(1+b(K−1)) equals σ²/N · [1 + b/(1+b(K−1))], not σ²/N · (1+b)/(1+b(K−1)). The subsequent variance comparisons use the correct form, but the displayed equality should be fixed.","section":"Section 2.2.1, Eq. (9)"},{"comment":"In the proof of Proposition 1, the definitions 'Qj = σ−2 j (c − τ 2σj)' and 'Rj,j′ = −τ 2σjσj′' contain missing inverse exponents; they should read Q_j = σ_j^{-2}(c − τ²σ_j^{-2}) and R_{j,j'} = −τ²σ_j^{-2}σ_{j'}^{-2} to match Proposition 1 and the later working.","section":"Appendix A"},{"comment":"The simulation results are reported only for one covariate matrix Z1, with the text stating that results for Z2–Z5 are similar. Since the paper claims stability across covariate matrices, numerical tables or supplementary figures for all five matrices would make this claim verifiable.","section":"Section 4"},{"comment":"The model and Proposition 1 assume that every subject participates in every experiment and that τ² and the σ_j² are known before allocation. The paper lists dropout as future work but does not discuss sensitivity to misspecified variance components; a brief remark on this limitation in Section 5 would be useful.","section":"Section 2.1 and Section 5"},{"comment":"The description 'using QR factorization on this uniformly generated vector to generate a set of mutually orthogonal vectors' is imprecise: the QR factorization of a single N-vector produces a full orthonormal basis only after completing the vector to a basis, for example via Householder reflections. The intended construction should be stated more carefully.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The analytical core of the paper is sound and worth publishing after revision. The main obstacle is the SDR algorithm's invalid orthogonality mechanism and the incorrect motivating statement about the gap tending to zero; the greedy algorithm's PSD proof also needs repair, though the underlying claim appears true. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core theoretical contribution here is real. Proposition 1 gives a clean precision matrix for treatment effects when K experiments share a subject pool with random effects, and Section 2.2 correctly shows collaborative design can beat independent design when allocations are covariate-balanced and orthogonal. Eq. (8) is derived, not hand-waved, and the worst-case lower bound in Appendix B is a nice addition. This is the first paper I know to treat collaborative design with covariates and a random-effects dependence structure. The greedy algorithm is sensible. Credit where due: the algebra is standard GLS and block-matrix work, internally consistent, and the simulation supports the claim that covariate-aware methods beat covariate-agnostic ones when covariate count is high. The soft spots are real, though not uniform. The big one: Section 3.2's SDR algorithm. The stress-test concern lands. Generating orthogonal v_j and rounding to x_j = sign((W*)^{1/2} v_j) does not preserve orthogonality. The N=3 example is enough to show the mechanism fails, and this matters because the D-efficiency gain in Eq. (8) explicitly rewards cross-experiment orthogonality. The greedy algorithm enforces orthogonality sequentially, so the central theoretical comparison is not invalidated, but the SDR algorithm's claimed mechanism is unsupported. The paper should either prove a different rounding scheme, add post-hoc orthogonalization, or present evidence that sign-rounded vectors are nearly orthogonal on average; also compare SDR against the independent-analysis baseline directly. Two smaller issues: variance components tau^2 and sigma_j^2 are assumed known, with no misspecification discussion; and the paper assumes full participation in all experiments, acknowledged as future work but a real limitation for platforms. No code is shipped, which hurts reproducibility for a methods paper. The Appendix A typos are minor but should be fixed. Who is this for? Researchers in optimal experimental design and statisticians working on A/B testing platforms will get value from the precision-matrix result and the greedy algorithm. The SDR part needs work before I would trust it. It deserves serious peer review, but a revision should address the SDR gap and provide code.","headline":"The core precision-matrix result is solid and the greedy algorithm is sensible; the SDR algorithm's orthogonality claim does not survive its rounding step.","tokens_in":726,"tokens_out":1499,"would_cite":true,"duration_ms":37467,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62K05","62J05"],"pacs":[],"model":"deepseek-v4-flash","headline":"When several controlled experiments draw on the same subject pool, designing and analysing them jointly gives more precise treatment-effect estimates than designing and analysing each experiment separately.","keywords":["D-optimality","collaborative design","controlled experiments","subject covariates","mixed-effects model","semi-definite programming","treatment-effect precision","orthogonal allocations"],"falsifier":"Fix $K=2$, $b>0$, any full-rank covariate matrix $Z$, and enumerate over covariate-balanced allocations with $x_1^\\top x_2=0$, computing $\\det(P(\\hat\\beta))^{1/2}$ from Proposition 1; if any such design fails to exceed $N/((1+b)\\sigma^2)$ under the equal-variance assumption, the claimed dominance of collaborative design is false. A second, more practical check: simulate with a fraction of subjects missing from one experiment and compare the collaborative and independent designs; a regime where independent design wins would show the complete-participation assumption is carrying the result.","tokens_in":19962,"feed_emoji":"🧪","tokens_out":8726,"duration_ms":79563,"temperature":0.7,"pith_summary":"The paper studies several separate controlled experiments that draw on the same subject pool, where each subject has covariates and a subject-specific random effect that induces dependence across experiments. It derives the precision matrix of the treatment-effect estimates and uses D-optimality to allocate subjects to treatment or control in every experiment. The central claim is that collaboratively designing and analyzing such experiments yields strictly more precise treatment-effect estimates than independent design and analysis, with a best-case D-efficiency of $N/\\sigma^2 \\cdot (1+b(K-1))/(1+bK)$ versus $N/((1+b)\\sigma^2)$; the gain appears as soon as there is more than one experiment and the subject-effect variance is positive. The paper also proposes a greedy SDP-based randomized algorithm and a cheaper single-SDP algorithm, and its simulations show these beat covariate-agnostic allocations when many covariates are present.","feed_headline":"Shared-subject experiments gain precision when designed jointly","feed_subtitle":"D-optimality shows precision grows with the number of experiments when allocations balance covariates and stay orthogonal.","key_machinery":"The central object is the precision matrix $P(\\hat\\beta)$ of the treatment-effect estimators, obtained from the GLS information matrix $X^\\top V^{-1}X$ through block-matrix inversion. Proposition 1 writes it as a matrix whose diagonal blocks $Q_j x_j^\\top P_{Z^\\perp}x_j$ reward covariate balance within an experiment and whose off-diagonal blocks $R_{j,j'}x_j^\\top P_{Z^\\perp}x_{j'}$ reward orthogonality across experiments. The greedy algorithm uses the Schur-complement determinant identity to split the joint D-optimality problem into $K$ binary quadratic programs, each solved by an SDP relaxation with randomized rounding that carries a $2/\\pi$ approximation guarantee; the single-SDP variant solves the covariate-balance relaxation once and generates allocations from $K$ random orthogonal vectors, sacrificing a little precision for much lower computational cost as $K$ grows.","core_discovery":"The paper models $K$ experiments jointly through a linear mixed-effects model $y_{ij}=\\beta_j x_{ij}+z_i^\\top\\gamma_j+u_i+\\epsilon_{ij}$, with $u_i\\sim N(0,\\tau^2)$ and $\\epsilon_{ij}\\sim N(0,\\sigma_j^2)$. Generalized least squares gives a precision matrix for the estimated treatment effects (Proposition 1) whose diagonal entries are driven by $x_j^\\top P_{Z^\\perp}x_j$, a covariate-balance measure, and whose off-diagonal entries are driven by $x_j^\\top P_{Z^\\perp}x_{j'}$, a cross-experiment orthogonality measure. Maximizing the determinant of this precision matrix therefore encourages both covariate balance within each experiment and orthogonality across experiments. In the equal-variance case $\\tau^2=b\\sigma^2$, the best covariate-balanced and orthogonal design has D-efficiency $N/\\sigma^2\\cdot (1+b(K-1))/(1+bK)$, which exceeds the independent-analysis efficiency $N/((1+b)\\sigma^2)$, and each individual treatment-effect variance is $\\sigma^2/N \\cdot (1+b)/(1+b(K-1))$, which decreases as $K$ grows; using the same covariate-balanced allocation in all experiments reproduces the independent-analysis variance, showing orthogonality is what creates the gain.","pith_inferences":["If subjects are allowed to skip some experiments, the block-covariance structure of $V$ breaks; a testable extension would model experiment-specific participation and ask whether a modified orthogonality-plus-balance criterion still dominates independent design.","The optimality criterion assumes $\\tau^2$ and $\\sigma_j^2$ are known at allocation time; a natural robustness check is to estimate these from historical data or a pilot and measure the D-efficiency loss under misspecification.","The single-SDP-plus-orthogonal-rounding scheme is structurally suited to online allocation: solve the covariate-balance SDP once, then assign arriving subjects by projecting their covariate vector onto the stored orthogonal directions, though this extension is not tested here.","The $2/\\pi$ guarantee applies to each quadratic subproblem, not to the overall D-optimality problem; users should read the approximation ratio as a per-step property unless a global guarantee is established."],"forward_implications":["Collaborative allocation gives strictly larger D-efficiency than independent allocation whenever $K>1$ and $b>0$, so research platforms that run many experiments on one subject pool can expect measurable precision gains from joint design.","The variance of each experiment's treatment-effect estimate decreases as the number of experiments grows, so collaboration sharpens experiment-by-experiment estimates and not just the joint confidence region.","Covariate balance alone is not enough: reusing one balanced allocation across all experiments yields the same per-experiment variance as independent analysis, so orthogonality across experiments is a required design property.","The greedy algorithm with SDP rounding and the single-SDP algorithm provide polynomial-time randomized allocations; in the simulated regimes considered, both outperform fully random and factorial covariate-agnostic allocations when covariates are numerous.","The gap between the single-SDP and greedy approaches tends to zero as the number of experiments increases, making the cheaper single-SDP method the practical choice for platforms running hundreds of experiments."],"supporting_citations":[{"why":"Provides the paired-experiment collaborative analysis framework this paper extends to the design phase with covariates.","marker":"Zhang et al. (2024)"},{"why":"Supplies D-optimality and D-efficiency as the design criterion used to formulate the allocation problem.","marker":"Pukelsheim (2006)"},{"why":"Gives the SDP-relaxation randomized rounding algorithm with a 2/π approximation guarantee used for each greedy subproblem.","marker":"Ben-Tal and Nemirovski (2001)"},{"why":"Origin of the semidefinite relaxation and randomized rounding technique that the subproblem solver relies on.","marker":"Goemans and Williamson (1995)"},{"why":"Introduces the single-experiment SDP relaxation approach and motivates the large-covariate simulation regime where covariate-aware allocations matter.","marker":"Bhat et al. (2020)"},{"why":"Provides the Plackett–Burman factorial designs used as a covariate-agnostic baseline in the simulations.","marker":"Plackett and Burman (1946)"},{"why":"Supplies the Hadamard-inequality based upper bound used to gauge how close the algorithms come to the optimal D-efficiency.","marker":"Różański et al. (2017)"}],"fun_headline_variants":["Joint design beats independent trials for shared subjects","Collaborative experiments gain precision over separate ones","Design experiments together for sharper treatment effects","Shared-subject design improves precision across trials"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every subject takes part in every experiment, so the shared subject random effect produces the same block-structured covariance across all $K$ experiments; together with variance components being known before allocation, this is what makes the derived precision matrix and the collaborative advantage valid.","fun_headline_variants_meta":{"raw":{"variants":["Joint design beats independent trials for shared subjects","Collaborative experiments gain precision over separate ones","Design experiments together for sharper treatment effects","Shared-subject design improves precision across trials"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000292,"raw_usage":{"total_tokens":1750,"prompt_tokens":1038,"completion_tokens":712,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":654,"completion_tokens_details":{"reasoning_tokens":658}},"tokens_in":654,"tokens_out":712,"duration_ms":8025,"temperature":1.0,"reasoning_tokens":658,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:13:22.457261+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Fix $K=2$, $b>0$, any full-rank covariate matrix $Z$, and enumerate over covariate-balanced allocations with $x_1^\\top x_2=0$, computing $\\det(P(\\hat\\beta))^{1/2}$ from Proposition 1; if any such design fails to exceed $N/((1+b)\\sigma^2)$ under the equal-variance assumption, the claimed dominance of collaborative design is false. A second, more practical check: simulate with a fraction of subjects missing from one experiment and compare the collaborative and independent designs; a regime where independent design wins would show the complete-participation assumption is carrying the result.","supporting_citations":[{"cited_title":"Collaborative Analysis for Paired A/B Testing Experiments","cited_arxiv_id":"2407.05400","evidence_quote":"Provides the paired-experiment collaborative analysis framework this paper extends to the design phase with covariates."},{"cited_title":"Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming","cited_arxiv_id":null,"evidence_quote":"Origin of the semidefinite relaxation and randomized rounding technique that the subproblem solver relies on."},{"cited_title":"Near-optimal ab testing","cited_arxiv_id":null,"evidence_quote":"Introduces the single-experiment SDP relaxation approach and motivates the large-covariate simulation regime where covariate-aware allocations matter."},{"cited_title":"The design of optimum multifactorial experiments","cited_arxiv_id":null,"evidence_quote":"Provides the Plackett–Burman factorial designs used as a covariate-agnostic baseline in the simulations."}],"review_version":1}