Pith. sign in

REVIEW 3 major objections 5 minor 39 references

Collaborative Design of Controlled Experiments in the Presence of Subject Covariates

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read When several controlled experiments draw on the same subject pool, designing and analysing them jointly gives more precise treatment-effect estimates than designing and analysing each experiment separately.

desk verdict The core precision-matrix result is solid and the greedy algorithm is sensible; the SDR algorithm's orthogonality claim does not survive its rounding step. read the letter →

arxiv 2412.10213 v1 pith:RO6UWHN7 submitted 2024-12-13 stat.ME

William Fisher , Qiong Zhang , Lulu Kang , Xinwei Deng This is my paper · ORCID
classification stat.ME MSC 62K0562J05
keywords D-optimalitycollaborativedesigncontrolledexperimentssubjectcovariatesmixed-effectsmodelsemi-definiteprogrammingtreatment-effectprecisionorthogonalallocations
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper studies several separate controlled experiments that draw on the same subject pool, where each subject has covariates and a subject-specific random effect that induces dependence across experiments. It derives the precision matrix of the treatment-effect estimates and uses D-optimality to allocate subjects to treatment or control in every experiment. The central claim is that collaboratively designing and analyzing such experiments yields strictly more precise treatment-effect estimates than independent design and analysis, with a best-case D-efficiency of $N/\sigma^2 \cdot (1+b(K-1))/(1+bK)$ versus $N/((1+b)\sigma^2)$; the gain appears as soon as there is more than one experiment and the subject-effect variance is positive. The paper also proposes a greedy SDP-based randomized algorithm and a cheaper single-SDP algorithm, and its simulations show these beat covariate-agnostic allocations when many covariates are present.

What carries the argument

The central object is the precision matrix $P(\hat\beta)$ of the treatment-effect estimators, obtained from the GLS information matrix $X^\top V^{-1}X$ through block-matrix inversion. Proposition 1 writes it as a matrix whose diagonal blocks $Q_j x_j^\top P_{Z^\perp}x_j$ reward covariate balance within an experiment and whose off-diagonal blocks $R_{j,j'}x_j^\top P_{Z^\perp}x_{j'}$ reward orthogonality across experiments. The greedy algorithm uses the Schur-complement determinant identity to split the joint D-optimality problem into $K$ binary quadratic programs, each solved by an SDP relaxation with randomized rounding that carries a $2/\pi$ approximation guarantee; the single-SDP variant solves the covariate-balance relaxation once and generates allocations from $K$ random orthogonal vectors, sacrificing a little precision for much lower computational cost as $K$ grows.

What would settle it

Fix $K=2$, $b>0$, any full-rank covariate matrix $Z$, and enumerate over covariate-balanced allocations with $x_1^\top x_2=0$, computing $\det(P(\hat\beta))^{1/2}$ from Proposition 1; if any such design fails to exceed $N/((1+b)\sigma^2)$ under the equal-variance assumption, the claimed dominance of collaborative design is false. A second, more practical check: simulate with a fraction of subjects missing from one experiment and compare the collaborative and independent designs; a regime where independent design wins would show the complete-participation assumption is carrying the result.

Watch

Extended reading notes

Core claim

The paper models $K$ experiments jointly through a linear mixed-effects model $y_{ij}=\beta_j x_{ij}+z_i^\top\gamma_j+u_i+\epsilon_{ij}$, with $u_i\sim N(0,\tau^2)$ and $\epsilon_{ij}\sim N(0,\sigma_j^2)$. Generalized least squares gives a precision matrix for the estimated treatment effects (Proposition 1) whose diagonal entries are driven by $x_j^\top P_{Z^\perp}x_j$, a covariate-balance measure, and whose off-diagonal entries are driven by $x_j^\top P_{Z^\perp}x_{j'}$, a cross-experiment orthogonality measure. Maximizing the determinant of this precision matrix therefore encourages both covariate balance within each experiment and orthogonality across experiments. In the equal-variance case $\tau^2=b\sigma^2$, the best covariate-balanced and orthogonal design has D-efficiency $N/\sigma^2\cdot (1+b(K-1))/(1+bK)$, which exceeds the independent-analysis efficiency $N/((1+b)\sigma^2)$, and each individual treatment-effect variance is $\sigma^2/N \cdot (1+b)/(1+b(K-1))$, which decreases as $K$ grows; using the same covariate-balanced allocation in all experiments reproduces the independent-analysis variance, showing orthogonality is what creates the gain.

Load-bearing premise

The load-bearing premise is that every subject takes part in every experiment, so the shared subject random effect produces the same block-structured covariance across all $K$ experiments; together with variance components being known before allocation, this is what makes the derived precision matrix and the collaborative advantage valid.

Editorial extensions

If this is right

  • Collaborative allocation gives strictly larger D-efficiency than independent allocation whenever $K>1$ and $b>0$, so research platforms that run many experiments on one subject pool can expect measurable precision gains from joint design.
  • The variance of each experiment's treatment-effect estimate decreases as the number of experiments grows, so collaboration sharpens experiment-by-experiment estimates and not just the joint confidence region.
  • Covariate balance alone is not enough: reusing one balanced allocation across all experiments yields the same per-experiment variance as independent analysis, so orthogonality across experiments is a required design property.
  • The greedy algorithm with SDP rounding and the single-SDP algorithm provide polynomial-time randomized allocations; in the simulated regimes considered, both outperform fully random and factorial covariate-agnostic allocations when covariates are numerous.
  • The gap between the single-SDP and greedy approaches tends to zero as the number of experiments increases, making the cheaper single-SDP method the practical choice for platforms running hundreds of experiments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If subjects are allowed to skip some experiments, the block-covariance structure of $V$ breaks; a testable extension would model experiment-specific participation and ask whether a modified orthogonality-plus-balance criterion still dominates independent design.
  • The optimality criterion assumes $\tau^2$ and $\sigma_j^2$ are known at allocation time; a natural robustness check is to estimate these from historical data or a pilot and measure the D-efficiency loss under misspecification.
  • The single-SDP-plus-orthogonal-rounding scheme is structurally suited to online allocation: solve the covariate-balance SDP once, then assign arriving subjects by projecting their covariate vector onto the stored orthogonal directions, though this extension is not tested here.
  • The $2/\pi$ guarantee applies to each quadratic subproblem, not to the overall D-optimality problem; users should read the approximation ratio as a per-step property unless a global guarantee is established.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper considers the D-optimal allocation of subjects to treatment and control when the same subjects participate in K separate controlled experiments and subject covariates are observed. A linear mixed-effects model with a subject-specific random effect is used, and the GLS precision matrix of the treatment-effect estimates is derived in Proposition 1. The authors show analytically that, under ideal covariate-balanced and mutually orthogonal allocations, collaborative design and analysis yields D-efficiency N/σ² · (1+b(K−1))/(1+bK) and per-experiment variance σ²/N · [1 + b/(1+b(K−1))], both preferable to the corresponding independent-analysis values. They then propose two randomized algorithms: a greedy method that sequentially solves quadratic binary subproblems via SDP relaxation with a 2/π guarantee, and a single-SDP method (SDR) intended to produce orthogonal allocations. A simulation study compares RAND, Plackett-Burman, SDR, and two greedy variants across N=96, K=4 or 8, τ=0.25 or 2, and p=10 or 70 covariates.

Significance. The central precision-matrix calculation is standard GLS algebra and is presented with enough detail to be checked; the closed-form comparison in Eqs. (5), (8), and (9) is a clean, internally consistent result that makes a convincing analytical case for collaborative design when exact covariate balance and cross-experiment orthogonality are achievable. The paper is also honest about its scope: it assumes complete subject participation and known variance components, and it lists relaxation of these assumptions as future work. If the algorithmic claims are repaired, this would be a useful contribution to the design of experiments on shared subject pools, with clear practical motivation from online experimentation platforms. The analytical upper-bound comparison in the simulation is a sensible way to benchmark without knowing the global optimum, and the paper explicitly reports that covariate-agnostic methods degrade as p grows.

major comments (3)
  1. [Section 3.2] The SDR rounding scheme does not deliver the orthogonality that the text claims it delivers. After solving the SDP, the allocation is x_j = sign((W*)^{1/2} v_j) using Eq. (18), where the v_j are mutually orthogonal unit vectors. The sign map is nonlinear and does not preserve orthogonality: for example, with N=3, v1=(1,1,1)/√3 and v2=(1,-2,1)/√6 are orthogonal, but x1=(1,1,1) and x2=(1,-1,1) have inner product 1, not 0. Since the D-efficiency advantage in Eq. (8) requires exact orthogonality and Section 3.2 states that the randomization scheme 'addresses orthogonality across experiments,' this mechanism is unsupported. The simulation does not report the achieved orthogonality of SDR designs or compare SDR against the independent-analysis baseline, so the claimed large-covariate advantage of SDR is not established. A correct rounding scheme or an explicit probabilistic analysis of near-orthogonality is needed.
  2. [Section 3.1, Eqs. (15)-(17)] The proof that the matrix M_j = (1/c)Q_j P⊥ − B(P*)^{-1} B^T is positive semidefinite is not valid as written. The text argues from Sylvester's criterion and from nonnegativity of the scalar Schur complement in Eq. (14), but Eq. (14) only gives x_j^T M_j x_j ≥ 0 for x_j ∈ {−1,1}^N, not for all real vectors, and Sylvester's criterion concerns principal minors of a fixed matrix. The PSD property is nevertheless true: writing the precision matrix as a Hadamard product of the PSD matrix S and the PSD matrix X^T P⊥ X, and replacing x_j by an arbitrary real vector in the Schur-complement argument, proves the required semidefiniteness. The proof should be rewritten because the 2/π performance guarantee in Eq. (17) depends on this property.
  3. [Section 1 and Section 3.2] The statement that 'the gap in precision attained by designing separately and designing collaboratively tends to zero as the number of experiments increases' is contradicted by the paper's own formulas. From Eq. (8), the best-case collaborative D-efficiency tends to N/σ² as K→∞, while the independent-analysis D-efficiency is N/((1+b)σ²), a positive gap. Likewise, the variance in Eq. (9) tends to σ²/N, while the independent-analysis variance is σ²/N(1+b). If the intended comparison is instead between the SDR and greedy algorithms, that claim should be stated precisely and proved. As written, this sentence gives an incorrect motivation for SDR.
minor comments (5)
  1. [Section 2.2.1, Eq. (9)] The second equality in Eq. (9) is a typo: σ²/N · (1+bK)/(1+b(K−1)) equals σ²/N · [1 + b/(1+b(K−1))], not σ²/N · (1+b)/(1+b(K−1)). The subsequent variance comparisons use the correct form, but the displayed equality should be fixed.
  2. [Appendix A] In the proof of Proposition 1, the definitions 'Qj = σ−2 j (c − τ 2σj)' and 'Rj,j′ = −τ 2σjσj′' contain missing inverse exponents; they should read Q_j = σ_j^{-2}(c − τ²σ_j^{-2}) and R_{j,j'} = −τ²σ_j^{-2}σ_{j'}^{-2} to match Proposition 1 and the later working.
  3. [Section 4] The simulation results are reported only for one covariate matrix Z1, with the text stating that results for Z2–Z5 are similar. Since the paper claims stability across covariate matrices, numerical tables or supplementary figures for all five matrices would make this claim verifiable.
  4. [Section 2.1 and Section 5] The model and Proposition 1 assume that every subject participates in every experiment and that τ² and the σ_j² are known before allocation. The paper lists dropout as future work but does not discuss sensitivity to misspecified variance components; a brief remark on this limitation in Section 5 would be useful.
  5. [Section 3.2] The description 'using QR factorization on this uniformly generated vector to generate a set of mutually orthogonal vectors' is imprecise: the QR factorization of a single N-vector produces a full orthonormal basis only after completing the vector to a basis, for example via Householder reflections. The intended construction should be stated more carefully.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the collaborative-versus-independent D-efficiency comparison is derived from the stated mixed-effects model with an external baseline; self-citations are motivational, not load-bearing.

full rationale

The paper's central claim is analytically derived rather than fitted. Proposition 1 derives the treatment-effect precision matrix from GLS under the stated mixed-effects model (1); Section 2.2 then computes the D-efficiency of covariate-balanced collaborative designs under best- and worst-case orthogonality. Equation (8), det(P(β_c))^{1/K}=N/σ²·(1+b(K−1))/(1+bK), is obtained by direct matrix-determinant calculation under the stated assumptions Z^T x_j=0 and x_j^T x_j'=0. The independent-analysis baseline in Section 2.2.2 is a separate model (10) in which the subject random effect is absorbed into the error, giving precision N/((1+b)σ²); it is not fitted from the collaborative model, so the comparison is not self-definitional. No fitted parameter is renamed as a prediction. The simulation upper bound equals Equation (8), but it is independently justified by Hadamard's inequality plus perfect covariate balancing, so benchmarking against it is not circular. The self-citations to Zhang et al. (2024) and other prior work by the same authors are used as motivation and related work, not as the basis of the derivation; the precision-matrix result and D-optimality problem are self-contained. Stated limitations, such as all subjects participating in every experiment and τ²/σ_j² being known, are explicit assumptions rather than hidden circular inputs. Separately, the SDR algorithm's claim in Section 3.2 that its randomization scheme 'addresses orthogonality across experiments' is not established by the rounding step—sign(W^{1/2}v_j) does not generally preserve orthogonality even when v_j are orthogonal—but this is a correctness/validation gap, not a circularity step.

Assumptions & free parameters 1 free parameters · 6 assumptions · 0 invented entities

The central claim rests on a mixed-effects model with known covariance, full participation of all subjects in all experiments, and standard linear-algebra tools. No new entities are introduced. The only user-supplied input is the variance ratio b = τ²/σ², which the design criterion requires.

free parameters (1)
  • b = τ²/σ² (variance ratio) = Varies by simulation setting: 0.0625 (τ=0.25, σ=1) and 4 (τ=2, σ=1); also 0.5, 1, 2 in figures
    The D-optimality criterion and all efficiency comparisons depend on the ratio of random-effect variance to error variance, which must be supplied before allocation. The paper treats it as known and does not estimate it from data.
assumptions (6)
  • domain assumption Mixed-effects model y_ij = β_j x_ij + z_iᵀγ_j + u_i + ε_ij with u_i ~ N(0, τ²) and ε_ij ~ N(0, σ_j²), all independent.
    Imposes the covariance structure used in Eq. (1) and Proposition 1.
  • domain assumption Every subject participates in every experiment.
    Stated in Section 2.1; the block form of V and the Kronecker-product structure of M3 rely on it.
  • domain assumption No treatment-covariate interactions and no cross-experiment treatment interactions.
    Stated in Section 2.1; these restrictions keep the model linear in the treatment effects.
  • domain assumption Variance components τ² and σ_j² are known before allocation.
    GLS precision (Eq. 2) and the D-optimality objective assume known V; misspecification is not addressed.
  • domain assumption Covariate matrix Z is full rank and includes an intercept column.
    Required for the projection P_Z⊥ and for invertibility of ZᵀZ in Proposition 1.
  • standard math Standard linear-algebra tools: Woodbury formula, block-matrix determinant identity, Sylvester's criterion, and the Ostrowski determinant bound.
    Used in the proofs of Proposition 1 and Appendix B.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Collaborative Design of Controlled Experiments in the Presence of Subject Covariates." pith.science (2026). https://pith.science/paper/RO6UWHN7

@misc{pith2026241210213,
  author       = {Pith},
  title        = {Pith review of: Collaborative Design of Controlled Experiments in the Presence of Subject Covariates},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RO6UWHN7}},
  note         = {Machine review of arXiv:2412.10213}
}
read the original abstract

We consider the optimal experimental design problem of allocating subjects to treatment or control when subjects participate in multiple, separate controlled experiments within a short time-frame and subject covariate information is available. Here, in addition to subject covariates, we consider the dependence among the responses coming from the subject's random effect across experiments. In this setting, the goal of the allocation is to provide precise estimates of treatment effects for each experiment. Deriving the precision matrix of the treatment effects and using D-optimality as our allocation criterion, we demonstrate the advantage of collaboratively designing and analyzing multiple experiments over traditional independent design and analysis, and propose two randomized algorithms to provide solutions to the D-optimality problem for collaborative design. The first algorithm decomposes the D-optimality problem into a sequence of subproblems, where each subproblem is a quadratic binary program that can be solved through a semi-definite relaxation based randomized algorithm with performance guarantees. The second algorithm involves solving a single semi-definite program, and randomly generating allocations for each experiment from the solution of this program. We showcase the performance of these algorithms through a simulation study, finding that our algorithms outperform covariate-agnostic methods when there are a large number of covariates.

Figures

Figures reproduced from arXiv: 2412.10213 by the authors.

Figure 1
Figure 1. Normalized D-efficiencies as functions of [PITH_FULL_IMAGE:figures/full_fig_p012_1.png] view at source ↗
Figure 2
Figure 2. Log-transformed normalized variances as functions of [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. This is in agreement with results on the performance of random, covariate-agnostic allocation in [PITH_FULL_IMAGE:figures/full_fig_p021_3.png] view at source ↗
Figures from the paper (2 more)
Figure 3
Figure 3. Figure 3: D-efficiencies of the methods RAND, PB, SDR, GREED-SDP, and GREED-BP across different [PITH_FULL_IMAGE:figures/full_fig_p022_3.png]
Figure 4
Figure 4. Figure 4: Variances of the first and last experiments of the methods RAND, PB, SDR, GREED-SDP, and [PITH_FULL_IMAGE:figures/full_fig_p023_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

39 extracted references · 33 canonical work pages

  1. [1]

    Matrix algebra, volume 1

    Karim M Abadir and Jan R Magnus. Matrix algebra, volume 1. Cambridge University Press, 2005

    2005

  2. [2]

    MOSEK Optimizer API for Python 10.1.31, 2024

    MOSEK ApS. MOSEK Optimizer API for Python 10.1.31, 2024. URL https://docs.mosek.com/latest/pythonapi/index.html

    work page 2024

  3. [3]

    Lectures on modern convex optimization: analysis, algorithms, and engineering applications

    Aharon Ben-Tal and Arkadi Nemirovski. Lectures on modern convex optimization: analysis, algorithms, and engineering applications. SIAM, 2001

    2001

  4. [4]

    The power of optimization over randomization in designing experiments involving small samples

    Dimitris Bertsimas, Mac Johnson, and Nathan Kallus. The power of optimization over randomization in designing experiments involving small samples. Operations Research, 63 0 (4): 0 868--876, 2015

    work page 2015

  5. [5]

    Near-optimal ab testing

    Nikhil Bhat, Vivek F Farias, Ciamac C Moallemi, and Deeksha Sinha. Near-optimal ab testing. Management Science, 66 0 (10): 0 4477--4495, 2020

    work page 2020

  6. [6]

    Convex optimization

    Stephen P Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004

    2004

  7. [7]

    Note on best possible bounds for determinants of matrices close to the identity matrix

    Richard P Brent, H Osborn Judy-anne, and Warren D Smith. Note on best possible bounds for determinants of matrices close to the identity matrix. Linear Algebra and its Applications, 466: 0 21--26, 2015

    work page 2015

  8. [8]

    Plane answers to complex questions: the theory of linear models

    Ronald Christensen. Plane answers to complex questions: the theory of linear models. Springer, 2002

    work page 2002

Show all 39 references
  1. [9]

    Maximum cut problem, max-cut

    Clayton W Commander. Maximum cut problem, max-cut. Encyclopedia of Optimization, 2, 2009

    2009

  2. [10]

    An introduction to generalized linear models

    Annette J Dobson and Adrian G Barnett. An introduction to generalized linear models. Chapman and Hall/CRC, 2018

    2018

  3. [11]

    Randomization and factorial experiments

    S Ehrenfeld and So Zacks. Randomization and factorial experiments. The Annals of Mathematical Statistics, 32 0 (1): 0 270--297, 1961

    1961

  4. [12]

    Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming

    Michel X Goemans and David P Williamson. Improved approximation algorithms for maximum cut and satisfiability problems using semidefinite programming. Journal of the ACM (JACM), 42 0 (6): 0 1115--1145, 1995

    1995

  5. [13]

    Gurobi Optimizer Reference Manual , 2023

    Gurobi Optimization, LLC . Gurobi Optimizer Reference Manual , 2023. URL https://www.gurobi.com

    2023

  6. [14]

    Accuracy and stability of numerical algorithms

    Nicholas J Higham. Accuracy and stability of numerical algorithms. SIAM, 2002

    2002

  7. [15]

    Covariate balancing propensity score

    Kosuke Imai and Marc Ratkovic. Covariate balancing propensity score. Journal of the Royal Statistical Society Series B: Statistical Methodology, 76 0 (1): 0 243--263, 2014

    2014

  8. [16]

    Online controlled experiments at large scale

    Ron Kohavi, Alex Deng, Brian Frasca, Toby Walker, Ya Xu, and Nils Pohlmann. Online controlled experiments at large scale. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 1168--1176, 2013

    2013

  9. [17]

    Experimentation and start-up performance: Evidence from a/b testing

    Rembrand Koning, Sharique Hasan, and Aaron Chatterji. Experimentation and start-up performance: Evidence from a/b testing. Management Science, 68 0 (9): 0 6434--6453, 2022

    2022

  10. [18]

    Statistical challenges in online controlled experiments: A review of a/b testing methodology

    Nicholas Larsen, Jonathan Stallrich, Srijan Sengupta, Alex Deng, Ron Kohavi, and Nathaniel T Stevens. Statistical challenges in online controlled experiments: A review of a/b testing methodology. The American Statistician, 78 0 (2): 0 135--149, 2024

    2024

  11. [19]

    Balancing covariates via propensity score weighting

    Fan Li, Kari Lock Morgan, and Alan M Zaslavsky. Balancing covariates via propensity score weighting. Journal of the American Statistical Association, 113 0 (521): 0 390--400, 2018

    2018

  12. [20]

    Optimal design of controlled experiments for personalized decision making in the presence of observational covariates

    Yezhuo Li, Qiong Zhang, Amin Khademi, and Boshi Yang. Optimal design of controlled experiments for personalized decision making in the presence of observational covariates. The New England Journal of Statistics in Data Science, 2023

    2023

  13. [21]

    Efficient d\_s -optimal designs for multivariate polynomial regression on the q -cube

    Yong B Lim and WJ Studden. Efficient d\_s -optimal designs for multivariate polynomial regression on the q -cube. The Annals of Statistics, 16 0 (3): 0 1225--1240, 1988

    1988

  14. [22]

    A pragmatic randomized controlled trial reports lack of efficacy of hydroxychloroquine on coronavirus disease 2019 viral kinetics

    Magnus Nakrem Lyngbakken, Jan-Erik Berdal, Arne Eskesen, Dag Kvale, Inge Christoffer Olsen, Corina Silvia Rueegg, Anbj rg Rangberg, Christine Monceyron Jonassen, Torbj rn Omland, Helge R sj , et al. A pragmatic randomized controlled trial reports lack of efficacy of hydroxychl...

    2019

  15. [23]

    On the design of experiments under spatial correlation

    RJ Martin. On the design of experiments under spatial correlation. Biometrika, 73 0 (2): 0 247--277, 1986

    1986

  16. [24]

    Rerandomization to improve covariate balance in experiments

    Kari Lock Morgan and Donald B Rubin. Rerandomization to improve covariate balance in experiments. The Annals of Statistics, 2012

    2012

  17. [25]

    A note on a method for generating points uniformly on n-dimensional spheres

    Mervin E Muller. A note on a method for generating points uniformly on n-dimensional spheres. Communications of the ACM, 2 0 (4): 0 19--20, 1959

    1959

  18. [26]

    Modern Control Engineering

    Katsuhiko Ogata. Modern Control Engineering. Prentice Hall, 2010

    2010

  19. [27]

    The assessment of learning infrastructure (ali) the theory, practice, and scalability of automated assessment

    Korinn S Ostrow, Doug Selent, Yan Wang, Eric G Van Inwegen, Neil T Heffernan, and Joseph Jay Williams. The assessment of learning infrastructure (ali) the theory, practice, and scalability of automated assessment. In Proceedings of the sixth international conference on learnin...

    2016

  20. [28]

    Sur l'approximation du d \'e terminant de Fredholm par les d \'e terminants des syst \`e mes d'equations lin \'e aires

    Alexander M Ostrowski. Sur l'approximation du d \'e terminant de Fredholm par les d \'e terminants des syst \`e mes d'equations lin \'e aires . Almqvist & Wiksell, 1938

    1938

  21. [29]

    The design of optimum multifactorial experiments

    Robin L Plackett and J Peter Burman. The design of optimum multifactorial experiments. Biometrika, 33 0 (4): 0 305--325, 1946

    1946

  22. [30]

    The principal minor test for semidefinite matrices

    John E Prussing. The principal minor test for semidefinite matrices. Journal of Guidance, Control, and Dynamics, 9 0 (1): 0 121--122, 1986

    1986

  23. [31]

    Optimal design of experiments

    Friedrich Pukelsheim. Optimal design of experiments. SIAM, 2006

    2006

  24. [32]

    More subtle versions of the hadamard inequality

    Micha R \'o \.z a \'n ski, Roman Witu a, and Edyta Hetmaniok. More subtle versions of the hadamard inequality. Linear Algebra and its Applications, 532: 0 500--511, 2017

    2017

  25. [33]

    How experience of use influences mass-market drivers’ willingness to consider a battery electric vehicle: A randomised controlled trial

    Stephen M Skippon, Neale Kinnear, Louise Lloyd, and Jenny Stannard. How experience of use influences mass-market drivers’ willingness to consider a battery electric vehicle: A randomised controlled trial. Transportation Research Part A: Policy and Practice, 92: 0 26--42, 2016

    2016

  26. [34]

    Numerical linear algebra

    Lloyd N Trefethen and David Bau. Numerical linear algebra. SIAM, 2022

    2022

  27. [35]

    From infrastructure to culture: A/b testing challenges in large scale social networks

    Ya Xu, Nanyu Chen, Addrian Fernandez, Omar Sinno, and Anmol Bhasin. From infrastructure to culture: A/b testing challenges in large scale social networks. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pages 2227--2236, 2015

    2015

  28. [36]

    Randomization and experimentation

    WJ Youden. Randomization and experimentation. Technometrics, 14 0 (1): 0 13--22, 1972

    1972

  29. [37]

    Locally optimal design for a/b tests in the presence of covariates and network dependence

    Qiong Zhang and Lulu Kang. Locally optimal design for a/b tests in the presence of covariates and network dependence. Technometrics, 64 0 (3): 0 358--369, 2022

    2022

  30. [38]

    Min-max optimal design of two-armed trials with side information

    Qiong Zhang, Amin Khademi, and Yongjia Song. Min-max optimal design of two-armed trials with side information. INFORMS Journal on Computing, 34 0 (1): 0 165--182, 2022

    2022

  31. [39]

    Collaborative analysis for paired a/b testing experiments

    Qiong Zhang, Lulu Kang, and Xinwei Deng. Collaborative analysis for paired a/b testing experiments. arXiv preprint arXiv:2407.05400, 2024

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.