REVIEW 3 major objections 4 minor 26 references
Repeated sampling of different individuals within the same clusters to improve precision of longitudinal estimators: the DISC design
T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A study design that samples the same clusters but different individuals at each wave produces difference-in-differences estimators with substantially lower variance than repeated cross-sectional sampling, with the gain captured by an…
desk verdict A correct, practically useful variance ratio for the DISC design; the main claim holds up, with the expected caveats about time-invariant cluster effects. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the variance identity comparing $\hat{\delta}_{\mathrm{RCS}}$ and $\hat{\delta}_{\mathrm{DISC}}$ under the model above. Because the same clusters appear at both time points under DISC, the cluster-level term $\alpha_i$ enters the before-after difference within each cluster as $\alpha_i - \alpha_i = 0$, leaving only individual noise; this cancellation yields $\operatorname{Var}(\hat{\delta}_{\mathrm{DISC}}) = 8\sigma^2/n$, while RCS retains both noise and cluster variation, giving $\operatorname{Var}(\hat{\delta}_{\mathrm{RCS}}) = 8(m\tau^2+\sigma^2)/n$. The ratio $1 + m\rho/(1-\rho)$, with $\rho = \tau^2/(\tau^2+\sigma^2)$, then converts the identity into a sample-size planning statement.
What would settle it
Simulate data with a cluster-by-time interaction, for example $\alpha_{it} \sim \mathrm{N}(0, \tau^2)$ with $\mathrm{Corr}(\alpha_{i1}, \alpha_{i2}) = 0.8$ under the model used in Section 3, and compare the empirical variances of $\hat{\delta}_{\mathrm{DISC}}$ and $\hat{\delta}_{\mathrm{RCS}}$; if the DISC variance rises above $8\sigma^2/n$ and the ratio falls below $1 + m\rho/(1-\rho)$, the time-invariant cluster-effect assumption is doing the work. The same test can be run with clusters entering and leaving the frame between waves.
Extended reading notes
Core claim
The central discovery is an exact variance comparison. Under the generative model $Y_{ijk} = \mu_j + \delta X_{ij} + \alpha_i + \epsilon_{ijk}$ with i.i.d. cluster effects $\alpha_i \sim \mathrm{N}(0, \tau^2)$ and individual noise $\epsilon_{ijk} \sim \mathrm{N}(0, \sigma^2)$, the simple DID estimator has variance $8(m\tau^2+\sigma^2)/n$ under RCS and $8\sigma^2/n$ under DISC, so the RCS variance is $1 + m\rho/(1-\rho)$ times larger. Simulations reproduce the formula, and a simulation with a doubly-robust DID estimator that adjusts for covariates shows the precision gain persists with covariate adjustment. The authors present the design as a practical middle ground: sample the same clusters each wave, take a different sample of individuals within each cluster, and analyze with existing longitudinal estimators.
Load-bearing premise
The load-bearing premise is that each cluster's effect is a single time-invariant additive term that is identical at both surveys and independent of treatment and sampling, so the entire DISC variance gain is the cancellation of that term — a premise that fails if cluster effects drift, clusters change composition, or new clusters enter the population.
Editorial extensions
If this is right
- With 40 clusters and 25 individuals per cluster, the repeated cross-sectional variance is 2.3 times larger than DISC at an intraclass correlation of 0.05, 3.8 times larger at 0.1, and 7.3 times larger at 0.2.
- Since variance is inversely proportional to sample size, a study powered at 90 percent could be run with roughly half the sample at an ICC of 0.05, and roughly one quarter of the sample at an ICC of 0.1, if DISC replaces RCS.
- The precision gain holds for a doubly-robust DID estimator that uses covariate information, not just for the simple unadjusted linear estimator, as shown in simulation.
- By a simple modification of the Section 3 variance argument, the same variance ratio applies to uncontrolled before-and-after comparisons, and the design transfers to cluster randomized trials, stepped wedge trials, and interrupted time series designs.
Reading between the lines
- Beyond the paper: the variance gain is bought at the price of wave-by-wave representativeness; when clusters enter, dissolve, or change composition during the study, the fixed cluster list describes the subpopulation present across the whole study, and the DISC estimand is cleaner for treatment effects but not for population-level change.
- Beyond the paper: a testable extension is the three-level sampling case using the paper's S-S-D and S-D-D notation, where one would predict an intermediate variance reduction whose size depends on which level's cluster effect cancels.
- Beyond the paper: because clusters are fixed across waves, cluster-level covariates can be measured once, so a natural comparison is whether analysis-phase adjustment for cluster covariates can recover part of the DISC gain without repeating cluster samples.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This manuscript proposes the DISC design, in which the same clusters are sampled at each wave but different individuals are drawn within clusters, and studies it in a two-period difference-in-differences setting. Under the linear mixed model in Eq. (3), the authors derive Var(δ̂_RCS)=8(mτ²+σ²)/n and Var(δ̂_DISC)=8σ²/n, yielding a variance ratio of 1 + mρ/(1−ρ). A simulation study with 1,000 replicates confirms the variance formulas for the simple estimator and shows similar gains for a doubly-robust DID estimator that adjusts for covariates. The paper also discusses extensions to other longitudinal designs. The mathematical derivation is correct under the stated assumptions, and the simulation results support the theory, although the simulation text and figure captions disagree on the ICC, and an illustration promised in the abstract is absent from the body.
Significance. If the result holds, the DISC design provides a practically meaningful precision improvement for cluster-sampled longitudinal and DID studies without requiring follow-up of the same individuals, and the analytic variance ratio is simple enough to be used in power calculations. The derivation is explicit and self-contained: the ICC enters only as a reparameterization of τ² and σ², and the simulation checks the formulas rather than fitting them. The authors also disclose the design's limitation when cluster populations change and connect the work to existing practice in cluster randomized trials. The contribution is incremental but useful for applied researchers planning cluster-based longitudinal studies.
major comments (3)
- [Section 4, Figures 3 and 4] The simulation text states 'THe value of τ was chosen to yield an ICC of 0.2,' but the captions of Figures 3 and 4 both state an ICC of 0.1. Since Figure 3 is presented as confirming the analytic variance formulas, the simulation parameter must be unambiguous; please correct the inconsistency and report the Monte Carlo standard errors for the simulated variances so the visual agreement can be assessed.
- [Abstract and body] The abstract's Results paragraph says 'we illustrate DISC sampling using a household survey dataset from South Africa,' but no such illustration appears in Sections 1-5 or Appendix B of the manuscript. As written, the abstract promises content the paper does not deliver; either add the application or remove the claim.
- [Section 3 and Discussion] The variance reduction in Eq. (5) relies entirely on the time-invariant cluster effect α_i in model (3) canceling when the same clusters are differenced. The Discussion's limitation paragraph mentions changes in the number or size of clusters but does not mention the equally important case where cluster-level effects drift over time; if α_i is replaced by α_i + γ_it with time-varying γ_it, the DISC variance gains an additional term and the advantage shrinks. Please add an explicit caveat that the claimed precision gain assumes stable cluster effects, not just stable cluster populations.
minor comments (4)
- [Section 4] The text contains a typo: 'THe value of τ' should read 'The value of τ'.
- [Figures 3 and 4] With 1,000 simulation replicates, the relative standard error of a variance estimate is roughly √(2/999) ≈ 4.5%; adding Monte Carlo standard errors or confidence bands around the simulated variance points would make the agreement with the analytic curves more interpretable.
- [Figure 2] The y-axis label 'Total variance' is used after scaling Var(δ̂_DISC)=1, so the plotted quantity is actually the variance ratio; please relabel the axis accordingly.
- [Section 5] The statement that the variance ratio increases as the number of clusters decreases would benefit from a brief reference back to the formula 1 + mρ/(1−ρ) with n = mC, making the dependence on C immediate.
Circularity Check
No significant circularity: the variance comparison follows algebraically from the stated generative model, with ICC entering only as a reparameterization and simulation providing an independent check.
full rationale
The paper's central derivation is self-contained. The estimator in Eq. (2) is a linear combination of cluster-level sums, and the generative model in Eq. (3) specifies Yijk = µj + δXij + αi + ϵijk with αi and ϵijk independent, zero-mean variance components. Appendix B then computes Var(δ̂_RCS) = 8(mτ² + σ²)/n and Var(δ̂_DISC) = 8σ²/n by direct variance algebra: under RCS, separately sampled clusters contribute both cluster-level variance mτ² and individual-level variance σ², while under DISC, the same clusters appear at both time points so the cluster effect αi cancels inside each cluster difference, leaving only ϵ terms. The variance ratio is then 1 + mτ²/σ², rewritten as 1 + mρ/(1 − ρ) using the definition ρ = τ²/(τ² + σ²). This ICC reparameterization is not a fitted quantity and does not smuggle in the result: the illustrative ratios 2.3 and 3.8 are evaluated at user-specified ICC values of 0.05 and 0.1, not estimated from any outcome data. The simulation study is an independent check: data are generated from Eq. (4) with known parameters, and the empirical variance across replicates is compared with the analytic formula. No fitted parameter is renamed as a prediction, and no load-bearing claim is justified solely by self-citation. The self-citations (Kenny and Wolock 2024 for the SimEngine simulation framework; Kenny et al. 2015 as a substantive example about health-facility distance) are not central to the variance derivation. The paper also explicitly discloses the key scope condition: the advantage relies on cluster membership and sizes being stable across the study period, and the Discussion acknowledges that changes in the number or sizes of clusters can make the second time point non-representative. For these reasons, no circular step is present; the minor reporting inconsistency between the simulation text (ICC = 0.2) and some figure captions (ICC = 0.1) affects presentation, not the logical structure of the derivation.
Assumptions & free parameters
assumptions (4)
- domain assumption Data generating model Y = μ_j + δX_ij + α_i + ε_ijk with α_i ~ N(0,τ²), ε_ijk ~ N(0,σ²), mutually independent.
- domain assumption Clusters are sampled with probability proportional to size, m individuals are sampled per cluster at each time point, and samples are independent across time with no missingness or non-response.
- ad hoc to paper The population of clusters is large enough that RCS samples have zero probability of overlapping clusters.
- domain assumption The DID identification conditions from Sant'Anna and Zhao (2020), including parallel trends assumptions, hold.
Cite this review
Pith. "Pith review of Repeated sampling of different individuals within the same clusters to improve precision of longitudinal estimators: the DISC design." pith.science (2026). https://pith.science/paper/ZAKAAB5A
@misc{pith2026241117905,
author = {Pith},
title = {Pith review of: Repeated sampling of different individuals within the same clusters to improve precision of longitudinal estimators: the DISC design},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZAKAAB5A}},
note = {Machine review of arXiv:2411.17905}
}
read the original abstract
Background: Longitudinal studies often involve repeated cluster sampling of a population at multiple time points, such as in difference-in-differences (DID) studies. Although cohort designs typically lead to more efficient estimators relative to repeated cross-sectional (RCS) designs, they are often impractical. Methods: We describe the DISC (Different Individuals, Same Clusters) design, a sampling scheme that improves the precision of estimators in these settings. The DISC design represents a hybrid between a cohort and an RCS design, in which the researcher takes a single sample of clusters at baseline, but takes different cross-sectional samples of individuals within clusters at each time point. Results: We show analytically that the DISC design yields DID estimators with much higher precision relative to an RCS design, particularly if cluster effects are present. For example, for a design with two surveys, 40 clusters, and 25 individuals per cluster, the variance of a commonly-used DID treatment effect estimator is 2.3 times higher in the RCS design for an intraclass correlation coefficient (ICC) of 0.05 and 3.8 times higher for an ICC of 0.1. We also present results of a simulation study comparing the RCS and DISC designs, using both a simple DID estimator and a more complex doubly-robust DID (DRDID) estimator that leverages covariate information, and show gains in precision for both estimators when using the DISC design. Additionally, we illustrate DISC sampling using a household survey dataset from South Africa. Conclusions: Use of the DISC design can result in estimators that have substantially lower variance than the analogous estimator resulting from an RCS design.
Figures
Reference graph
Works this paper leans on
-
[1]
Difference-in-differences with multiple time periods
Brantly Callaway and Pedro HC Sant’Anna. Difference-in-differences with multiple time periods. Journal of econometrics, 225 0 (2): 0 200--230, 2021
2021
-
[2]
Survey sampling: theory and methods
Arijit Chaudhuri and Horst Stenger. Survey sampling: theory and methods. CRC Press, 2005
work page 2005
-
[3]
Paula Diehr, Donald C Martin, Thomas Koepsell, Allen Cheadle, Bruce M Psaty, and Edward H Wagner. Optimal survey design for community intervention evaluations: cohort or cross-sectional? Journal of clinical epidemiology, 48 0 (12): 0 1461--1472, 1995
work page 1995
-
[4]
Peter J Diggle, Patrick J Heagerty, Kung-Yee Liang, and Scott L Zeger. Analysis of Longitudinal Data. Oxford University Press, 2002
work page 2002
-
[5]
Methods for evaluating changes in health care policy: the difference-in-differences approach
Justin B Dimick and Andrew M Ryan. Methods for evaluating changes in health care policy: the difference-in-differences approach. Jama, 312 0 (22): 0 2401--2402, 2014
work page 2014
-
[6]
Madeleine Short Fabic, YoonJoung Choi, and Sandra Bird. A systematic review of demographic and health surveys: data availability and utilization for research. Bulletin of the World Health Organization, 90: 0 604--612, 2012
work page 2012
-
[7]
Henry A Feldman and Sonja M McKinlay. Cohort versus cross-sectional design in large field trials: precision, sample size, and a unifying model. Statistics in medicine, 13 0 (1): 0 61--78, 1994
work page 1994
-
[8]
The stepped wedge cluster randomised trial: rationale, design, analysis, and reporting
Karla Hemming, Terry P Haines, Peter J Chilton, Alan J Girling, and Richard J Lilford. The stepped wedge cluster randomised trial: rationale, design, analysis, and reporting. Bmj, 350, 2015
work page 2015
Show all 26 references
-
[9]
A tutorial on sample size calculation for multiple-period cluster randomized parallel, cross-over and stepped-wedge trials using the shiny crt calculator
Karla Hemming, Jessica Kasza, Richard Hooper, Andrew Forbes, and Monica Taljaard. A tutorial on sample size calculation for multiple-period cluster randomized parallel, cross-over and stepped-wedge trials using the shiny crt calculator. International journal of epidemiology, 4...
2020
-
[10]
Designing cross-sectional and cohort studies
Stephen B Hulley, Steven R Cummings, Thomas B Newman, WS Browner, and DG Grady. Designing cross-sectional and cohort studies. Designing clinical research, 4: 0 85--96, 2013
2013
-
[11]
The essential role of pair matching in cluster-randomized experiments, with application to the mexican universal health insurance evaluation
Kosuke Imai, Gary King, and Clayton Nall. The essential role of pair matching in cluster-randomized experiments, with application to the mexican universal health insurance evaluation. Statistical Science, 24 0 (1), 2009
2009
-
[12]
Estimands in cluster-randomized trials: choosing analyses that answer the right question
Brennan C Kahan, Fan Li, Andrew J Copas, and Michael O Harhay. Estimands in cluster-randomized trials: choosing analyses that answer the right question. International Journal of Epidemiology, 52 0 (1): 0 107--118, 2023
2023
-
[13]
SimEngine : A modular framework for statistical simulations in R
Avi Kenny and Charles J Wolock. SimEngine : A modular framework for statistical simulations in R . arXiv preprint arXiv:2403.05698, 2024
2024
-
[14]
Remoteness and maternal and child health service utilization in rural liberia: A population--based survey
Avi Kenny, Gaurab Basu, Madeleine Ballard, Thomas Griffiths, Katherine Kentoffio, Jean Bosco Niyonzima, G Andrew Sechler, Stephen Selinsky, Rajesh R Panjabi, Mark J Siedner, et al. Remoteness and maternal and child health service utilization in rural liberia: A population--bas...
2015
-
[15]
Intra-cluster correlations from the clustered outcome dataset bank to inform the design of longitudinal cluster trials
Elizabeth Korevaar, Jessica Kasza, Monica Taljaard, Karla Hemming, Terry Haines, Elizabeth L Turner, Jennifer A Thompson, James P Hughes, and Andrew B Forbes. Intra-cluster correlations from the clustered outcome dataset bank to inform the design of longitudinal cluster trials...
2021
-
[16]
A cluster-randomized trial to improve stroke care in hospitals
K Lakshminarayan, C Borbas, B McLaughlin, NE Morris, G Vazquez, RV Luepker, and DC Anderson. A cluster-randomized trial to improve stroke care in hospitals. Neurology, 74 0 (20): 0 1634--1642, 2010
2010
-
[17]
Practical methods for design and analysis of complex surveys
Risto Lehtonen and Erkki Pahkinen. Practical methods for design and analysis of complex surveys. John Wiley & Sons, 2004
2004
-
[18]
Analysis of complex survey samples
Thomas Lumley. Analysis of complex survey samples. Journal of statistical software, 9: 0 1--19, 2004
2004
-
[19]
Bayesian inference under cluster sampling with probability proportional to size
Susanna Makela, Yajuan Si, and Andrew Gelman. Bayesian inference under cluster sampling with probability proportional to size. Statistics in medicine, 37 0 (26): 0 3849--3868, 2018
2018
-
[20]
Observational research methods
CJ Mann. Observational research methods. research design ii: cohort, cross sectional, and case-control studies. Emergency medicine journal, 20 0 (1): 0 54--60, 2003
2003
-
[21]
Use of interrupted time series analysis in evaluating health care quality improvements
Robert B Penfold and Fang Zhang. Use of interrupted time series analysis in evaluating health care quality improvements. Academic pediatrics, 13 0 (6): 0 S38--S44, 2013
2013
-
[22]
Multilevel modelling of complex survey data
Sophia Rabe-Hesketh and Anders Skrondal. Multilevel modelling of complex survey data. Journal of the Royal Statistical Society Series A: Statistics in Society, 169 0 (4): 0 805--827, 2006
2006
-
[23]
Resampling inference with complex survey data
Jon NK Rao and CFJ Wu. Resampling inference with complex survey data. Journal of the american statistical association, 83 0 (401): 0 231--241, 1988
1988
-
[24]
Generalized difference-in-differences
David B Richardson, Ting Ye, and Eric J Tchetgen Tchetgen. Generalized difference-in-differences. Epidemiology, 34 0 (2): 0 167--174, 2023
2023
-
[25]
On sampling with probability proportional to size
Bengt Ros \'e n. On sampling with probability proportional to size. Journal of statistical planning and inference, 62 0 (2): 0 159--191, 1997
1997
-
[26]
Doubly robust difference-in-differences estimators
Pedro HC Sant’Anna and Jun Zhao. Doubly robust difference-in-differences estimators. Journal of econometrics, 219 0 (1): 0 101--122, 2020
2020
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.