Pith. sign in

REVIEW 3 major objections 4 minor 26 references

Repeated sampling of different individuals within the same clusters to improve precision of longitudinal estimators: the DISC design

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Pith's one-line read A study design that samples the same clusters but different individuals at each wave produces difference-in-differences estimators with substantially lower variance than repeated cross-sectional sampling, with the gain captured by an…

desk verdict A correct, practically useful variance ratio for the DISC design; the main claim holds up, with the expected caveats about time-invariant cluster effects. read the letter →

arxiv 2411.17905 v3 pith:ZAKAAB5A submitted 2024-11-26 stat.ME

classification stat.ME MSC 62D05
keywords difference-in-differencesDISCdesignrepeatedcross-sectionalsamplingcohortclusterintraclasscorrelationestimatorvariancelongitudinalsurveys
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a hybrid longitudinal sampling design, DISC (Different Individuals, Same Clusters), that redraws a fresh cross-sectional sample of individuals within each selected cluster at every wave, while returning to the same clusters. The paper claims that for difference-in-differences estimators, this one change removes the contribution of time-invariant cluster effects from the estimator's variance, making DISC substantially more precise than a repeated cross-sectional (RCS) design under the same total sample size. The claimed gain is summarized by a closed-form variance ratio $1 + m\rho/(1-\rho)$, which grows with both cluster size and the intraclass correlation $\rho$. If the claim holds, a study that cannot follow individuals over time can still reduce DID estimator variance substantially by committing to a fixed set of clusters across waves.

What carries the argument

The load-bearing object is the variance identity comparing $\hat{\delta}_{\mathrm{RCS}}$ and $\hat{\delta}_{\mathrm{DISC}}$ under the model above. Because the same clusters appear at both time points under DISC, the cluster-level term $\alpha_i$ enters the before-after difference within each cluster as $\alpha_i - \alpha_i = 0$, leaving only individual noise; this cancellation yields $\operatorname{Var}(\hat{\delta}_{\mathrm{DISC}}) = 8\sigma^2/n$, while RCS retains both noise and cluster variation, giving $\operatorname{Var}(\hat{\delta}_{\mathrm{RCS}}) = 8(m\tau^2+\sigma^2)/n$. The ratio $1 + m\rho/(1-\rho)$, with $\rho = \tau^2/(\tau^2+\sigma^2)$, then converts the identity into a sample-size planning statement.

What would settle it

Simulate data with a cluster-by-time interaction, for example $\alpha_{it} \sim \mathrm{N}(0, \tau^2)$ with $\mathrm{Corr}(\alpha_{i1}, \alpha_{i2}) = 0.8$ under the model used in Section 3, and compare the empirical variances of $\hat{\delta}_{\mathrm{DISC}}$ and $\hat{\delta}_{\mathrm{RCS}}$; if the DISC variance rises above $8\sigma^2/n$ and the ratio falls below $1 + m\rho/(1-\rho)$, the time-invariant cluster-effect assumption is doing the work. The same test can be run with clusters entering and leaving the frame between waves.

Watch

Extended reading notes

Core claim

The central discovery is an exact variance comparison. Under the generative model $Y_{ijk} = \mu_j + \delta X_{ij} + \alpha_i + \epsilon_{ijk}$ with i.i.d. cluster effects $\alpha_i \sim \mathrm{N}(0, \tau^2)$ and individual noise $\epsilon_{ijk} \sim \mathrm{N}(0, \sigma^2)$, the simple DID estimator has variance $8(m\tau^2+\sigma^2)/n$ under RCS and $8\sigma^2/n$ under DISC, so the RCS variance is $1 + m\rho/(1-\rho)$ times larger. Simulations reproduce the formula, and a simulation with a doubly-robust DID estimator that adjusts for covariates shows the precision gain persists with covariate adjustment. The authors present the design as a practical middle ground: sample the same clusters each wave, take a different sample of individuals within each cluster, and analyze with existing longitudinal estimators.

Load-bearing premise

The load-bearing premise is that each cluster's effect is a single time-invariant additive term that is identical at both surveys and independent of treatment and sampling, so the entire DISC variance gain is the cancellation of that term — a premise that fails if cluster effects drift, clusters change composition, or new clusters enter the population.

Editorial extensions

If this is right

  • With 40 clusters and 25 individuals per cluster, the repeated cross-sectional variance is 2.3 times larger than DISC at an intraclass correlation of 0.05, 3.8 times larger at 0.1, and 7.3 times larger at 0.2.
  • Since variance is inversely proportional to sample size, a study powered at 90 percent could be run with roughly half the sample at an ICC of 0.05, and roughly one quarter of the sample at an ICC of 0.1, if DISC replaces RCS.
  • The precision gain holds for a doubly-robust DID estimator that uses covariate information, not just for the simple unadjusted linear estimator, as shown in simulation.
  • By a simple modification of the Section 3 variance argument, the same variance ratio applies to uncontrolled before-and-after comparisons, and the design transfers to cluster randomized trials, stepped wedge trials, and interrupted time series designs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the variance gain is bought at the price of wave-by-wave representativeness; when clusters enter, dissolve, or change composition during the study, the fixed cluster list describes the subpopulation present across the whole study, and the DISC estimand is cleaner for treatment effects but not for population-level change.
  • Beyond the paper: a testable extension is the three-level sampling case using the paper's S-S-D and S-D-D notation, where one would predict an intermediate variance reduction whose size depends on which level's cluster effect cancels.
  • Beyond the paper: because clusters are fixed across waves, cluster-level covariates can be measured once, so a natural comparison is whether analysis-phase adjustment for cluster covariates can recover part of the DISC gain without repeating cluster samples.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This manuscript proposes the DISC design, in which the same clusters are sampled at each wave but different individuals are drawn within clusters, and studies it in a two-period difference-in-differences setting. Under the linear mixed model in Eq. (3), the authors derive Var(δ̂_RCS)=8(mτ²+σ²)/n and Var(δ̂_DISC)=8σ²/n, yielding a variance ratio of 1 + mρ/(1−ρ). A simulation study with 1,000 replicates confirms the variance formulas for the simple estimator and shows similar gains for a doubly-robust DID estimator that adjusts for covariates. The paper also discusses extensions to other longitudinal designs. The mathematical derivation is correct under the stated assumptions, and the simulation results support the theory, although the simulation text and figure captions disagree on the ICC, and an illustration promised in the abstract is absent from the body.

Significance. If the result holds, the DISC design provides a practically meaningful precision improvement for cluster-sampled longitudinal and DID studies without requiring follow-up of the same individuals, and the analytic variance ratio is simple enough to be used in power calculations. The derivation is explicit and self-contained: the ICC enters only as a reparameterization of τ² and σ², and the simulation checks the formulas rather than fitting them. The authors also disclose the design's limitation when cluster populations change and connect the work to existing practice in cluster randomized trials. The contribution is incremental but useful for applied researchers planning cluster-based longitudinal studies.

major comments (3)
  1. [Section 4, Figures 3 and 4] The simulation text states 'THe value of τ was chosen to yield an ICC of 0.2,' but the captions of Figures 3 and 4 both state an ICC of 0.1. Since Figure 3 is presented as confirming the analytic variance formulas, the simulation parameter must be unambiguous; please correct the inconsistency and report the Monte Carlo standard errors for the simulated variances so the visual agreement can be assessed.
  2. [Abstract and body] The abstract's Results paragraph says 'we illustrate DISC sampling using a household survey dataset from South Africa,' but no such illustration appears in Sections 1-5 or Appendix B of the manuscript. As written, the abstract promises content the paper does not deliver; either add the application or remove the claim.
  3. [Section 3 and Discussion] The variance reduction in Eq. (5) relies entirely on the time-invariant cluster effect α_i in model (3) canceling when the same clusters are differenced. The Discussion's limitation paragraph mentions changes in the number or size of clusters but does not mention the equally important case where cluster-level effects drift over time; if α_i is replaced by α_i + γ_it with time-varying γ_it, the DISC variance gains an additional term and the advantage shrinks. Please add an explicit caveat that the claimed precision gain assumes stable cluster effects, not just stable cluster populations.
minor comments (4)
  1. [Section 4] The text contains a typo: 'THe value of τ' should read 'The value of τ'.
  2. [Figures 3 and 4] With 1,000 simulation replicates, the relative standard error of a variance estimate is roughly √(2/999) ≈ 4.5%; adding Monte Carlo standard errors or confidence bands around the simulated variance points would make the agreement with the analytic curves more interpretable.
  3. [Figure 2] The y-axis label 'Total variance' is used after scaling Var(δ̂_DISC)=1, so the plotted quantity is actually the variance ratio; please relabel the axis accordingly.
  4. [Section 5] The statement that the variance ratio increases as the number of clusters decreases would benefit from a brief reference back to the formula 1 + mρ/(1−ρ) with n = mC, making the dependence on C immediate.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the variance comparison follows algebraically from the stated generative model, with ICC entering only as a reparameterization and simulation providing an independent check.

full rationale

The paper's central derivation is self-contained. The estimator in Eq. (2) is a linear combination of cluster-level sums, and the generative model in Eq. (3) specifies Yijk = µj + δXij + αi + ϵijk with αi and ϵijk independent, zero-mean variance components. Appendix B then computes Var(δ̂_RCS) = 8(mτ² + σ²)/n and Var(δ̂_DISC) = 8σ²/n by direct variance algebra: under RCS, separately sampled clusters contribute both cluster-level variance mτ² and individual-level variance σ², while under DISC, the same clusters appear at both time points so the cluster effect αi cancels inside each cluster difference, leaving only ϵ terms. The variance ratio is then 1 + mτ²/σ², rewritten as 1 + mρ/(1 − ρ) using the definition ρ = τ²/(τ² + σ²). This ICC reparameterization is not a fitted quantity and does not smuggle in the result: the illustrative ratios 2.3 and 3.8 are evaluated at user-specified ICC values of 0.05 and 0.1, not estimated from any outcome data. The simulation study is an independent check: data are generated from Eq. (4) with known parameters, and the empirical variance across replicates is compared with the analytic formula. No fitted parameter is renamed as a prediction, and no load-bearing claim is justified solely by self-citation. The self-citations (Kenny and Wolock 2024 for the SimEngine simulation framework; Kenny et al. 2015 as a substantive example about health-facility distance) are not central to the variance derivation. The paper also explicitly discloses the key scope condition: the advantage relies on cluster membership and sizes being stable across the study period, and the Discussion acknowledges that changes in the number or sizes of clusters can make the second time point non-representative. For these reasons, no circular step is present; the minor reporting inconsistency between the simulation text (ICC = 0.2) and some figure captions (ICC = 0.1) affects presentation, not the logical structure of the derivation.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central derivation rests on a random-effects model with an additive cluster effect, independent individual residuals, and exact reuse of the same clusters across waves with fresh individuals. No parameters are fitted to data; simulation values are scenario settings. No new physical or statistical entities are introduced beyond the name DISC for an existing sampling scheme.

assumptions (4)
  • domain assumption Data generating model Y = μ_j + δX_ij + α_i + ε_ijk with α_i ~ N(0,τ²), ε_ijk ~ N(0,σ²), mutually independent.
    Introduced in Eq. (3) and used for both variance derivations in Section 3 and Appendix B; the cancellation of cluster effects in DISC depends on additive, time-invariant α_i.
  • domain assumption Clusters are sampled with probability proportional to size, m individuals are sampled per cluster at each time point, and samples are independent across time with no missingness or non-response.
    Stated in Section 3; the variance formulas treat within-cluster draws as independent and cluster sizes as fixed.
  • ad hoc to paper The population of clusters is large enough that RCS samples have zero probability of overlapping clusters.
    Appendix B calls this a simplifying assumption to ease calculations; in finite populations the RCS variance would need a finite-population correction.
  • domain assumption The DID identification conditions from Sant'Anna and Zhao (2020), including parallel trends assumptions, hold.
    Section 3 states that δ is identified as the ATT under such causal conditions; the variance calculations target the estimator in Eq. (2) and do not establish identification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Repeated sampling of different individuals within the same clusters to improve precision of longitudinal estimators: the DISC design." pith.science (2026). https://pith.science/paper/ZAKAAB5A

@misc{pith2026241117905,
  author       = {Pith},
  title        = {Pith review of: Repeated sampling of different individuals within the same clusters to improve precision of longitudinal estimators: the DISC design},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZAKAAB5A}},
  note         = {Machine review of arXiv:2411.17905}
}
read the original abstract

Background: Longitudinal studies often involve repeated cluster sampling of a population at multiple time points, such as in difference-in-differences (DID) studies. Although cohort designs typically lead to more efficient estimators relative to repeated cross-sectional (RCS) designs, they are often impractical. Methods: We describe the DISC (Different Individuals, Same Clusters) design, a sampling scheme that improves the precision of estimators in these settings. The DISC design represents a hybrid between a cohort and an RCS design, in which the researcher takes a single sample of clusters at baseline, but takes different cross-sectional samples of individuals within clusters at each time point. Results: We show analytically that the DISC design yields DID estimators with much higher precision relative to an RCS design, particularly if cluster effects are present. For example, for a design with two surveys, 40 clusters, and 25 individuals per cluster, the variance of a commonly-used DID treatment effect estimator is 2.3 times higher in the RCS design for an intraclass correlation coefficient (ICC) of 0.05 and 3.8 times higher for an ICC of 0.1. We also present results of a simulation study comparing the RCS and DISC designs, using both a simple DID estimator and a more complex doubly-robust DID (DRDID) estimator that leverages covariate information, and show gains in precision for both estimators when using the DISC design. Additionally, we illustrate DISC sampling using a household survey dataset from South Africa. Conclusions: Use of the DISC design can result in estimators that have substantially lower variance than the analogous estimator resulting from an RCS design.

Figures

Figures reproduced from arXiv: 2411.17905 by the authors.

Figure 1
Figure 1. Visual representation of the DISC design (panel (c)), compared to the repeated cross [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Total analytical variance comparing DISC and RCS designs as a function of the intraclass [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Theoretical variance and empirical (simulation-based) variance of the linear treatment [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Estimated total variance under DISC and RCS designs, comparing a simple unadjusted [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 24 canonical work pages

  1. [1]

    Difference-in-differences with multiple time periods

    Brantly Callaway and Pedro HC Sant’Anna. Difference-in-differences with multiple time periods. Journal of econometrics, 225 0 (2): 0 200--230, 2021

  2. [2]

    Survey sampling: theory and methods

    Arijit Chaudhuri and Horst Stenger. Survey sampling: theory and methods. CRC Press, 2005

  3. [3]

    Optimal survey design for community intervention evaluations: cohort or cross-sectional? Journal of clinical epidemiology, 48 0 (12): 0 1461--1472, 1995

    Paula Diehr, Donald C Martin, Thomas Koepsell, Allen Cheadle, Bruce M Psaty, and Edward H Wagner. Optimal survey design for community intervention evaluations: cohort or cross-sectional? Journal of clinical epidemiology, 48 0 (12): 0 1461--1472, 1995

  4. [4]

    Analysis of Longitudinal Data

    Peter J Diggle, Patrick J Heagerty, Kung-Yee Liang, and Scott L Zeger. Analysis of Longitudinal Data. Oxford University Press, 2002

  5. [5]

    Methods for evaluating changes in health care policy: the difference-in-differences approach

    Justin B Dimick and Andrew M Ryan. Methods for evaluating changes in health care policy: the difference-in-differences approach. Jama, 312 0 (22): 0 2401--2402, 2014

  6. [6]

    A systematic review of demographic and health surveys: data availability and utilization for research

    Madeleine Short Fabic, YoonJoung Choi, and Sandra Bird. A systematic review of demographic and health surveys: data availability and utilization for research. Bulletin of the World Health Organization, 90: 0 604--612, 2012

  7. [7]

    Cohort versus cross-sectional design in large field trials: precision, sample size, and a unifying model

    Henry A Feldman and Sonja M McKinlay. Cohort versus cross-sectional design in large field trials: precision, sample size, and a unifying model. Statistics in medicine, 13 0 (1): 0 61--78, 1994

  8. [8]

    The stepped wedge cluster randomised trial: rationale, design, analysis, and reporting

    Karla Hemming, Terry P Haines, Peter J Chilton, Alan J Girling, and Richard J Lilford. The stepped wedge cluster randomised trial: rationale, design, analysis, and reporting. Bmj, 350, 2015

Show all 26 references
  1. [9]

    A tutorial on sample size calculation for multiple-period cluster randomized parallel, cross-over and stepped-wedge trials using the shiny crt calculator

    Karla Hemming, Jessica Kasza, Richard Hooper, Andrew Forbes, and Monica Taljaard. A tutorial on sample size calculation for multiple-period cluster randomized parallel, cross-over and stepped-wedge trials using the shiny crt calculator. International journal of epidemiology, 4...

  2. [10]

    Designing cross-sectional and cohort studies

    Stephen B Hulley, Steven R Cummings, Thomas B Newman, WS Browner, and DG Grady. Designing cross-sectional and cohort studies. Designing clinical research, 4: 0 85--96, 2013

  3. [11]

    The essential role of pair matching in cluster-randomized experiments, with application to the mexican universal health insurance evaluation

    Kosuke Imai, Gary King, and Clayton Nall. The essential role of pair matching in cluster-randomized experiments, with application to the mexican universal health insurance evaluation. Statistical Science, 24 0 (1), 2009

  4. [12]

    Estimands in cluster-randomized trials: choosing analyses that answer the right question

    Brennan C Kahan, Fan Li, Andrew J Copas, and Michael O Harhay. Estimands in cluster-randomized trials: choosing analyses that answer the right question. International Journal of Epidemiology, 52 0 (1): 0 107--118, 2023

  5. [13]

    SimEngine : A modular framework for statistical simulations in R

    Avi Kenny and Charles J Wolock. SimEngine : A modular framework for statistical simulations in R . arXiv preprint arXiv:2403.05698, 2024

  6. [14]

    Remoteness and maternal and child health service utilization in rural liberia: A population--based survey

    Avi Kenny, Gaurab Basu, Madeleine Ballard, Thomas Griffiths, Katherine Kentoffio, Jean Bosco Niyonzima, G Andrew Sechler, Stephen Selinsky, Rajesh R Panjabi, Mark J Siedner, et al. Remoteness and maternal and child health service utilization in rural liberia: A population--bas...

  7. [15]

    Intra-cluster correlations from the clustered outcome dataset bank to inform the design of longitudinal cluster trials

    Elizabeth Korevaar, Jessica Kasza, Monica Taljaard, Karla Hemming, Terry Haines, Elizabeth L Turner, Jennifer A Thompson, James P Hughes, and Andrew B Forbes. Intra-cluster correlations from the clustered outcome dataset bank to inform the design of longitudinal cluster trials...

  8. [16]

    A cluster-randomized trial to improve stroke care in hospitals

    K Lakshminarayan, C Borbas, B McLaughlin, NE Morris, G Vazquez, RV Luepker, and DC Anderson. A cluster-randomized trial to improve stroke care in hospitals. Neurology, 74 0 (20): 0 1634--1642, 2010

  9. [17]

    Practical methods for design and analysis of complex surveys

    Risto Lehtonen and Erkki Pahkinen. Practical methods for design and analysis of complex surveys. John Wiley & Sons, 2004

  10. [18]

    Analysis of complex survey samples

    Thomas Lumley. Analysis of complex survey samples. Journal of statistical software, 9: 0 1--19, 2004

  11. [19]

    Bayesian inference under cluster sampling with probability proportional to size

    Susanna Makela, Yajuan Si, and Andrew Gelman. Bayesian inference under cluster sampling with probability proportional to size. Statistics in medicine, 37 0 (26): 0 3849--3868, 2018

  12. [20]

    Observational research methods

    CJ Mann. Observational research methods. research design ii: cohort, cross sectional, and case-control studies. Emergency medicine journal, 20 0 (1): 0 54--60, 2003

  13. [21]

    Use of interrupted time series analysis in evaluating health care quality improvements

    Robert B Penfold and Fang Zhang. Use of interrupted time series analysis in evaluating health care quality improvements. Academic pediatrics, 13 0 (6): 0 S38--S44, 2013

  14. [22]

    Multilevel modelling of complex survey data

    Sophia Rabe-Hesketh and Anders Skrondal. Multilevel modelling of complex survey data. Journal of the Royal Statistical Society Series A: Statistics in Society, 169 0 (4): 0 805--827, 2006

  15. [23]

    Resampling inference with complex survey data

    Jon NK Rao and CFJ Wu. Resampling inference with complex survey data. Journal of the american statistical association, 83 0 (401): 0 231--241, 1988

  16. [24]

    Generalized difference-in-differences

    David B Richardson, Ting Ye, and Eric J Tchetgen Tchetgen. Generalized difference-in-differences. Epidemiology, 34 0 (2): 0 167--174, 2023

  17. [25]

    On sampling with probability proportional to size

    Bengt Ros \'e n. On sampling with probability proportional to size. Journal of statistical planning and inference, 62 0 (2): 0 159--191, 1997

  18. [26]

    Doubly robust difference-in-differences estimators

    Pedro HC Sant’Anna and Jun Zhao. Doubly robust difference-in-differences estimators. Journal of econometrics, 219 0 (1): 0 101--122, 2020

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.