REVIEW 3 major objections 5 minor 1 cited by
Analysis of longitudinal data with destructive sampling using linear mixed models
T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read This paper claims that grouping similarly performing observational units into pseudo-units within each experimental unit—and fitting a linear mixed model with a random effect per pseudo-unit—yields the lowest mean square error among four…
desk verdict The paper's central device—grouping units by their own outcomes—makes the proposed random effect a function of the response, so the main theorem and simulation gains do not support the conclusion. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The pseudo-observational unit is the machine: at every measurement time, experimental units' observations are ranked by outcome and cut into two groups by the median, so each experimental unit has a complete time series of group-level subjects. The random effect $\eta_{r(i)}$ is then estimable because the same group index appears at every time. The proof of Lemma 1 works through a Cholesky decomposition of the random-effect design matrix, showing that the residual sum of squares for the proposed model is the smallest because its richer random-effect structure absorbs more variability than the random-intercept or averaged alternatives.
What would settle it
Simulate destructive-sampling data, but build the pseudo-groups from a covariate that is independent of the response, or at random, instead of from a median split, and compare the proposed model's MSE and p-values with the complete-data references; if its advantage disappears, the result is driven by outcome-based grouping rather than by the mixed-model structure.
Extended reading notes
Core claim
The central discovery is that the unobservable observational-unit effect in the original mixed model can be replaced by an estimable random effect attached to pseudo-observational units: within each experimental unit, the observations at each time are sorted by their response and assigned to one of two groups, a high and a low group, and the same group index is treated as a longitudinal subject. The proposed model is $y_{irk} = x_{irk}\beta + b_i + \eta_{r(i)} + \varepsilon_{irk}$, with $b_i$ and $\eta_{r(i)}$ independent normal random effects. Lemma 1 proves that when the covariate matrix contains only dichotomous variables, the mean square error ordering is $\mathrm{MSE}(0) \le \mathrm{MSE}(iii) \le \mathrm{MSE}(i) \le \mathrm{MSE}(ii)$, where (0) is the proposed model, (iii) the random-intercept model, (i) the averaged pseudo-panel model, and (ii) the fixed-effects model. The simulations then show the proposed model producing not only the smallest MSE but also p-values for treatment, time, and interaction effects closest to the reference values from the full unsampled data, and the Colombian test-score application reports the best fit by MSE, correlation, and pseudo-$R^2$.
Load-bearing premise
The method stands on the premise that dividing each experimental unit's measurements into high and low groups and reusing those groups as the repeated units produces random effects that behave like independent noise; if the grouping is instead just a data-driven reordering of the outcome, the estimated variances and error rates are not what they claim to be.
Editorial extensions
If this is right
- A destructive-sampling longitudinal dataset can be fitted with standard mixed-model software once pseudo-units are constructed, with no imputation of missing individual trajectories.
- With binary covariates, the proposed model is guaranteed under the paper's assumptions to have no larger mean square error than fixed-effects, random-intercept, and averaged pseudo-panel competitors.
- For testing main effects and interactions, the proposed model's p-values track the complete-data reference more closely than the other methods in the simulated scenarios, which matters for detecting treatment and time differences.
- In the Saber 11 application, the model changes the practical conclusion for the gender-by-area interaction: it is significant under the fixed-effects model but not under the proposed mixed model.
- The grouping can be refined beyond two groups using quartiles, percentiles, or auxiliary student characteristics, an extension the paper describes but does not fully test.
Reading between the lines
- The paper's own application reports a residual homoscedasticity failure for the proposed model's errors, and its conclusions admit that the simulated temporal autocorrelation is not recovered by the pseudo-groups; both are signs that the generative assumptions behind the model are already strained in the favorable case.
- Because pseudo-groups are formed from the response variable, the reported MSE and pseudo-$R^2$ advantages are in-sample comparisons; a hold-out or cross-validated comparison would separate the value of the mixed-model structure from the information leaked by outcome-based grouping.
- The construction does not require the grouping variable to be the outcome: if a stable auxiliary covariate such as gender, cohort, or batch is available, forming pseudo-units from it would give a stronger validation and, if it worked, a less circular version of the method.
- With more than two pseudo-groups per experimental unit, the model could capture more of the distribution of the destroyed observational units; this is a natural extension the paper mentions but leaves untested.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a linear mixed model for longitudinal data under destructive sampling of observational units: experimental units (e.g., schools) are observed at all times, while each observational unit (e.g., a student) is measured only once. The authors construct "pseudo-observational units" by ranking observations within each experimental unit and time and splitting them into groups based on the response value. They then fit a mixed model with random effects for these pseudo-units and compare its mean squared error (MSE) with three alternatives: a Deaton pseudo-panel model, a fixed-effects model, and a random-intercept model. A lemma (Lemma 1) is stated to prove that the proposed model has the smallest MSE. The comparison is supported by a simulation study and an application to Colombian Saber 11 test scores, where the proposed model reports the lowest MSE and highest pseudo-R².
Significance. If the proposed methodology were valid, it would address a real gap in models for destructive-sampling longitudinal designs, where individual observational units cannot be followed over time but experimental units can. The authors formulate a relevant problem, provide R code in a supplementary file, and illustrate the approach on an important educational dataset. However, the core construction is circular and the main theoretical claim is not established. The pseudo-observational units are defined by ranking and splitting the response variable itself, so the random effects in Eq. (2) are functions of the outcome, not independent latent variables. Consequently, the variance components, fitted values, MSE comparisons, and goodness-of-fit measures are not identified in the usual mixed-model sense. The reported advantages of the proposed model are plausibly artifacts of this data-dependent grouping, not evidence of a genuinely better-fitting model. The paper's central contribution therefore does not meet the standard for publication in a statistical methodology journal.
major comments (3)
- [Section 2.1, Eq. (2), Table 1] The pseudo-observational units are constructed by sorting the observed response y within each experimental unit at each time and then grouping the sorted values (e.g., median split in the application). Therefore the random-effects design matrix Zη and the random effects η_r(i) in Eq. (2) are deterministic functions of the outcome variable. This contradicts the assumption stated immediately before Eq. (2) that η_r(i) are iid N(0, σ²_η) and independent of the errors ǫ_irk. Because group membership encodes the realized values of y, the model is a post-hoc partition of the data rather than a generative mixed model. A concrete test of this circularity would be to form groups using an auxiliary variable (such as gender) or a quantile split computed on an independent subsample; under the current construction, the estimated σ²_η and the resulting MSE are not invariant to the number of groups or the cut points, confirming that the random effects are not exchangeable latent effects.
- [Appendix A, proof of Lemma 1] The proof compares residual sums of squares from models fitted to different data: the proposed and fixed-effects models are fit to individual observations y_ijk, while the Deaton model in Eq. (5) is fit to group averages ¯y_i·k. The statement in the appendix that "since the variables that conform X are all binary, the estimators of β are the same in the 4 estimation models" is false for the Deaton model because averaging changes the response and the error covariance. Even for the models fit to individual data, the inequality SS_Resid^(0) < SS_Resid^(iii) follows from the fact that adding columns to the random-effects design matrix Z (from Z_b to [Z_b, Z_η]) cannot increase the in-sample residual sum of squares in a linear mixed model fit; this is a general algebraic property, not a demonstration that the proposed model recovers the true η_j(i) structure. Consequently Lemma 1's MSE ordering is not proven, and the comparison with the averaged Deaton model is not meaningful as written.
- [Section 3.5 and Section 4, Table 9] The simulation results (Figures 2–4) and the application results (Table 9) evaluate MSE and pseudo-R² on the same data used to create the response-based groupings. Because the random effects in the proposed model are constructed from the outcome values, the fitted values are mechanically closer to the observations, inflating the correlation and pseudo-R² (e.g., pseudo-R² = 0.70 in Table 9). No out-of-sample or cross-validation assessment is provided, so the claim of superior goodness-of-fit is not supported. A fair comparison would require forming the pseudo-units using covariates or on a training subsample, and then computing predictive MSE on a validation subsample; the current design cannot distinguish genuine improvement from overfitting to the realized response ordering.
minor comments (5)
- [Section 2.2, Eq. (4)] The likelihood expression in Eq. (4) appears to be misprinted or truncated: the numerator has an ambiguous norm and no matching exponent, and the matrices are not clearly defined. Please provide a correct formula or a precise citation to the source.
- [Section 3.5, paragraph after Eq. (12)] The text claims that "the estimates of the coefficients Ai, Tk, ATij are the same" across the four models, but this is not generally true for the Deaton model in Eq. (11), which is fitted to group averages rather than individual observations; please clarify the conditions under which this equality holds.
- [Section 4, Eq. (13)] The notation is inconsistent: the text refers to random effects "bn and η_r(nm)", while Eq. (13) writes "bn + η_nm"; please harmonize the subscripts.
- [Tables 6 and 7] The "Period" row in both tables shows a single degree of freedom, yet the application uses six measurement years (2013–2018) and Eq. (13) specifies l = 1,...,6; please reconcile the degrees of freedom or explain how the period variable is coded.
- [Table 1] The column header "POUT1" and the entries "PUO1′" and "PUO2′" appear to have transposed letters, and the rows for EU2 do not show the same POU labeling as EU1; please correct the table for readability.
Circularity Check
The pseudo-observational units in Eq. (2) are defined by sorting the response itself, so the random effect η_{r(i)} is a function of y; Lemma 1, the simulation MSE comparisons, and the application's pseudo-R² are therefore largely constructed by this outcome-based grouping rather than by recovering an independent latent effect.
-
self definitional
[Section 2.1, after Eq. (2), construction of pseudo-observational units]
"An example of these pseudo-observational units is shown in the P OU column of Table 1, where two of these are constructed, 1 ′ and 2 ′ by ordering the response variable in ascending order each time to capture the correlation."
This is the definitional step: the pseudo-unit index r is assigned by ranking the outcome y within each experimental unit and time. The model in Eq. (2) then includes a random effect η_{r(i)} for each rank-defined group. Because the grouping partitions y into high and low values, η_{r(i)} is not an independent latent effect and cannot be treated as independent of the errors as assumed after Eq. (2). The fitted values of the proposed model use y itself to decide which group mean each observation is pulled toward, so the subsequent MSE comparisons are built into the definition of the model.
-
self definitional
[Appendix A, proof of Lemma 1]
"since the matrix ZZZ contains both the indicator variables of the random effects of each experimental unit bi and the random effects of each subgroup η1(i), . . . , ηGi(i), it can be written ZZZ = [ ZbZbZb . . . ZηZηZη] ... Therefore SS(0)_Resid < SS(iii)_Resid."
The proof treats Zη as a fixed, exogenous random-effects design matrix and concludes that adding more random-effect columns must reduce the residual sum of squares. But Zη is not exogenous: its columns are constructed from the response y by the ordering described in Section 2.1. The inequality MSE(0) ≤ MSE(iii) is therefore an algebraic artifact of adding outcome-defined columns to the design matrix, not evidence that the proposed generative model is true or that its MSE advantage would survive independent replication.
2 more flagged steps
-
self definitional
[Section 3.5, discussion of Figure 3]
"the observed values of the proposed model are divided into two groups (one upper and one lower) to estimate a random effect ηr(i), trying to make an analogy to the initially simulated ηj(i) coefficients, and that is, the reason why two blue lines are displayed in such figure. Additionally, it is observed that through the proposed model, a better fit is obtained on the observed points."
In the simulation study, the comparison is made after splitting the realized responses into upper and lower halves and fitting η_{r(i)} to those halves. The blue fitted lines in Figure 3 are thus essentially conditional means of the two y-defined clusters. The lower MSE shown in Figures 2 and 4 is the in-sample fit of a clustering of the outcome itself, not a demonstration that the model recovers the originally simulated η_{j(i)} effects.
-
fitted input called prediction
[Section 4, model (13) setup and Table 9]
"what is done in the proposed model is to divide the students into those with good performance and those with poor performance based on the median scores in each school."
The application reports proposed MSE 39.61 and pseudo-R² 0.70, with pseudo-R² defined as the squared correlation between fitted values and the original math scores. Those same math scores are used to form the good/poor median split that defines the random effect η*_{nm}. The pseudo-R² therefore measures how well the model reproduces its own outcome-derived grouping of y. This is an in-sample fit statistic, not predictive validation, and presenting it as evidence of the proposed methodology's superiority is a fitted-input-as-prediction step.
full rationale
The paper does not rely on a self-citation chain or an imported uniqueness theorem; the external benchmarks (Deaton's pseudo-panel, fixed-effects, MANOVA, and the lme4/nlme software) are standard. The circularity is internal to the model construction. Section 2.1 creates pseudo-observational units by sorting the response variable, so the random effect η_{r(i)} in Eq. (2) is a function of y while the model's assumptions and Lemma 1 treat it as an exogenous random effect. Appendix A proves the central MSE inequality by adding columns to the random-effects design matrix, but those columns are rank-of-y indicators, so the inequality is an in-sample matrix-inclusion consequence rather than a recovered modeling advantage. The same mechanism drives the simulation results (upper/lower splits of observed y) and the application's median-split pseudo-R² of 0.70. Because the paper explicitly allows more groups via quartiles or percentiles, the in-sample MSE could be reduced further simply by refining the y-sorting, which confirms that the reported superiority is substantially constructed by the outcome-based definition of the pseudo-units. The proper finding is therefore partial-to-strong circularity: the central comparative claim reduces, by the paper's own construction, to fitting outcome-defined groups and then reporting in-sample fit on the same outcome. Score 8.
Assumptions & free parameters
free parameters (2)
- Number of pseudo-groups G_i per experimental unit =
2 (in all simulations and application)
- Grouping rule (cut point) =
median of the response within each experimental unit and time
assumptions (5)
- domain assumption The response follows the linear mixed model of Equation (1) with independent normal random effects.
- ad hoc to paper Pseudo-observational units formed by sorting the response within each experimental unit at each time produce random effects with exchangeable, normal, independent structure (Equation 2).
- ad hoc to paper The fixed effect estimates β are the same in all four compared models.
- ad hoc to paper Residual sums of squares from models fitted to individual observations and to group averages can be compared via the decomposition in Appendix A.
- domain assumption The complete-data model in Equation (8) is the correct reference for p-value comparisons.
invented entities (1)
-
Pseudo-observational units (POUs)
Cite this review
Pith. "Pith review of Analysis of longitudinal data with destructive sampling using linear mixed models." pith.science (2026). https://pith.science/paper/LXH6NRMY
@misc{pith2026241116153,
author = {Pith},
title = {Pith review of: Analysis of longitudinal data with destructive sampling using linear mixed models},
year = {2026},
howpublished = {\url{https://pith.science/paper/LXH6NRMY}},
note = {Machine review of arXiv:2411.16153}
}
read the original abstract
This paper proposes an analysis methodology for the case where there is longitudinal data with destructive sampling of observational units, which come from experimental units that are measured at all times of the analysis. A mixed linear model is proposed and compared with regression models with fixed and mixed effects, among which is a similar that is used for data called pseudo-panel, and one of multivariate analysis of variance, which are common in statistics. To compare the models, the mean square error was used, demonstrating the advantage of the proposed methodology. In addition, an application was made to real-life data that refers to the scores in the Saber 11 tests applied to students in Colombia to see the advantage of using this methodology in practical scenarios.
Forward citations
Cited by 1 Pith paper
-
Plans for acceptance sampling by attributes when observations are destructive
For destructive testing, ISO 2859-2 plans do not bound the Bayesian risk of an unsatisfactory remaining lot; new plans and a remaining-lot-size tabulation are provided.
Reference graph
Works this paper leans on
- [1]
- [2]
- [3]
-
[4]
G. Canavire-Bacarreza , M. Robles (2017). Non-parametric analysis of poverty duration using repeated cross section: an application for Peru . Applied Economics, 49, no. 22, pp. 2141--2152
work page 2017
-
[5]
C. Covacevich , A. Mann , C. Santos , J. Champaud (2021). Indicators of teenage career readiness: An analysis of longitudinal data from eight countries. OECD
work page 2021
-
[6]
A. Deaton (1985). Panel data from time series of cross-sections. Journal of Econometrics , 30, no. 1-2, pp. 109--126
work page 1985
-
[7]
W. T. Federer , F. King (2007). Variations on split plot and split block experiment designs, vol. 654. John Wiley and Sons, Hoboken
work page 2007
-
[8]
W. H. Finch , J. E. Bolin , K. Kelley (2016). Multilevel modeling using R. Chapman and Hall/CRC, Boca Raton
work page 2016
Show all 31 references
-
[9]
Gardes , G
F. Gardes , G. J. Duncan , P. Gaubert , M. Gurgand , C. Starzec (2005). Panel and pseudo-panel estimation of cross-sectional and time series elasticities of food consumption: The case of US and polish data . Journal of Business and Economic Statistics, 23, no. 2, pp. 242--253
2005
-
[10]
Himaz , H
R. Himaz , H. Aturupane (2016). Returns to education in Sri Lanka : a pseudo-panel approach . Education Economics, 24, no. 3, pp. 300--311
2016
-
[11]
Hinkelmann (2011)
K. Hinkelmann (2011). Design and analysis of experiments, special designs and applications, vol. 3. John Wiley and Sons, Blacksburg
2011
-
[12]
Melo , L
O. Melo , L. L \'o pez , S. Melo (2007). Dise \ n o de experimentos: m \'e todos y aplicaciones . Editorial Universidad Nacional de Colombia. Bogot \'a
2007
-
[13]
Datos abiertos Colombia
MinTic (2020). Datos abiertos Colombia . url https://www.datos.gov.co . Accedido 01-08-2019
2020
-
[14]
Pinheiro , D
J. Pinheiro , D. Bates (2006). Mixed-effects models in S and S-PLUS. Springer Science & Business Media, New York
2006
-
[15]
Pinheiro , D
J. Pinheiro , D. Bates , S. DebRoy , D. Sarkar , R Core Team (2018). nlme : Linear and Nonlinear Mixed Effects Models . ://CRAN.R-project.org/package=nlme. R package version 3.1-131
2018
-
[16]
F. A. Quintana , W. O. Johnson , L. E. Waetjen , E. B. Gold (2016). Bayesian nonparametric longitudinal data analysis. Journal of the American Statistical Association, 111, no. 515, pp. 1168--1181
2016
-
[17]
R: A Language and Environment for Statistical Computing
R Core Team (2024). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. ://www.R-project.org/
2024
-
[18]
Schabenberger , C
O. Schabenberger , C. A. Gotway (2017). Statistical methods for spatial data analysis. Chapman and Hall/CRC, Boca Raton
2017
-
[19]
H. Shi , J. Dong , L. Wang , J. Cao (2021). Functional principal component analysis for longitudinal data with informative dropout. Statistics in Medicine, 40, no. 3, pp. 712--724
2021
-
[20]
Sprietsma (2012)
M. Sprietsma (2012). Computers as pedagogical tools in Brazil : a pseudo-panel analysis . Education Economics, 20, no. 1, pp. 19--32
2012
-
[21]
A. O. Tovar , I. G. Zulaica , V. N \'u \ n ez-Ant \'o n (2012). Analysis of pseudo-panel data with dependent samples. Journal of Applied Statistics, 39, no. 9, pp. 1921--1937
2012
-
[22]
International standard classification of education: Isced 2011
UNESCO (2012). International standard classification of education: Isced 2011. Comparative Social Research, 30
2012
-
[23]
B. P. Urdinola , C. Ospino (2015). Long-term consequences of adolescent fertility: The colombian case. Demographic Research, 32, pp. 1487--1518
2015
-
[24]
Verbeek (2008)
M. Verbeek (2008). Pseudo-panels and repeated cross-sections. In The econometrics of panel data, Springer, pp. 369--383
2008
-
[25]
Verbeek , T
M. Verbeek , T. Nijman (1993). Minimum mse estimation of a regression model with fixed effects from a series of cross-sections. Journal of Econometrics, 59, no. 1-2, pp. 125--136
1993
-
[26]
C. Wan , W. Zhong , W. Zhang , C. Zou (2023). Multikink quantile regression for longitudinal data with application to progesterone data analysis. Biometrics, 79, no. 2, pp. 747--760
2023
-
[27]
W. W. Wei (2006). Time series analysis. In The Oxford Handbook of Quantitative Methods in Psychology: Vol. 2
2006
-
[28]
B. T. West , K. B. Welch , A. T. Galecki (2014). Linear mixed models: a practical guide using Statistical Software . CRC Press, Boca Raton
2014
-
[29]
H. Zou , D. Zeng , L. Xiao , S. Luo (2023). Bayesian inference and dynamic prediction for multivariate longitudinal and survival data. The Annals of Applied Statistics, 17, no. 3, pp. 2574--2595
2023
-
[30]
, " * write output.state after.block = add.period write newline
ENTRY address author booktitle chapter edition editor eid howpublished institution journal key month note number organization pages publisher school series title type url volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.se...
-
[31]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.