{"id":"86da3881-2989-4900-b909-4d0d049489f5","arxiv_id":"2411.16153","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A linear mixed model with outcome-defined pseudo-groups is proposed for destructive-sampling longitudinal data, but its apparent MSE advantage is a circular artifact.","lead":"This paper proposes a mixed linear model for longitudinal data where each individual is measured only once, by forming artificial 'pseudo-units' from high and low performers within each school or experimental unit. The model is claimed to outperform standard fixed-effects, pseudo-panel, and multivariate models, but its advantage is largely an artifact of grouping by the outcome itself.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The pseudo-unit grouping is defined by sorting the response within each time, so η_r(i) in Eq. (2) is a function of y and cannot be treated as an independent random effect; Lemma 1, the simulations, and the reported pseudo-R² all inherit this circularity.","rationale":"After reading the manuscript as a constructive exercise, I find the reader's identification correct. The central comparison is in-sample MSE across models whose flexibility differs and whose proposed model uses a grouping derived from the outcome. Even if Lemma 1's algebra were correct, it would only show that adding random-effect columns can reduce residual sums of squares; it would not establish any population-level advantage because the extra columns are formed from y. The random-intercept model itself would have lower in-sample SSE with more groups, which is not a meaningful claim. The proof is also internally questionable: it states the fixed-effect estimators are equal across all models because X is binary, but Deaton's model uses within-unit-period averages and a different error structure, so β estimates need not coincide; and the residual-SS inequality for Deaton does not follow from the preceding decomposition. These issues are independent of whether the response-based grouping is acceptable. The proposed concrete test would directly separate the contribution of sorting on y from the contribution of modeling a genuine repeated pseudo-unit effect. Because the central claim as stated is unsupported and the methodology's distinguishing step is the source of the circularity, the verdict should remain REJECT.","tokens_in":19039,"tokens_out":7135,"duration_ms":79994,"concrete_test":"Permutation test: Re-run the Section 3 simulation with the same data-generating process and median-split algorithm, but before fitting the proposed model randomly permute the pseudo-group labels within each experimental unit and time (G=2 remains fixed). Repeat 1,000 times to form a null distribution of in-sample MSE for the proposed model. If the observed MSE in Figures 2/4 falls below the 5th percentile, the advantage comes from sorting on y rather than from an identifiable random effect; if it does not, the concern is mitigated. An additional out-of-sample check would hold out one time point per experimental unit, estimate the model without it, and compare predictive MSE.","verdict_should_be":"REJECT","load_bearing_attack":"Section 2.1 builds pseudo-observational units by ordering the response variable in ascending order each time (Table 1, POUs 1' and 2'), and the application splits students by median score within each school (Section 4). Consequently the columns of Zη in Eq. (2) are selected from y itself: the random effect η_r(i) is assigned based on y_irk, so it is not independent of the errors ǫ_irk as assumed before Eq. (3). Every quantity that depends on the fitted values, including the MSE in Lemma 1, the simulation comparisons in Figures 2-4, and the pseudo-R²=0.70 in Table 9, is computed in-sample on the same response values used to create the groups. Lemma 1's proof (Appendix A) treats Z=[Zb Zη] as a fixed design matrix and argues solely that a larger random-effects matrix gives smaller residual SS; it never addresses the data-dependent construction of Zη, and the claim that binary X forces identical β estimates across the four models is false for Deaton's aggregated model (Eq. 11). The reported lower MSE is therefore attributable to grouping observations by their own outcomes, not to recovering any latent η_j(i) structure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a linear mixed model for longitudinal data under destructive sampling of observational units: experimental units (e.g., schools) are observed at all times, while each observational unit (e.g., a student) is measured only once. The authors construct \"pseudo-observational units\" by ranking observations within each experimental unit and time and splitting them into groups based on the response value. They then fit a mixed model with random effects for these pseudo-units and compare its mean squared error (MSE) with three alternatives: a Deaton pseudo-panel model, a fixed-effects model, and a random-intercept model. A lemma (Lemma 1) is stated to prove that the proposed model has the smallest MSE. The comparison is supported by a simulation study and an application to Colombian Saber 11 test scores, where the proposed model reports the lowest MSE and highest pseudo-R².","tokens_in":19398,"tokens_out":5045,"duration_ms":49966,"significance":"If the proposed methodology were valid, it would address a real gap in models for destructive-sampling longitudinal designs, where individual observational units cannot be followed over time but experimental units can. The authors formulate a relevant problem, provide R code in a supplementary file, and illustrate the approach on an important educational dataset. However, the core construction is circular and the main theoretical claim is not established. The pseudo-observational units are defined by ranking and splitting the response variable itself, so the random effects in Eq. (2) are functions of the outcome, not independent latent variables. Consequently, the variance components, fitted values, MSE comparisons, and goodness-of-fit measures are not identified in the usual mixed-model sense. The reported advantages of the proposed model are plausibly artifacts of this data-dependent grouping, not evidence of a genuinely better-fitting model. The paper's central contribution therefore does not meet the standard for publication in a statistical methodology journal.","major_comments":[{"comment":"The pseudo-observational units are constructed by sorting the observed response y within each experimental unit at each time and then grouping the sorted values (e.g., median split in the application). Therefore the random-effects design matrix Zη and the random effects η_r(i) in Eq. (2) are deterministic functions of the outcome variable. This contradicts the assumption stated immediately before Eq. (2) that η_r(i) are iid N(0, σ²_η) and independent of the errors ǫ_irk. Because group membership encodes the realized values of y, the model is a post-hoc partition of the data rather than a generative mixed model. A concrete test of this circularity would be to form groups using an auxiliary variable (such as gender) or a quantile split computed on an independent subsample; under the current construction, the estimated σ²_η and the resulting MSE are not invariant to the number of groups or the cut points, confirming that the random effects are not exchangeable latent effects.","section":"Section 2.1, Eq. (2), Table 1"},{"comment":"The proof compares residual sums of squares from models fitted to different data: the proposed and fixed-effects models are fit to individual observations y_ijk, while the Deaton model in Eq. (5) is fit to group averages ¯y_i·k. The statement in the appendix that \"since the variables that conform X are all binary, the estimators of β are the same in the 4 estimation models\" is false for the Deaton model because averaging changes the response and the error covariance. Even for the models fit to individual data, the inequality SS_Resid^(0) < SS_Resid^(iii) follows from the fact that adding columns to the random-effects design matrix Z (from Z_b to [Z_b, Z_η]) cannot increase the in-sample residual sum of squares in a linear mixed model fit; this is a general algebraic property, not a demonstration that the proposed model recovers the true η_j(i) structure. Consequently Lemma 1's MSE ordering is not proven, and the comparison with the averaged Deaton model is not meaningful as written.","section":"Appendix A, proof of Lemma 1"},{"comment":"The simulation results (Figures 2–4) and the application results (Table 9) evaluate MSE and pseudo-R² on the same data used to create the response-based groupings. Because the random effects in the proposed model are constructed from the outcome values, the fitted values are mechanically closer to the observations, inflating the correlation and pseudo-R² (e.g., pseudo-R² = 0.70 in Table 9). No out-of-sample or cross-validation assessment is provided, so the claim of superior goodness-of-fit is not supported. A fair comparison would require forming the pseudo-units using covariates or on a training subsample, and then computing predictive MSE on a validation subsample; the current design cannot distinguish genuine improvement from overfitting to the realized response ordering.","section":"Section 3.5 and Section 4, Table 9"}],"minor_comments":[{"comment":"The likelihood expression in Eq. (4) appears to be misprinted or truncated: the numerator has an ambiguous norm and no matching exponent, and the matrices are not clearly defined. Please provide a correct formula or a precise citation to the source.","section":"Section 2.2, Eq. (4)"},{"comment":"The text claims that \"the estimates of the coefficients Ai, Tk, ATij are the same\" across the four models, but this is not generally true for the Deaton model in Eq. (11), which is fitted to group averages rather than individual observations; please clarify the conditions under which this equality holds.","section":"Section 3.5, paragraph after Eq. (12)"},{"comment":"The notation is inconsistent: the text refers to random effects \"bn and η_r(nm)\", while Eq. (13) writes \"bn + η_nm\"; please harmonize the subscripts.","section":"Section 4, Eq. (13)"},{"comment":"The \"Period\" row in both tables shows a single degree of freedom, yet the application uses six measurement years (2013–2018) and Eq. (13) specifies l = 1,...,6; please reconcile the degrees of freedom or explain how the period variable is coded.","section":"Tables 6 and 7"},{"comment":"The column header \"POUT1\" and the entries \"PUO1′\" and \"PUO2′\" appear to have transposed letters, and the rows for EU2 do not show the same POU labeling as EU1; please correct the table for readability.","section":"Table 1"}],"recommendation":"reject","confidential_remarks":"The problem addressed in this paper is relevant, and the authors have attempted to provide simulation code and a real-data application. However, the proposed methodology is built on a circular construction (grouping by the outcome), the main lemma is not proven, and the reported advantages are likely artifacts of in-sample fitting. In my view, these are fundamental issues that cannot be remedied by revision within the scope of the current manuscript. I recommend rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe reader's report and the stress-test note are accurate. I read the paper and largely agree.\n\nThe paper tackles a real problem: repeated measurements on experimental units (schools) when observational units (students) are destroyed at each time. That framing is worthwhile. The authors correctly note that you cannot estimate student-level random effects when each student appears once. Their idea—form pseudo-observational units by sorting students within each school and time into high/low groups and add a group random effect—is new in a narrow sense; I haven't seen it in the pseudo-panel literature. The simulation study is extensive, and the Saber 11 application is relevant.\n\nUnfortunately, the central device is circular. The groups are defined by ranking the response variable itself. So the column of Z that carries η_r(i) is a deterministic function of y. The model assumes η_r(i) is independent of the errors, but it isn't; it is selected to be correlated with y by construction. Every quantity that follows—the MSE ordering in Lemma 1, the simulation boxplots, and the pseudo-R² of 0.70—is computed in-sample on the same response values that created the grouping. The pseudo-R² is essentially a median-split artifact.\n\nThe proof of Lemma 1 in Appendix A does not address this. It treats Z as a fixed design matrix and compares residual sums of squares across models fitted to different response vectors. The sentence claiming binary X gives the same β estimates in all four models is simply false for Deaton's aggregated model (Equation 11). So the paper's headline result is unsupported.\n\nThe paper does some things well. The comparison framework—fixed, Deaton, MANOVA, proposed—is sensible as an experimental design. The p-value plots in Figures 5–7 are a reasonable way to calibrate tests under destructive sampling. The real-data analysis honestly reports the Bartlett test failure on residuals. But these positives surround a flawed core.\n\nFor a practitioner, this paper is not usable as written: the proposed model's variance components and p-values are not identifiable once the grouping depends on y. For a methodologist, it is a useful example of why you cannot construct random effects from the outcome. I would not cite it, and I would not send it to peer review in current form; the central flaw is fundamental rather than fixable by reanalysis. A serious referee would likely reject on the same grounds. The underlying problem—destructive sampling with intact experimental units—deserves better work, and the authors could pivot to using auxiliary variables (e.g., gender, prior grades) to form groups.\n\nSerious thinker? No—the model contradicts its own independence assumption. But the authors are not being disingenuous; they missed the circularity.","headline":"The paper's central device—grouping units by their own outcomes—makes the proposed random effect a function of the response, so the main theorem and simulation gains do not support the conclusion.","tokens_in":100,"tokens_out":3026,"would_cite":false,"duration_ms":64769,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62J05","62J10"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that grouping similarly performing observational units into pseudo-units within each experimental unit—and fitting a linear mixed model with a random effect per pseudo-unit—yields the lowest mean square error among four…","keywords":["destructive sampling","longitudinal data","linear mixed models","pseudo-observational units","pseudo-panel data","mean square error","multivariate analysis of variance","Saber 11 test scores"],"falsifier":"Simulate destructive-sampling data, but build the pseudo-groups from a covariate that is independent of the response, or at random, instead of from a median split, and compare the proposed model's MSE and p-values with the complete-data references; if its advantage disappears, the result is driven by outcome-based grouping rather than by the mixed-model structure.","tokens_in":18864,"feed_emoji":"📊","tokens_out":9427,"duration_ms":84166,"temperature":0.7,"pith_summary":"Standard longitudinal models assume the same observational units are measured repeatedly, but many real datasets destroy each unit at measurement: a student takes a national test once, a leaf is picked once, a device is burned once. The paper's proposal keeps the experimental unit (the school) across all times and manufactures pseudo-observational units inside it—at each time, students are split by their score into high and low groups, and those groups are followed as if they were repeated subjects. Fitting the usual linear mixed model to these pseudo-units adds a random effect for each group, and the paper claims this recovers the variability that destructive sampling hides. Across simulations and an application to Colombian Saber 11 mathematics scores, this model has the lowest mean square error and the best fit among the methods compared, and its hypothesis-test p-values stay closest to what the complete-data model would have given. If the claim holds, any destructive-sampling longitudinal dataset can be analyzed without imputation or individual follow-up.","feed_headline":"Median-split pseudo-groups beat four standard longitudinal fits","feed_subtitle":"When every subject is measured once, grouping similar outcomes into pseudo-units lets mixed models win on MSE and fit.","key_machinery":"The pseudo-observational unit is the machine: at every measurement time, experimental units' observations are ranked by outcome and cut into two groups by the median, so each experimental unit has a complete time series of group-level subjects. The random effect $\\eta_{r(i)}$ is then estimable because the same group index appears at every time. The proof of Lemma 1 works through a Cholesky decomposition of the random-effect design matrix, showing that the residual sum of squares for the proposed model is the smallest because its richer random-effect structure absorbs more variability than the random-intercept or averaged alternatives.","core_discovery":"The central discovery is that the unobservable observational-unit effect in the original mixed model can be replaced by an estimable random effect attached to pseudo-observational units: within each experimental unit, the observations at each time are sorted by their response and assigned to one of two groups, a high and a low group, and the same group index is treated as a longitudinal subject. The proposed model is $y_{irk} = x_{irk}\\beta + b_i + \\eta_{r(i)} + \\varepsilon_{irk}$, with $b_i$ and $\\eta_{r(i)}$ independent normal random effects. Lemma 1 proves that when the covariate matrix contains only dichotomous variables, the mean square error ordering is $\\mathrm{MSE}(0) \\le \\mathrm{MSE}(iii) \\le \\mathrm{MSE}(i) \\le \\mathrm{MSE}(ii)$, where (0) is the proposed model, (iii) the random-intercept model, (i) the averaged pseudo-panel model, and (ii) the fixed-effects model. The simulations then show the proposed model producing not only the smallest MSE but also p-values for treatment, time, and interaction effects closest to the reference values from the full unsampled data, and the Colombian test-score application reports the best fit by MSE, correlation, and pseudo-$R^2$.","pith_inferences":["The paper's own application reports a residual homoscedasticity failure for the proposed model's errors, and its conclusions admit that the simulated temporal autocorrelation is not recovered by the pseudo-groups; both are signs that the generative assumptions behind the model are already strained in the favorable case.","Because pseudo-groups are formed from the response variable, the reported MSE and pseudo-$R^2$ advantages are in-sample comparisons; a hold-out or cross-validated comparison would separate the value of the mixed-model structure from the information leaked by outcome-based grouping.","The construction does not require the grouping variable to be the outcome: if a stable auxiliary covariate such as gender, cohort, or batch is available, forming pseudo-units from it would give a stronger validation and, if it worked, a less circular version of the method.","With more than two pseudo-groups per experimental unit, the model could capture more of the distribution of the destroyed observational units; this is a natural extension the paper mentions but leaves untested."],"forward_implications":["A destructive-sampling longitudinal dataset can be fitted with standard mixed-model software once pseudo-units are constructed, with no imputation of missing individual trajectories.","With binary covariates, the proposed model is guaranteed under the paper's assumptions to have no larger mean square error than fixed-effects, random-intercept, and averaged pseudo-panel competitors.","For testing main effects and interactions, the proposed model's p-values track the complete-data reference more closely than the other methods in the simulated scenarios, which matters for detecting treatment and time differences.","In the Saber 11 application, the model changes the practical conclusion for the gender-by-area interaction: it is significant under the fixed-effects model but not under the proposed mixed model.","The grouping can be refined beyond two groups using quartiles, percentiles, or auxiliary student characteristics, an extension the paper describes but does not fully test."],"supporting_citations":[{"why":"Supplies the pseudo-panel averaged baseline model that the proposed model is compared against and extends.","marker":"Deaton (1985)"},{"why":"Provides the mixed-model likelihood, REML estimation machinery, and residual assumptions used to fit the proposed model.","marker":"Pinheiro and Bates (2006)"},{"why":"Supplies the Cholesky decomposition used in the proof of Lemma 1 for the MSE ordering.","marker":"Bates et al. (2015b)"},{"why":"Provides the lme4 software used to estimate the fitted mixed models in the simulations and application.","marker":"Bates et al. (2015a)"},{"why":"Identifies the averaged pseudo-panel model as a split-plot design, grounding the variance-component comparison.","marker":"Federer and King (2007)"},{"why":"Supplies the ANOVA notation and estimable-constraint conventions used to define the simulation model.","marker":"Melo et al. (2007)"}],"fun_headline_variants":["Split-and-average trick wins MSE in destructive-sampling longitudinal data","Pseudo-panel trick: sorting outcomes into high/low beats fixed and mixed baselines","When each subject is measured once, grouping responses into pseudo-units wins on MSE","Sort responses into high/low pseudo-groups to win MSE in destructive sampling","Median-splitting each time point into pseudo-units yields best longitudinal MSE"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method stands on the premise that dividing each experimental unit's measurements into high and low groups and reusing those groups as the repeated units produces random effects that behave like independent noise; if the grouping is instead just a data-driven reordering of the outcome, the estimated variances and error rates are not what they claim to be.","fun_headline_variants_meta":{"raw":{"variants":["Split-and-average trick wins MSE in destructive-sampling longitudinal data","Pseudo-panel trick: sorting outcomes into high/low beats fixed and mixed baselines","When each subject is measured once, grouping responses into pseudo-units wins on MSE","Sort responses into high/low pseudo-groups to win MSE in destructive sampling","Median-splitting each time point into pseudo-units yields best longitudinal MSE"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000863,"raw_usage":{"total_tokens":3735,"prompt_tokens":929,"completion_tokens":2806,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":2706}},"tokens_in":545,"tokens_out":2806,"duration_ms":16330,"temperature":1.0,"reasoning_tokens":2706,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:29:42.008695+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate destructive-sampling data, but build the pseudo-groups from a covariate that is independent of the response, or at random, instead of from a median split, and compare the proposed model's MSE and p-values with the complete-data references; if its advantage disappears, the result is driven by outcome-based grouping rather than by the mixed-model structure.","supporting_citations":[],"review_version":1}