{"id":"729aa394-273c-4deb-ba06-247600cbac84","arxiv_id":"2608.00073","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"K-means clustering on six intensity and temporal features plus intra-cluster stratified sampling reduces cross-subset imaging and temporal imbalance versus random splitting in a 149-patient longitudinal brain MRI cohort.","lead":"Randomly shuffling a small medical-imaging dataset into training and test groups can accidentally put very different patients in each group, hurting fair evaluation. This paper proposes an unsupervised grouping-and-splitting protocol that balances image intensity and follow-up timing, cutting the largest intensity gap between groups from over 34% to under 2% in 149 brain-MRI patients.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported bias reduction is measured on the same features used for stratification, so it is partly by construction; the paper's own §4.4 concedes downstream benefit is unproven, leaving the central covariate-shift claim unsupported.","rationale":"The reader's weakest assumption is that balancing the six summary statistics is a sufficient proxy for preventing covariate shift; the paper itself leaves downstream benefit as an open question. I agree and sharpen the point: because the reported imbalance reduction is measured on the exact features used for stratification, the headline result is partly tautological. The authors are transparent about this limitation in §4.4, which is a credit to the paper, but it leaves the central practical claim untested. The proposed concrete test — measuring imbalance on held-out covariates not used in clustering — directly targets whether the proxy holds. Since the paper already frames downstream benefit as an open empirical question, this concern does not change the reader's CONDITIONAL verdict; it reinforces the need for the suggested additional validation.","tokens_in":11558,"tokens_out":4637,"duration_ms":55094,"concrete_test":"Run the SOP and the naive random shuffling baseline on the same N=149 cohort, and for each of the 10 seeds and 3 split ratios in §3.3, compute the cross-subset imbalance (Eq. 7) for a set of held-out covariates not in the six-dimensional vector — e.g., age, sex, tumor volume, or clinical score — in addition to the six original features. If the SOP does not significantly reduce imbalance on these out-of-feature covariates relative to random shuffling, the premise that balancing the six summary statistics prevents evaluation-boundary covariate shift is falsified. Ideally, also compare downstream model validation/test performance gaps.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the SOP reduces maximum cross-subset intensity bias from 34.1% to under 2.1% is supported only for the six features in the clustering vector (Eq. 4). Because the SOP explicitly optimizes these features via K-means and proportionate stratified sampling, the reduction in cross-subset imbalance for I_max and Δt is a direct consequence of the optimization target, not an independent demonstration of covariate balance. The authors acknowledge in §4.4 that whether stratified partitioning translates into a measurable downstream evaluation benefit is dataset-dependent and that balancing on a fixed set of summary features can under-represent rare sub-populations. Thus the practical purpose of the SOP — protecting the training/validation/test boundary from covariate shift — rests on an untested proxy. If that proxy fails, the SOP may balance irrelevancies while leaving genuine shift (e.g., in age, lesion volume, scanner, or clinical variables) untouched.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that naive random partitioning of small longitudinal medical imaging cohorts introduces covariate shift and temporal sampling imbalance across train/validation/test subsets. It introduces a Tripartite Dataset Analytics Framework that audits spatial grid consistency, extracts a six-dimensional per-subject fingerprint (Imin, Imax, Imean, Istd, Δtmax, M), and maps longitudinal intervals. On top of this, it formalizes an 'unsupervised spatio-temporal cohort-balancing SOP' that applies elbow-optimized K-means clustering in this feature space (Eq. 4–6) followed by intra-cluster proportionate stratified sampling. On a 149-patient longitudinal contrast-enhanced T1-weighted brain MRI cohort, the paper reports that the SOP reduces the maximum cross-subset intensity bias from 34.1% under random shuffling to under 2.1%, with Monte Carlo stress tests across ten seeds and three split ratios showing tighter alignment than random partitioning. The paper explicitly states in §4.4 that downstream model benefit is not established.","tokens_in":11783,"tokens_out":3541,"duration_ms":42108,"significance":"If the balance claim were validated on features independent of the clustering objective and if downstream evaluation benefit were demonstrated, the SOP would be a useful, reproducible, and auditable pre-training cohort engineering step. The proposed pipeline is coherent and addresses a real gap: small longitudinal cohorts are often split without regard to distributional alignment. The paper also ships a concrete algorithmic recipe (Eq. 4–6) and an explicit stress-testing protocol. However, the current empirical support is substantially weakened by the fact that the reported balance metrics are computed on the same feature vector used to construct clusters and stratified splits, making the headline reduction partly a mathematical consequence of the optimization. The paper's own §4.4 concedes that the link to downstream model reliability is an open question. This limits the significance of the present results to a methodology proposal with an incomplete validation.","major_comments":[{"comment":"The headline claim that the SOP reduces maximum cross-subset intensity bias from 34.1% to under 2.1% is measured on the same six features (Imin, Imax, Imean, Istd, Δtmax, M) that are used to construct the K-means clusters (Eq. 4–6) and to perform proportionate stratified sampling. The clustering objective explicitly minimizes within-cluster variance over these features, and the sampling step enforces cluster balance across subsets. Therefore, the reduction in cross-subset imbalance for Imax and Δt is a direct property of the optimization target, not an independent demonstration of covariate balance. To support the claim that the SOP mitigates covariate shift, the evaluation must include held-out covariates not used in the clustering vector (e.g., age, sex, lesion volume, scanner, clinical variables) or a downstream model whose performance is compared across SOP-based and random splits.","section":"§3.1, §3.2, Table 1, Eq. (4)"},{"comment":"The 34.1% versus 2.1% figures are presented in Table 1 as point estimates from a single split, with no confidence intervals or standard errors. The stress tests in Figures 6 and 7 are descriptive (violin plots of deviations) and do not report formal uncertainty quantification or statistical comparisons. The claim that the SOP is 'seed-invariant and scale-invariant' would be much more convincing if the authors reported, across the ten seeds and three split configurations, the distribution of the maximum cross-subset bias (e.g., mean and 95% CI) and a paired statistical test against random shuffling. As it stands, the reader cannot assess whether the 2.1% figure is robust or a favorable draw.","section":"Abstract, §3.2, §3.3"},{"comment":"The paper explicitly states that 'whether stratified partitioning of this kind translates into a measurable downstream evaluation benefit is itself dataset-dependent' and that balancing on a fixed set of summary features can under-represent rare sub-populations. These are not merely future-work remarks; they directly qualify the abstract's assertion that random partitioning 'exposes downstream models to out-of-distribution evaluation.' The central practical motivation of the SOP—protecting the evaluation boundary—is therefore supported only by a proxy whose connection to model performance is untested. The authors should either temper the abstract's causal language or add an experiment (e.g., a segmentation or classification task with a standard architecture) that compares evaluation stability under SOP-based versus random splits.","section":"§4.4"}],"minor_comments":[{"comment":"The text says 'minimum intensities converged tightly at a fixed value of −0.338 (σ≈0)' but the earlier figure legend uses a window [−0.337, 1]. Please reconcile these values and clarify whether the quoted number is exact or rounded.","section":"§2.2"},{"comment":"The paper calls the vector 'six-dimensional', but Imin has near-zero variance in this cohort (§2.2). While retaining it for generalizability is reasonable, the effective dimensionality is five; a brief note would avoid overstating the feature diversity.","section":"§3.1, Eq. (4)"},{"comment":"Equation (7) writes Δ = ||μ_subset − μ_population||, but the text describes it as an 'absolute deviation' along a single feature. If the norm is over the feature vector, this is inconsistent with the univariate descriptions in Figure 6 and 7. Please define the norm explicitly and use a consistent notation.","section":"§3.3, Eq. (7)"},{"comment":"The seed set is written as 'S ∈ {12,42,...,9999}'; the ellipsis is ambiguous. Please list the exact ten seed values or describe the sampling rule.","section":"§3.3"},{"comment":"Figure 2 uses violin plots with boxplots and jittered points, but the caption does not define the kernel bandwidth or whether the violin width is scaled by sample size. Adding this information would improve reproducibility.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The central methodological concern is the circularity between the clustering features and the reported balance metrics. This is not a case of internal inconsistency; the method is coherent and the paper is honest about its limitations. However, the empirical claim as presented—a bias reduction from 34.1% to 2.1%—is largely a mathematical consequence of optimizing on those very features. I would not reject the paper, because the SOP is a reasonable proposal that could be made publishable with an independent validation on held-out covariates or a downstream task, and with proper uncertainty quantification. I recommend major revision rather than rejection, but the revision must include such validation, not merely a response in the discussion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this is not a big new algorithm, it's a carefully argued SOP for splitting small longitudinal medical cohorts, and it does what it claims — but the headline bias reduction is measured on the very features the method optimizes. The authors know this and say downstream benefit is an open question (§4.4).\n\nWhat's actually new and good: the tripartite audit (spatial grid, intensity fingerprint, temporal trajectory) is a sensible pre-processing discipline, and the clustering-plus-proportionate-stratified-sampling recipe is cleanly formalized for variable-length longitudinal MRI. The paper is clearly written, the pipeline is reproducible in principle, and the stress testing across multiple split ratios and 5-fold CV gives a fair picture of stability relative to random shuffling. Credit where due: the authors are explicit about limitations, don't overclaim, and the open-question framing is appropriate.\n\nSoft spots, in proportion: the central 34.1% vs 2.1% is partly by construction — the features balanced are the features clustered on. That doesn't make it worthless (it's the definition of stratified sampling), but it's a validation on the optimization target, not an independent check of covariate balance. No downstream model is trained, so the practical claim that this protects evaluation boundaries rests on the untested proxy that those six summary features are the shift that matters. The comparison baseline is only naive random shuffling; a simpler stratified baseline (e.g., on just Imax or just Δt) would clarify how much the full six-dimensional vector buys you. Also, N=149 single-center, single-modality, and the 2.1% figure comes from one split without confidence intervals — minor, but worth fixing.\n\nWho this is for: anyone working with small longitudinal clinical imaging cohorts who wants a defensible way to create train/validation/test splits. It's not a breakthrough, but it's a solid methods note that could improve reproducibility in practice. A serious referee should see it; the main requested revisions would be a downstream model or out-of-feature validation, a simpler stratification baseline, and error bars on the headline numbers.\n\nRecommendation: send to peer review. The core idea is sound, the writing is honest, and the open questions are clearly scoped rather than hidden.","headline":"A clear, honest methods paper that shows stratified splitting reduces imbalance on the stratification features; the main open question is whether that matters downstream, which the authors themselves flag.","tokens_in":12256,"tokens_out":1463,"would_cite":true,"duration_ms":18653,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Random splitting of clinical cohorts secretly creates artificial distribution shifts; this paper replaces it with unsupervised, phenotype-aware stratified partitioning that shrinks cross-subset intensity bias from 34.1% to under 2.1%.","keywords":["cohort partitioning","longitudinal imaging","stratified sampling","covariate shift","K-means clustering","MRI intensity","temporal trajectories","domain shift"],"falsifier":"Train a longitudinal model on a cohort split both ways and compare performance on a truly independent external test set: if the stratified split does not reduce the gap between internal validation and external test performance, or if a held-out covariate like age or lesion burden remains unbalanced across the stratified subsets, the core claim is falsified.","tokens_in":11458,"feed_emoji":"🧠","tokens_out":3237,"duration_ms":34461,"temperature":0.7,"pith_summary":"The paper argues that the conventional shuffle-and-train habit in medical imaging is unsafe for small longitudinal cohorts, because random splits routinely push rare intensity profiles and irregular follow-up schedules into one subset, creating an artificial domain shift at the evaluation boundary. It proposes a standard operating procedure: audit the cohort's spatial grid, intensity distributions, and temporal trajectories; map each patient to a six-dimensional fingerprint; cluster patients into phenotypic strata with K-means; and then sample proportionally within each cluster to build train, validation, and test sets. On 149 patients with longitudinal contrast-enhanced brain MRI, this reduces the maximum cross-subset intensity imbalance from 34.1% under random splitting to under 2.1%, and keeps follow-up intervals aligned around the population mean. The authors position the protocol as a preemptive, auditable form of domain-shift mitigation that can be combined with later intensity-normalization and adaptation methods.","feed_headline":"Stratified splits shrink MRI cohort bias from 34% to 2%","feed_subtitle":"Unsupervised clustering plus proportionate sampling keeps intensity and follow-up stats aligned across train, validation, and test.","key_machinery":"The engine is a six-dimensional subject fingerprint: four intensity statistics (minimum, maximum, mean, and standard deviation of voxel intensities in the baseline scan) plus two temporal features (maximum follow-up interval in hours and number of follow-up encounters). These are z-score standardized, clustered by K-means with the number of clusters chosen by an elbow/curvature criterion (K=4 for this cohort), and then each cluster is proportionally sampled so that every subset reproduces the cohort's cluster mix. This intra-cluster proportionate stratified sampling is what guarantees that rare hyperintense phenotypes and long, densely sampled follow-up trajectories appear in every subset in","core_discovery":"The central claim is that covariate shift in longitudinal clinical datasets can be engineered away at the partitioning stage, without labels, by aligning summary statistics of intensity and temporal sampling across subsets. The paper shows that naive random allocation on a 149-patient cohort produces validation and test sets whose mean maximum intensity exceeds the training mean by more than 34%, whereas the proposed six-dimensional, K-means-based stratified protocol confines the same deviation to under 2.1% while equalizing follow-up span and sequence length across splits. The balancing is achieved deterministically and survives Monte Carlo stress tests over ten seeds and three split ratios","pith_inferences":["If the six summary statistics are not sufficient to capture the covariates that actually drive model performance (for example, tumor volume, age, or scanner site), balancing them may not remove the evaluation risk; the paper itself flags downstream benefit as open.","A natural test would be to run a downstream longitudinal model under both random and stratified splits on a task with a known covariate, and check whether stratified splits improve true generalization or merely decorrelate the reported metrics.","The framework could be extended to multi-modal or multi-center cohorts by adding modality-level intensity fingerprints and site identities to the clustering space, which would also let it serve as a harmonization-audit tool."],"forward_implications":["Splitting a longitudinal cohort without this kind of stratification systematically under-represents high-intensity outlier patients in training and over-exposes them at evaluation; the paper quantifies this as a >34% intensity shift.","The protocol reduces cross-subset intensity bias to <2.1% and temporal follow-up alignment to within roughly 0.5 hours across holdout and 5-fold cross-validation regimes.","Because the features are unsupervised baseline statistics and temporal intervals, the procedure avoids label leakage while making the partition auditable.","The framework is split-ratio agnostic and supports stratified k-fold cross-validation, not just a single holdout.","The authors recommend running spatial grid audits and intensity fingerprinting before any model training, and applying augmentation only after stratification."],"fun_headline_variants":["Clustering-based splits cut MRI cohort bias from 34% to 2.1%","Unsupervised stratification reduces MRI intensity bias by 94%","Smart data partitioning trims MRI intensity bias below 2.1%","K-means splits align MRI follow-up, cut bias to 2.1%"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The protocol assumes that aligning six summary statistics—four intensity moments and two temporal measures—across subsets is a sufficient proxy for eliminating covariate shift; if the real driver of a model's failure is some covariate not captured by those six numbers, the balanced partitions may not protect the evaluation.","fun_headline_variants_meta":{"raw":{"variants":["Clustering-based splits cut MRI cohort bias from 34% to 2.1%","Unsupervised stratification reduces MRI intensity bias by 94%","Smart data partitioning trims MRI intensity bias below 2.1%","K-means splits align MRI follow-up, cut bias to 2.1%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000691,"raw_usage":{"total_tokens":2977,"prompt_tokens":764,"completion_tokens":2213,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":508,"completion_tokens_details":{"reasoning_tokens":2129}},"tokens_in":508,"tokens_out":2213,"duration_ms":20565,"temperature":1.0,"reasoning_tokens":2129,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T01:22:14.071808+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a longitudinal model on a cohort split both ways and compare performance on a truly independent external test set: if the stratified split does not reduce the gap between internal validation and external test performance, or if a held-out covariate like age or lesion burden remains unbalanced across the stratified subsets, the core claim is falsified.","supporting_citations":[],"review_version":1}