{"id":"70294706-8c95-47a4-a393-c71650d04ba0","arxiv_id":"2411.17709","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new multi-hospital EEG dataset shows that data diversity hurts pathology detection, but scaling up data volume, especially for attention-based models, can nearly compensate.","lead":"The authors tested how the size and diversity of EEG training data change the accuracy of 11 machine-learning models for detecting brain pathology, using a new 55,787-recording corpus from 39 hospitals plus the Temple University Hospital benchmark. They find that while multi-hospital data lower accuracy, more recordings help, especially for transformer-based models, and a combined meta-model approaches the performance seen on clean single-site data.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 91% asymptote is the load-bearing joint: Eq. (2) is an unvalidated 3-parameter fit to five nested points, and it is compared to finite-sample TUH rather than to a homogeneous asymptotic baseline.","rationale":"The paper is a valuable empirical study: the 6-fold cross-validation design, the TUH reproduction check in Table 3, and the monotone improvement across ELM1-ELM19 are strong evidence that more heterogeneous data improve performance. My concern is not with the comparative trends, nor with the claim that MIL and transformer models outperform simpler models on large data. The weak point is the extrapolation in Section 4.3 and the comparison baseline used in the Conclusions. Eq. (2) is fitted to five nested means; the authors explicitly label the prediction as 'to some extent, speculative.' More importantly, the sentence that the asymptotic result is 'comparable to the results for homogeneous datasets' compares an infinite-data heterogeneous ceiling with a 2,993-recording TUH point. If homogeneous data also scale with quantity, the gap may never close; the available data alone show a 3-point gap at 55,787 recordings. The meta-model's role adds a secondary risk: the logistic-regression blending stage is not described as nested in the 6-fold cross-validation, so the values feeding the fit could be optimistically biased. Separately, the abstract's claim that ELM19 is the 'largest publicly available EEG corpus' conflicts with the data availability statement, which says the data are available upon reasonable request, not via public download; this does not directly weaken the asymptotic argument but it makes independent verification harder. None of this overturns the measured comparative results, but it means the headline 'quantity can compensate diversity' is a conjecture, not a demonstrated result. The proposed test, fitting a homogeneous scaling curve with the same pipeline and performing leave-ELM19-out validation of the heterogeneous curve, would directly decide whether the conjectured asymptote is credible. I therefore keep the reader's CONDITIONAL verdict: the paper should either soften the asymptotic conclusion or supply the homogeneous scaling control.","tokens_in":20794,"tokens_out":10155,"duration_ms":94752,"concrete_test":"Run an asymptotic-control check: (1) fit Eq. (2) on nested homogeneous subsets (TUH, or ideally Kiessner et al.'s larger homogeneous corpus) through the same 6-fold pipeline to estimate a homogeneous AUC_infinity; (2) refit the heterogeneous curve on ELM1-ELM8 and predict the held-out ELM19 meta AUC. The compensation claim survives only if the homogeneous AUC_infinity is within combined error of the heterogeneous AUC_infinity=91.3 +/- 1.1 and the predicted ELM19 AUC is within 2 SEs of the observed 88.7 +/- 0.5. This one protocol tests both the functional-form extrapolation and the unmatched-baseline gap simultaneously.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that quantity can compensate diversity is carried by Section 4.3, and three problems stack there. First, Eq. (2), AUC(n)=AUC_infinity - alpha*n^(-beta), is fitted to only five ELM subset means (n=2,993 to 55,787), and the subsets are nested, so the five points are not independent; ordinary least squares on mean values with a 3-parameter curve gives R^2 near 1 without evidence for the functional form. The paper itself concedes the law is 'not fully rooted in fundamental principles' and calls the predictions 'to some extent, speculative.' Second, the conclusion that the asymptotic result is 'comparable to the results for homogeneous datasets' compares the fitted heterogeneous asymptote (91.3 +/- 1.1) with the finite-sample TUH meta AUC (91.7 +/- 1.1 on n=2,993), not with a homogeneous asymptotic estimate. If TUH or similar homogeneous data also improve with size, the asymptotic gap may remain; the observed ELM19 gap (88.7 +/- 0.5 vs 91.7 +/- 1.1) is closed only by assuming the particular curve. Third, the meta-model that produces the best curve is the least documented: the logistic-regression blending step is not described as nested inside the 6-fold protocol, so leakage into the test fold cannot be ruled out, and this directly affects the data points used for the extrapolation. All three issues point at the same extrapolation, so the headline claim is not yet supported at the stated confidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares machine-learning models for EEG pathology detection on the small homogeneous TUH dataset (2,993 recordings) and the larger multi-hospital ELM19 dataset (55,787 recordings), including several nested subsets. The authors find that model performance drops with data heterogeneity but improves with data quantity, and they fit a saturation power law (Eq. 2) to estimate asymptotic AUC. Their central claim is that the meta-model could reach roughly 91% AUC with infinite heterogeneous data, comparable to finite-sample TUH performance, implying that very large data volumes can compensate for diversity.","tokens_in":21106,"tokens_out":5482,"duration_ms":43687,"significance":"If the asymptotic claim held, the paper would provide a valuable scaling law and a strong argument for building very large multi-hospital EEG corpora. The study has genuine strengths: it reproduces Gemein et al. (2020) on TUH within one standard error (Table 3), uses a stratified 6-fold protocol with a separate validation fold, reports uncertainties, and introduces a large multi-hospital dataset that is a potential community resource. The observed degradation from TUH to SZC to ELM1 and the improvement across ELM subsets are well supported by the present results. However, the headline quantity-compensates-diversity conclusion rests on an extrapolation that is not yet validated, and the meta-model protocol is incompletely specified.","major_comments":[{"comment":"The asymptotic AUC_infinity values, in particular META AUC_infinity = 91.3 +/- 1.1, are fitted parameters of a three-parameter power law fitted to five nested ELM subset means (n = 2,993 to 55,787). Because the subsets are nested, the five points are not independent, and the functional form is not independently validated; the high R^2 values in Table 4 do not establish that the extrapolation to infinity is correct. The paper itself states that the law is 'not fully rooted in fundamental principles' and the predictions are 'to some extent, speculative' (Sections 4.3 and 5). The observed ELM19 gap (META 88.7 vs TUH 91.7, Table B.5) is closed only by this extrapolation, so the central claim that 'very large amounts of data can eventually compensate for data heterogeneity' (Conclusions) is not supported at the stated confidence.","section":"Section 4.3, Eq. (2), Table 4"},{"comment":"The conclusion that the asymptotic result is 'comparable to the results for homogeneous datasets' compares the fitted heterogeneous asymptote (AUC_infinity = 91.3 +/- 1.1) with the finite-sample TUH meta-model AUC (91.7 +/- 1.1, Table B.5). No homogeneous asymptotic curve is provided; if TUH-like data also improve with dataset size, as suggested by Kiessner et al. (2024) for ACC, the asymptotic gap between homogeneous and heterogeneous performance may remain. Please provide a like-for-like comparison, e.g., a scaling fit on homogeneous subsets of larger data, or explicitly frame the claim as a conditional extrapolation.","section":"Section 4.3 and Conclusions"},{"comment":"The meta-model is described as logistic regression on the three component-model probabilities, but the fitting procedure is not specified as nested inside the 6-fold cross-validation. It is not stated whether the logistic regression weights are trained only on predictions from the training folds (or with an inner CV) and then evaluated on the test fold, or whether the probabilities from all folds, including the test fold, are used to fit the regressor. If the latter, the META AUC values in Table B.5 and the extrapolation in Section 4.3 would be affected by leakage. Please clarify the exact protocol and, if necessary, rerun the analysis with proper nesting.","section":"Sections 3.3.3 and 3.4"}],"minor_comments":[{"comment":"The abstract calls the ELM19 dataset 'the largest publicly available EEG corpus,' but the data availability statement says it is available 'upon reasonable request.' Please reconcile this with the actual sharing terms and specify the access conditions.","section":"Abstract and Data availability"},{"comment":"NDB values are reported to three significant figures, but they are derived from extrapolated fits with large uncertainties; consider reporting orders of magnitude or confidence intervals instead.","section":"Table 4"},{"comment":"Equation (2) is written for ACC, and Section 4.3 says the fit was also performed for AUC 'substituting AUC in place of ACC.' Please write the equation generically or define both forms to avoid confusion.","section":"Section 4.3, Eq. (2)"},{"comment":"The captions of Figures 12 and 13 mention the vertical NDB line, but the definition of NDB appears only in the main text; please define it in the captions or refer explicitly to the relevant paragraph.","section":"Figures 12 and 13 captions"}],"recommendation":"major_revision","confidential_remarks":"I recommend major revision. The dataset and the careful reproduction of prior results are valuable. The main concern is the extrapolation-based headline claim; the authors should either add a homogeneous scaling baseline, validate the power law with additional model checks (e.g., predicting the held-out largest subset or fitting non-nested subsets), or soften the Conclusions to explicitly label the compensation claim as a hypothesis. The meta-model leakage issue must be resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper before citing it as evidence that data quantity can compensate for clinical EEG heterogeneity. The comparative parts are solid; the asymptotic headline is not.\n\nWhat is genuinely new: the ELM19 corpus, 55,787 recordings from 39 hospitals, and a systematic comparison of 11 models across homogeneous and heterogeneous data. The TUH reproduction check lands within one standard error of Gemein et al., which is a good sanity test. The cross-validation design is careful: stratified six-fold splits, validation fold for early stopping, test fold untouched, and standard errors reported. The ranking findings are believable and useful: heterogeneity degrades all models, richer architectures (MIL, attention, transformer) benefit most from scale, and the GBE+MINetP+TransNetP meta-model wins.\n\nThe soft spots all sit in Section 4.3. The 91% asymptotic AUC is a three-parameter power law fitted to five nested dataset sizes. The points are not independent, and the functional form is not tested against alternatives. The paper openly admits the law is \"not fully rooted in fundamental principles\" and that the predictions are \"to some extent, speculative.\" That matters because the conclusion compares the fitted heterogeneous asymptote (91.3 ± 1.1) with finite-sample TUH (91.7 ± 1.1), not with a homogeneous asymptotic baseline. If TUH also improves with more data, the gap may persist. The observed data alone still show META at 88.7 on ELM19 versus 91.7 on TUH.\n\nTwo further issues weaken the paper as a benchmark contribution. First, the abstract calls ELM19 \"the largest publicly available EEG corpus,\" but the data availability statement says it is available \"upon reasonable request\" from Elmiko. That is not public in the usual sense, and it undercuts the reproducibility value. Second, the meta-model logistic regression is under-documented: it is not explicit that the stacking weights are trained inside each cross-validation fold, so leakage into the test fold cannot be ruled out. That directly affects the META points that drive the extrapolation.\n\nNone of this invalidates the comparative trends, which I trust more than the asymptote. The paper is a serious empirical contribution that deserves a real referee. I would ask for code and metadata release, a clarified meta-model protocol, and a reframing of Section 4.3 so the extrapolation is clearly labeled as a hypothesis, not a result. With those changes it could be a useful reference for anyone building clinical EEG screening datasets.","headline":"Worth reading for the new multi-hospital EEG benchmark and careful comparisons, but the headline 'quantity compensates diversity' claim rests on an extrapolated power-law fit that the authors themselves call speculative.","tokens_in":21694,"tokens_out":1589,"would_cite":true,"duration_ms":16805,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that data volume can compensate for data heterogeneity in EEG pathology detection, with the best models extrapolating to an asymptotic AUC of about 91%, roughly the level they already reach on homogeneous single-hospital…","keywords":["EEG","pathology detection","data heterogeneity","scaling laws","saturation power law","transformer","multiple-instance learning","meta-model"],"falsifier":"Retrain the meta-model and TransNetP under the same protocol on a heterogeneous corpus two to four times the size of ELM19 (roughly 110,000 to 220,000 recordings) and compare the observed AUC with the band predicted by the power law fitted on the five smaller subsets. If the observed values fall below that band, or if refitting the power law moves the asymptotic estimate $AUC_\\infty$ down significantly, the claim that very large data volumes compensate for heterogeneity is falsified. A cheaper check is to refit the curve while omitting each dataset size in turn: the paper already notes that one model's fit (miNetN) failed to converge, so the stability of the 91% value across these refits is itself informative.","tokens_in":20587,"feed_emoji":"🧠","tokens_out":11337,"duration_ms":82784,"temperature":0.7,"pith_summary":"The paper asks whether machine-learning models trained on EEG recordings from many different hospitals can ever match models trained on a single, uniform hospital dataset. Using the small, homogeneous benchmark TUH and a newly released corpus of 55,787 recordings from 39 hospitals, the authors show that heterogeneity — varied pathologies, recording protocols, and labeling standards — sharply cuts performance when dataset sizes are equal. They then argue that this penalty can be paid off with volume: fitting a saturation power law to models trained on progressively larger subsets points to an asymptotic AUC near 91% for their best model, roughly the level the same models already reach on homogeneous data. The practical consequence, if true, is that collecting very large, diverse EEG corpora is a viable route toward reliable pathology screening.","feed_headline":"EEG models on messy multi-hospital data can hit 91% AUC","feed_subtitle":"A 55,787-recording, 39-hospital corpus shows volume may outweigh clinical diversity.","key_machinery":"The load-bearing object is the saturation power law $AUC(n) = AUC_\\infty - \\alpha n^{-\\beta}$, fitted by least squares to the mean cross-validated AUC of each model on the five nested ELM subsets (ELM1, ELM2, ELM4, ELM8, ELM19). The fitted asymptote $AUC_\\infty$ is the theoretical performance at infinite data, and the better-behaved AUC fits carry the conclusion. The second piece of machinery is the dataset design: the ELM subsets are stratified by hospital, sex, and pathology so that only size changes between them, while the SZC subset matches TUH in size and class balance but comes from a single hospital with a broader range of conditions. The meta-model, a logistic regression over the outputs of the gradient-boosted ensemble, the attention network MINetP, and the transformer TransNetP, is the best-performing system and the one whose asymptote (91.3% ± 1.1% AUC) drives the headline conclusion.","core_discovery":"The central claim is that data quantity can compensate for data diversity in EEG pathology detection. The paper introduces ELM19, a 55,787-recording corpus from 39 hospitals that is about 19 times the size of the TUH benchmark, and compares eleven models: a random forest, a Riemannian-geometry classifier, a gradient-boosted ensemble, several multiple-instance neural networks with attention, a transformer-based network, and a meta-model that blends the best of these. Every model's AUC drops sharply from TUH to a size-matched single-hospital subset (SZC) and again to a size-matched multi-hospital subset (ELM1), isolating the costs of broader pathology and of multi-site protocol variation. As the multi-hospital data grow from ELM1 to ELM19, performance climbs steadily, and the attention- and transformer-based networks keep improving with no sign of saturation. Extrapolating the fitted power law $AUC(n) = AUC_\\infty - \\alpha n^{-\\beta}$ to infinite data gives an asymptote of about 91% AUC for the meta-model, comparable to its 91.7% AUC on homogeneous TUH data, which supports the conclusion that very large volumes of heterogeneous data may eventually match homogeneous-data performance. The paper itself cautions that these predictions are to some extent speculative.","pith_inferences":["The 91% asymptote is an extrapolation from five dataset sizes, so the cleanest test is empirical: when a heterogeneous corpus of roughly 100,000–400,000 recordings becomes available, refit the power law and check whether observed AUC lands on the predicted curve, or whether the asymptote moves.","The power-law form is not derived from any theory of learning, so the 'quantity compensates diversity' claim should be tested on other clinical signal tasks, such as ECG or polysomnography, where cross-site heterogeneity can be measured the same way.","The paper's three-way comparison (TUH, SZC, ELM1) separates two heterogeneity costs, broader pathology versus multi-site protocol variation; if the multi-site cost is the larger one at fixed size, then standardizing recording protocols may be a cheaper lever than collecting ever more data, a reading the paper's data support but do not develop.","If labeling noise sets the ceiling, adding data without improving label quality should show diminishing returns; a direct check would be to measure inter-rater agreement on a subsample of ELM19 and compare it with the fitted asymptote."],"forward_implications":["If the 91% asymptote holds, attention- and transformer-based models trained on corpora of a few hundred thousand heterogeneous recordings should reach roughly the same accuracy as models trained on homogeneous single-hospital data.","On small homogeneous data, architecture barely matters: nearly all models perform equivalently on TUH, and statistically meaningful differences between models emerge only as the dataset grows.","The gradient-boosted ensemble saturates early because it is limited to handcrafted features, while networks that learn features from raw signals keep improving; the meta-model that blends both outperforms every single model on the largest dataset (88.7% AUC on ELM19).","The fitted $N_{DB}$ values imply that tens to hundreds of thousands of recordings are needed for the best models to approach their limits, which justifies continued collection of large multi-hospital EEG corpora.","If the asymptotic ceiling is set by inter-rater labeling noise rather than by data heterogeneity, then labeling recordings with averaged ratings from several experts (fuzzy labels) is the proposed route to push past the current ceiling."],"supporting_citations":[{"why":"Supplies the reference models, the 6-second frame preprocessing, and the TUH results the paper reproduces as a sanity check.","marker":"Gemein et al. (2020)"},{"why":"Supplies the saturation power law and the asymptotic accuracy estimates that the paper extends to heterogeneous data.","marker":"Kiessner et al. (2024)"},{"why":"Defines the TUH corpus used as the homogeneous benchmark, including its single-institution origin and labeling standards.","marker":"Obeid and Picone (2016)"},{"why":"Supplies the EEGNet convolutional architecture used as the shared frame encoder in all the paper's neural models.","marker":"Lawhern et al. (2018)"},{"why":"Supplies the attention-based aggregation mechanism used in the MINet attention module.","marker":"Ilse et al. (2018)"},{"why":"Supplies the self-attention transformer blocks used in the TransNet model.","marker":"Vaswani (2017)"},{"why":"Supplies the gradient-boosting algorithm used for the GBE ensemble, the classical arm of the meta-model.","marker":"Dorogush et al. (2018)"},{"why":"Supplies the multiple-instance architecture with feature aggregation that underlies MINet.","marker":"Wang et al. (2018)"},{"why":"Introduces the multiple-instance learning paradigm used to train the neural networks on whole recordings.","marker":"Dietterich et al. (1997)"}],"fun_headline_variants":["EEG pathology: volume compensates for data diversity","Largest EEG dataset shows quantity overcomes diversity","Multi-hospital EEG data: more recordings lift accuracy","Transformer models thrive as EEG data scales up","Meta-model reaches 91% AUC with huge EEG corpus"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline conclusion rests on the assumption that the saturation power law $AUC(n) = AUC_\\infty - \\alpha n^{-\\beta}$ correctly describes how performance grows with dataset size, and that fitting it to just five dataset sizes, the largest being 55,787 recordings, supports extrapolation to infinite data; if the curve is misspecified or bends differently, quantity may not fully compensate for diversity.","fun_headline_variants_meta":{"raw":{"variants":["EEG pathology: volume compensates for data diversity","Largest EEG dataset shows quantity overcomes diversity","Multi-hospital EEG data: more recordings lift accuracy","Transformer models thrive as EEG data scales up","Meta-model reaches 91% AUC with huge EEG corpus"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000282,"raw_usage":{"total_tokens":1681,"prompt_tokens":970,"completion_tokens":711,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":586,"completion_tokens_details":{"reasoning_tokens":638}},"tokens_in":586,"tokens_out":711,"duration_ms":7003,"temperature":1.0,"reasoning_tokens":638,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:24:39.675485+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the meta-model and TransNetP under the same protocol on a heterogeneous corpus two to four times the size of ELM19 (roughly 110,000 to 220,000 recordings) and compare the observed AUC with the band predicted by the power law fitted on the five smaller subsets. If the observed values fall below that band, or if refitting the power law moves the asymptotic estimate $AUC_\\infty$ down significantly, the claim that very large data volumes compensate for heterogeneity is falsified. A cheaper check is to refit the curve while omitting each dataset size in turn: the paper already notes that one model's fit (miNetN) failed to converge, so the stability of the 91% value across these refits is itself informative.","supporting_citations":[],"review_version":1}