REVIEW 4 major objections 4 minor 3 references
Intracranial EEG foundation models are data-constrained: under a fixed compute budget, a small model trained long on many unique recordings beats a briefly trained billion-parameter model.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 14:47 UTC pith:GJNFUU5M
load-bearing objection A serious, honest EFM scaling study whose 'data-constrained' conclusion is plausible but partly confounded by an unstable wide-shallow model regime and a weak EEG fit. the 4 major comments →
DIVER-1: Scaling Intracranial EEG Foundation Models for Transferable Representations
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Central claim: electrophysiology foundation models are data-limited, not capacity-limited. At any fixed compute level, smaller and mid-sized DIVER-1 variants reach lower pretraining loss than larger models, and billion-parameter models only catch up after far more epochs, if ever. The authors fit a data-constrained scaling law L(N,D) = A/(N')^α + B/(D')^β + E, with effective token and parameter counts that decay under repetition, across data scale, subject count, model size (13M–1.83B), and training duration on 5,310 hours of iEEG and 56,769 hours of EEG. Fitted half-lives — R*_N ≈ 3.4 for added parameters vs R*_D ≈ 9 for repeated data in the 1s iEEG model — show capacity saturating quickly
What carries the argument
The load-bearing object is the data-constrained scaling law L(N,D) = A/(N')^α + B/(D')^β + E, where N' and D' are "effective" parameter and token counts that compress with repetition: D' = U_D + U_D·R*_D·(1 − e^{−R_D/R*_D}), and likewise for N'. The fitted half-lives — R*_D for repeated data, R*_N for added parameters — convert raw loss curves into the conclusion that data and training duration, not size, are the productive scaling axes. Around this, the DIVER-1 encoder contributes three named components: any-variate attention, which embeds spatio-temporal structure directly into attention scores so the model stays valid under arbitrary electrode orderings; the spatio-temporal conditional po
Load-bearing premise
The load-bearing premise is the count of unique tokens per training sample — 540, computed as 32 channels × 30 time steps × an assumed 0.75 × 0.75 subsampling retention, without subtracting the 50% masking ratio; if models actually see only about half that many tokens (≈270), the fitted scaling exponents, half-lives, and the compute-optimal frontier all shift.
What would settle it
Refit the scaling law using the realized number of unmasked tokens each model actually processed (track the actual Beta(3,1) subsamples and the 50% masking: ≈270 tokens per sample instead of 540). If the corrected exponents and half-lives move the compute-optimal frontier so that a large model trained briefly wins at equal FLOPs — or shift the predicted optimal model size materially — the data-constrained conclusion fails. A cheaper probe: the EEG fit is already acknowledged as unstable (R² = 0.50, E = 0.0004 "difficult to justify"), so the claim rests on the iEEG fits; check whether they surv
If this is right
- Labs with limited compute should spend their budget on curating more unique recordings and training small models for many epochs: the fitted compute-optimal frontier says a mid-size model trained long wins at equal FLOPs over a briefly trained large model.
- Repeated passes over the same data are nearly as valuable as new data for a while — the 1-second iEEG model keeps learning from repeats up to a half-life of roughly 9 epochs — which makes training duration a legitimate, quantitative scaling knob.
- The LLM-derived "bigger is better" rule should not be transplanted to electrophysiology; the paper offers a data-constrained scaling playbook for choosing model size and epochs before spending compute.
- One flexible backbone can transfer across recording types and age groups: a 12.7M-parameter model pretrained on adult ECoG/SEEG reaches top AUROC on pediatric Neuroprobe decoding tasks and MAYO seizure detection without having seen those corpora.
- Architecture choices compound with the scaling gains: controlled ablations show any-variate attention, RoPE, STCPE, and multi-domain reconstruction each contribute to transfer, while the fixed 3D electrode-position embedding can hurt under age-related geometry shifts.
Where Pith is reading between the lines
- Editorial inference: the strongest practical consequence the authors leave implicit is that public recording-sharing and data curation may be higher-leverage investments than architecture or hardware for near-term EFM progress, since the fitted regime makes unique, diverse recordings the scarce factor.
- Editorial inference: if the fitted data exponent β reflects the intrinsic dimension of ephys data (β ≈ 2/d), then measuring β for a candidate new recording type could predict in advance whether adding it to a corpus will pay — a testable selection rule the paper does not spell out.
- Editorial inference: the absolute numbers of the playbook are shakier than its direction, because the unique-token count (540 per sample) does not apply the 50% masking ratio; refitting with realized visible-token counts would sharpen the frontier without, in all likelihood, reversing the ordering.
- Editorial inference: the same scaling framework can be tested on MEG, EMG, or fMRI; the paper itself flags that its conclusions may not transfer there, so a comparison of fitted half-lives across modalities would show how general the data-constrained regime really is.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces DIVER-1, a family of self-supervised EEG/iEEG foundation models built on any-variate electrode-time attention, spatio-temporal conditional positional encodings, and multi-domain masked reconstruction. The authors pretrain models up to 1.82B parameters on a large corpus (5,310 hours of iEEG and 59,613 hours of combined EEG/iEEG) and evaluate on Neuroprobe, MAYO, and three EEG benchmarks, reporting state-of-the-art or competitive results. The central scientific claim is a scaling-law analysis showing that electrophysiology foundation models are 'data-constrained': at fixed compute, smaller models trained for longer outperform briefly trained large models, so data scale and training duration should be prioritized over parameter count.
Significance. If the scaling conclusions are correct, the paper provides the first controlled multi-axis scaling study for EEG/iEEG foundation models and gives actionable guidance for compute-limited labs. The paper's strengths include a large and diverse pretraining corpus, open code, controlled comparisons on public benchmarks, ablations, and an unusually candid appendix that reports fit instabilities and baseline reproduction issues. However, the central scaling-law conclusion rests on a few load-bearing modeling choices that need to be verified or corrected before the 'data-constrained' claim can be accepted.
major comments (4)
- [Appendix B.4 / C.2 / Table 2] The parameter-count scaling axis is confounded by architectural aspect ratio. All model-size sweeps fix depth at 12 and vary only width, so DIVER XXL has d_model=3072 with an aspect ratio of 256. Appendix C.2 itself states that this aspect ratio 'places it outside the region where loss remains stable.' The fitted small R*_N values in Table 8, and the Section 4.1 Finding 3 conclusion that 'at any fixed compute level, smaller and mid-sized models achieve lower loss than larger models,' are therefore partly attributable to an unstable, poorly shaped model rather than to a domain property. Please add at least one depth-balanced comparison (e.g., 24-layer d_model=1536 vs. 12-layer d_model=3072 at matched parameters/FLOPs) or explicitly restrict the claim to the fixed-depth architecture family and refit without the unstable sizes.
- [Appendix C.5 / Table 9 / Figure 2] The EEG scaling-law fit is too weak to support the abstract's and Figure 2's claim that the same data-constrained behavior holds across both EEG and iEEG. Table 9 reports R^2=0.50 for the EEG fit, and the authors note that the fitted value E=0.0004 is 'difficult to justify on theoretical grounds' and that losses rise around epochs 4-8 due to extracting checkpoints from 32-epoch runs. A fit with R^2=0.50 and a known schedule artifact cannot sustain the quantitative claim that EEG scaling follows the same law. Refit EEG with comparable training schedules/convergence conditions and report sensitivity; otherwise present EEG conclusions as qualitative only.
- [Appendix C.1 / Eq. (7)-(8)] The unique-token count U_D is internally inconsistent. The text states that tokens per sample include the 'proportion of unmasked patches,' but the calculation 32×30×0.75×0.75=540 applies only the Beta(3,1) channel/time subsampling rates and does not multiply by the 50% mask ratio. This directly affects U_D and hence D' in Eq. (8), and it also affects the U_N term in N' because U_N is defined as a function of U_D. Even if a constant factor on U_D were absorbed by the fitted prefactor B for the data axis, the effect on U_N is not obviously multiplicative. Please correct the token-count calculation or explicitly justify why the mask ratio is excluded, and quantify the sensitivity of Tables 8-9 and Figure 3(p) to this choice.
- [Section 4.1 / Figure 3(p) / Appendices C.3 and C.5] The 'predicted' IsoLoss contours in Figure 3(p) are not independent predictions. They are generated from the same fitted scaling-law parameters (Tables 8-9) obtained from the same training runs used to produce the empirical contours. Calling them 'predicted' and stating in the introduction that the framework 'quantitatively predicts the compute-efficient allocation' overstates the evidence. The paper should either validate the fitted law on held-out (N, epochs) configurations or on a new compute budget, or relabel the right panel as a fitted interpolation.
minor comments (4)
- [Section 4.2 / Tables 15, 24-25] The main-text EEG comparisons in Table 15 use CBraMod and LaBraM numbers as reported in their original papers, while Appendix E.3 shows that CBraMod's reported numbers are not reproduced with its own public code (e.g., MentalArithmetic 0.726 reported vs. 0.619 reproduced). Please report reproduced baseline values in the main comparison or add a clear note directing readers to the one-to-one controlled comparison in Table 25.
- [Appendix C.3] The text says the 0.1s patch model yields a 'comparatively small value of B (0.3925),' but Table 8 reports B=1.1550 for DIVER-/I/0.1s. Please correct the inconsistency.
- [Section 4.1, Finding 4] The sentence 'EFMs exhibit smaller R*_N values (3.39 for 1s patch sizes and 0.72 for 1s)' should say '0.72 for 0.1s patch sizes.'
- [Appendix C.1] The notation 'DIVER Small/I/0.1s trained for 2 epochs' in the subject-scaling description is confusing because subject scaling is a dataset-composition experiment; clarify which model and data subset were used.
Circularity Check
No significant circularity; the scaling-law 'predictions' are in-sample fitted contours, but the central claims rest on direct empirical loss curves and held-out benchmarks.
full rationale
The scaling-law analysis fits Eq. 7–8 to pretraining loss curves and then plots both empirical and 'predicted' IsoLoss contours (Fig. 3p). Calling the fitted contours 'predicted' is generous, since they are in-sample evaluations of the fitted law, but the paper transparently describes them as fitted in Appendix C.3 ('contours predicted by the fitted scaling law'), and the core 'data-constrained' conclusion does not depend on this label: it is directly supported by the empirical loss curves (Fig. 3d,h), the interpolated empirical IsoLoss contours, and downstream scaling results (Fig. 3j–o). The fitted exponents and R* values in Tables 8–9 are empirical estimates, not assumptions smuggled in by self-citation; the framework is cited to Muennighoff et al. (2023) and the architecture components are cited to independent prior work. The authors themselves flag the main validity risks: the EEG fit is weak (R²=0.50, E=0.0004 'difficult to justify'), and the XXL model's high aspect ratio 'places it outside the region where loss remains stable' (App. C.2). These are correctness/confounding concerns about the parameter-count axis, not circular derivations. No step in the paper defines a predicted quantity in terms of the same quantity used to define an input, and the benchmark evaluations (Neuroprobe, MAYO, EEG tasks) are external and held out, giving independent content to the empirical claims. Score 1 reflects the minor overstatement of calling fitted contours 'predictions,' not actual circularity.
Axiom & Free-Parameter Ledger
free parameters (4)
- Scaling-law coefficients A, B, E, α, β, R*_D, R*_N =
iEEG 1s: A=19.217, B=57.065, E=0.0092, α=0.3773, β=0.3504, R*_D=9.5372, R*_N=3.385; EEG R²=0.50
- Beta(3,1) resampling parameters and caps C'≤32, N'≤30 =
Beta(3,1); caps 32 channels, 30 patches
- QAQC thresholds =
3.33% clipping threshold; 50% channel-loss threshold
- Multi-domain reconstruction loss weights =
(λ_raw, λ_FFT, λ_STFT) = (1, 0.1, 1) for 1s; (1, 1, 0) for 0.1s
axioms (6)
- domain assumption Data-constrained scaling law L(N,D) = A/N'^α + B/D'^β + E holds for EEG/iEEG with the Muennighoff functional form.
- domain assumption U_D = 636,480 samples × 540 tokens/sample accurately represents unique training tokens.
- domain assumption Pretext reconstruction loss is a valid proxy for downstream transfer.
- domain assumption MNI-registered 3D electrode coordinates are meaningful and transferable across the adult-to-pediatric shift.
- domain assumption Pretraining on adult ECoG/SEEG transfers to pediatric SEEG benchmarks without special adaptation.
- domain assumption MAYO 1s samples can be concatenated to 6s windows and train/test swapped without changing seizure-detection semantics.
read the original abstract
Intracranial EEG (iEEG) provides direct, millisecond-scale recordings of human neural activity, but reusable representation learning is difficult because electrode layouts, anatomical coverage, referencing schemes, and recording conditions vary across patients and centers. We introduce DIVER-1, a self-supervised iEEG foundation model for variable-input recordings that combines any-variate electrode-time attention, spatio-temporal resampling, input-conditioned positional embeddings, and multi-domain masked reconstruction without assuming a fixed electrode montage. We pretrain two variants, DIVER-1-0.1s and DIVER-1-1s, on 5,310 hours of ECoG and SEEG spanning 352k channel-hours, roughly 54x the BrainTreeBank-based pretraining volume. We evaluate DIVER-1 on two held-out benchmarks: Neuroprobe for naturalistic cognitive decoding and MAYO for seizure detection. On leakage-aware Neuroprobe, DIVER-1-0.1s outperforms prior evaluated iEEG foundation models despite using no BrainTreeBank recordings, the corpus underlying Neuroprobe, during pretraining; it also exceeds the linear spectrogram decoder in mean AUROC and remains competitive with stronger nonlinear baselines, a level prior evaluated iEEG foundation models did not reach. DIVER-1-1s also achieves the top AUROC on MAYO seizure detection. Finally, we conduct, to our knowledge, the first controlled compute-aware scaling study for self-supervised iEEG pretraining, sweeping data scale, subject count, training duration, and model size up to 1.8B parameters. Our results indicate a data-constrained regime: expanding unique recordings and training sufficiently long are more reliable scaling axes than increasing parameter count alone. Code is available at link.
Figures
Reference graph
Works this paper leans on
-
[2019]
or bayesian hyperparameter optimization. The search space was defined as±1order of mag- nitude around the best grid search configurations (range:×0.1to×10), with 50 trials conducted to systematically explore this refined hyperparameter space. The optimal hyperparameter settings identified through Optuna optimization are presented in Table 7. Table 7:Optim...
2048
-
[2025]
Scaled Epochs on Our Data
and BrainBERT (Wang et al., 2023)) are pretrained on the BrainTreebank (BTB) (Wang et al., 2024a) datasets—the precursor to Neuroprobe. Therefore, we compare downstream performance when we use BTB exclusively (Figure 13). We trained DIVER T iny/I/0.1smodels with BTB and size-variations of our self-collected iEEG datasets. For the model trained on our self...
2023
-
[2026]
URLhttps://openreview.net/forum?id=5Xwm8e6vbh. S Bbrinkm and W Cukierski. Upenn and mayo clinic’s seizure detection challenge, 2014. Geeling Chau, Christopher Wang, Sabera Talukder, Vighnesh Subramaniam, Saraswati Soedarmadji, Yisong Yue, Boris Katz, and Andrei Barbu. Population transformer: Learning population-level representations of neural activity. Ar...
Pith/arXiv arXiv 2014
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.