Pith. sign in

REVIEW 4 major objections 4 minor 3 references

Intracranial EEG foundation models are data-constrained: under a fixed compute budget, a small model trained long on many unique recordings beats a briefly trained billion-parameter model.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-03 14:47 UTC pith:GJNFUU5M

load-bearing objection A serious, honest EFM scaling study whose 'data-constrained' conclusion is plausible but partly confounded by an unstable wide-shallow model regime and a weak EEG fit. the 4 major comments →

arxiv 2512.19097 v3 pith:GJNFUU5M submitted 2025-12-22 cs.LG cs.AI

DIVER-1: Scaling Intracranial EEG Foundation Models for Transferable Representations

classification cs.LG cs.AI
keywords intracranial EEG foundation modelelectrophysiology scaling lawsdata-constrained scalingself-supervised pretrainingmasked reconstructionany-variate attentionseizure detectionneural decoding
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper tries to establish that self-supervised models of brain recordings — intracranial EEG and scalp EEG — live in a "data-constrained" regime whose scaling rules differ from large language models. Under a fixed compute budget, the authors claim, training a small model for many epochs on many unique recordings beats briefly training a billion-parameter model: data scale first, training duration second, parameter count last. To show this, they pretrain a model family (DIVER-1, 13M–1.83B parameters) on the largest curated EEG/iEEG corpus to date and fit a data-constrained scaling law across data size, subject count, model size, and epochs. If the claim holds, labs with modest GPU budgets should spend resources on curating more recordings and training small models longer rather than chasing bigger architectures, which would change how the field allocates compute.

Core claim

Central claim: electrophysiology foundation models are data-limited, not capacity-limited. At any fixed compute level, smaller and mid-sized DIVER-1 variants reach lower pretraining loss than larger models, and billion-parameter models only catch up after far more epochs, if ever. The authors fit a data-constrained scaling law L(N,D) = A/(N')^α + B/(D')^β + E, with effective token and parameter counts that decay under repetition, across data scale, subject count, model size (13M–1.83B), and training duration on 5,310 hours of iEEG and 56,769 hours of EEG. Fitted half-lives — R*_N ≈ 3.4 for added parameters vs R*_D ≈ 9 for repeated data in the 1s iEEG model — show capacity saturating quickly

What carries the argument

The load-bearing object is the data-constrained scaling law L(N,D) = A/(N')^α + B/(D')^β + E, where N' and D' are "effective" parameter and token counts that compress with repetition: D' = U_D + U_D·R*_D·(1 − e^{−R_D/R*_D}), and likewise for N'. The fitted half-lives — R*_D for repeated data, R*_N for added parameters — convert raw loss curves into the conclusion that data and training duration, not size, are the productive scaling axes. Around this, the DIVER-1 encoder contributes three named components: any-variate attention, which embeds spatio-temporal structure directly into attention scores so the model stays valid under arbitrary electrode orderings; the spatio-temporal conditional po

Load-bearing premise

The load-bearing premise is the count of unique tokens per training sample — 540, computed as 32 channels × 30 time steps × an assumed 0.75 × 0.75 subsampling retention, without subtracting the 50% masking ratio; if models actually see only about half that many tokens (≈270), the fitted scaling exponents, half-lives, and the compute-optimal frontier all shift.

What would settle it

Refit the scaling law using the realized number of unmasked tokens each model actually processed (track the actual Beta(3,1) subsamples and the 50% masking: ≈270 tokens per sample instead of 540). If the corrected exponents and half-lives move the compute-optimal frontier so that a large model trained briefly wins at equal FLOPs — or shift the predicted optimal model size materially — the data-constrained conclusion fails. A cheaper probe: the EEG fit is already acknowledged as unstable (R² = 0.50, E = 0.0004 "difficult to justify"), so the claim rests on the iEEG fits; check whether they surv

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Labs with limited compute should spend their budget on curating more unique recordings and training small models for many epochs: the fitted compute-optimal frontier says a mid-size model trained long wins at equal FLOPs over a briefly trained large model.
  • Repeated passes over the same data are nearly as valuable as new data for a while — the 1-second iEEG model keeps learning from repeats up to a half-life of roughly 9 epochs — which makes training duration a legitimate, quantitative scaling knob.
  • The LLM-derived "bigger is better" rule should not be transplanted to electrophysiology; the paper offers a data-constrained scaling playbook for choosing model size and epochs before spending compute.
  • One flexible backbone can transfer across recording types and age groups: a 12.7M-parameter model pretrained on adult ECoG/SEEG reaches top AUROC on pediatric Neuroprobe decoding tasks and MAYO seizure detection without having seen those corpora.
  • Architecture choices compound with the scaling gains: controlled ablations show any-variate attention, RoPE, STCPE, and multi-domain reconstruction each contribute to transfer, while the fixed 3D electrode-position embedding can hurt under age-related geometry shifts.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the strongest practical consequence the authors leave implicit is that public recording-sharing and data curation may be higher-leverage investments than architecture or hardware for near-term EFM progress, since the fitted regime makes unique, diverse recordings the scarce factor.
  • Editorial inference: if the fitted data exponent β reflects the intrinsic dimension of ephys data (β ≈ 2/d), then measuring β for a candidate new recording type could predict in advance whether adding it to a corpus will pay — a testable selection rule the paper does not spell out.
  • Editorial inference: the absolute numbers of the playbook are shakier than its direction, because the unique-token count (540 per sample) does not apply the 50% masking ratio; refitting with realized visible-token counts would sharpen the frontier without, in all likelihood, reversing the ordering.
  • Editorial inference: the same scaling framework can be tested on MEG, EMG, or fMRI; the paper itself flags that its conclusions may not transfer there, so a comparison of fitted half-lives across modalities would show how general the data-constrained regime really is.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces DIVER-1, a family of self-supervised EEG/iEEG foundation models built on any-variate electrode-time attention, spatio-temporal conditional positional encodings, and multi-domain masked reconstruction. The authors pretrain models up to 1.82B parameters on a large corpus (5,310 hours of iEEG and 59,613 hours of combined EEG/iEEG) and evaluate on Neuroprobe, MAYO, and three EEG benchmarks, reporting state-of-the-art or competitive results. The central scientific claim is a scaling-law analysis showing that electrophysiology foundation models are 'data-constrained': at fixed compute, smaller models trained for longer outperform briefly trained large models, so data scale and training duration should be prioritized over parameter count.

Significance. If the scaling conclusions are correct, the paper provides the first controlled multi-axis scaling study for EEG/iEEG foundation models and gives actionable guidance for compute-limited labs. The paper's strengths include a large and diverse pretraining corpus, open code, controlled comparisons on public benchmarks, ablations, and an unusually candid appendix that reports fit instabilities and baseline reproduction issues. However, the central scaling-law conclusion rests on a few load-bearing modeling choices that need to be verified or corrected before the 'data-constrained' claim can be accepted.

major comments (4)
  1. [Appendix B.4 / C.2 / Table 2] The parameter-count scaling axis is confounded by architectural aspect ratio. All model-size sweeps fix depth at 12 and vary only width, so DIVER XXL has d_model=3072 with an aspect ratio of 256. Appendix C.2 itself states that this aspect ratio 'places it outside the region where loss remains stable.' The fitted small R*_N values in Table 8, and the Section 4.1 Finding 3 conclusion that 'at any fixed compute level, smaller and mid-sized models achieve lower loss than larger models,' are therefore partly attributable to an unstable, poorly shaped model rather than to a domain property. Please add at least one depth-balanced comparison (e.g., 24-layer d_model=1536 vs. 12-layer d_model=3072 at matched parameters/FLOPs) or explicitly restrict the claim to the fixed-depth architecture family and refit without the unstable sizes.
  2. [Appendix C.5 / Table 9 / Figure 2] The EEG scaling-law fit is too weak to support the abstract's and Figure 2's claim that the same data-constrained behavior holds across both EEG and iEEG. Table 9 reports R^2=0.50 for the EEG fit, and the authors note that the fitted value E=0.0004 is 'difficult to justify on theoretical grounds' and that losses rise around epochs 4-8 due to extracting checkpoints from 32-epoch runs. A fit with R^2=0.50 and a known schedule artifact cannot sustain the quantitative claim that EEG scaling follows the same law. Refit EEG with comparable training schedules/convergence conditions and report sensitivity; otherwise present EEG conclusions as qualitative only.
  3. [Appendix C.1 / Eq. (7)-(8)] The unique-token count U_D is internally inconsistent. The text states that tokens per sample include the 'proportion of unmasked patches,' but the calculation 32×30×0.75×0.75=540 applies only the Beta(3,1) channel/time subsampling rates and does not multiply by the 50% mask ratio. This directly affects U_D and hence D' in Eq. (8), and it also affects the U_N term in N' because U_N is defined as a function of U_D. Even if a constant factor on U_D were absorbed by the fitted prefactor B for the data axis, the effect on U_N is not obviously multiplicative. Please correct the token-count calculation or explicitly justify why the mask ratio is excluded, and quantify the sensitivity of Tables 8-9 and Figure 3(p) to this choice.
  4. [Section 4.1 / Figure 3(p) / Appendices C.3 and C.5] The 'predicted' IsoLoss contours in Figure 3(p) are not independent predictions. They are generated from the same fitted scaling-law parameters (Tables 8-9) obtained from the same training runs used to produce the empirical contours. Calling them 'predicted' and stating in the introduction that the framework 'quantitatively predicts the compute-efficient allocation' overstates the evidence. The paper should either validate the fitted law on held-out (N, epochs) configurations or on a new compute budget, or relabel the right panel as a fitted interpolation.
minor comments (4)
  1. [Section 4.2 / Tables 15, 24-25] The main-text EEG comparisons in Table 15 use CBraMod and LaBraM numbers as reported in their original papers, while Appendix E.3 shows that CBraMod's reported numbers are not reproduced with its own public code (e.g., MentalArithmetic 0.726 reported vs. 0.619 reproduced). Please report reproduced baseline values in the main comparison or add a clear note directing readers to the one-to-one controlled comparison in Table 25.
  2. [Appendix C.3] The text says the 0.1s patch model yields a 'comparatively small value of B (0.3925),' but Table 8 reports B=1.1550 for DIVER-/I/0.1s. Please correct the inconsistency.
  3. [Section 4.1, Finding 4] The sentence 'EFMs exhibit smaller R*_N values (3.39 for 1s patch sizes and 0.72 for 1s)' should say '0.72 for 0.1s patch sizes.'
  4. [Appendix C.1] The notation 'DIVER Small/I/0.1s trained for 2 epochs' in the subject-scaling description is confusing because subject scaling is a dataset-composition experiment; clarify which model and data subset were used.

Circularity Check

0 steps flagged

No significant circularity; the scaling-law 'predictions' are in-sample fitted contours, but the central claims rest on direct empirical loss curves and held-out benchmarks.

full rationale

The scaling-law analysis fits Eq. 7–8 to pretraining loss curves and then plots both empirical and 'predicted' IsoLoss contours (Fig. 3p). Calling the fitted contours 'predicted' is generous, since they are in-sample evaluations of the fitted law, but the paper transparently describes them as fitted in Appendix C.3 ('contours predicted by the fitted scaling law'), and the core 'data-constrained' conclusion does not depend on this label: it is directly supported by the empirical loss curves (Fig. 3d,h), the interpolated empirical IsoLoss contours, and downstream scaling results (Fig. 3j–o). The fitted exponents and R* values in Tables 8–9 are empirical estimates, not assumptions smuggled in by self-citation; the framework is cited to Muennighoff et al. (2023) and the architecture components are cited to independent prior work. The authors themselves flag the main validity risks: the EEG fit is weak (R²=0.50, E=0.0004 'difficult to justify'), and the XXL model's high aspect ratio 'places it outside the region where loss remains stable' (App. C.2). These are correctness/confounding concerns about the parameter-count axis, not circular derivations. No step in the paper defines a predicted quantity in terms of the same quantity used to define an input, and the benchmark evaluations (Neuroprobe, MAYO, EEG tasks) are external and held out, giving independent content to the empirical claims. Score 1 reflects the minor overstatement of calling fitted contours 'predictions,' not actual circularity.

Axiom & Free-Parameter Ledger

4 free parameters · 6 axioms · 0 invented entities

No new physical entities are postulated. The invented-entity list is empty; STCPE and DIVER-1 are architectural artifacts, not hypothesized entities with independent falsifiable handles.

free parameters (4)
  • Scaling-law coefficients A, B, E, α, β, R*_D, R*_N = iEEG 1s: A=19.217, B=57.065, E=0.0092, α=0.3773, β=0.3504, R*_D=9.5372, R*_N=3.385; EEG R²=0.50
    Seven parameters fit to 20-31 loss measurements; used to compute the central 'data-constrained' conclusion and compute-optimal frontier.
  • Beta(3,1) resampling parameters and caps C'≤32, N'≤30 = Beta(3,1); caps 32 channels, 30 patches
    Hand-chosen sampling schedule; directly enters the unique-token estimate U_D in Appendix C.1.
  • QAQC thresholds = 3.33% clipping threshold; 50% channel-loss threshold
    Manual data-exclusion thresholds deciding what recordings enter pretraining; can bias corpus composition.
  • Multi-domain reconstruction loss weights = (λ_raw, λ_FFT, λ_STFT) = (1, 0.1, 1) for 1s; (1, 1, 0) for 0.1s
    Hand-set weights; affect what representations are learned and thus downstream transfer.
axioms (6)
  • domain assumption Data-constrained scaling law L(N,D) = A/N'^α + B/D'^β + E holds for EEG/iEEG with the Muennighoff functional form.
    Eqs. 7-8 assumed a priori in Appendix C.3/C.5; the paper fits, not derives, this functional form.
  • domain assumption U_D = 636,480 samples × 540 tokens/sample accurately represents unique training tokens.
    C.1 estimate uses expected Beta(3,1) subsample proportions but does not account for the 50% mask ratio; a factor-of-2 error would change fitted exponents.
  • domain assumption Pretext reconstruction loss is a valid proxy for downstream transfer.
    Scaling conclusions use pretext loss as the main y-axis (Fig 3a-h); downstream curves are noisier and sometimes contradictory for iEEG model size.
  • domain assumption MNI-registered 3D electrode coordinates are meaningful and transferable across the adult-to-pediatric shift.
    Positional embeddings in Eqs. 1-2 assume cross-subject coordinate comparability; the authors themselves find removing the absolute position embedding improves Neuroprobe transfer (Table 18).
  • domain assumption Pretraining on adult ECoG/SEEG transfers to pediatric SEEG benchmarks without special adaptation.
    Neuroprobe evaluation uses an adult-to-child distribution shift; results suggest it works, but a larger shift would weaken benchmark conclusions.
  • domain assumption MAYO 1s samples can be concatenated to 6s windows and train/test swapped without changing seizure-detection semantics.
    Appendix B.3: original 1s samples were concatenated to 6s and train/test sets swapped because the test set was larger; this alters the original challenge protocol.

pith-pipeline@v1.3.0-alltime-deepseek · 45154 in / 15163 out tokens · 135022 ms · 2026-08-03T14:47:59.666068+00:00 · methodology

0 comments
read the original abstract

Intracranial EEG (iEEG) provides direct, millisecond-scale recordings of human neural activity, but reusable representation learning is difficult because electrode layouts, anatomical coverage, referencing schemes, and recording conditions vary across patients and centers. We introduce DIVER-1, a self-supervised iEEG foundation model for variable-input recordings that combines any-variate electrode-time attention, spatio-temporal resampling, input-conditioned positional embeddings, and multi-domain masked reconstruction without assuming a fixed electrode montage. We pretrain two variants, DIVER-1-0.1s and DIVER-1-1s, on 5,310 hours of ECoG and SEEG spanning 352k channel-hours, roughly 54x the BrainTreeBank-based pretraining volume. We evaluate DIVER-1 on two held-out benchmarks: Neuroprobe for naturalistic cognitive decoding and MAYO for seizure detection. On leakage-aware Neuroprobe, DIVER-1-0.1s outperforms prior evaluated iEEG foundation models despite using no BrainTreeBank recordings, the corpus underlying Neuroprobe, during pretraining; it also exceeds the linear spectrogram decoder in mean AUROC and remains competitive with stronger nonlinear baselines, a level prior evaluated iEEG foundation models did not reach. DIVER-1-1s also achieves the top AUROC on MAYO seizure detection. Finally, we conduct, to our knowledge, the first controlled compute-aware scaling study for self-supervised iEEG pretraining, sweeping data scale, subject count, training duration, and model size up to 1.8B parameters. Our results indicate a data-constrained regime: expanding unique recordings and training sufficiently long are more reliable scaling axes than increasing parameter count alone. Code is available at link.

Figures

Figures reproduced from arXiv: 2512.19097 by Ahhyun Lucy Lee, Chun Kee Chung, Danny Dongyeop Han, David Keetae Park, Jihyun Bang, Jiook Cha, Jubin Choi, Sebin Lee, Seong Jin Lee, Seungju Lee, Shinjae Yoo, Taeyang Lee, Yonghyeon Gwon.

Figure 1
Figure 1. Figure 1: Overview of DIVER-1 architecture and pretraining. DIVER-1 is pretrained on a large EEG and iEEG data corpus. After preprocessing, input patches are randomly masked and enhanced by adding modality, spectral, and CNN-based patch embeddings, along with STCPE. The enhanced patches are processed through MOIRAI blocks and trained to reconstruct missing patches across multiple signal domains (time series, spectru… view at source ↗
Figure 2
Figure 2. Figure 2: Contrasting scaling laws between DIVER-1 and LLMs. Each colored region shows how much each factor should be scaled (assuming compute optimal scaling) (multiplicative contri￾bution) when total compute increases (with epoch fixed to two). Both DIVER-1 variants (left (iEEG) and middle (EEG)) demonstrates that data is the critical bottleneck for EFMs, requiring proportion￾ally much larger increases in dataset … view at source ↗
Figure 3
Figure 3. Figure 3: Scaling laws and downstream performance of DIVER-1. (a-h) Scaling law validation: DIVER-1 follows data-constrained scaling laws across four dimensions for iEEG (a-d) and EEG (e-h) modalities. Loss decreases predictably with increased (a,e) compute (training FLOPs), (b,f) dataset size (number of tokens), (c,g) model size (parameters), and (d,h) training epochs, with strong log-log fits. iEEG experiments (a-… view at source ↗
Figure 4
Figure 4. Figure 4: Verification of the µP implementation. The L1 norm of activation vectors (y-axis) is plotted against model width (x-axis) for five training timesteps (t=1 to t=5) across four different widths (256, 512, 768, 1024). (Top Row) With standard parameterization, activation norms are un￾stable and diverge as model width increases. (Bottom Row) In contrast, our µP implementation yields stable activation norms that… view at source ↗
Figure 5
Figure 5. Figure 5: Loss curves of the DIVER-/I/1s model family. Test loss across epochs is shown [PITH_FULL_IMAGE:figures/full_fig_p024_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Loss curves of the DIVER-/I/0.1s model family. Test loss across epochs is shown. 24 [PITH_FULL_IMAGE:figures/full_fig_p024_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Loss curves of the DIVER-/E/1s model family for each dataset size. Test loss across epochs is shown. Unlike the iEEG experiments, where separate models were trained for each epoch, all EEG models were trained for a fixed 32-epoch schedule. Early-epoch losses were extracted from interme￾diate checkpoints of these longer runs. Consequently, each dataset size yields five loss curves, one for each model scale … view at source ↗
Figure 8
Figure 8. Figure 8: IsoLoss contours for DIVER-/I/0.1s model family: (Left) Twenty models with 0.1s patches were trained across varying epochs and parameter counts. Iso-loss contours are obtained by linear interpolation between measured data points. (Right) Corresponding contours predicted by the fitted scaling law. The fading line denotes the minimum-loss configuration for each compute budget. contributions to overall perfor… view at source ↗
Figure 9
Figure 9. Figure 9: Scaling law extended results for the DIVER-/I/1s family. Compute scaling and model size scaling plots are given for models trained for 2, 4, 8, 16, 32, and 64 epochs.(a) Compute scaling and (b) model size scaling plots are given for models trained for 2, 8, 32, and 64 epochs. (c) Epoch scaling plot of the models reported in [PITH_FULL_IMAGE:figures/full_fig_p028_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Scaling law extended results for the DIVER-/I/0.1s family. (a) Compute scaling and (b) model size scaling plots are given for models trained for 2, 8, 32, and 64 epochs. (c) Epoch scaling plot of the models reported in [PITH_FULL_IMAGE:figures/full_fig_p029_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Scaling law extended results for the DIVER-/E/1s family on 10% of EEG dataset. (a) Compute scaling and (b) model size scaling plots are given for test loss values extracted at epochs 2, 4, 8, 16, and 32 from models trained with a maximum of 32 epochs. For epochs 16 and 32, the fitted slope was effectively zero, so the corresponding fitted lines were omitted from the visualization. (c) Epoch scaling plot o… view at source ↗
Figure 12
Figure 12. Figure 12: Scaling law extended results for the DIVER-/E/1s family on 20% of EEG dataset. (a) Compute scaling and (b) model size scaling plots are given for test loss values extracted at epochs 2, 4, 8, 16, and 32 from models trained with a maximum of 32 epochs. (c) Epoch scaling plot of the models reported in reported in [PITH_FULL_IMAGE:figures/full_fig_p032_12.png] view at source ↗
Figure 13
Figure 13. Figure 13: Pretraining dataset and size effects on performance: BrainTreebank vs. Self-Collected [PITH_FULL_IMAGE:figures/full_fig_p041_13.png] view at source ↗
Figure 14
Figure 14. Figure 14: Visualizations of representations on neuroprobe downstream tasks. Each plot shows test [PITH_FULL_IMAGE:figures/full_fig_p043_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Scaled attention rollout weights mapped onto the cortical surface for four Neuro￾probe downstream tasks. Each panel shows DIVERTiny/I/0.1s scaled attention weights extracted via attention rollout and aggregated in 100 ms bins from 0 ms to 1 s relative to stimulus onset. For each task, the top row shows the left hemisphere and the bottom row shows the right hemisphere; brighter colors indicate larger scale… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

3 extracted references · 1 linked inside Pith

  1. [2019]

    or bayesian hyperparameter optimization. The search space was defined as±1order of mag- nitude around the best grid search configurations (range:×0.1to×10), with 50 trials conducted to systematically explore this refined hyperparameter space. The optimal hyperparameter settings identified through Optuna optimization are presented in Table 7. Table 7:Optim...

  2. [2025]

    Scaled Epochs on Our Data

    and BrainBERT (Wang et al., 2023)) are pretrained on the BrainTreebank (BTB) (Wang et al., 2024a) datasets—the precursor to Neuroprobe. Therefore, we compare downstream performance when we use BTB exclusively (Figure 13). We trained DIVER T iny/I/0.1smodels with BTB and size-variations of our self-collected iEEG datasets. For the model trained on our self...

  3. [2026]

    foundation

    URLhttps://openreview.net/forum?id=5Xwm8e6vbh. S Bbrinkm and W Cukierski. Upenn and mayo clinic’s seizure detection challenge, 2014. Geeling Chau, Christopher Wang, Sabera Talukder, Vighnesh Subramaniam, Saraswati Soedarmadji, Yisong Yue, Boris Katz, and Andrei Barbu. Population transformer: Learning population-level representations of neural activity. Ar...