Pith. sign in

REVIEW 3 major objections 5 minor 275 references

ZUNA1.1 shows that a single diffusion autoencoder can repair and up-sample EEG of variable length (0.5–30 s), with any number of channels at arbitrary scalp locations and arbitrary temporal gaps, matching a fixed-window specialist and beati

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 09:46 UTC pith:JYEL66RB

load-bearing objection Solid engineering follow-up with a real generalization gap between the abstract's 'arbitrary scalp locations' and the evidence. the 3 major comments →

arxiv 2607.27308 v1 pith:JYEL66RB submitted 2026-07-29 cs.LG q-bio.NC

ZUNA1.1: A more flexible EEG foundation model for Denoising and Super-resolution

Christopher Warner , Jonas Mago , JR Huml , Beren Millidge This is my paper
classification cs.LG q-bio.NC
keywords EEG foundation modeldiffusion autoencoderchannel reconstructionspherical spline interpolation4D-RoPEvariable-length EEGdenoisingsuper-resolution
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces ZUNA1.1, a 380M-parameter diffusion autoencoder that performs EEG reconstruction — denoising, channel infilling, and up-sampling — on variable-length recordings from 0.5 to 30 seconds, with arbitrary channel counts at arbitrary scalp locations. Its central claim is that a single model, trained on roughly 3.5 million channel-hours with a diverse mixture of dropout patterns, can replace task-specific pipelines: it matches the previous fixed-5-second ZUNA1 model on that model's home turf and clearly outperforms classical spherical-spline interpolation. The authors argue this makes the model directly useful for real-world EEG workflows, where recordings have variable duration, consumer headsets have sparse layouts, and electrodes fail in spatially or temporally correlated bursts.

Core claim

ZUNA1.1 is a transformer-based encoder–decoder diffusion autoencoder for masked EEG reconstruction, trained with a rectified-flow loss. The model's defining capability is flexibility: because it encodes electrode positions with a discretized 4D rotary positional embedding over (x, y, z, t), it can in principle process arbitrary sequences up to 30 seconds, any number of channels at any scalp locations, and reconstruct arbitrary temporal intervals within channels as well as whole channels. The paper reports that ZUNA1.1 performs at least on par with its predecessor ZUNA1 on fixed-5-second reconstruction benchmarks, while substantially outperforming spherical spline interpolation, with the gap

What carries the argument

The load-bearing mechanism is the discretized 4D-RoPE positional encoding: each channel token carries a 3D scalp coordinate (x, y, z) and a coarse time index t (0.125 s per token), so attention geometry is defined by physical electrode location and time rather than by a fixed montage or window size. On top of this, variable-length training packs randomly cropped windows (0.5–30 s) into batches with sample-aware attention masks via flex attention, and eight channel-dropout schemes (four structural, four layout-based, including consumer-headset montages) teach the model to infill diverse corruption patterns. Quality-aware loading and two precomputed filter variants provide implicit augmentatio

Load-bearing premise

The central flexibility claims rest on the discretized 4D-RoPE positional encoding extrapolating to scalp positions and temporal window lengths that were not seen during training; if that extrapolation fails, the model is only as flexible as its training distribution.

What would settle it

Take a consumer headset montage that was never in the training set (e.g., a 6-electrode layout with electrode positions lying outside the 100-bin discretization grid used in training), record real EEG, delete all channels, and ask the model to reconstruct a 256-channel high-density montage; if NMSE degrades sharply compared to reconstructions from a standard 10–20 montage, the arbitrary-location claim is falsified. A simpler check: feed the model a 30-second window longer than any it has seen in evaluation and see whether the error grows discontinuously at the training-length boundary.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A single model can replace montage-specific pipelines: a consumer four-electrode headset could be up-sampled to a dense research montage without retraining.
  • Temporal gaps in a channel, not just whole-channel loss, are covered: the model can reconstruct short intervals inside a channel using surrounding time points and other channels.
  • Variable-length inference removes the need to chop EEG into fixed 5-second segments, avoiding boundary artifacts in real recordings.
  • Because the model beats spherical spline interpolation by a widening margin as dropout increases, it offers practical robustness for noisy or partially failing recordings.
  • The authors observe that the hardest layout schemes (e.g., reconstructing 256 channels from four electrodes) show little improvement over training, implying some information is irrecoverably lost by such aggressive down-sampling.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If 4D-RoPE generalizes as claimed, the same architecture could be applied to other spatially embedded biosignal arrays (e.g., high-density EMG or ECoG grids) without modification, since the positional encoding is agnostic to signal type.
  • The paper's own ablations show reconstruction accuracy and downstream classification accuracy diverge as training proceeds, suggesting that a reconstruction objective alone may not yield the best latent representations; a hybrid objective may be needed.
  • The robustness of the flexibility claim is untested at the extremes: all evaluations use fixed 5-second windows on standard montages, so the true 'arbitrary channel layout' claim awaits a held-out test with a never-seen consumer device or an out-of-distribution scalp coordinate.
  • The open release of weights at 380M parameters makes it feasible to fine-tune or distill the model for real-time streaming applications, a direction not explored in the paper.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. ZUNA1.1 is a 380M-parameter transformer-based diffusion autoencoder for EEG reconstruction. It extends the earlier ZUNA1 model with variable-length training (0.5–30 s windows), a mixture of eight channel-dropout schemes, quality-aware preprocessing with thresholds applied at load time, and a larger training corpus (≈3.5M channel-hours). The paper claims that ZUNA1.1 can reconstruct variable-length sequences with an arbitrary number of channels at arbitrary scalp locations and arbitrary temporal intervals, that it is at least on par with ZUNA1, and that it substantially outperforms spherical spline interpolation. Reconstruction is evaluated on four held-out datasets (ANPHY-Sleep, BerlinBCI, BCI2000, AAD) under channel dropout and region occlusion, plus a duration sweep and structural dropout schemes. The model and code are released open source.

Significance. If the flexibility claims are substantiated, ZUNA1.1 would be a practically useful contribution: a single open-source model for EEG denoising, super-resolution, and temporal infilling, trained at scale and evaluated against a standard baseline. The paper's strengths include a detailed training recipe, honest discussion of observed limitations, and comparisons against spherical spline across multiple datasets with error bars in several figures. However, the central novelty—spatial and temporal flexibility—is not fully validated by the experiments as presented. The evaluation of 'arbitrary scalp locations' is limited to montages that are in-distribution, the only variable-length result lacks error bars and a baseline, and the 'at least on par' claim holds only on average, not uniformly. These gaps are load-bearing because they concern the abstract's primary claims.

major comments (3)
  1. [§II, §III.D, Fig 4] The claim of 'an arbitrary number of EEG channels at arbitrary scalp locations' (Abstract and §II) is not supported by the evaluation. All reconstruction experiments in §IV.A use the four datasets' own montages (Figs 5, 6) at fixed 5 s samples, which are explicitly at the heart of the training distribution. No held-out test reconstructs from a previously unseen sparse consumer montage to a high-density layout, and no test evaluates coordinates outside the 100-bin-per-axis 4D-RoPE discretization. The paper's own Fig 4 and §III.D show that the four layout dropout schemes plateaued at high loss and were dropped from Stage 3, indicating spatial extrapolation is the least-learned capability. Please provide an explicit held-out montage experiment (e.g., a consumer layout not used in training) or qualify the abstract and §II claims to match the evaluated regime.
  2. [Fig 7b, §IV.A] The only evidence for variable-length capability is Fig 7b (NMSE vs sample duration at 15% token dropout). This panel has no error bars, no spline baseline, and the caption merely says 'Evaluation dataset' without identifying the dataset or sample size. Since variable-length reconstruction is a central contribution, this figure needs error bars, a named dataset, and ideally a comparison to spline or to ZUNA1 to establish that the model generalizes across durations. All other figures use 5 s samples, so the temporal-range claim rests entirely on this single panel. In addition, 'arbitrary temporal intervals within channels' is not directly tested: the structural schemes (Fig 8) drop channels or time blocks, but no experiment isolates infilling a specific temporal interval in a single channel with other channels intact.
  3. [Table IV, §IV.A] The abstract and §I state that ZUNA1.1 is 'at least on par with our earlier ZUNA1 model', but Table IV shows this is only true on average. ZUNA1.1 is worse than ZUNA1 on BerlinBCI at all four dropout levels (e.g., 50%: 0.737±0.022 vs 0.674±0.017; 90%: 1.486±0.069 vs 1.135±0.025) and on AAD at 20%, 50%, and 75% dropout. The paper acknowledges this in §IV.A ('on average but not uniformly'), but the abstract does not carry that qualification. Either reword the abstract to explicitly state 'on average' or report an appropriate aggregate statistical test (e.g., paired comparison across datasets/conditions) to justify unqualified parity.
minor comments (5)
  1. [§III.B, Eq. (1)] The notation 'e2(ρc,s−3.5)' is ambiguous. Please use a clearer exponential form, e.g., exp(2(ρc,s − 3.5)), or define the function.
  2. [Fig 7 caption] The caption says 'Evaluation dataset' without naming it. Specify which dataset and how many samples/trials were used, and include error bars.
  3. [Table I and Fig 4] The scheme named 'full-time-pt-random' in Table I appears as 'full-time-pt-random' in Fig 4; ensure consistent naming. Also, Fig 4's y-axis label 'NMSE' could benefit from a note on the evaluation set and the fixed 5 s window.
  4. [§IV.A] The sentence beginning 'Note that 5 second samples are also at the heart of ZUNA1.1’s training distribution' is relevant to the interpretation of all results. It should appear earlier, in the experimental setup, so that the in-distribution nature of the main evaluations is transparent.
  5. [Appendix C, Tables V–VI] The checkpoint labels 'No MMD-50K', 'MMD–50K', 'Hybrid–400K', and 'Hybrid–575K' are not defined in the main text or table captions. A sentence explaining these training stages would improve reproducibility.

Circularity Check

0 steps flagged

No load-bearing circularity; minor self-citation and test-selection caveats only.

full rationale

The paper's central claims are empirical, not derived by fitting a theory to its own outputs. ZUNA1.1 is trained on a 3.5M channel-hour corpus with variable-length windows and a mixture of eight dropout schemes (Sections III.A-III.D), and the main comparisons are NMSE on held-out recordings against an external spherical-spline baseline (Perrin et al., 1989) and a separately trained predecessor ZUNA1 (Warner et al., 2026). No equation in the paper defines a predicted quantity in terms of the same fitted values used to produce it; the rectified-flow diffusion objective and the evaluation metric are standard, and no 'uniqueness theorem' or ansatz is imported from the authors' prior work to forbid alternative architectures. The architecture is inherited from the authors' own ZUNA1 release via citation, but this is an ordinary incremental-model self-citation rather than a load-bearing circular premise; the new contributions (variable-length packing, quality-aware preprocessing, filter variants, and the dropout mixture) are specified in the present paper. Two caveats are genuine but non-circular correctness risks: the 'frozen evaluation dataset' in Fig. 4 appears to have been used to tune the Stage 2/Stage 3 curriculum, so Figure 7 is not a fully independent test of the final training recipe; and the abstract's 'arbitrary scalp locations' is not tested on a held-out montage, with the paper itself reporting that the layout schemes plateaued and were dropped from Stage 3 (Section III.D). These are generalization and test-hygiene concerns, not reductions by construction. I therefore find no self-definitional or fitted-input/prediction circularity; the score of 2 reflects one minor self-citation and mild evaluation-selection caveats, not a circular derivation.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 0 invented entities

The central claim is supported by empirical evaluation rather than derivation, but hand-set quality thresholds, training-bin weights, and the curriculum schedule are free parameters that shape model behavior. No new physical entities are introduced.

free parameters (6)
  • min_quality_any / min_quality_mean thresholds = 0.1 / 0.3
    Hand-set load-time thresholds deciding which channel-seconds enter training; no sensitivity analysis is reported (Section III-B).
  • Window-length bin weights = 0.20 / 0.30 / 0.30 / 0.20
    Sampling distribution over 0.5-1, 1-5, 5-10, 10-30s windows chosen by hand to emphasize the 1-10s regime (Section III-A).
  • Dropout curriculum scheme weights and p_drop = Stage weights in Table I; p_drop 0.9 -> 0.99
    Scheme weights and difficulty were adjusted at 250k/440k steps based on validation-loss plateaus (Section III-D).
  • Non-dropped token loss weight = 0.1
    Observed-token loss downweighted to 10% of dropped-token weight to concentrate on infilling; chosen without a reported sweep (Section III-E).
  • MMD loss weight = 1e-2 -> 0 at 400k steps
    MMD regularizer turned off after ablations improved classification; affects latent representations and downstream behavior (Appendix C).
  • Inference ODE steps and initial noise scale = 50 steps; 0.1
    Decoder sampling hyperparameters used for all evaluations; no sensitivity analysis is given (Appendix A).
axioms (4)
  • domain assumption 4D-RoPE over discretized (x,y,z,t) generalizes to arbitrary scalp locations and sequence lengths up to 30s
    Underlies the arbitrary-location/length capability claim; only trained on existing montages and crops, with no held-out position extrapolation test (Section II, Figure 7b).
  • domain assumption The eight synthetic dropout schemes adequately approximate real-world EEG corruption patterns
    Used to justify the training curriculum; no evaluation on naturally corrupted recordings is reported (Section III-D).
  • domain assumption Per-window z-score normalization plus two filter variants gives sufficient implicit augmentation to prevent overfitting over six epochs
    The paper claims no overfitting in validation loss but does not show detailed loss curves (Section III-D).
  • domain assumption NMSE on four held-out datasets is an adequate proxy for reconstruction quality across the claimed flexible use cases
    Main evaluation is on 5s samples; temporal and arbitrary-montage generalization is only partially covered (Section IV).

pith-pipeline@v1.3.0-daily-deepseek · 15136 in / 13469 out tokens · 115366 ms · 2026-08-01T09:46:57.046736+00:00 · methodology

0 comments
read the original abstract

We introduce ZUNA1.1, a 380M-parameter diffusion autoencoder for flexible EEG signal reconstruction. ZUNA1.1 is capable of reconstructing variable length sequences of up to 30s, with an arbitrary number of EEG channels at arbitrary scalp locations, and can reconstruct arbitrary temporal intervals within channels in addition to reconstructing entire channels. We demonstrate that ZUNA1.1 performs at least on par with our earlier ZUNA1 model, while being far more flexible and capable of handling a wide range of reconstruction tasks. ZUNA1.1 continues to substantially outperform standard EEG denoising and reconstruction methods such as spherical spline interpolation, which is ubiquitously deployed in the MNE package. The ZUNA1.1 model is released open source under the permissive Apache 2.0 license.

Figures

Figures reproduced from arXiv: 2607.27308 by Beren Millidge, Christopher Warner, Jonas Mago, JR Huml.

Figure 1
Figure 1. Figure 1: Per-region reconstruction accuracy (topographic). Average NMSE by scalp region for ZUNA1.1, ZUNA1 (labeled with the ∆ relative to ZUNA1.1), and spherical-spline, averaged over the four datasets. All electrodes in one region are deleted and reconstructed from the remaining seven regions. Lower (greener) is better. more common pattern is bursts of errors, often individual or correlated across many channels d… view at source ↗
Figure 2
Figure 2. Figure 2: ZUNA1.1 architecture. A transformer-based diffusion autoencoder. The main modifications to the ZUNA1 architecture are adding sandwich norm and QK-norm to improve training stability. This allowed us to remove Adaptive Loss Weighting from the decoder RF loss term. TABLE I Channel-dropout schemes and their sampling weights across the three training stages. Stage 1 (0–250k steps) samples all eight schemes with… view at source ↗
Figure 3
Figure 3. Figure 3: The four channel-dropout schemes retained throughout training. Each panel shows one training sample: channels run along the vertical axis and coarse-time windows (τ = 0.125 s) along the horizontal axis. Observed tokens (unshaded traces) are passed to the encoder; the shaded spans are set to zero in the encoder and reconstructed by the decoder based on observed signals. full-channel-random removes entire ch… view at source ↗
Figure 4
Figure 4. Figure 4: Reconstruction performance for each dropout scheme throughout training. Median NMSE (lower is better) on the frozen evaluation dataset for each dropout scheme. See Table I for details on how the dropout schemes were adjusted during training [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Reconstruction accuracy as channels drop. Reconstruction NMSE versus channel-dropout rate on four held-out datasets (ANPHY-Sleep, BerlinBCI, BCI2000, AAD), comparing ZUNA1.1, ZUNA1, and MNE spherical-spline interpolation; lower is better. ZUNA1.1 matches or improves on ZUNA1, and both clearly outperform spline interpolation, with the gap widening as more channels are removed. Evaluation restricted to 5 s s… view at source ↗
Figure 6
Figure 6. Figure 6: Per-region reconstruction error. The region-occlusion evaluation of fig. 1 shown as grouped bars: average NMSE across the four datasets with error bars giving the propagated standard deviation. Lower is better. stability issues worth recording. Rare extreme samples produced sudden gradient-norm spikes followed by loss excursions, and in a handful of cases NaN losses. We addressed these with three safeguard… view at source ↗
Figure 7
Figure 7. Figure 7: Reconstruction performance for 4 structural dropout schemes on masked tokens in Evaluation dataset. NMSE (lower is better). Description in [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: ZUNA1.1 vs MNE Spherical Spline - reconstruction performance vs dropout percent on mixed evaluation dataset for structural dropout schemes V. DISCUSSION ZUNA1.1 builds upon and substantially extends our original ZUNA1 model, making it far more flexible and ultimately usable in real-world situations by EEG scientists and practitioners. The new dropout training schemes we introduced into the training of ZUNA… view at source ↗
Figure 9
Figure 9. Figure 9: Model ranking on downstream tasks. Models are ordered by their mean normalized rank, displayed on the right hand side of the boxes. Lower is better. ZUNA1.1 is shown for No MMD-50K. TABLE VII Classification accuracy for ZUNA1.1 (No MMD) compared with NeuralBench EEG foundation models and baseline methods. Dataset ZUNA1.1 BENDR BIOT CBraMod LaBraM LUNA REVE Dummy Handcrafted Artifact 0.431±0.016 0.502±0.017… view at source ↗
Figure 10
Figure 10. Figure 10: Balanced accuracy for linear classifiers attached at different ZUNA representation layers. We ran this experiment to probe the representational quality [PITH_FULL_IMAGE:figures/full_fig_p015_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

275 extracted references · 76 linked inside Pith

  1. [1]

    2021 , eprint=

    CogView: Mastering Text-to-Image Generation via Transformers , author=. 2021 , eprint=

    2021

  2. [2]

    2021 , eprint=

    Masked Autoencoders Are Scalable Vision Learners , author=. 2021 , eprint=

    2021

  3. [3]

    2026 , eprint=

    EEG-FM-Bench: A Comprehensive Benchmark for the Systematic Evaluation and Diagnostic Analyses of EEG Foundation Models , author=. 2026 , eprint=

    2026

  4. [4]

    arXiv preprint arXiv:2605.14698 , year=

    NeuroAtlas: Benchmarking Foundation Models for Clinical EEG and Brain-Computer Interfaces , author=. arXiv preprint arXiv:2605.14698 , year=

    Pith/arXiv arXiv

  5. [5]

    2020 , eprint=

    Query-Key Normalization for Transformers , author=. 2020 , eprint=

    2020

  6. [6]

    2024 , eprint=

    Consistency Models Made Easy , author=. 2024 , eprint=

    2024

  7. [7]

    IEEE Transactions on Neural Systems and Rehabilitation Engineering , year=

    DeepSleepNet: A Model for Automatic Sleep Stage Scoring based on Raw Single-Channel EEG , author=. IEEE Transactions on Neural Systems and Rehabilitation Engineering , year=

  8. [8]

    arXiv preprint arXiv:2602.18478 , year=

    ZUNA: Flexible EEG Superresolution with Position-Aware Diffusion Autoencoders , author=. arXiv preprint arXiv:2602.18478 , year=. 2602.18478 , archivePrefix=

    arXiv

  9. [9]

    arXiv preprint arXiv:2605.08495 , year=

    NeuralBench: A Unifying Framework to Benchmark NeuroAI Models , author=. arXiv preprint arXiv:2605.08495 , year=

    Pith/arXiv arXiv

  10. [10]

    Advances in neural information processing systems , volume=

    Attention is all you need , author=. Advances in neural information processing systems , volume=

  11. [11]

    2025 , eprint=

    Can Language Models Falsify? Evaluating Algorithmic Reasoning with Counterexample Creation , author=. 2025 , eprint=

    2025

  12. [12]

    2026 , eprint=

    Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMs , author=. 2026 , eprint=

    2026

  13. [13]

    arXiv preprint arXiv:2512.13961 , pages=

    Olmo 3 , author=. arXiv preprint arXiv:2512.13961 , pages=

    Pith/arXiv arXiv

  14. [14]

    2026 , howpublished =

    2026

  15. [15]

    2025 , url=

    INTELLECT-3: Technical Report , author=. 2025 , url=

    2025

  16. [16]

    arXiv preprint arXiv:2507.06261 , year=

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities , author=. arXiv preprint arXiv:2507.06261 , year=

    Pith/arXiv arXiv

  17. [17]

    2025 , eprint=

    DAPO: An Open-Source LLM Reinforcement Learning System at Scale , author=. 2025 , eprint=

    2025

  18. [18]

    DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models , author=

  19. [19]

    2025 , eprint=

    Qwen3 Technical Report , author=. 2025 , eprint=

    2025

  20. [20]

    2025 , month = aug, day =

    2025

  21. [21]

    2025 , eprint=

    CodeI/O: Condensing Reasoning Patterns via Code Input-Output Prediction , author=. 2025 , eprint=

    2025

  22. [22]

    2026 , eprint =

    Maximum Likelihood Reinforcement Learning , author =. 2026 , eprint =

    2026

  23. [23]

    Hugging Face repository , howpublished =

    CodeForces , author=. Hugging Face repository , howpublished =. 2025 , publisher =

    2025

  24. [24]

    2025 , eprint=

    CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesis , author=. 2025 , eprint=

    2025

  25. [25]

    arXiv preprint arXiv:2402.00838 , year=

    Olmo: Accelerating the science of language models , author=. arXiv preprint arXiv:2402.00838 , year=

    Pith/arXiv arXiv

  26. [26]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    High-resolution image synthesis with latent diffusion models , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  27. [27]

    arXiv preprint arXiv:2406.03476 , year=

    Does your data spark joy? Performance gains from domain upsampling at the end of training , author=. arXiv preprint arXiv:2406.03476 , year=

    Pith/arXiv arXiv

  28. [28]

    arXiv preprint arXiv:2406.07887 , year=

    An Empirical Study of Mamba-based Language Models , author=. arXiv preprint arXiv:2406.07887 , year=

    Pith/arXiv arXiv

  29. [29]

    International conference on machine learning , pages=

    Transformers are rnns: Fast autoregressive transformers with linear attention , author=. International conference on machine learning , pages=. 2020 , organization=

    2020

  30. [30]

    arXiv preprint arXiv:2312.00752 , year=

    Mamba: Linear-time sequence modeling with selective state spaces , author=. arXiv preprint arXiv:2312.00752 , year=

    Pith/arXiv arXiv

  31. [31]

    arXiv preprint arXiv:2402.01771 , year=

    BlackMamba: Mixture of Experts for State-Space Models , author=. arXiv preprint arXiv:2402.01771 , year=

    Pith/arXiv arXiv

  32. [32]

    https://huggingface.co/Zyphra/Mamba-370M , year=

    Zyphra , title=. https://huggingface.co/Zyphra/Mamba-370M , year=

  33. [33]

    https://huggingface.co/datasets/BAAI/Infinity-Instruct , year=

    BAAI , title=. https://huggingface.co/datasets/BAAI/Infinity-Instruct , year=

  34. [34]

    https://huggingface.co/datasets/Intel/orca\_dpo\_pairs , year=

    BAAI , title=. https://huggingface.co/datasets/Intel/orca\_dpo\_pairs , year=

  35. [35]

    HuggingFace repository , howpublished =

    OpenOrca: An Open Dataset of GPT Augmented FLAN Reasoning Traces , author =. HuggingFace repository , howpublished =. 2023 , publisher =

    2023

  36. [36]

    arXiv preprint arXiv:2307.09288 , year=

    Llama 2: Open foundation and fine-tuned chat models , author=. arXiv preprint arXiv:2307.09288 , year=

    Pith/arXiv arXiv

  37. [37]

    International Conference on Machine Learning , pages=

    Pythia: A suite for analyzing large language models across training and scaling , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  38. [38]

    2026 , eprint=

    GLM-5: from Vibe Coding to Agentic Engineering , author=. 2026 , eprint=

    2026

  39. [39]

    arXiv preprint arXiv:2312.06550 , year=

    Llm360: Towards fully transparent open-source llms , author=. arXiv preprint arXiv:2312.06550 , year=

    Pith/arXiv arXiv

  40. [40]

    arXiv preprint arXiv:2404.06395 , year=

    MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies , author=. arXiv preprint arXiv:2404.06395 , year=

    Pith/arXiv arXiv

  41. [41]

    arXiv preprint arXiv:2402.16819 , year=

    Nemotron-4 15B Technical Report , author=. arXiv preprint arXiv:2402.16819 , year=

    Pith/arXiv arXiv

  42. [42]

    arXiv preprint arXiv:2404.07413 , year=

    JetMoE: Reaching Llama2 Performance with 0.1 M Dollars , author=. arXiv preprint arXiv:2404.07413 , year=

    Pith/arXiv arXiv

  43. [43]

    arXiv preprint arXiv:2310.06825 , year=

    Mistral 7B , author=. arXiv preprint arXiv:2310.06825 , year=

    Pith/arXiv arXiv

  44. [44]

    arXiv preprint arXiv:2403.08295 , year=

    Gemma: Open models based on gemini research and technology , author=. arXiv preprint arXiv:2403.08295 , year=

    Pith/arXiv arXiv

  45. [45]

    2024 , eprint=

    Gemma 2: Improving Open Language Models at a Practical Size , author=. 2024 , eprint=

    2024

  46. [46]

    CoRR , volume =

    Alexey Dosovitskiy and Lucas Beyer and Alexander Kolesnikov and Dirk Weissenborn and Xiaohua Zhai and Thomas Unterthiner and Mostafa Dehghani and Matthias Minderer and Georg Heigold and Sylvain Gelly and Jakob Uszkoreit and Neil Houlsby , title =. CoRR , volume =. 2020 , url =. 2010.11929 , timestamp =

    Pith/arXiv arXiv 2020

  47. [47]

    Tom B. Brown and Benjamin Mann and Nick Ryder and Melanie Subbiah and Jared Kaplan and Prafulla Dhariwal and Arvind Neelakantan and Pranav Shyam and Girish Sastry and Amanda Askell and Sandhini Agarwal and Ariel Herbert. Language Models are Few-Shot Learners , journal =. 2020 , url =. 2005.14165 , timestamp =

    Pith/arXiv arXiv 2020

  48. [48]

    2022 , eprint=

    Training Compute-Optimal Large Language Models , author=. 2022 , eprint=

    2022

  49. [49]

    Signal Transduction and Targeted Therapy , year=

    Yang, Zhenyu and Zeng, Xiaoxi and Zhao, Yi and Chen, Runsheng , title=. Signal Transduction and Targeted Therapy , year=. doi:10.1038/s41392-023-01381-z , url=

    doi:10.1038/s41392-023-01381-z

  50. [50]

    2022 , eprint=

    Scaling Language Models: Methods, Analysis & Insights from Training Gopher , author=. 2022 , eprint=

    2022

  51. [51]

    2024 , eprint=

    Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models , author=. 2024 , eprint=

    2024

  52. [52]

    2024 , eprint=

    Gated Linear Attention Transformers with Hardware-Efficient Training , author=. 2024 , eprint=

    2024

  53. [53]

    arXiv preprint arXiv:2312.11805 , year=

    Gemini: a family of highly capable multimodal models , author=. arXiv preprint arXiv:2312.11805 , year=

    Pith/arXiv arXiv

  54. [54]

    arXiv preprint arXiv:2404.14219 , year=

    Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone , author=. arXiv preprint arXiv:2404.14219 , year=

    Pith/arXiv arXiv

  55. [55]

    Advances in Neural Information Processing Systems , volume=

    Doremi: Optimizing data mixtures speeds up language model pretraining , author=. Advances in Neural Information Processing Systems , volume=

  56. [56]

    Advances in Neural Information Processing Systems , volume=

    D4: Improving llm pretraining via document de-duplication and diversification , author=. Advances in Neural Information Processing Systems , volume=

  57. [57]

    arXiv preprint arXiv:2401.16380 , year=

    Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling , author=. arXiv preprint arXiv:2401.16380 , year=

    Pith/arXiv arXiv

  58. [58]

    arXiv preprint arXiv:2309.05463 , year=

    Textbooks are all you need ii: phi-1.5 technical report , author=. arXiv preprint arXiv:2309.05463 , year=

    Pith/arXiv arXiv

  59. [59]

    arXiv preprint arXiv:2212.14052 , year=

    Hungry hungry hippos: Towards language modeling with state space models , author=. arXiv preprint arXiv:2212.14052 , year=

    Pith/arXiv arXiv

  60. [60]

    arXiv preprint arXiv:2111.00396 , year=

    Efficiently modeling long sequences with structured state spaces , author=. arXiv preprint arXiv:2111.00396 , year=

    Pith/arXiv arXiv

  61. [61]

    arXiv preprint arXiv:2403.17844 , year=

    Mechanistic Design and Scaling of Hybrid Architectures , author=. arXiv preprint arXiv:2403.17844 , year=

    Pith/arXiv arXiv

  62. [62]

    Journal of Machine Learning Research , volume=

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity , author=. Journal of Machine Learning Research , volume=

  63. [63]

    International conference on machine learning , pages=

    Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale , author=. International conference on machine learning , pages=. 2022 , organization=

    2022

  64. [64]

    GPT-J-6B: A 6 billion parameter autoregressive language model , author=

  65. [65]

    Advances in Neural Information Processing Systems , volume=

    Direct preference optimization: Your language model is secretly a reward model , author=. Advances in Neural Information Processing Systems , volume=

  66. [66]

    Advances in neural information processing systems , volume=

    Combining recurrent, convolutional, and continuous-time models with linear state space layers , author=. Advances in neural information processing systems , volume=

  67. [67]

    arXiv preprint arXiv:2403.08763 , year=

    Simple and scalable strategies to continually pre-train large language models , author=. arXiv preprint arXiv:2403.08763 , year=

    Pith/arXiv arXiv

  68. [68]

    Advances in neural information processing systems , volume=

    Hippo: Recurrent memory with optimal polynomial projections , author=. Advances in neural information processing systems , volume=

  69. [69]

    2023 , publisher =

    OpenHermes 2.5: An Open Dataset of Synthetic Data for Generalist LLM Assistants , author =. 2023 , publisher =

    2023

  70. [70]

    2024 , month =

    Bamba: Inference-Efficient Hybrid Mamba2 Model , author =. 2024 , month =

    2024

  71. [71]

    arXiv preprint arXiv:1412.6980 , year=

    Adam: A method for stochastic optimization , author=. arXiv preprint arXiv:1412.6980 , year=

    Pith/arXiv arXiv

  72. [72]

    arXiv preprint arXiv:2106.09685 , year=

    Lora: Low-rank adaptation of large language models , author=. arXiv preprint arXiv:2106.09685 , year=

    Pith/arXiv arXiv

  73. [73]

    arXiv preprint arXiv:1712.00409 , year=

    Deep learning scaling is predictable, empirically , author=. arXiv preprint arXiv:1712.00409 , year=

    Pith/arXiv arXiv

  74. [74]

    arXiv preprint arXiv:2402.04248 , year=

    Can mamba learn how to learn? a comparative study on in-context learning tasks , author=. arXiv preprint arXiv:2402.04248 , year=

    Pith/arXiv arXiv

  75. [75]

    arXiv preprint arXiv:2406.08464 , year=

    Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing , author=. arXiv preprint arXiv:2406.08464 , year=

    Pith/arXiv arXiv

  76. [76]

    arXiv preprint arXiv:1807.03819 , year=

    Universal transformers , author=. arXiv preprint arXiv:1807.03819 , year=

    Pith/arXiv arXiv

  77. [77]

    PLoS Computational Biology , volume=

    Hybrid predictive coding: Inferring, fast and slow , author=. PLoS Computational Biology , volume=. 2023 , publisher=

    2023

  78. [78]

    Current Opinion in Neurobiology , volume=

    Going in circles is the way forward: the role of recurrence in visual inference , author=. Current Opinion in Neurobiology , volume=. 2020 , publisher=

    2020

  79. [79]

    arXiv preprint arXiv:2101.00027 , year=

    The pile: An 800gb dataset of diverse text for language modeling , author=. arXiv preprint arXiv:2101.00027 , year=

    Pith/arXiv arXiv

  80. [80]

    arXiv preprint arXiv:2306.01116 , year=

    The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only , author=. arXiv preprint arXiv:2306.01116 , year=

    Pith/arXiv arXiv

Showing first 80 references.