Pith. sign in

REVIEW 3 major objections 5 minor 275 references

ZUNA1.1: A more flexible EEG foundation model for Denoising and Super-resolution

T0 review · 3 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ZUNA1.1 shows that a single diffusion autoencoder can repair and up-sample EEG of variable length (0.5–30 s), with any number of channels at arbitrary scalp locations and arbitrary temporal gaps, matching a fixed-window specialist and beati

desk verdict Solid engineering follow-up with a real generalization gap between the abstract's 'arbitrary scalp locations' and the evidence. read the letter →

arxiv 2607.27308 v1 pith:JYEL66RB submitted 2026-07-29 cs.LG q-bio.NC

Christopher Warner , Jonas Mago , JR Huml , Beren Millidge This is my paper · ORCID
classification cs.LGq-bio.NC
keywords EEGfoundationmodeldiffusionautoencoderchannelreconstructionsphericalsplineinterpolation4D-RoPEvariable-lengthdenoisingsuper-resolution
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces ZUNA1.1, a 380M-parameter diffusion autoencoder that performs EEG reconstruction — denoising, channel infilling, and up-sampling — on variable-length recordings from 0.5 to 30 seconds, with arbitrary channel counts at arbitrary scalp locations. Its central claim is that a single model, trained on roughly 3.5 million channel-hours with a diverse mixture of dropout patterns, can replace task-specific pipelines: it matches the previous fixed-5-second ZUNA1 model on that model's home turf and clearly outperforms classical spherical-spline interpolation. The authors argue this makes the model directly useful for real-world EEG workflows, where recordings have variable duration, consumer headsets have sparse layouts, and electrodes fail in spatially or temporally correlated bursts.

What carries the argument

The load-bearing mechanism is the discretized 4D-RoPE positional encoding: each channel token carries a 3D scalp coordinate (x, y, z) and a coarse time index t (0.125 s per token), so attention geometry is defined by physical electrode location and time rather than by a fixed montage or window size. On top of this, variable-length training packs randomly cropped windows (0.5–30 s) into batches with sample-aware attention masks via flex attention, and eight channel-dropout schemes (four structural, four layout-based, including consumer-headset montages) teach the model to infill diverse corruption patterns. Quality-aware loading and two precomputed filter variants provide implicit augmentatio

What would settle it

Take a consumer headset montage that was never in the training set (e.g., a 6-electrode layout with electrode positions lying outside the 100-bin discretization grid used in training), record real EEG, delete all channels, and ask the model to reconstruct a 256-channel high-density montage; if NMSE degrades sharply compared to reconstructions from a standard 10–20 montage, the arbitrary-location claim is falsified. A simpler check: feed the model a 30-second window longer than any it has seen in evaluation and see whether the error grows discontinuously at the training-length boundary.

Watch

Extended reading notes

Core claim

ZUNA1.1 is a transformer-based encoder–decoder diffusion autoencoder for masked EEG reconstruction, trained with a rectified-flow loss. The model's defining capability is flexibility: because it encodes electrode positions with a discretized 4D rotary positional embedding over (x, y, z, t), it can in principle process arbitrary sequences up to 30 seconds, any number of channels at any scalp locations, and reconstruct arbitrary temporal intervals within channels as well as whole channels. The paper reports that ZUNA1.1 performs at least on par with its predecessor ZUNA1 on fixed-5-second reconstruction benchmarks, while substantially outperforming spherical spline interpolation, with the gap

Load-bearing premise

The central flexibility claims rest on the discretized 4D-RoPE positional encoding extrapolating to scalp positions and temporal window lengths that were not seen during training; if that extrapolation fails, the model is only as flexible as its training distribution.

Editorial extensions

If this is right

  • A single model can replace montage-specific pipelines: a consumer four-electrode headset could be up-sampled to a dense research montage without retraining.
  • Temporal gaps in a channel, not just whole-channel loss, are covered: the model can reconstruct short intervals inside a channel using surrounding time points and other channels.
  • Variable-length inference removes the need to chop EEG into fixed 5-second segments, avoiding boundary artifacts in real recordings.
  • Because the model beats spherical spline interpolation by a widening margin as dropout increases, it offers practical robustness for noisy or partially failing recordings.
  • The authors observe that the hardest layout schemes (e.g., reconstructing 256 channels from four electrodes) show little improvement over training, implying some information is irrecoverably lost by such aggressive down-sampling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If 4D-RoPE generalizes as claimed, the same architecture could be applied to other spatially embedded biosignal arrays (e.g., high-density EMG or ECoG grids) without modification, since the positional encoding is agnostic to signal type.
  • The paper's own ablations show reconstruction accuracy and downstream classification accuracy diverge as training proceeds, suggesting that a reconstruction objective alone may not yield the best latent representations; a hybrid objective may be needed.
  • The robustness of the flexibility claim is untested at the extremes: all evaluations use fixed 5-second windows on standard montages, so the true 'arbitrary channel layout' claim awaits a held-out test with a never-seen consumer device or an out-of-distribution scalp coordinate.
  • The open release of weights at 380M parameters makes it feasible to fine-tune or distill the model for real-time streaming applications, a direction not explored in the paper.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. ZUNA1.1 is a 380M-parameter transformer-based diffusion autoencoder for EEG reconstruction. It extends the earlier ZUNA1 model with variable-length training (0.5–30 s windows), a mixture of eight channel-dropout schemes, quality-aware preprocessing with thresholds applied at load time, and a larger training corpus (≈3.5M channel-hours). The paper claims that ZUNA1.1 can reconstruct variable-length sequences with an arbitrary number of channels at arbitrary scalp locations and arbitrary temporal intervals, that it is at least on par with ZUNA1, and that it substantially outperforms spherical spline interpolation. Reconstruction is evaluated on four held-out datasets (ANPHY-Sleep, BerlinBCI, BCI2000, AAD) under channel dropout and region occlusion, plus a duration sweep and structural dropout schemes. The model and code are released open source.

Significance. If the flexibility claims are substantiated, ZUNA1.1 would be a practically useful contribution: a single open-source model for EEG denoising, super-resolution, and temporal infilling, trained at scale and evaluated against a standard baseline. The paper's strengths include a detailed training recipe, honest discussion of observed limitations, and comparisons against spherical spline across multiple datasets with error bars in several figures. However, the central novelty—spatial and temporal flexibility—is not fully validated by the experiments as presented. The evaluation of 'arbitrary scalp locations' is limited to montages that are in-distribution, the only variable-length result lacks error bars and a baseline, and the 'at least on par' claim holds only on average, not uniformly. These gaps are load-bearing because they concern the abstract's primary claims.

major comments (3)
  1. [§II, §III.D, Fig 4] The claim of 'an arbitrary number of EEG channels at arbitrary scalp locations' (Abstract and §II) is not supported by the evaluation. All reconstruction experiments in §IV.A use the four datasets' own montages (Figs 5, 6) at fixed 5 s samples, which are explicitly at the heart of the training distribution. No held-out test reconstructs from a previously unseen sparse consumer montage to a high-density layout, and no test evaluates coordinates outside the 100-bin-per-axis 4D-RoPE discretization. The paper's own Fig 4 and §III.D show that the four layout dropout schemes plateaued at high loss and were dropped from Stage 3, indicating spatial extrapolation is the least-learned capability. Please provide an explicit held-out montage experiment (e.g., a consumer layout not used in training) or qualify the abstract and §II claims to match the evaluated regime.
  2. [Fig 7b, §IV.A] The only evidence for variable-length capability is Fig 7b (NMSE vs sample duration at 15% token dropout). This panel has no error bars, no spline baseline, and the caption merely says 'Evaluation dataset' without identifying the dataset or sample size. Since variable-length reconstruction is a central contribution, this figure needs error bars, a named dataset, and ideally a comparison to spline or to ZUNA1 to establish that the model generalizes across durations. All other figures use 5 s samples, so the temporal-range claim rests entirely on this single panel. In addition, 'arbitrary temporal intervals within channels' is not directly tested: the structural schemes (Fig 8) drop channels or time blocks, but no experiment isolates infilling a specific temporal interval in a single channel with other channels intact.
  3. [Table IV, §IV.A] The abstract and §I state that ZUNA1.1 is 'at least on par with our earlier ZUNA1 model', but Table IV shows this is only true on average. ZUNA1.1 is worse than ZUNA1 on BerlinBCI at all four dropout levels (e.g., 50%: 0.737±0.022 vs 0.674±0.017; 90%: 1.486±0.069 vs 1.135±0.025) and on AAD at 20%, 50%, and 75% dropout. The paper acknowledges this in §IV.A ('on average but not uniformly'), but the abstract does not carry that qualification. Either reword the abstract to explicitly state 'on average' or report an appropriate aggregate statistical test (e.g., paired comparison across datasets/conditions) to justify unqualified parity.
minor comments (5)
  1. [§III.B, Eq. (1)] The notation 'e2(ρc,s−3.5)' is ambiguous. Please use a clearer exponential form, e.g., exp(2(ρc,s − 3.5)), or define the function.
  2. [Fig 7 caption] The caption says 'Evaluation dataset' without naming it. Specify which dataset and how many samples/trials were used, and include error bars.
  3. [Table I and Fig 4] The scheme named 'full-time-pt-random' in Table I appears as 'full-time-pt-random' in Fig 4; ensure consistent naming. Also, Fig 4's y-axis label 'NMSE' could benefit from a note on the evaluation set and the fixed 5 s window.
  4. [§IV.A] The sentence beginning 'Note that 5 second samples are also at the heart of ZUNA1.1’s training distribution' is relevant to the interpretation of all results. It should appear earlier, in the experimental setup, so that the in-distribution nature of the main evaluations is transparent.
  5. [Appendix C, Tables V–VI] The checkpoint labels 'No MMD-50K', 'MMD–50K', 'Hybrid–400K', and 'Hybrid–575K' are not defined in the main text or table captions. A sentence explaining these training stages would improve reproducibility.

Circularity Check

0 steps flagged · score 2.0 of 10

No load-bearing circularity; minor self-citation and test-selection caveats only.

full rationale

The paper's central claims are empirical, not derived by fitting a theory to its own outputs. ZUNA1.1 is trained on a 3.5M channel-hour corpus with variable-length windows and a mixture of eight dropout schemes (Sections III.A-III.D), and the main comparisons are NMSE on held-out recordings against an external spherical-spline baseline (Perrin et al., 1989) and a separately trained predecessor ZUNA1 (Warner et al., 2026). No equation in the paper defines a predicted quantity in terms of the same fitted values used to produce it; the rectified-flow diffusion objective and the evaluation metric are standard, and no 'uniqueness theorem' or ansatz is imported from the authors' prior work to forbid alternative architectures. The architecture is inherited from the authors' own ZUNA1 release via citation, but this is an ordinary incremental-model self-citation rather than a load-bearing circular premise; the new contributions (variable-length packing, quality-aware preprocessing, filter variants, and the dropout mixture) are specified in the present paper. Two caveats are genuine but non-circular correctness risks: the 'frozen evaluation dataset' in Fig. 4 appears to have been used to tune the Stage 2/Stage 3 curriculum, so Figure 7 is not a fully independent test of the final training recipe; and the abstract's 'arbitrary scalp locations' is not tested on a held-out montage, with the paper itself reporting that the layout schemes plateaued and were dropped from Stage 3 (Section III.D). These are generalization and test-hygiene concerns, not reductions by construction. I therefore find no self-definitional or fitted-input/prediction circularity; the score of 2 reflects one minor self-citation and mild evaluation-selection caveats, not a circular derivation.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim is supported by empirical evaluation rather than derivation, but hand-set quality thresholds, training-bin weights, and the curriculum schedule are free parameters that shape model behavior. No new physical entities are introduced.

free parameters (6)
  • min_quality_any / min_quality_mean thresholds = 0.1 / 0.3
    Hand-set load-time thresholds deciding which channel-seconds enter training; no sensitivity analysis is reported (Section III-B).
  • Window-length bin weights = 0.20 / 0.30 / 0.30 / 0.20
    Sampling distribution over 0.5-1, 1-5, 5-10, 10-30s windows chosen by hand to emphasize the 1-10s regime (Section III-A).
  • Dropout curriculum scheme weights and p_drop = Stage weights in Table I; p_drop 0.9 -> 0.99
    Scheme weights and difficulty were adjusted at 250k/440k steps based on validation-loss plateaus (Section III-D).
  • Non-dropped token loss weight = 0.1
    Observed-token loss downweighted to 10% of dropped-token weight to concentrate on infilling; chosen without a reported sweep (Section III-E).
  • MMD loss weight = 1e-2 -> 0 at 400k steps
    MMD regularizer turned off after ablations improved classification; affects latent representations and downstream behavior (Appendix C).
  • Inference ODE steps and initial noise scale = 50 steps; 0.1
    Decoder sampling hyperparameters used for all evaluations; no sensitivity analysis is given (Appendix A).
assumptions (4)
  • domain assumption 4D-RoPE over discretized (x,y,z,t) generalizes to arbitrary scalp locations and sequence lengths up to 30s
    Underlies the arbitrary-location/length capability claim; only trained on existing montages and crops, with no held-out position extrapolation test (Section II, Figure 7b).
  • domain assumption The eight synthetic dropout schemes adequately approximate real-world EEG corruption patterns
    Used to justify the training curriculum; no evaluation on naturally corrupted recordings is reported (Section III-D).
  • domain assumption Per-window z-score normalization plus two filter variants gives sufficient implicit augmentation to prevent overfitting over six epochs
    The paper claims no overfitting in validation loss but does not show detailed loss curves (Section III-D).
  • domain assumption NMSE on four held-out datasets is an adequate proxy for reconstruction quality across the claimed flexible use cases
    Main evaluation is on 5s samples; temporal and arbitrary-montage generalization is only partially covered (Section IV).

how reviews work

0 comments
Cite this review

Pith. "Pith review of ZUNA1.1: A more flexible EEG foundation model for Denoising and Super-resolution." pith.science (2026). https://pith.science/paper/JYEL66RB

@misc{pith2026260727308,
  author       = {Pith},
  title        = {Pith review of: ZUNA1.1: A more flexible EEG foundation model for Denoising and Super-resolution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JYEL66RB}},
  note         = {Machine review of arXiv:2607.27308}
}
read the original abstract

We introduce ZUNA1.1, a 380M-parameter diffusion autoencoder for flexible EEG signal reconstruction. ZUNA1.1 is capable of reconstructing variable length sequences of up to 30s, with an arbitrary number of EEG channels at arbitrary scalp locations, and can reconstruct arbitrary temporal intervals within channels in addition to reconstructing entire channels. We demonstrate that ZUNA1.1 performs at least on par with our earlier ZUNA1 model, while being far more flexible and capable of handling a wide range of reconstruction tasks. ZUNA1.1 continues to substantially outperform standard EEG denoising and reconstruction methods such as spherical spline interpolation, which is ubiquitously deployed in the MNE package. The ZUNA1.1 model is released open source under the permissive Apache 2.0 license.

Figures

Figures reproduced from arXiv: 2607.27308 by the authors.

Figure 1
Figure 1. Per-region reconstruction accuracy (topographic). Average NMSE by scalp region for ZUNA1.1, ZUNA1 (labeled with the ∆ relative to ZUNA1.1), and spherical-spline, averaged over the four datasets. All electrodes in one region are deleted and reconstructed from the remaining seven regions. Lower (greener) is better. more common pattern is bursts of errors, often individual or correlated across many channels due to temp… view at source ↗
Figure 2
Figure 2. ZUNA1.1 architecture. A transformer-based diffusion autoencoder. The main modifications to the ZUNA1 architecture are adding sandwich norm and QK-norm to improve training stability. This allowed us to remove Adaptive Loss Weighting from the decoder RF loss term. TABLE I Channel-dropout schemes and their sampling weights across the three training stages. Stage 1 (0–250k steps) samples all eight schemes with the 1/8 w… view at source ↗
Figure 3
Figure 3. The four channel-dropout schemes retained throughout training. Each panel shows one training sample: channels run along the vertical axis and coarse-time windows (τ = 0.125 s) along the horizontal axis. Observed tokens (unshaded traces) are passed to the encoder; the shaded spans are set to zero in the encoder and reconstructed by the decoder based on observed signals. full-channel-random removes entire channels; fu… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Reconstruction performance for each dropout scheme throughout training. Median NMSE (lower is better) on the frozen evaluation dataset for each dropout scheme. See Table I for details on how the dropout schemes were adjusted during training [PITH_FULL_IMAGE:figures/fu…
Figure 5
Figure 5. Figure 5: Reconstruction accuracy as channels drop. Reconstruction NMSE versus channel-dropout rate on four held-out datasets (ANPHY-Sleep, BerlinBCI, BCI2000, AAD), comparing ZUNA1.1, ZUNA1, and MNE spherical-spline interpolation; lower is better. ZUNA1.1 matches or improves on…
Figure 6
Figure 6. Figure 6: Per-region reconstruction error. The region-occlusion evaluation of fig. 1 shown as grouped bars: average NMSE across the four datasets with error bars giving the propagated standard deviation. Lower is better. stability issues worth recording. Rare extreme samples pro…
Figure 7
Figure 7. Figure 7: Reconstruction performance for 4 structural dropout schemes on masked tokens in Evaluation dataset. NMSE (lower is better). Description in [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]
Figure 8
Figure 8. Figure 8: ZUNA1.1 vs MNE Spherical Spline - reconstruction performance vs dropout percent on mixed evaluation dataset for structural dropout schemes V. DISCUSSION ZUNA1.1 builds upon and substantially extends our original ZUNA1 model, making it far more flexible and ultimately u…
Figure 9
Figure 9. Figure 9: Model ranking on downstream tasks. Models are ordered by their mean normalized rank, displayed on the right hand side of the boxes. Lower is better. ZUNA1.1 is shown for No MMD-50K. TABLE VII Classification accuracy for ZUNA1.1 (No MMD) compared with NeuralBench EEG fo…
Figure 10
Figure 10. Figure 10: Balanced accuracy for linear classifiers attached at different ZUNA representation layers. We ran this experiment to probe the representational quality [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

275 extracted references · 76 linked inside Pith

  1. [1]

    2021 , eprint=

    CogView: Mastering Text-to-Image Generation via Transformers , author=. 2021 , eprint=

    2021

  2. [2]

    2021 , eprint=

    Masked Autoencoders Are Scalable Vision Learners , author=. 2021 , eprint=

    2021

  3. [3]

    2026 , eprint=

    EEG-FM-Bench: A Comprehensive Benchmark for the Systematic Evaluation and Diagnostic Analyses of EEG Foundation Models , author=. 2026 , eprint=

    2026

  4. [4]

    arXiv preprint arXiv:2605.14698 , year=

    NeuroAtlas: Benchmarking Foundation Models for Clinical EEG and Brain-Computer Interfaces , author=. arXiv preprint arXiv:2605.14698 , year=

    arXiv

  5. [5]

    2020 , eprint=

    Query-Key Normalization for Transformers , author=. 2020 , eprint=

    2020

  6. [6]

    2024 , eprint=

    Consistency Models Made Easy , author=. 2024 , eprint=

    2024

  7. [7]

    IEEE Transactions on Neural Systems and Rehabilitation Engineering , year=

    DeepSleepNet: A Model for Automatic Sleep Stage Scoring based on Raw Single-Channel EEG , author=. IEEE Transactions on Neural Systems and Rehabilitation Engineering , year=

  8. [8]

    arXiv preprint arXiv:2602.18478 , year=

    ZUNA: Flexible EEG Superresolution with Position-Aware Diffusion Autoencoders , author=. arXiv preprint arXiv:2602.18478 , year=. 2602.18478 , archivePrefix=

    arXiv

Show all 275 references
  1. [9]

    arXiv preprint arXiv:2605.08495 , year=

    NeuralBench: A Unifying Framework to Benchmark NeuroAI Models , author=. arXiv preprint arXiv:2605.08495 , year=

    arXiv

  2. [10]

    Advances in neural information processing systems , volume=

    Attention is all you need , author=. Advances in neural information processing systems , volume=

  3. [11]

    2025 , eprint=

    Can Language Models Falsify? Evaluating Algorithmic Reasoning with Counterexample Creation , author=. 2025 , eprint=

    2025

  4. [12]

    2026 , eprint=

    Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMs , author=. 2026 , eprint=

    2026

  5. [13]

    arXiv preprint arXiv:2512.13961 , pages=

    Olmo 3 , author=. arXiv preprint arXiv:2512.13961 , pages=

    arXiv

  6. [14]

    2026 , howpublished =

    2026

  7. [15]

    2025 , url=

    INTELLECT-3: Technical Report , author=. 2025 , url=

    2025

  8. [16]

    arXiv preprint arXiv:2507.06261 , year=

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities , author=. arXiv preprint arXiv:2507.06261 , year=

    arXiv

  9. [17]

    2025 , eprint=

    DAPO: An Open-Source LLM Reinforcement Learning System at Scale , author=. 2025 , eprint=

    2025

  10. [18]

    DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models , author=

  11. [19]

    2025 , eprint=

    Qwen3 Technical Report , author=. 2025 , eprint=

    2025

  12. [20]

    2025 , month = aug, day =

    2025

  13. [21]

    2025 , eprint=

    CodeI/O: Condensing Reasoning Patterns via Code Input-Output Prediction , author=. 2025 , eprint=

    2025

  14. [22]

    2026 , eprint =

    Maximum Likelihood Reinforcement Learning , author =. 2026 , eprint =

    2026

  15. [23]

    Hugging Face repository , howpublished =

    CodeForces , author=. Hugging Face repository , howpublished =. 2025 , publisher =

    2025

  16. [24]

    2025 , eprint=

    CodeARC: Benchmarking Reasoning Capabilities of LLM Agents for Inductive Program Synthesis , author=. 2025 , eprint=

    2025

  17. [25]

    arXiv preprint arXiv:2402.00838 , year=

    Olmo: Accelerating the science of language models , author=. arXiv preprint arXiv:2402.00838 , year=

    arXiv

  18. [26]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    High-resolution image synthesis with latent diffusion models , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  19. [27]

    arXiv preprint arXiv:2406.03476 , year=

    Does your data spark joy? Performance gains from domain upsampling at the end of training , author=. arXiv preprint arXiv:2406.03476 , year=

    arXiv

  20. [28]

    arXiv preprint arXiv:2406.07887 , year=

    An Empirical Study of Mamba-based Language Models , author=. arXiv preprint arXiv:2406.07887 , year=

    arXiv

  21. [29]

    International conference on machine learning , pages=

    Transformers are rnns: Fast autoregressive transformers with linear attention , author=. International conference on machine learning , pages=. 2020 , organization=

    2020

  22. [30]

    arXiv preprint arXiv:2312.00752 , year=

    Mamba: Linear-time sequence modeling with selective state spaces , author=. arXiv preprint arXiv:2312.00752 , year=

    arXiv

  23. [31]

    arXiv preprint arXiv:2402.01771 , year=

    BlackMamba: Mixture of Experts for State-Space Models , author=. arXiv preprint arXiv:2402.01771 , year=

    arXiv

  24. [32]

    https://huggingface.co/Zyphra/Mamba-370M , year=

    Zyphra , title=. https://huggingface.co/Zyphra/Mamba-370M , year=

  25. [33]

    https://huggingface.co/datasets/BAAI/Infinity-Instruct , year=

    BAAI , title=. https://huggingface.co/datasets/BAAI/Infinity-Instruct , year=

  26. [34]

    https://huggingface.co/datasets/Intel/orca\_dpo\_pairs , year=

    BAAI , title=. https://huggingface.co/datasets/Intel/orca\_dpo\_pairs , year=

  27. [35]

    HuggingFace repository , howpublished =

    OpenOrca: An Open Dataset of GPT Augmented FLAN Reasoning Traces , author =. HuggingFace repository , howpublished =. 2023 , publisher =

    2023

  28. [36]

    arXiv preprint arXiv:2307.09288 , year=

    Llama 2: Open foundation and fine-tuned chat models , author=. arXiv preprint arXiv:2307.09288 , year=

    arXiv

  29. [37]

    International Conference on Machine Learning , pages=

    Pythia: A suite for analyzing large language models across training and scaling , author=. International Conference on Machine Learning , pages=. 2023 , organization=

    2023

  30. [38]

    2026 , eprint=

    GLM-5: from Vibe Coding to Agentic Engineering , author=. 2026 , eprint=

    2026

  31. [39]

    arXiv preprint arXiv:2312.06550 , year=

    Llm360: Towards fully transparent open-source llms , author=. arXiv preprint arXiv:2312.06550 , year=

    arXiv

  32. [40]

    arXiv preprint arXiv:2404.06395 , year=

    MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies , author=. arXiv preprint arXiv:2404.06395 , year=

    arXiv

  33. [41]

    arXiv preprint arXiv:2402.16819 , year=

    Nemotron-4 15B Technical Report , author=. arXiv preprint arXiv:2402.16819 , year=

    arXiv

  34. [42]

    arXiv preprint arXiv:2404.07413 , year=

    JetMoE: Reaching Llama2 Performance with 0.1 M Dollars , author=. arXiv preprint arXiv:2404.07413 , year=

    arXiv

  35. [43]

    arXiv preprint arXiv:2310.06825 , year=

    Mistral 7B , author=. arXiv preprint arXiv:2310.06825 , year=

    arXiv

  36. [44]

    arXiv preprint arXiv:2403.08295 , year=

    Gemma: Open models based on gemini research and technology , author=. arXiv preprint arXiv:2403.08295 , year=

    arXiv

  37. [45]

    2024 , eprint=

    Gemma 2: Improving Open Language Models at a Practical Size , author=. 2024 , eprint=

    2024

  38. [46]

    CoRR , volume =

    Alexey Dosovitskiy and Lucas Beyer and Alexander Kolesnikov and Dirk Weissenborn and Xiaohua Zhai and Thomas Unterthiner and Mostafa Dehghani and Matthias Minderer and Georg Heigold and Sylvain Gelly and Jakob Uszkoreit and Neil Houlsby , title =. CoRR , volume =. 2020 , url =...

    2020 arXiv

  39. [47]

    Tom B. Brown and Benjamin Mann and Nick Ryder and Melanie Subbiah and Jared Kaplan and Prafulla Dhariwal and Arvind Neelakantan and Pranav Shyam and Girish Sastry and Amanda Askell and Sandhini Agarwal and Ariel Herbert. Language Models are Few-Shot Learners , journal =. 2020 ...

    2020 arXiv

  40. [48]

    2022 , eprint=

    Training Compute-Optimal Large Language Models , author=. 2022 , eprint=

    2022

  41. [49]

    Signal Transduction and Targeted Therapy , year=

    Yang, Zhenyu and Zeng, Xiaoxi and Zhao, Yi and Chen, Runsheng , title=. Signal Transduction and Targeted Therapy , year=. doi:10.1038/s41392-023-01381-z , url=

    doi

  42. [50]

    2022 , eprint=

    Scaling Language Models: Methods, Analysis & Insights from Training Gopher , author=. 2022 , eprint=

    2022

  43. [51]

    2024 , eprint=

    Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models , author=. 2024 , eprint=

    2024

  44. [52]

    2024 , eprint=

    Gated Linear Attention Transformers with Hardware-Efficient Training , author=. 2024 , eprint=

    2024

  45. [53]

    arXiv preprint arXiv:2312.11805 , year=

    Gemini: a family of highly capable multimodal models , author=. arXiv preprint arXiv:2312.11805 , year=

    arXiv

  46. [54]

    arXiv preprint arXiv:2404.14219 , year=

    Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone , author=. arXiv preprint arXiv:2404.14219 , year=

    arXiv

  47. [55]

    Advances in Neural Information Processing Systems , volume=

    Doremi: Optimizing data mixtures speeds up language model pretraining , author=. Advances in Neural Information Processing Systems , volume=

  48. [56]

    Advances in Neural Information Processing Systems , volume=

    D4: Improving llm pretraining via document de-duplication and diversification , author=. Advances in Neural Information Processing Systems , volume=

  49. [57]

    arXiv preprint arXiv:2401.16380 , year=

    Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modeling , author=. arXiv preprint arXiv:2401.16380 , year=

    arXiv

  50. [58]

    arXiv preprint arXiv:2309.05463 , year=

    Textbooks are all you need ii: phi-1.5 technical report , author=. arXiv preprint arXiv:2309.05463 , year=

    arXiv

  51. [59]

    arXiv preprint arXiv:2212.14052 , year=

    Hungry hungry hippos: Towards language modeling with state space models , author=. arXiv preprint arXiv:2212.14052 , year=

    arXiv

  52. [60]

    arXiv preprint arXiv:2111.00396 , year=

    Efficiently modeling long sequences with structured state spaces , author=. arXiv preprint arXiv:2111.00396 , year=

    arXiv

  53. [61]

    arXiv preprint arXiv:2403.17844 , year=

    Mechanistic Design and Scaling of Hybrid Architectures , author=. arXiv preprint arXiv:2403.17844 , year=

    arXiv

  54. [62]

    Journal of Machine Learning Research , volume=

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity , author=. Journal of Machine Learning Research , volume=

  55. [63]

    International conference on machine learning , pages=

    Deepspeed-moe: Advancing mixture-of-experts inference and training to power next-generation ai scale , author=. International conference on machine learning , pages=. 2022 , organization=

    2022

  56. [64]

    GPT-J-6B: A 6 billion parameter autoregressive language model , author=

  57. [65]

    Advances in Neural Information Processing Systems , volume=

    Direct preference optimization: Your language model is secretly a reward model , author=. Advances in Neural Information Processing Systems , volume=

  58. [66]

    Advances in neural information processing systems , volume=

    Combining recurrent, convolutional, and continuous-time models with linear state space layers , author=. Advances in neural information processing systems , volume=

  59. [67]

    arXiv preprint arXiv:2403.08763 , year=

    Simple and scalable strategies to continually pre-train large language models , author=. arXiv preprint arXiv:2403.08763 , year=

    arXiv

  60. [68]

    Advances in neural information processing systems , volume=

    Hippo: Recurrent memory with optimal polynomial projections , author=. Advances in neural information processing systems , volume=

  61. [69]

    2023 , publisher =

    OpenHermes 2.5: An Open Dataset of Synthetic Data for Generalist LLM Assistants , author =. 2023 , publisher =

    2023

  62. [70]

    2024 , month =

    Bamba: Inference-Efficient Hybrid Mamba2 Model , author =. 2024 , month =

    2024

  63. [71]

    arXiv preprint arXiv:1412.6980 , year=

    Adam: A method for stochastic optimization , author=. arXiv preprint arXiv:1412.6980 , year=

    arXiv

  64. [72]

    arXiv preprint arXiv:2106.09685 , year=

    Lora: Low-rank adaptation of large language models , author=. arXiv preprint arXiv:2106.09685 , year=

    arXiv

  65. [73]

    arXiv preprint arXiv:1712.00409 , year=

    Deep learning scaling is predictable, empirically , author=. arXiv preprint arXiv:1712.00409 , year=

    arXiv

  66. [74]

    arXiv preprint arXiv:2402.04248 , year=

    Can mamba learn how to learn? a comparative study on in-context learning tasks , author=. arXiv preprint arXiv:2402.04248 , year=

    arXiv

  67. [75]

    arXiv preprint arXiv:2406.08464 , year=

    Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing , author=. arXiv preprint arXiv:2406.08464 , year=

    arXiv

  68. [76]

    arXiv preprint arXiv:1807.03819 , year=

    Universal transformers , author=. arXiv preprint arXiv:1807.03819 , year=

    arXiv

  69. [77]

    PLoS Computational Biology , volume=

    Hybrid predictive coding: Inferring, fast and slow , author=. PLoS Computational Biology , volume=. 2023 , publisher=

    2023

  70. [78]

    Current Opinion in Neurobiology , volume=

    Going in circles is the way forward: the role of recurrence in visual inference , author=. Current Opinion in Neurobiology , volume=. 2020 , publisher=

    2020

  71. [79]

    arXiv preprint arXiv:2101.00027 , year=

    The pile: An 800gb dataset of diverse text for language modeling , author=. arXiv preprint arXiv:2101.00027 , year=

    arXiv

  72. [80]

    arXiv preprint arXiv:2306.01116 , year=

    The RefinedWeb dataset for Falcon LLM: outperforming curated corpora with web data, and web data only , author=. arXiv preprint arXiv:2306.01116 , year=

    arXiv

  73. [81]

    Journal of machine learning research , volume=

    Exploring the limits of transfer learning with a unified text-to-text transformer , author=. Journal of machine learning research , volume=

  74. [82]

    Luca Soldaini and Kyle Lo , year = 2023, title =

    2023

  75. [83]

    2024 , eprint=

    Is Mamba Capable of In-Context Learning? , author=. 2024 , eprint=

    2024

  76. [84]

    2024 , eprint=

    Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality , author=. 2024 , eprint=

    2024

  77. [85]

    2024 , eprint=

    Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks , author=. 2024 , eprint=

    2024

  78. [86]

    2024 , eprint=

    Repeat After Me: Transformers are Better than State Space Models at Copying , author=. 2024 , eprint=

    2024

  79. [87]

    2024 , eprint=

    Jamba: A Hybrid Transformer-Mamba Language Model , author=. 2024 , eprint=

    2024

  80. [88]

    2024 , eprint=

    Jamba-1.5: Hybrid Transformer-Mamba Models at Scale , author=. 2024 , eprint=

    2024

  81. [89]

    Zyphra , title =. 2024

    2024

  82. [90]

    2024 , eprint=

    Nemotron-4 15B Technical Report , author=. 2024 , eprint=

    2024

  83. [91]

    2021 , eprint=

    Measuring Massive Multitask Language Understanding , author=. 2021 , eprint=

    2021

  84. [92]

    Meta , title =. 2024

    2024

  85. [93]

    2024 , eprint=

    Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrence , author=. 2024 , eprint=

    2024

  86. [94]

    2023 , eprint=

    RWKV: Reinventing RNNs for the Transformer Era , author=. 2023 , eprint=

    2023

  87. [95]

    2023 , eprint=

    Retentive Network: A Successor to Transformer for Large Language Models , author=. 2023 , eprint=

    2023

  88. [96]

    Ben Wang and Aran Komatsuzaki , title =. 2021

    2021

  89. [97]

    NVIDIA , title =. 2023

    2023

  90. [98]

    2018 , eprint=

    Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge , author=. 2018 , eprint=

    2018

  91. [99]

    arXiv preprint arXiv:2001.08361 , year=

    Scaling laws for neural language models , author=. arXiv preprint arXiv:2001.08361 , year=

    2001 arXiv

  92. [100]

    Advances in Neural Information Processing Systems , volume=

    Scaling mlps: A tale of inductive bias , author=. Advances in Neural Information Processing Systems , volume=. 2023 , url=

    2023

  93. [101]

    International Conference on Learning Representations , year=

    Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer , author=. International Conference on Learning Representations , year=

  94. [102]

    Neural computation , volume=

    Adaptive mixtures of local experts , author=. Neural computation , volume=. 1991 , publisher=

    1991

  95. [103]

    arXiv preprint arXiv:2308.04014 , year=

    Continual Pre-Training of Large Language Models: How to (re) warm your model? , author=. arXiv preprint arXiv:2308.04014 , year=

    arXiv

  96. [104]

    The 61st Annual Meeting Of The Association For Computational Linguistics , year=

    Downstream Datasets Make Surprisingly Good Pretraining Corpora , author=. The 61st Annual Meeting Of The Association For Computational Linguistics , year=

  97. [105]

    Cognition , volume=

    Learning and development in neural networks: The importance of starting small , author=. Cognition , volume=. 1993 , publisher=

    1993

  98. [106]

    Proceedings of the 26th annual international conference on machine learning , pages=

    Curriculum learning , author=. Proceedings of the 26th annual international conference on machine learning , pages=. 2009 , url=

    2009

  99. [107]

    2026 , eprint=

    Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs , author=. 2026 , eprint=

    2026

  100. [108]

    2025 , howpublished =

    2025

  101. [109]

    Proceedings of the 27th International Joint Conference on Artificial Intelligence , pages=

    Curriculum learning for natural answer generation , author=. Proceedings of the 27th International Joint Conference on Artificial Intelligence , pages=. 2018 , url=

    2018

  102. [110]

    Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , year=

    Efficient Memory Management for Large Language Model Serving with PagedAttention , author=. Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , year=

  103. [111]

    2023 , eprint=

    FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning , author=. 2023 , eprint=

    2023

  104. [112]

    2023 , eprint=

    The Falcon Series of Open Language Models , author=. 2023 , eprint=

    2023

  105. [113]

    arXiv preprint arXiv:2306.11644 , year=

    Textbooks are all you need , author=. arXiv preprint arXiv:2306.11644 , year=

    arXiv

  106. [114]

    Cell , author =

    The. Cell , author =. 2020 , note =. doi:10.1016/j.cell.2020.10.024 , abstract =

    2020 doi

  107. [115]

    International Conference on Learning Representations , author =

    Relating transformers to models and neural representations of the hippocampal formation , url =. International Conference on Learning Representations , author =

  108. [116]

    2025 , eprint =

    Defeating the Training-Inference Mismatch via FP16 , author =. 2025 , eprint =

    2025

  109. [117]

    doi:10.5281/zenodo.10256836 , url =

    Gao, Leo and Tow, Jonathan and Abbasi, Baber and Biderman, Stella and Black, Sid and DiPofi, Anthony and Foster, Charles and Golding, Laurence and Hsu, Jeffrey and Le Noac'h, Alain and Li, Haonan and McDonell, Kyle and Muennighoff, Niklas and Ociepa, Chris and Phang, Jason and...

    doi

  110. [118]

    arxiv_s2orc_parsed , author=

  111. [119]

    Long Short-Term Memory , year =

    Sepp Hochreiter and Jürgen Schmidhuber , journal =. Long Short-Term Memory , year =

  112. [120]

    2014 , eprint=

    Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling , author=. 2014 , eprint=

    2014

  113. [121]

    2022 , url=

    NVIDIA H100 tensor core gpu architecture , author=. 2022 , url=

    2022

  114. [122]

    Williams, Samuel and Waterman, Andrew and Patterson, David , title =. Commun. ACM , month = apr, pages =. 2009 , issue_date =. doi:10.1145/1498765.1498785 , abstract =

    2009

  115. [123]

    2024 , eprint=

    An Empirical Study of Mamba-based Language Models , author=. 2024 , eprint=

    2024

  116. [124]

    2405.16712 , archivePrefix=

    Paolo Glorioso and Quentin Anthony and Yury Tokpanov and James Whittington and Jonathan Pilault and Adam Ibrahim and Beren Millidge , year=. 2405.16712 , archivePrefix=

    arXiv

  117. [125]

    2023 , eprint=

    RoFormer: Enhanced Transformer with Rotary Position Embedding , author=. 2023 , eprint=

    2023

  118. [126]

    2024 , eprint=

    The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale , author=. 2024 , eprint=

    2024

  119. [127]

    Dimakis and Yair Carmon and Achal Dave and Ludwig Schmidt and Vaishaal Shankar , year=

    Jeffrey Li and Alex Fang and Georgios Smyrnis and Maor Ivgi and Matt Jordan and Samir Gadre and Hritik Bansal and Etash Guha and Sedrick Keh and Kushal Arora and Saurabh Garg and Rui Xin and Niklas Muennighoff and Reinhard Heckel and Jean Mercat and Mayee Chen and Suchin Gurur...

    arXiv

  120. [128]

    2020 , eprint=

    ZeRO: Memory Optimizations Toward Training Trillion Parameter Models , author=. 2020 , eprint=

    2020

  121. [129]

    2024 , month =

    Yury Tokpanov, Paolo Glorioso and Ayush Dattagupta and Vibhu Jawa and Ryan Wolf and Vikranth Jeyakumar and Arham Mehta, Quentin Anthony and Beren Millidge , title =. 2024 , month =

    2024

  122. [130]

    2406.01981 , archivePrefix=

    Yury Tokpanov and Beren Millidge and Paolo Glorioso and Jonathan Pilault and Adam Ibrahim and James Whittington and Quentin Anthony , year=. 2406.01981 , archivePrefix=

    arXiv

  123. [131]

    2024 , eprint=

    Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research , author=. 2024 , eprint=

    2024

  124. [132]

    2023 , eprint=

    StarCoder: may the source be with you! , author=. 2023 , eprint=

    2023

  125. [133]

    2024 , eprint=

    Direct Preference Optimization: Your Language Model is Secretly a Reward Model , author=. 2024 , eprint=

    2024

  126. [134]

    2023 , eprint=

    Enhancing Chat Language Models by Scaling High-quality Instructional Conversations , author=. 2023 , eprint=

    2023

  127. [135]

    2024 , eprint=

    UltraFeedback: Boosting Language Models with Scaled AI Feedback , author=. 2024 , eprint=

    2024

  128. [136]

    Hugging Face repository , howpublished =

    Shengyi Costa Huang and Agustín Piqueres and Kashif Rasul and Philipp Schmid and Daniel Vila and Lewis Tunstall , title =. Hugging Face repository , howpublished =. 2024 , publisher =

    2024

  129. [137]

    2023 , eprint=

    Instruction-Following Evaluation for Large Language Models , author=. 2023 , eprint=

    2023

  130. [138]

    2023 , eprint=

    Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena , author=. 2023 , eprint=

    2023

  131. [139]

    2023 , eprint=

    QLoRA: Efficient Finetuning of Quantized LLMs , author=. 2023 , eprint=

    2023

  132. [140]

    2022 , eprint=

    LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale , author=. 2022 , eprint=

    2022

  133. [141]

    PEFT: State-of-the-art Parameter-Efficient Fine-Tuning methods , author =

  134. [142]

    2024 , eprint=

    The Case for Co-Designing Model Architectures with Hardware , author=. 2024 , eprint=

    2024

  135. [143]

    2024 , eprint=

    The Llama 3 Herd of Models , author=. 2024 , eprint=

    2024

  136. [144]

    2025 , eprint=

    Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routers , author=. 2025 , eprint=

    2025

  137. [145]

    arXiv preprint arXiv:2501.12948 , year=

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning , author=. arXiv preprint arXiv:2501.12948 , year=

    arXiv

  138. [146]

    arXiv preprint arXiv:2507.20534 , year=

    Kimi K2: Open Agentic Intelligence , author=. arXiv preprint arXiv:2507.20534 , year=

    arXiv

  139. [147]

    arXiv preprint arXiv:2310.01889 , year=

    Ring attention with blockwise transformers for near-infinite context , author=. arXiv preprint arXiv:2310.01889 , year=

    arXiv

  140. [148]

    arXiv preprint arXiv:2405.13792 , year=

    xrag: Extreme context compression for retrieval-augmented generation with one token , author=. arXiv preprint arXiv:2405.13792 , year=

    arXiv

  141. [149]

    arXiv preprint arXiv:2307.06945 , year=

    In-context autoencoder for context compression in a large language model , author=. arXiv preprint arXiv:2307.06945 , year=

    arXiv

  142. [150]

    arXiv preprint arXiv:2410.15252 , year=

    Lossless KV Cache Compression to 2\ author=. arXiv preprint arXiv:2410.15252 , year=

    arXiv

  143. [151]

    SSRN Electronic Journal , year=

    Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers , author=. SSRN Electronic Journal , year=

  144. [152]

    arXiv preprint arXiv:2412.08890 , year=

    Lexico: Extreme KV Cache Compression via Sparse Coding over Universal Dictionaries , author=. arXiv preprint arXiv:2412.08890 , year=

    arXiv

  145. [153]

    URL https://arxiv

    Understanding r1-zero-like training: A critical perspective, 2025 , author=. URL https://arxiv. org/abs/2503.20783 , year=

    2025 arXiv

  146. [154]

    arXiv preprint arXiv:2412.03213 , year=

    Clusterkv: Manipulating llm kv cache in semantic space for recallable compression , author=. arXiv preprint arXiv:2412.03213 , year=

    arXiv

  147. [155]

    arXiv preprint arXiv:2502.11089 , year=

    Native sparse attention: Hardware-aligned and natively trainable sparse attention , author=. arXiv preprint arXiv:2502.11089 , year=

    arXiv

  148. [156]

    Neurocomputing , volume=

    Roformer: Enhanced transformer with rotary position embedding , author=. Neurocomputing , volume=. 2024 , publisher=

    2024

  149. [157]

    arXiv preprint arXiv:2305.13245 , year=

    Gqa: Training generalized multi-query transformer models from multi-head checkpoints , author=. arXiv preprint arXiv:2305.13245 , year=

    arXiv

  150. [158]

    arXiv preprint arXiv:1911.02150 , year=

    Fast transformer decoding: One write-head is all you need , author=. arXiv preprint arXiv:1911.02150 , year=

    1911 arXiv

  151. [159]

    arXiv preprint arXiv:2408.04093 , year=

    Tree attention: Topology-aware decoding for long-context attention on gpu clusters , author=. arXiv preprint arXiv:2408.04093 , year=

    arXiv

  152. [160]

    arXiv preprint arXiv:2509.19128 , year=

    Pipelinerl: Faster on-policy reinforcement learning for long sequence generation , author=. arXiv preprint arXiv:2509.19128 , year=

  153. [161]

    arXiv preprint arXiv:2511.17127 , year=

    Training Foundation Models on a Full-Stack AMD Platform: Compute, Networking, and System Design , author=. arXiv preprint arXiv:2511.17127 , year=

  154. [162]

    arXiv preprint arXiv:2509.26626 , year=

    Recursive self-aggregation unlocks deep thinking in large language models , author=. arXiv preprint arXiv:2509.26626 , year=

  155. [163]

    arXiv preprint arXiv:2510.06557 , year=

    The markovian thinker: Architecture-agnostic linear scaling of reasoning , author=. arXiv preprint arXiv:2510.06557 , year=

  156. [164]

    arXiv preprint arXiv:2601.05593 , year=

    PaCoRe: Learning to Scale Test-Time Compute with Parallel Coordinated Reasoning , author=. arXiv preprint arXiv:2601.05593 , year=

  157. [165]

    arXiv preprint arXiv:2505.22617 , year=

    The entropy mechanism of reinforcement learning for reasoning language models , author=. arXiv preprint arXiv:2505.22617 , year=

    arXiv

  158. [166]

    arXiv preprint arXiv:2505.11711 , year=

    Reinforcement learning finetunes small subnetworks in large language models , author=. arXiv preprint arXiv:2505.11711 , year=

  159. [167]

    arXiv preprint arXiv:2602.04879 , year=

    Rethinking the Trust Region in LLM Reinforcement Learning , author=. arXiv preprint arXiv:2602.04879 , year=

    arXiv

  160. [168]

    arXiv preprint arXiv:2510.13786 , year=

    The art of scaling reinforcement learning compute for llms , author=. arXiv preprint arXiv:2510.13786 , year=

    arXiv

  161. [170]

    arXiv preprint arXiv:2510.03264 , year=

    Front-loading reasoning: The synergy between pretraining and post-training data , author=. arXiv preprint arXiv:2510.03264 , year=

  162. [171]

    arXiv preprint arXiv:2602.07729 , year=

    Do We Need Adam? Surprisingly Strong and Sparse Reinforcement Learning with SGD in LLMs , author=. arXiv preprint arXiv:2602.07729 , year=

  163. [173]

    https://brianbartoldson.wordpress.com/2026/05/04/cheaply-approximating-kl-against-an-ema-an-async-rl-hack/ , year=

    Cheaply approximating kl against an ema: an async rl hack , author=. https://brianbartoldson.wordpress.com/2026/05/04/cheaply-approximating-kl-against-an-ema-an-async-rl-hack/ , year=

    2026

  164. [174]

    arXiv preprint arXiv:2506.13585 , year=

    Minimax-m1: Scaling test-time compute efficiently with lightning attention , author=. arXiv preprint arXiv:2506.13585 , year=

    arXiv

  165. [175]

    arXiv preprint arXiv:2411.15124 , year=

    T " ulu 3: Pushing frontiers in open language model post-training , author=. arXiv preprint arXiv:2411.15124 , year=

    arXiv

  166. [176]

    2024 , eprint=

    Falcon Mamba: The First Competitive Attention-free 7B Language Model , author=. 2024 , eprint=

    2024

  167. [177]

    2023 , eprint=

    Ring Attention with Blockwise Transformers for Near-Infinite Context , author=. 2023 , eprint=

    2023

  168. [178]

    arXiv e-prints , keywords =

    YaRN: Efficient Context Window Extension of Large Language Models. arXiv e-prints , keywords =. doi:10.48550/arXiv.2309.00071 , archivePrefix =. 2309.00071 , primaryClass =

    doi arXiv

  169. [179]

    Anthony, Quentin and Millidge, Beren and Glorioso, Paolo and Tokpanov, Yury , howpublished =

  170. [180]

    GitHub repository , url =

    Bekman, Stas , title =. GitHub repository , url =. 2023-2024 , publisher =

    2023

  171. [181]

    2024 , eprint=

    FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision , author=. 2024 , eprint=

    2024

  172. [182]

    arXiv preprint arXiv:2411.15242 , year=

    The Zamba2 Suite: Technical Report , author=. arXiv preprint arXiv:2411.15242 , year=

    arXiv

  173. [183]

    arXiv preprint arXiv:2412.19437 , year=

    Deepseek-v3 technical report , author=. arXiv preprint arXiv:2412.19437 , year=

    arXiv

  174. [184]

    Pacific Journal of Mathematics , volume=

    Concerning nonnegative matrices and doubly stochastic matrices , author=. Pacific Journal of Mathematics , volume=. 1967 , publisher=

    1967

  175. [185]

    2020 , eprint=

    GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding , author=. 2020 , eprint=

    2020

  176. [186]

    2025 , eprint=

    DeepSeek-V3 Technical Report , author=. 2025 , eprint=

    2025

  177. [187]

    2024 , eprint=

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context , author=. 2024 , eprint=

    2024

  178. [188]

    2024 , eprint=

    DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models , author=. 2024 , eprint=

    2024

  179. [189]

    2024 , eprint=

    Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset , author=. 2024 , eprint=

    2024

  180. [190]

    2025 , eprint=

    MoBA: Mixture of Block Attention for Long-Context LLMs , author=. 2025 , eprint=

    2025

  181. [191]

    Length Generalization of Causal Transformers without Position Encoding , doi =

    Wang, Jie and Ji, Tao and Wu, Yuanbin and Yan, Hang and Gui, Tao and Huang, Xuanjing and Wang, Xiaoling , year =. Length Generalization of Causal Transformers without Position Encoding , doi =

  182. [192]

    DeepSeek-V3.2-Exp: Boosting Long-Context Efficiency with DeepSeek Sparse Attention , author=

  183. [193]

    2024 , eprint=

    Rail-only: A Low-Cost High-Performance Network for Training LLMs with Trillion Parameters , author=. 2024 , eprint=

    2024

  184. [194]

    2024 , howpublished =

    2024

  185. [195]

    2025 , eprint=

    Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws , author=. 2025 , eprint=

    2025

  186. [196]

    2025 , eprint=

    Compressed Convolutional Attention: Efficient Attention in a Compressed Latent Space , author=. 2025 , eprint=

    2025

  187. [197]

    2020 , eprint=

    Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism , author=. 2020 , eprint=

    2020

  188. [198]

    Primus: A Flexible and High-Performance Training Framework for Large-Scale Foundation Model Training and Inference , year =

  189. [199]

    DK , journal=

    Anthony, Quentin and Michalowicz, Benjamin and Hatef, Jacob and Xu, Lang and Abduljabbar, Mustafa and Shafi, Aamir and Subramoni, Hari and Panda, Dhabaleswar K. DK , journal=. Understanding and Characterizing Communication Characteristics for Distributed Transformer Models , year=

  190. [200]

    2021 , eprint=

    Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM , author=. 2021 , eprint=

    2021

  191. [201]

    arXiv preprint arXiv:2507.17702 , year=

    Towards greater leverage: Scaling laws for efficient mixture-of-experts language models , author=. arXiv preprint arXiv:2507.17702 , year=

  192. [202]

    Advances in neural information processing systems , volume=

    Denseformer: Enhancing information flow in transformers via depth weighted averaging , author=. Advances in neural information processing systems , volume=

  193. [203]

    arXiv preprint arXiv:2410.02660 , year=

    How to train long-context language models (effectively) , author=. arXiv preprint arXiv:2410.02660 , year=

  194. [204]

    arXiv preprint arXiv:2510.06826 , year=

    Mid-Training of Large Language Models: A Survey , author=. arXiv preprint arXiv:2510.06826 , year=

  195. [205]

    arXiv preprint arXiv:2505.09388 , year=

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    arXiv

  196. [206]

    arXiv preprint arXiv:2409.02060 , year=

    Olmoe: Open mixture-of-experts language models , author=. arXiv preprint arXiv:2409.02060 , year=

    arXiv

  197. [207]

    arXiv preprint arXiv:2503.19786 , year=

    Gemma 3 technical report , author=. arXiv preprint arXiv:2503.19786 , year=

    arXiv

  198. [208]

    arXiv preprint arXiv:2504.06214 , year=

    From 128k to 4m: Efficient training of ultra-long context large language models , author=. arXiv preprint arXiv:2504.06214 , year=

    arXiv

  199. [209]

    arXiv preprint arXiv:2502.16982 , year=

    Muon is scalable for LLM training , author=. arXiv preprint arXiv:2502.16982 , year=

    arXiv

  200. [210]

    arXiv preprint arXiv:2508.15884 , year=

    Jet-Nemotron: Efficient Language Model with Post Neural Architecture Search , author=. arXiv preprint arXiv:2508.15884 , year=

  201. [211]

    arXiv preprint arXiv:2505.23737 , year=

    On the convergence analysis of muon , author=. arXiv preprint arXiv:2505.23737 , year=

    arXiv

  202. [212]

    URL https://kellerjordan

    Muon: An optimizer for hidden layers in neural networks, 2024 , author=. URL https://kellerjordan. github. io/posts/muon , volume=

    2024

  203. [213]

    IEEE Control Systems Magazine , volume=

    PID control , author=. IEEE Control Systems Magazine , volume=

  204. [214]

    arXiv preprint arXiv:2505.06708 , year=

    Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free , author=. arXiv preprint arXiv:2505.06708 , year=

    arXiv

  205. [215]

    2021 , eprint=

    A Large Batch Optimizer Reality Check: Traditional, Generic Optimizers Suffice Across Batch Sizes , author=. 2021 , eprint=

    2021

  206. [216]

    Apex: A PyTorch Extension: Tools for easy mixed precision and distributed training in Pytorch , url =

  207. [217]

    2025 , eprint=

    Muon is Scalable for LLM Training , author=. 2025 , eprint=

    2025

  208. [218]

    SIGCOMM Comput

    Al-Fares, Mohammad and Loukissas, Alexander and Vahdat, Amin , title =. SIGCOMM Comput. Commun. Rev. , month = aug, pages =. 2008 , issue_date =. doi:10.1145/1402946.1402967 , abstract =

    2008

  209. [219]

    2008 , isbn =

    Al-Fares, Mohammad and Loukissas, Alexander and Vahdat, Amin , title =. 2008 , isbn =. doi:10.1145/1402958.1402967 , booktitle =

    2008

  210. [220]

    2025 , month =

    7060X6 Series 800G Data Center Switches: Data Sheet , author =. 2025 , month =

    2025

  211. [221]

    Composable Kernel: Performance Portable Programming Model for Machine Learning Tensor Operators , year =

  212. [222]

    2024 , eprint=

    Demystifying the Communication Characteristics for Distributed Transformer Models , author=. 2024 , eprint=

    2024

  213. [223]

    Understanding

    Kolla, Jayacharan and Alizadeh, Pedram and Lee, Gilbert , year =. Understanding

  214. [224]

    2019 , eprint=

    GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism , author=. 2019 , eprint=

    2019

  215. [225]

    Bakouch, Elie and Ben Allal, Loubna and Lozhkov, Anton and Tazi, Nouamane and Tunstall, Lewis and Patiño, Carlos Miguel and Beeching, Edward and Roucher, Aymeric and Reedi, Aksel Joonas and Gallouédec, Quentin and Rasul, Kashif and Habib, Nathan and Fourrier, Clémentine and Ky...

  216. [226]

    2025 , url =

    toothacher17 , title =. 2025 , url =

    2025

  217. [227]

    2024 , note =

    rocm-systems: Super Repository for ROCm Systems Projects , url =. 2024 , note =

    2024

  218. [228]

    2024 , note =

    nvbandwidth: A Tool for Bandwidth Measurements on NVIDIA GPUs , url =. 2024 , note =

    2024

  219. [229]

    Deakin, Tom and McIntosh-Smith, Simon , month = apr, title =

  220. [230]

    de , title =

    Moody, Adam and Bronevetsky, Greg and Mohror, Kathryn and Supinski, Bronis R. de , title =. 2010 , isbn =. doi:10.1109/SC.2010.18 , booktitle =

    2010 doi

  221. [231]

    Evaluating Multi-Level Checkpointing for Distributed Deep Neural Network Training , year=

    Anthony, Quentin and Dai, Donglai , booktitle=. Evaluating Multi-Level Checkpointing for Distributed Deep Neural Network Training , year=

  222. [232]

    arXiv preprint arXiv:2511.07317 , year=

    RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environments , author=. arXiv preprint arXiv:2511.07317 , year=

    arXiv

  223. [233]

    Proceedings of the Twentieth European Conference on Computer Systems (EuroSys 2025) , pages=

    HybridFlow: A Flexible and Efficient RLHF Framework , author=. Proceedings of the Twentieth European Conference on Computer Systems (EuroSys 2025) , pages=. 2025 , doi=

    2025

  224. [234]

    1980 , isbn=

    Applications of Item Response Theory to Practical Testing Problems , author=. 1980 , isbn=

    1980

  225. [235]

    Biometrika , volume=

    On the Likelihood that One Unknown Probability Exceeds Another in View of the Evidence of Two Samples , author=. Biometrika , volume=. 1933 , doi=

    1933

  226. [236]

    arXiv preprint arXiv:2505.12284v3 , year=

    Shorten After You're Right: Lazy Length Penalties for Reasoning RL , author=. arXiv preprint arXiv:2505.12284v3 , year=

  227. [237]

    arXiv preprint arXiv:2506.05256 , year=

    Just Enough Thinking: Efficient Reasoning with Adaptive Length Penalties Reinforcement Learning , author=. arXiv preprint arXiv:2506.05256 , year=

    arXiv

  228. [238]

    Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Models , author=

  229. [239]

    2024 , eprint=

    HelpSteer2: Open-source dataset for training top-performing reward models , author=. 2024 , eprint=

    2024

  230. [240]

    2025 , eprint=

    HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages , author=. 2025 , eprint=

    2025

  231. [241]

    arXiv preprint arXiv:2407.01082 , year =

    Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputs , author =. arXiv preprint arXiv:2407.01082 , year =. doi:10.48550/arXiv.2407.01082 , url =

    doi

  232. [242]

    arXiv preprint arXiv:2510.02387 , year =

    CWM: An Open-Weights LLM for Research on Code Generation with World Models , author =. arXiv preprint arXiv:2510.02387 , year =. doi:10.48550/arXiv.2510.02387 , url =

    doi

  233. [243]

    arXiv preprint arXiv:2603.10055 , year =

    Training Language Models via Neural Cellular Automata , author =. arXiv preprint arXiv:2603.10055 , year =. doi:10.48550/arXiv.2603.10055 , url =

    doi

  234. [244]

    2019 , isbn =

    An Introduction to Kolmogorov Complexity and Its Applications , author =. 2019 , isbn =. doi:10.1007/978-3-030-11298-1 , url =

    2019 doi

  235. [245]

    2024 , eprint=

    Fewer Truncations Improve Language Modeling , author=. 2024 , eprint=

    2024

  236. [246]

    arXiv preprint arXiv:2509.04259 , year =

    RL's Razor: Why Online Reinforcement Learning Forgets Less , author =. arXiv preprint arXiv:2509.04259 , year =. doi:10.48550/arXiv.2509.04259 , url =

    doi arXiv

  237. [247]

    2026 , eprint=

    Kimi K2: Open Agentic Intelligence , author=. 2026 , eprint=

    2026

  238. [248]

    2021 , eprint=

    BENDR: Using transformers and a contrastive self-supervised learning task to learn from massive amounts of EEG data , author=. 2021 , eprint=

    2021

  239. [249]

    2024 , eprint=

    SleepFM: Multi-modal representation learning for sleep across brain activity, ECG and respiratory signals , author=. 2024 , eprint=

    2024

  240. [250]

    2024 , eprint=

    CBraMod: A criss-cross brain foundation model for EEG decoding , author=. 2024 , eprint=

    2024

  241. [251]

    Advances in Neural Information Processing Systems , year=

    Brant: Foundation model for intracranial neural signal , author=. Advances in Neural Information Processing Systems , year=

  242. [252]

    2025 , eprint=

    LUNA: A latent unified network architecture for topology-agnostic EEG modeling , author=. 2025 , eprint=

    2025

  243. [253]

    Neural Networks , year=

    EEGMamba: A foundation model for EEG using selective state-space modeling , author=. Neural Networks , year=

  244. [254]

    2025 , eprint=

    CSBrain: A cross-scale spatiotemporal brain foundation model for EEG decoding , author=. 2025 , eprint=

    2025

  245. [255]

    2025 , eprint=

    REVE: Representation for EEG with versatile embeddings , author=. 2025 , eprint=

    2025

  246. [256]

    2026 , eprint=

    Hypnos: A multimodal autoregressive foundation model for sleep , author=. 2026 , eprint=

    2026

  247. [257]

    Advances in Neural Information Processing Systems , year=

    EEGPT: Pretrained transformer for universal and reliable representation of EEG signals , author=. Advances in Neural Information Processing Systems , year=

  248. [258]

    2026 , eprint=

    Laya: A latent-predictive EEG foundation model , author=. 2026 , eprint=

    2026

  249. [259]

    2026 , eprint=

    NeuroAtlas: A large-scale benchmark for EEG foundation models , author=. 2026 , eprint=

    2026

  250. [260]

    International journal of environmental research and public health , volume=

    A deep learning method approach for sleep stage classification with EEG spectrogram , author=. International journal of environmental research and public health , volume=. 2022 , publisher=

    2022

  251. [261]

    Frontiers in Human Neuroscience , volume=

    Machine learning approaches for MDD detection and emotion decoding using EEG signals , author=. Frontiers in Human Neuroscience , volume=. 2020 , publisher=

    2020

  252. [262]

    IEEE Transactions on Biomedical Engineering , volume=

    STAnet: A spatiotemporal attention network for decoding auditory spatial attention from EEG , author=. IEEE Transactions on Biomedical Engineering , volume=. 2022 , publisher=

    2022

  253. [263]

    Scientific Reports , volume=

    Decoding the cognitive states of attention and distraction in a real-life setting using EEG , author=. Scientific Reports , volume=. 2022 , publisher=

    2022

  254. [264]

    Perrin and J

    F. Perrin and J. Pernier and O. Bertrand and J.F. Echallier , keywords =. Spherical splines for scalp potential and current density mapping , journal =. 1989 , issn =. doi:https://doi.org/10.1016/0013-4694(89)90180-6 , url =

    1989 doi

  255. [265]

    arXiv preprint arXiv:2507.11783 , year=

    Eeg foundation models: A critical review of current progress and future directions , author=. arXiv preprint arXiv:2507.11783 , year=

  256. [266]

    Advances in Neural Information Processing Systems , volume=

    Biot: Biosignal transformer for cross-data learning in the wild , author=. Advances in Neural Information Processing Systems , volume=

  257. [267]

    arXiv preprint arXiv:2405.18765 , year=

    Large brain model for learning generic representations with tremendous EEG data in BCI , author=. arXiv preprint arXiv:2405.18765 , year=

    arXiv

  258. [268]

    arXiv preprint arXiv:2402.10251 , year=

    Brainwave: A brain signal foundation model for clinical applications , author=. arXiv preprint arXiv:2402.10251 , year=

    arXiv

  259. [269]

    arXiv preprint arXiv:2409.00101 , year=

    NeuroLM: A universal multi-task foundation model for bridging the gap between language and EEG signals , author=. arXiv preprint arXiv:2409.00101 , year=

    arXiv

  260. [270]

    2024 IEEE International Symposium on Biomedical Imaging (ISBI) , pages=

    Neuro-gpt: Towards a foundation model for eeg , author=. 2024 IEEE International Symposium on Biomedical Imaging (ISBI) , pages=. 2024 , organization=

    2024

  261. [271]

    Neural Networks , volume=

    An unsupervised EEG decoding system for human emotion recognition , author=. Neural Networks , volume=. 2019 , publisher=

    2019

  262. [272]

    2, 2022-06-27 , author=

    A path towards autonomous machine intelligence version 0.9. 2, 2022-06-27 , author=. Open Review , volume=

    2022

  263. [273]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Self-supervised learning from images with a joint-embedding predictive architecture , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  264. [274]

    arXiv preprint arXiv:1807.03748 , year=

    Representation learning with contrastive predictive coding , author=. arXiv preprint arXiv:1807.03748 , year=

    arXiv

  265. [275]

    arXiv preprint arXiv:1911.05722 , year=

    Momentum contrast for unsupervised visual representation learning , author=. arXiv preprint arXiv:1911.05722 , year=

    1911 arXiv

  266. [276]

    International conference on machine learning , pages=

    A simple framework for contrastive learning of visual representations , author=. International conference on machine learning , pages=. 2020 , organization=

    2020

  267. [277]

    International conference on machine learning , pages=

    Learning transferable visual models from natural language supervision , author=. International conference on machine learning , pages=. 2021 , organization=

    2021

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.