Pith. sign in

REVIEW 6 major objections 4 minor 22 references

NucEL: Single-Nucleotide ELECTRA-Style Genomic Pre-training for Efficient and Interpretable Representations

T0 review · 6 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read NucEL shows ELECTRA-style DNA pre-training matches 25x-larger models, using a 93M-parameter discriminator that supervises every nucleotide token.

desk verdict NucEL is a solid, well-executed ELECTRA-style genomic pretraining paper with useful ablations, but the headline 'rivals 25x larger models' rests on a narrow, noisy margin and unre-run baselines. read the letter →

arxiv 2508.13191 v1 pith:U7G76RKB submitted 2025-08-15 q-bio.GN

Ke Ding , Brian Parker , Jiayu Wen This is my paper · ORCID
classification q-bio.GN
keywords genomiclanguagemodelELECTRAreplacedtokendetectionsingle-nucleotidetokenizationpre-trainingregulatorygenomicsparameterefficiencyattentioninterpretability
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

NucEL argues that the standard masked-language-model objective used by genomic foundation models is wasteful: it supervises only the 15% of tokens that are masked. The paper replaces it with ELECTRA-style replaced token detection (RTD), where a generator proposes replacements and a discriminator must label every token as original or swapped, giving dense supervision at all positions and a roughly 6.7x supervision ratio over MLM. Built with single-nucleotide tokens and a 93M-parameter hybrid-attention transformer, pre-trained only on the human genome, NucEL reports the best average scores on the GUE, Genomic Benchmarks, and NT suites, matching or beating models up to 2.5B parameters. The claim is that a small, efficient model trained with dense token-level supervision extracts more biological signal per parameter than larger MLM models.

What carries the argument

Replaced Token Detection (RTD), borrowed from ELECTRA: a small generator transformer masks and predicts tokens, then a larger discriminator transformer decides for every position whether the token is original or replaced. The discriminator's binary cross-entropy over all N tokens, weighted at λ=50, is the training signal; the total loss is L_generator + 50·L_discriminator. The model pairs RTD with single-nucleotide tokenization (a 27-token vocabulary) and ModernBERT-style hybrid attention—128-token local windows with global attention every third layer—to keep long sequences tractable.

What would settle it

Train a matched 93M-parameter NucEL twin with standard MLM (15% masking) instead of RTD, using identical single-nucleotide tokenization, hybrid attention, data, and compute; if the MLM twin scores within error bars on GUE and NT tasks, the RTD objective is not the cause of the reported gains.

Watch

Extended reading notes

Core claim

On its own terms, the discovery is that ELECTRA's replaced token detection transfers from natural language to DNA and becomes a superior pre-training objective for compact genomic models. The paper demonstrates that a 93M-parameter discriminator trained to detect generator-swapped nucleotides, with single-nucleotide tokenization and hybrid local-global attention, achieves an average score of 75.16 on GUE, an average MCC of 0.664 on the NT benchmark (slightly above NT-Multi-2.5B's 0.661), and an accuracy of 89.9% on Genomic Benchmarks, while using roughly 27x fewer parameters than the largest NT variant. The model also produces sharper attention maps on a synthetic motif-order task, with 65%

Load-bearing premise

The reported state-of-the-art numbers assume the published baseline scores were produced under comparable fine-tuning protocols, seeds, and data splits; the paper adopts prior protocols but does not re-run those baselines.

Editorial extensions

If this is right

  • ELECTRA-style dense supervision could replace MLM as the default pre-training objective for sequence-level genomic foundation models.
  • Single-nucleotide tokenization paired with RTD removes the need for k-mer vocabularies, enabling base-level interpretability and mutation-level analyses without sacrificing benchmark performance.
  • Human-only pre-training can generalize to mouse, yeast, and viral tasks as well as multi-species models, suggesting substantial conserved regulatory signal in the human genome.
  • The reported parameter-efficiency gains (matching 2.5B models at 1/27 the size) would make genomic foundation models practical for groups without massive compute budgets.
  • The attention signal-to-noise results indicate that RTD-trained models localize regulatory motifs more precisely, which could improve motif-discovery pipelines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extending the same RTD recipe to multi-species pre-training is an obvious next test; if dense supervision scales across clades, cross-species scores like CVC could improve above the current human-only baseline.
  • Because RTD supervises every position, NucEL is a natural candidate for variant-effect scoring and fine-grained mutagenesis tasks, which are not tested in this paper.
  • The synthetic motif-order experiment suggests attention maps from RTD models could serve as a motif-discovery prior, but only a real chromatin or ChIP-seq validation would confirm that extension.
  • The paper's own limitation section notes pretraining is confined to human DNA; whether clade-specific elements degrade in non-human tasks is an unaddressed boundary of the claimed cross-species transfer.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 4 minor

Summary. The paper introduces NucEL, a 93M-parameter genomic language model pre-trained on the human genome with an ELECTRA-style replaced-token-detection (RTD) objective, single-nucleotide tokenization, and a ModernBERT-inspired hybrid local/global attention architecture. The authors claim that this combination provides dense token-level supervision, improved sample efficiency, and fine-grained interpretability. They report state-of-the-art or competitive results on the GUE, Genomic Benchmarks, and Nucleotide Transformer benchmark suites, including an average GUE score of 75.16, an average GB accuracy of 89.9%, and an average NT MCC of 0.664, the latter slightly above the 2.5B-parameter Nucleotide Transformer. Additional analyses cover tokenization ablations, an efficiency-performance tradeoff plot, embedding quality across gene biotypes, and an attention SNR study on a synthetic motif-order task.

Significance. If the empirical claims hold, NucEL would be a meaningful contribution: it is, to the authors' knowledge, the first ELECTRA-style genomic foundation model, and it demonstrates a plausible path to reducing the pretraining cost of genomic transformers while preserving or improving downstream performance. The paper also ships useful ablations (tokenization, training objective) and a synthetic interpretability task that goes beyond simple motif enrichment. The strengths are the clear framing of RTD's dense supervision advantage (Section 3.1) and the use of seed-averaged results on GB and NT (Tables 2 and 3). However, the central SOTA and efficiency claims rest on comparisons to published baseline numbers under protocols that are not fully specified, on point estimates without variance in Table 1, and on unsupported petaflop-day numbers in Figure 2(B). These issues currently prevent the paper from fully substantiating its headline claims.

major comments (6)
  1. [§4.1, Table 1] GUE results are reported as averages over three seeds but without any standard deviation or per-seed values. The claim that NucEL achieves the 'highest overall average performance of 75.16' is therefore a point estimate compared to published baseline numbers, and no significance or stability information is provided. Given that the margins on several tasks are small (e.g., TF-H 67.64 vs. DNABERT-2 70.10; CVC 70.29 vs. NT-multi 73.04), the top-2 and SOTA claims are not yet verifiable. Please report per-task means and variances, and clarify whether the baseline numbers are re-run or taken from the original papers.
  2. [§4.3, Table 3] The central parameter-efficiency claim—that NucEL 'slightly exceeds' NT-Multi-2.5B (average MCC 0.664 vs. 0.661)—is a difference of 0.003, which is far smaller than the per-task standard deviations reported in the same table (e.g., H3K27me3 ±0.016, H3K4me3 ±0.019). No paired test, confidence interval, or variance of the average is given, and the baseline numbers are taken from published work. The claim that NucEL 'matches' or 'slightly exceeds' a 2.5B model is not statistically supported by the evidence as presented.
  3. [§4.1-§4.3] The fine-tuning protocols for NucEL are not specified. The text says 'Following DNABERT2's protocol' (§4.1), 'Adopting Caduceus's protocol' (§4.2), and 'Following Dalla-Torre et al.' (§4.3), but gives no optimizer, learning rate, batch size, epochs, sequence length, or classification-head details for NucEL on any benchmark. Combined with the lack of released code or data, this makes the comparison to published baselines unauditable. Minor differences in preprocessing, split selection, early stopping, or hyperparameter search can plausibly shift scores by the margins reported here.
  4. [§4.5, Figure 2(B)] The efficiency-performance tradeoff plot uses 'GPU time (petaflop-days)' on the x-axis, but neither the text nor the appendix defines how these numbers were computed or sourced for NucEL or the baseline models. The claim that NT-multi-2.5B requires 'over 100 times more computational resources' is unsupported by any table or citation. Since efficiency is a central contribution, these values should be tabulated with assumptions (hardware, FLOPs per token, training tokens, etc.) or removed.
  5. [§5 and §4.1] The Limitations section states that NucEL 'demonstrates strong zero-shot generalization to non-human species,' but the cross-species tasks in Table 1 (TF-M, EMP, CVC) are fine-tuned on those species, not evaluated zero-shot. This is a mislabeling. The results support transfer via fine-tuning, not zero-shot generalization, and the overstatement should be corrected.
  6. [Multiple ('see Appendix')] The manuscript repeatedly defers key details to 'Appendix'—architecture specifics (§3.2), fine-tuning details (§3.4), benchmark descriptions (§4), and the synthetic motif-order task (§4.7)—but no appendix is present in the arXiv v1. This missing material is load-bearing for reproducibility and for assessing the synthetic interpretability experiment. Please supply the appendix or inline the necessary details.
minor comments (4)
  1. [Table 4] The embedding-quality F1 scores are reported as point estimates with no standard deviation or number of samples. The micro/weighted F1 differences between NucEL and NT2-100m are very small (0.7603 vs. 0.7579), so the claim of 'superior separation' would benefit from error bars and a description of how the embeddings were extracted and classified.
  2. [§4.4, Figure 2(A)] The tokenization ablation plot reports 'performance on human GUE datasets' but does not specify which tasks are averaged, whether error bars are shown, or how BPE/k-mer tokenizers were implemented for the RTD objective. Clarify the aggregation and provide per-tokenizer details.
  3. [§4.7] The motif-order task is described as 'fine-tuned to near-perfect accuracy,' but no accuracy values or fine-tuning details are given. Since the interpretability claim rests on comparing models at similar performance, report the actual task performance for both models.
  4. [§3.3] The 'supervision ratio of approximately 6.67' is simply 1/0.15, a definitional token-count ratio. It would be helpful to clarify that this is not an empirical sample-efficiency gain but a theoretical advantage of RTD over MLM in supervision coverage.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: NucEL's pre-training objective is definitional, and its performance claims rest on external benchmarks and published baselines.

full rationale

The paper's derivation chain is self-contained with respect to circularity. The RTD objective (Eqs. 1-3) is a standard definition imported from ELECTRA with λ=50; the 6.67 supervision ratio in §3.1 is a simple arithmetic consequence of RTD supervising N tokens versus MLM's 0.15N, not a fitted parameter or a prediction derived from its own outputs. The model architecture combines externally published components (ModernBERT hybrid attention, flash attention, single-nucleotide tokenization), and no load-bearing premise is justified by self-citation: the references are to independent prior work (Clark et al. 2020; Warner et al. 2024; Dalla-Torre et al. 2025; Schiff et al. 2024). Downstream results are compared to published baseline numbers under the stated protocols (§4.1-4.3); this creates a reproducibility/selection concern about auditability and close margins (e.g., 0.664 vs 0.661 on NT), but it is not circularity because the baselines are external evidence, not outputs of NucEL's own fitted parameters. No equation reduces to itself, no fitted value is renamed as a prediction, and no uniqueness or ansatz is imported from same-author citations. Therefore no circular step is present.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central performance claim rests on standard training hyperparameters and publicly used benchmarks rather than on fitted physical parameters. Key choices such as the discriminator loss weight (λ=50), masking ratio (implied 15%), sequence length, and training epochs are human-selected and not optimized against the reported test sets. The main domain assumptions are that RTD provides useful signal for DNA and that published benchmark baselines are directly comparable. No invented entities are introduced.

free parameters (4)
  • discriminator loss weight λ = 50.0
    Total loss is L_generator + 50 * L_discriminator (Eq. 1). Taken from ELECTRA rather than fitted; affects balance of generator/discriminator learning.
  • generator masking ratio r = not stated (likely 0.15)
    The MLM masking fraction for the generator is implied by the 'r=0.15' comparison in §3.1 but never explicitly given for NucEL's generator.
  • pretraining sequence length = 1024 bp
    Random 1024 bp segments sampled from 1224 bp overlapping windows (§3.3).
  • training epochs = 50
    Pretrained for 50 epochs on the human genome (§3.4).
assumptions (3)
  • domain assumption Replaced-token detection provides a useful pretraining signal for DNA sequences
    The whole approach assumes that learning to distinguish real from generator-replaced bases induces biologically meaningful representations. Supported empirically but not proven.
  • domain assumption Benchmark datasets (GUE, GB, NT) and published baseline numbers are reliable and comparable
    Performance comparisons over these benchmarks form the evidence for SOTA claims (§4).
  • domain assumption Single-nucleotide tokenization preserves enough local context for the discriminator to detect replacements
    Assumes individual bases carry detectable statistical signal; in low-complexity DNA regions replacements may be hard to detect.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NucEL: Single-Nucleotide ELECTRA-Style Genomic Pre-training for Efficient and Interpretable Representations." pith.science (2026). https://pith.science/paper/U7G76RKB

@misc{pith2026250813191,
  author       = {Pith},
  title        = {Pith review of: NucEL: Single-Nucleotide ELECTRA-Style Genomic Pre-training for Efficient and Interpretable Representations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U7G76RKB}},
  note         = {Machine review of arXiv:2508.13191}
}
read the original abstract

Pre-training large language models on genomic sequences is a powerful approach for learning biologically meaningful representations. Masked language modeling (MLM) methods, such as DNABERT and Nucleotide Transformer (NT), achieve strong performance but suffer from partial token supervision, pre-training/fine-tuning mismatches, and high computational costs. We introduce NucEL, the first ELECTRA-style pre-training framework for genomic foundation models, addressing these limitations. Using a discriminator to identify tokens altered by a generator, NucEL provides comprehensive token-level supervision across all sequence positions, improving efficiency over the partial supervision of MLM. Incorporating ModernBERT's hybrid local-global attention and flash attention, NucEL offers an optimized BERT architecture for genomic modeling. Unlike 6-mer tokenization, NucEL uses single-nucleotide tokens for fine-grained resolution, boosting both efficiency and interpretability. Pre-trained on the human genome, NucEL achieves state-of-the-art results on diverse downstream tasks -- regulatory element identification (e.g., promoters, enhancers), transcription factor binding prediction, open chromatin classification, and histone modification profiling -- surpassing similarly sized MLM-based models and rivaling models 25x larger, such as NT. Ablation studies highlight optimal tokenization and masking strategies for ELECTRA-style DNA pre-training. Attention analysis reveals NucEL's superior capture of biologically relevant motifs compared to NT, providing insights into hierarchical learning and regulatory element modeling. These findings demonstrate ELECTRA-style pre-training as an efficient, effective strategy for genomic representation learning with broad implications for genomic research.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 8 canonical work pages

  1. [1]

    Accurate structure prediction of biomolecular interactions with alphafold 3

    Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J Ballard, Joshua Bambrick, et al. Accurate structure prediction of biomolecular interactions with alphafold 3. Nature, 630 0 (8016): 0 493--500, 2024

    2024

  2. [2]

    Predicting the sequence specificities of dna-and rna-binding proteins by deep learning

    Babak Alipanahi, Andrew Delong, Matthew T Weirauch, and Brendan J Frey. Predicting the sequence specificities of dna-and rna-binding proteins by deep learning. Nature biotechnology, 33 0 (8): 0 831--838, 2015

    work page 2015

  3. [3]

    Effective gene expression prediction from sequence by integrating long-range interactions

    Z iga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R Ledsam, Agnieszka Grabska-Barwinska, Kyle R Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R Kelley. Effective gene expression prediction from sequence by integrating long-range interactions. Nature methods, 18 0 (10): 0 1196--1203, 2021

    2021

  4. [4]

    Longformer: The long-document transformer

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020

    arXiv 2004

  5. [5]

    Electra: Pre-training text encoders as discriminators rather than generators

    Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555, 2020

    arXiv 2003

  6. [6]

    Nucleotide transformer: building and evaluating robust foundation models for human genomics

    Hugo Dalla-Torre, Liam Gonzalez, Javier Mendoza-Revilla, Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Bernardo P de Almeida, and Hassan Sirelkhatim. Nucleotide transformer: building and evaluating robust foundation models for human genomics. Nature Methods, 22 0 (2): 0 287--297, 2025. ISSN 1548-7091. UR...

    work page 2025

  7. [7]

    Flashattention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691, 2023

    arXiv 2023

  8. [8]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171--4186, 2019

    2019

Show all 22 references
  1. [9]

    Effect of tokenization on transformers for biological sequences

    Edo Dotan, Gal Jaschek, Tal Pupko, and Yonatan Belinkov. Effect of tokenization on transformers for biological sequences. Bioinformatics, 40 0 (4): 0 btae196, 2024

    2024

  2. [10]

    Genomic benchmarks: a collection of datasets for genomic sequence classification

    Katar \' na Gre s ov \'a , Vlastimil Martinek, David C ech \'a k, Petr S ime c ek, and Panagiotis Alexiou. Genomic benchmarks: a collection of datasets for genomic sequence classification. BMC Genomic Data, 24 0 (1): 0 25, 2023

    2023

  3. [11]

    Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing

    Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543, 2021

    2021 arXiv

  4. [12]

    Long short-term memory

    Sepp Hochreiter and J \"u rgen Schmidhuber. Long short-term memory. Neural computation, 9 0 (8): 0 1735--1780, 1997

    1997

  5. [13]

    Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language in genome

    Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V Davuluri. Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language in genome. Bioinformatics, 37 0 (15): 0 2112--2120, 2021. ISSN 1367-4803

    2021

  6. [14]

    Highly accurate protein structure prediction with alphafold

    John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Z \' dek, Anna Potapenko, et al. Highly accurate protein structure prediction with alphafold. nature, 596 0 (7873): 0 583--589, 2021

    2021

  7. [15]

    Deep learning

    Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521 0 (7553): 0 436--444, 2015

    2015

  8. [16]

    Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution

    Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow, Callum Birch-Sykes, Stefano Massaroli, Aman Patel, Clayton Rabideau, and Yoshua Bengio. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. Advances in neural information process...

    2023

  9. [17]

    Hyena hierarchy: Towards larger convolutional language models

    Michael Poli, Stefano Massaroli, Eric Nguyen, Daniel Y Fu, Tri Dao, Stephen Baccus, Yoshua Bengio, Stefano Ermon, and Christopher R \'e . Hyena hierarchy: Towards larger convolutional language models. In International Conference on Machine Learning, pages 28043--28078. PMLR, 2023

    2023

  10. [18]

    Dna language model grover learns sequence context in the human genome

    Melissa Sanabria, Jonas Hirsch, Pierre M Joubert, and Anna R Poetsch. Dna language model grover learns sequence context in the human genome. Nature Machine Intelligence, 6 0 (8): 0 911--923, 2024. ISSN 2522-5839

    2024

  11. [19]

    Unlocking gene regulation with sequence-to-function models

    Alexander Sasse, Maria Chikina, and Sara Mostafavi. Unlocking gene regulation with sequence-to-function models. Nature methods, 21 0 (8): 0 1374--1377, 2024

    2024

  12. [20]

    Caduceus: Bi-directional equivariant long-range dna sequence modeling

    Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov. Caduceus: Bi-directional equivariant long-range dna sequence modeling. arXiv preprint arXiv:2403.03234, 2024

    2024 arXiv

  13. [21]

    Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and inference

    Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, and Tom Aarsen. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetu...

    2024 arXiv

  14. [22]

    Dnabert-2: Efficient foundation model and benchmark for multi-species genome

    Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana Davuluri, and Han Liu. Dnabert-2: Efficient foundation model and benchmark for multi-species genome. arXiv preprint arXiv:2306.15006, 2023

    2023 arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.