REVIEW 16 cited by
DNABERT-2: Efficient Foundation Model and Benchmark For Multi-Species Genome
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Decoding the linguistic intricacies of the genome is a crucial problem in biology, and pre-trained foundational models such as DNABERT and Nucleotide Transformer have made significant strides in this area. Existing works have largely hinged on k-mer, fixed-length permutations of A, T, C, and G, as the token of the genome language due to its simplicity. However, we argue that the computation and sample inefficiencies introduced by k-mer tokenization are primary obstacles in developing large genome foundational models. We provide conceptual and empirical insights into genome tokenization, building on which we propose to replace k-mer tokenization with Byte Pair Encoding (BPE), a statistics-based data compression algorithm that constructs tokens by iteratively merging the most frequent co-occurring genome segment in the corpus. We demonstrate that BPE not only overcomes the limitations of k-mer tokenization but also benefits from the computational efficiency of non-overlapping tokenization. Based on these insights, we introduce DNABERT-2, a refined genome foundation model that adapts an efficient tokenizer and employs multiple strategies to overcome input length constraints, reduce time and memory expenditure, and enhance model capability. Furthermore, we identify the absence of a comprehensive and standardized benchmark for genome understanding as another significant impediment to fair comparative analysis. In response, we propose the Genome Understanding Evaluation (GUE), a comprehensive multi-species genome classification dataset that amalgamates $36$ distinct datasets across $9$ tasks, with input lengths ranging from $70$ to $10000$. Through comprehensive experiments on the GUE benchmark, we demonstrate that DNABERT-2 achieves comparable performance to the state-of-the-art model with $21 \times$ fewer parameters and approximately $92 \times$ less GPU time in pre-training.
Forward citations
Cited by 16 Pith papers
-
Evaluating DNA function understanding in genomic language models using evolutionarily implausible sequences
A new benchmark shows that genomic language models mostly fail to detect loss-of-function mutations in synthetic, evolutionarily implausible DNA, with accuracy tied to how likely the model finds the sequence.
-
Deep and Probabilistic Models for Gene Regulatory Network Inference
A dual-stage GRN inference framework combining a sequence-based transformer prior (GLM-Prior) with a probabilistic matrix-factorization model (PMF-GRN) for uncertainty-aware network reconstruction.
-
Rethinking Genomic Modeling Through Optical Character Recognition
Rendering DNA as OCR-style page images and training a vision-language model on reading/grounding/retrieval/completion tasks outperforms sequence-based genomic models on tested benchmarks with ~20x fewer effective tokens.
-
BioArc: Discovering Optimal Neural Architectures for Biological Foundation Models
BioArc uses neural architecture search to discover small hybrid models for DNA and protein that outperform far larger pretrained foundation models on several benchmarks.
-
NucEL: Single-Nucleotide ELECTRA-Style Genomic Pre-training for Efficient and Interpretable Representations
NucEL shows ELECTRA-style replaced-token pretraining on single-nucleotide DNA tokens reaches state-of-the-art regulatory genomics performance with far fewer parameters.
-
GeneBreaker: Jailbreak Attacks against DNA Language Models with Pathogenicity Guidance
GeneBreaker, a new attack framework, steers DNA language models to generate sequences with over 90% identity to human pathogens, with success rates up to 60% on the largest Evo2 model.
-
HAD: Hybrid Architecture Distillation Outperforms Teacher in Genomic Sequence Modeling
A compact hybrid GDN+attention model distilled from Nucleotide Transformer v2 outperforms similarly sized models and, on several tasks, its 500x larger teacher.
-
OmniGenBench: A Modular Platform for Reproducible Genomic Foundation Models Benchmarking
OmniGenBench packages five genomic benchmark suites, 31+ foundation models, automated evaluation, and interpretability tools into standardized one-command workflows.
-
JEPA-DNA: Grounding Genomic Foundation Models through Joint-Embedding Predictive Architectures
Adding a JEPA latent objective to DNABERT-2 improves several genomics benchmark scores, but the paper lacks the control needed to prove the JEPA objective—not extra pretraining—is responsible.
-
Evaluation of Coding Schemes for Transformer-based Gene Sequence Modeling
BPE tokenization and rotary position embeddings usually outperform k-mers and other positional encodings in from-scratch Transformer DNA classifiers, but the advantage is task-dependent.
-
BMFM-DNA: A SNP-aware DNA foundation model to capture variant effects
Encoding human genetic variants as special characters during DNA foundation model pre-training is claimed to improve downstream task performance, with small margins and a confounded comparison.
-
Minimalist Softmax Attention Provably Learns Constrained Boolean Functions
With teacher forcing that reveals pairwise products of the relevant bits, one gradient step lets a single-head attention recover the support of a k-bit AND/OR; the paper's claimed end-to-end hardness lower bound is in...
-
Generative Artificial Intelligence in Bioinformatics: A Systematic Review of Models, Applications, and Methodological Advances
Across the 68 papers it surveys, domain-specialized generative models usually outperform general-purpose LLMs on biological tasks, and agentic/conversational workflows are the least-covered topics.
-
Fast and Scalable Gene Embedding Search: A Comparative Study of FAISS and ScaNN
On 400 bp microbial gene fragments, embedding-based nearest-neighbor search with FAISS and ScaNN outperforms nucleotide MMseqs2 in speed and accuracy; tuned FAISS configurations also beat ScaNN.
-
EnTao-GPM: DNA Foundation Model for Predicting the Germline Pathogenic Mutations
EnTao-GPM, a fine-tuned DNA model built on the authors' TrinityDNA, reports AUC up to 0.963 for germline pathogenic mutation prediction on a ClinVar holdout.
-
Leveraging Natural Language Processing to Unravel the Mystery of Life: A Review of NLP Approaches in Genomics, Transcriptomics, and Proteomics
A review maps how NLP architectures from word2vec to Evo 2 are applied to DNA, RNA, protein, and genome sequences, with tokenization and context choices shaping performance.
Discussion (0). Sign in to comment.