REVIEW 9 cited by
Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large-scale sequence modeling has sparked rapid advances that now extend into biology and genomics. However, modeling genomic sequences introduces challenges such as the need to model long-range token interactions, the effects of upstream and downstream regions of the genome, and the reverse complementarity (RC) of DNA. Here, we propose an architecture motivated by these challenges that builds off the long-range Mamba block, and extends it to a BiMamba component that supports bi-directionality, and to a MambaDNA block that additionally supports RC equivariance. We use MambaDNA as the basis of Caduceus, the first family of RC equivariant bi-directional long-range DNA language models, and we introduce pre-training and fine-tuning strategies that yield Caduceus DNA foundation models. Caduceus outperforms previous long-range models on downstream benchmarks; on a challenging long-range variant effect prediction task, Caduceus exceeds the performance of 10x larger models that do not leverage bi-directionality or equivariance.
Forward citations
Cited by 9 Pith papers
-
Geometric Hyena Networks for Large-scale Equivariant Learning
Geometric Hyena is an equivariant long-convolutional architecture that captures global geometric context with sub-quadratic complexity and outperforms equivariant transformer baselines on several RNA and protein predi...
-
pLSTM: parallelizable Linear Source Transition Mark networks
pLSTM extends linear recurrent networks to general directed acyclic graphs with a parallelizable scheme and two stabilization modes for long-range propagation.
-
Evaluating DNA function understanding in genomic language models using evolutionarily implausible sequences
A new benchmark shows that genomic language models mostly fail to detect loss-of-function mutations in synthetic, evolutionarily implausible DNA, with accuracy tied to how likely the model finds the sequence.
-
NucEL: Single-Nucleotide ELECTRA-Style Genomic Pre-training for Efficient and Interpretable Representations
NucEL shows ELECTRA-style replaced-token pretraining on single-nucleotide DNA tokens reaches state-of-the-art regulatory genomics performance with far fewer parameters.
-
SPACE: Your Genomic Profile Predictor is a Powerful DNA Foundation Model
SPACE shows that supervised prediction of genomic profiles such as chromatin accessibility and histone marks produces competitive DNA representations, with a mixture-of-experts architecture that improves cross-species...
-
HAD: Hybrid Architecture Distillation Outperforms Teacher in Genomic Sequence Modeling
A compact hybrid GDN+attention model distilled from Nucleotide Transformer v2 outperforms similarly sized models and, on several tasks, its 500x larger teacher.
-
OmniGenBench: A Modular Platform for Reproducible Genomic Foundation Models Benchmarking
OmniGenBench packages five genomic benchmark suites, 31+ foundation models, automated evaluation, and interpretability tools into standardized one-command workflows.
-
Evaluation of Coding Schemes for Transformer-based Gene Sequence Modeling
BPE tokenization and rotary position embeddings usually outperform k-mers and other positional encodings in from-scratch Transformer DNA classifiers, but the advantage is task-dependent.
-
Fast and Scalable Gene Embedding Search: A Comparative Study of FAISS and ScaNN
On 400 bp microbial gene fragments, embedding-based nearest-neighbor search with FAISS and ScaNN outperforms nucleotide MMseqs2 in speed and accuracy; tuned FAISS configurations also beat ScaNN.
Discussion (0). Continue with ORCID to comment.