REVIEW 9 cited by
HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Genomic (DNA) sequences encode an enormous amount of information for gene regulation and protein synthesis. Similar to natural language models, researchers have proposed foundation models in genomics to learn generalizable features from unlabeled genome data that can then be fine-tuned for downstream tasks such as identifying regulatory elements. Due to the quadratic scaling of attention, previous Transformer-based genomic models have used 512 to 4k tokens as context (<0.001% of the human genome), significantly limiting the modeling of long-range interactions in DNA. In addition, these methods rely on tokenizers or fixed k-mers to aggregate meaningful DNA units, losing single nucleotide resolution where subtle genetic variations can completely alter protein function via single nucleotide polymorphisms (SNPs). Recently, Hyena, a large language model based on implicit convolutions was shown to match attention in quality while allowing longer context lengths and lower time complexity. Leveraging Hyena's new long-range capabilities, we present HyenaDNA, a genomic foundation model pretrained on the human reference genome with context lengths of up to 1 million tokens at the single nucleotide-level - an up to 500x increase over previous dense attention-based models. HyenaDNA scales sub-quadratically in sequence length (training up to 160x faster than Transformer), uses single nucleotide tokens, and has full global context at each layer. We explore what longer context enables - including the first use of in-context learning in genomics. On fine-tuned benchmarks from the Nucleotide Transformer, HyenaDNA reaches state-of-the-art (SotA) on 12 of 18 datasets using a model with orders of magnitude less parameters and pretraining data. On the GenomicBenchmarks, HyenaDNA surpasses SotA on 7 of 8 datasets on average by +10 accuracy points. Code at https://github.com/HazyResearch/hyena-dna.
Forward citations
Cited by 9 Pith papers
-
Decoding Phenotypes: A Framework for Fusing Genomic Language Models and Neuroimaging
GeneFuse fuses genomic language model embeddings with MRI via multi-scale FiLM modulation and uncertainty-gated residuals, improving dementia screening AUROC by 0.12 over image-only on the ADNI cohort.
-
OmniGenBench: A Modular Platform for Reproducible Genomic Foundation Models Benchmarking
OmniGenBench packages five genomic benchmark suites, 31+ foundation models, automated evaluation, and interpretability tools into standardized one-command workflows.
-
Multi-megabase scale genome interpretation with genetic language models
Phenformer uses frozen Enformer embeddings of 512 gene windows to predict disease risk and cell type involvement, outperforming PRS baselines restricted to the same genomic regions.
-
Screening of Biosecurity Features in Metagenomic Data with Evo 2 Probes
Frozen Evo 2 layer-26 embeddings carry linearly accessible AMR and virulence signals that simple probes read out with high held-out AUC, including on simulated short reads.
-
Correcting Prompt Dependence in LLM Benchmarks: A Bayesian Hierarchical Model with Embedding-Space Clustering
A Bayesian model that groups similar LLM test prompts into clusters gives better predictive scores than a no-clustering baseline but does not prove that it truly corrects prompt dependence.
-
When repeats drive the vocabulary: a Byte-Pair Encoding analysis of T2T primate genomes
BPE tokenizers trained on nine T2T primate genomes share only 11,569 of 512,000 tokens, and the vocabulary is dominated by short repeats rather than phylogenetic signal.
-
eccDNAMamba: A Pre-Trained Model for Ultra-Long eccDNA Sequence Analysis
eccDNAMamba is a bidirectional Mamba-2 model with circular sequence augmentation and span masking that classifies eccDNA better than existing genomic foundation models on two benchmarks.
-
The Reality of AI and Biorisk
A review of the evidence finds no current, statistically significant uplift in biorisk from LLMs or AI biological tools, but the underlying studies are too nascent to justify strong conclusions.
-
The Good, The Efficient and the Inductive Biases: Exploring Efficiency in Deep Learning Through the Use of Inductive Biases
A dissertation synthesizing the author's papers on continuous kernel convolutions and symmetry-preserving architectures, claiming these inductive biases improve deep learning efficiency.
Discussion (0). Continue with ORCID to comment.