RNA-FM embeddings, fed into a lightweight CNN with binarized ATAC-seq accessibility labels, outperform published Cas12 gRNA activity baselines on the Kim et al. HT1 test set.
HELM: Hierarchical Encoding for mRNA Language Modeling
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Messenger RNA (mRNA) plays a crucial role in protein synthesis, with its codon structure directly impacting biological properties. While Language Models (LMs) have shown promise in analyzing biological sequences, existing approaches fail to account for the hierarchical nature of mRNA's codon structure. We introduce Hierarchical Encoding for mRNA Language Modeling (HELM), a novel pre-training strategy that incorporates codon-level hierarchical structure into language model training. HELM modulates the loss function based on codon synonymity, aligning the model's learning process with the biological reality of mRNA sequences. We evaluate HELM on diverse mRNA datasets and tasks, demonstrating that HELM outperforms standard language model pre-training as well as existing foundation model baselines on seven diverse downstream property prediction tasks and an antibody region annotation tasks on average by around 8%. Additionally, HELM enhances the generative capabilities of language model, producing diverse mRNA sequences that better align with the underlying true data distribution compared to non-hierarchical baselines.
fields
q-bio.GN 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Multimodal Modeling of CRISPR-Cas12 Activity Using Foundation Models and Chromatin Accessibility Data
RNA-FM embeddings, fed into a lightweight CNN with binarized ATAC-seq accessibility labels, outperform published Cas12 gRNA activity baselines on the Kim et al. HT1 test set.