Pith. sign in

REVIEW 4 major objections 5 minor 32 references

BioLangFusion: Multimodal Fusion of DNA, mRNA, and Protein Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fusing frozen DNA, mRNA, and protein language models at the codon level outperforms the best single-modality baseline on five molecular property prediction tasks.

desk verdict The codon-level alignment idea is worth a look, but the paper's central claim of consistent gains is contradicted by its own Table 1 and needs major revision. read the letter →

arxiv 2506.08936 v1 pith:2NZOAKTA submitted 2025-06-10 cs.LG

Amina Mollaysa , Artem Moskale , Pushpak Pati , Tommaso Mansi , Mangal Prakash , Rui Liao This is my paper
classification cs.LG
keywords multimodalfusionlanguagemodelsDNAmRNAproteinmolecularpropertypredictioncodonalignmentattentionpooling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

BioLangFusion sets out to show that DNA, mRNA, and protein sequences carry complementary, jointly useful information that can be combined without retraining any large foundation model. It aligns frozen embeddings from a DNA model (6-mer tokens), an mRNA model (single-nucleotide tokens), and a protein model (amino-acid tokens) onto a shared codon grid, then attaches one of three small fusion heads. On five molecular property prediction tasks—vaccine mRNA degradation, fungal expression, E. coli expression classification, mRNA stability, and antibody expression—the paper reports that the fusion approach beats the strongest unimodal baselines, with entropy-regularized attention pooling the most consistently strong variant. The appeal is practical: no end-to-end retraining, a small parameter overhead, and attention weights that indicate which modality the model actually uses for each task.

What carries the argument

The load-bearing object is the codon-level alignment frame, in which every position $t$ of the three embedding tensors is taken to name the same biological codon. The DNA embeddings (originally over 6-mers) are upsampled with a stride-2 transposed convolution; the mRNA embeddings (per nucleotide) are downsampled by non-overlapping mean pooling over windows of three; the protein embeddings already sit on the frame. On top of that frame sit three fusion heads: (i) codon-level concatenation, with a modality-specific MLP projection for the 4107-dimensional DNA embedding so it does not dominate the 640-dimensional RNA and 320-dimensional protein vectors; (ii) entropy-regularized gated attention pooling, which treats the three modalities as a bag of instances, computes a softmax weight per modality from mean-pooled summaries, and adds a negative-entropy term to keep the weights from collapsing to uniform; and (iii) cross-modal multi-head attention, which concatenates the projected modalities along the time axis into a joint context $C = [H_{\mathrm{DNA}}; H_{\mathrm{RNA}}; H_{\mathrm{Prot}}] \in \mathbb{R}^{3T' \times d}$, lets each modality query that context, and merges the updated streams by residual averaging plus layer normalization. All three feed the same fixed TextCNN prediction head, so the reported differences come from the fusion mechanism itself.

What would settle it

Permute the order of codon-level positions in the DNA and protein embeddings relative to the mRNA embedding (or vice versa) while keeping each modality's own sequence order intact, and re-run the five tasks with the same fusion heads. If randomly misaligned fusion matches the codon-aligned fusion, the biological alignment is not the source of the gains; if performance drops, the alignment frame is confirmed as the carrier of the improvement.

Watch

Extended reading notes

Core claim

The paper's central claim is that a shared codon-level frame is enough to make frozen single-modality embeddings behave like a multimodal representation: after upsampling the DNA model's 6-mer tokens and downsampling the mRNA model's per-nucleotide tokens onto the protein frame of length $T' = T/3$, each aligned position nominally corresponds to the same biological codon across all three modalities. On that frame, the authors test three fusion heads and report that fusion beats the best unimodal baseline across the five tasks, with entropy-regularized attention pooling the most consistent winner (e.g., Spearman 0.864 vs 0.841 on CoV-Vac and accuracy 52.107% vs 49.017% on E. coli) while cross-modal multi-head attention takes the lead on E. coli classification. The claim is deliberately economical: the base models are frozen, no knowledge distillation or weight merging is needed, and the only trainable parts are the alignment projections, fusion head, and a fixed TextCNN predictor. Along the way, the attention weights are shown to track expected biology—mRNA emphasized for mRNA stability, protein emphasized for E. coli expression.

Load-bearing premise

The load-bearing assumption is that every mRNA position is translated, so the protein sequence has exactly one-third the length of the transcript and the $T/3$ codon grid puts the same biological location in correspondence across DNA, mRNA, and protein; the datasets include untranslated regions where that correspondence breaks.

Editorial extensions

If this is right

  • Adding any of the three fusion heads to frozen DNA, mRNA, and protein embeddings can improve molecular property prediction over the best single-modality baseline, with no end-to-end retraining of the base models.
  • Entropy-regularized attention pooling is the most consistently strong fusion head, so it is the natural default for low-budget multimodal setups on similar tasks.
  • The learned modality attention weights provide an interpretable diagnostic: they highlight mRNA on mRNA-stability prediction and protein on E. coli expression classification, suggesting the fused model can expose which biological layer drives a phenotype.
  • Codon-level alignment itself contributes to the gains: the paper's ablation shows vanilla concatenation without alignment scores 0.818 vs 0.831 on CoV-Vac and 46.208% vs 50.280% on E. coli, so the alignment step is not inert.
  • Cross-modal token-level attention is not uniformly better than sequence-level modality attention; it wins on E. coli classification but trails on most other tasks, implying token-level cross-talk is only sometimes worth its extra complexity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not restrict alignment to the annotated coding sequence; a natural follow-up is to compare the full-transcript $T/3$ alignment against a CDS-only alignment, which would reveal whether gains come from true codon correspondence or from pooling extra UTR-derived signal.
  • The recipe is model-agnostic: the same fusion heads could be applied to other frozen encoders (e.g., larger ESM-2 checkpoints or codon-aware mRNA models), a test the paper leaves open since it uses one checkpoint per modality.
  • The attention weights are presented as interpretation; a causal check would be to mask or corrupt one modality at inference and confirm that prediction accuracy drops most on tasks where that modality's learned attention is highest.
  • If the gains survive random permutation of codon positions across modalities, the 'biologically meaningful correspondence' explanation would be in doubt; if they vanish, the alignment frame is doing real work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. BioLangFusion proposes fusing frozen embeddings from DNA, mRNA, and protein language models by first aligning them at a claimed codon-level resolution and then applying one of three fusion heads: concatenation, entropy-regularized gated attention pooling (MIL), or cross-modal multi-head attention. The method is evaluated on five molecular property prediction datasets (CoV-Vac, Fungal, E. coli, mRNA Stab., Ab1) against several unimodal baselines. The abstract, introduction, and Section 3 claim that the fusion variants consistently outperform the best single-modality baselines, with additional interpretability analysis of learned attention weights.

Significance. If the claims were fully supported, the paper would offer a lightweight, modular recipe for combining existing biological foundation models without retraining, which could be practically useful. The interpretability analysis through modality attention is a nice qualitative addition, and the paper explicitly lists the frozen backbones, fusion dimensions, and training hyperparameters. However, the central empirical claim is contradicted by the paper's own Table 1 under any straightforward reading, and the biological alignment assumption in Section 2.1 is questionable for mRNA sequences with untranslated regions. The paper provides no error bars, significance tests, or code, so the quantitative evidence is currently too weak to establish the claimed advantage.

major comments (4)
  1. [Section 3, Table 1] The statement 'Across all the tasks, fusion-based models consistently outperform the best single-modality baselines' is not supported by the reported numbers. Taking the best unimodal baseline per column (RNA-FM for CoV-Vac, E. coli, mRNA Stab., Ab1; Nucleotide Transformer for Fungal), the fusion variants lose in 6 of 15 comparisons: Concatenation underperforms on CoV-Vac (0.831 vs 0.841), E. coli (50.280 vs 52.949), and mRNA Stab. (0.539 vs 0.553); Cross-Attention underperforms on CoV-Vac (0.828 vs 0.841) and mRNA Stab. (0.550 vs 0.553); MIL+Entropy underperforms on E. coli (52.107 vs 52.949). The claim can only survive if one selects the best fusion variant per task after seeing the test set, which is not stated as the evaluation protocol. The claim must be rewritten to reflect the observed pattern (e.g., some fusion variants help some tasks) and supported with uncertainty quantification.
  2. [Section 2.1, Eq. (1)] The alignment procedure assumes that every mRNA position is coding: the protein frame is defined as T' = T/3, where T is the full mRNA length, and the mRNA embedding is downsampled by non-overlapping 3-nucleotide mean pooling. However, the datasets in Table 2 are full mRNA sequences, which include 5' and 3' untranslated regions; the actual protein-coding sequence is generally shorter than T/3 and is not aligned to the start of the mRNA. Consequently, the 'codon-level' correspondence between the pooled RNA tokens and the protein residues is biologically incorrect for these sequences. The authors need to either translate the coding sequence and align only the CDS, or explicitly model UTRs separately; otherwise the central alignment mechanism does not do what the paper claims.
  3. [Section 2.1, Table 2] The preprocessing step truncates RNA sequences longer than 1000 tokens, yet Table 2 reports max lengths up to 3066 and shows large drops between raw and used counts (Fungal: 7056 to 3138; E. coli: 6348 to 4450; mRNA Stab.: 41123 to 23929). These numbers are inconsistent with a straightforward truncation to 1000 tokens (the post-truncation max length would be 1000), and the table caption does not state whether 'Max Length' refers to raw or used sequences. More importantly, the paper does not analyze how truncation interacts with the T/3 alignment or whether the discarded long sequences are systematically different. The authors should report post-truncation statistics, explain the discrepancy, and justify that the truncated subset remains representative.
  4. [Section 3, Table 1 and Appendix A.4] No error bars, confidence intervals, or significance tests are reported, and the evaluation appears to use a single split with fixed training runs. Several reported gains are very small (e.g., +0.003 Spearman on Ab1 for concatenation, +0.010 on mRNA Stab. for MIL+Entropy), so without repeated seeds or statistical testing they cannot be distinguished from noise. The authors should add multiple random seeds, report mean and standard deviation, and, where appropriate, paired significance tests (e.g., Wilcoxon or bootstrap) against the best unimodal baseline.
minor comments (5)
  1. [Abstract and Section 1] There is a typo in the abstract: 'a an mRNA sequence' should be 'an mRNA sequence'; also 'can destabilizing an mRNA hairpin' in the Introduction should be 'can destabilize an mRNA hairpin'.
  2. [Appendix A.3] The text contains typos and grammatical errors, e.g., 'Moverove' should be 'Moreover', 'having shared projection layers accross modality' should be 'across modalities', and 'brings performance gain only on one (Ab1) out of the five data set' should be 'datasets'.
  3. [References] Several references are duplicated or incomplete: Dalla-Torre et al. appears as 2022, 2023, and 2024 entries; Elnaggar et al. as 2021a and 2021b; Lin et al. as 2022a and 2022b; and some entries contain placeholder-like DOIs (e.g., '10.1101/2023.01.01.123456'). The reference list should be consolidated and verified.
  4. [Eq. (1) and List 1] The transposed convolution for DNA upsampling is described inconsistently: Eq. (1) says k=2, s=2, while List 1 states ConvTranspose1D(kernel=3, stride=2, padding=2) and List 2 states ConvTranspose1D(kernel=3, stride=2, padding=2). Please reconcile the kernel size and padding so the implementation matches the method description.
  5. [Table 3] The Nucleotide Transformer v2 100M model is listed with an embedding dimension of 4,107, which is unexpectedly large for a 100M-parameter model and differs from typical reported hidden dimensions for this model. Please verify this number and, if it is a concatenation of multiple representations, state that explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the fusion benchmark is evaluated on held-out data; the only self-citation is motivational and non-load-bearing.

full rationale

BioLangFusion is an empirical benchmark paper rather than a derivation from first principles. The central claim — that codon-level fusion of frozen DNA, mRNA, and protein embeddings improves downstream molecular property prediction — is supported by test-set numbers in Table 1 and ablations in Table 4. The fusion parameters and prediction heads are trained on training splits, and the entropy weight λ is selected on validation from {0.01, 0.5, 1}. No fitted constant is renamed as a prediction: the reported Spearman and accuracy values are computed on held-out splits, so no 'prediction' reduces by construction to a training-set fit. The modality alignment in Eq. (1) is a modeling choice: transposed-conv upsampling of DNA 6-mers and mean pooling of mRNA nucleotides to a T/3 protein frame define what the paper means by codon-level correspondence. It may be biologically wrong for UTR-containing mRNAs, but an incorrect or over-simplified assumption is not circularity. The paper cites the authors' own prior work (Prakash et al., 2024) only for motivation, e.g., 'DNA or protein FMs have also shown utility for mRNA-specific tasks,' and to justify the choice of representative unimodal models; it is not invoked as a uniqueness theorem or as evidence for the fusion results. The external benchmarks (CoV-Vac, Fungal, E. coli, mRNA Stab., Ab1) come from other groups' datasets. The skeptic's observation that Table 1 contradicts the 'consistently outperform' wording — several fusion variants lose to the best unimodal baseline on several tasks — is an empirical-support or correctness issue, not a circularity issue, and per the rubric it does not raise the circularity score.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central empirical claim rests on a codon-level alignment assumption, a hand-chosen length cutoff, and the assumption that frozen unimodal embeddings provide complementary signal. No new physical entities are proposed. The entropy weight and projection dimensions are the main free parameters; the paper does not disclose d'DNA.

free parameters (3)
  • Entropy regularization weight lambda = chosen from {0.01, 0.5, 1} via validation performance
    Controls the strength of the entropy penalty in the MIL fusion loss (Section 2.2.2). It is selected per task based on validation performance, so it is a fitted hyperparameter that affects the reported results.
  • Sequence length cutoff of 1000 tokens = 1000
    Sequences longer than 1000 nucleotides appear to have been discarded, not truncated, since Table 2 shows used counts far below raw counts for Fungal and mRNA Stab. This hand-chosen cutoff determines which samples are evaluated.
  • DNA projection dimension d'DNA = unspecified in the text
    Section 2.2.1 projects the 4107-dimensional Nucleotide Transformer embedding down to d'DNA before concatenation, but the value is never given, so the fused representation size is unclear.
assumptions (3)
  • domain assumption All mRNA tokens are within the coding sequence, so the protein frame length T/3 corresponds to actual codons
    Section 2.1 aligns DNA and mRNA embeddings to the protein frame of length T'=T/3. This is only valid for CDS-only sequences; the datasets are described as full mRNA sequences, which typically contain untranslated regions.
  • domain assumption Pretrained unimodal embeddings contain task-relevant information that can be combined by linear projections and pooling without retraining the base models
    The central claim that frozen-encoder fusion outperforms unimodal baselines presupposes the embeddings carry complementary, non-redundant signal. The paper does not quantify redundancy or demonstrate that the base models are individually well-calibrated on these tasks.
  • ad hoc to paper Dropping sequences longer than 1000 nucleotides preserves a representative dataset
    Table 2 shows only 3138 of 7056 fungal sequences and 23929 of 41123 mRNA stability sequences are used. If long genes have different biology, the reported comparisons on the short-sequence subset may not generalize.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BioLangFusion: Multimodal Fusion of DNA, mRNA, and Protein Language Models." pith.science (2026). https://pith.science/paper/2NZOAKTA

@misc{pith2026250608936,
  author       = {Pith},
  title        = {Pith review of: BioLangFusion: Multimodal Fusion of DNA, mRNA, and Protein Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2NZOAKTA}},
  note         = {Machine review of arXiv:2506.08936}
}
read the original abstract

We present BioLangFusion, a simple approach for integrating pre-trained DNA, mRNA, and protein language models into unified molecular representations. Motivated by the central dogma of molecular biology (information flow from gene to transcript to protein), we align per-modality embeddings at the biologically meaningful codon level (three nucleotides encoding one amino acid) to ensure direct cross-modal correspondence. BioLangFusion studies three standard fusion techniques: (i) codon-level embedding concatenation, (ii) entropy-regularized attention pooling inspired by multiple-instance learning, and (iii) cross-modal multi-head attention -- each technique providing a different inductive bias for combining modality-specific signals. These methods require no additional pre-training or modification of the base models, allowing straightforward integration with existing sequence-based foundation models. Across five molecular property prediction tasks, BioLangFusion outperforms strong unimodal baselines, showing that even simple fusion of pre-trained models can capture complementary multi-omic information with minimal overhead.

Figures

Figures reproduced from arXiv: 2506.08936 by the authors.

Figure 1
Figure 1. BioLangFusion Architecture Overview: pretrained DNA, RNA, and protein embeddings are aligned at the codon level and fused using biologically motivated strategies then passed to a prediction head for downstream molecular property prediction. introduce a lightweight gated attention mechanism that dy￾namically weights each modality based on its relevance to the task. For simplicity, we denote the aligned embeddings as:… view at source ↗
Figure 2
Figure 2. Modality-wise attention weights with and without en￾tropy regularization from MIL based fusion model 4. Conclusion In this paper, we explore strategies to fuse pretrained DNA, mRNA, and protein foundation models, capturing their bio￾logical interconnectivity. We introduce BIOLANGFUSION, a lightweight, plug-and-play suite of fusion strategies for integrating pretrained DNA, mRNA, and protein FMs. It aligns embeddings… view at source ↗
Figure 3
Figure 3. Model architectures for the three BioLangFusion fusion strategies. Codon-aware concatenation: DNA embeddings are upsampled via transposed convolution and mRNA embeddings downsampled via non-overlapping mean pooling to align with the protein framing; the three aligned embeddings are then concatenated at each codon position and projected before entering the TextCNN head. Entropy￾regularized gated attention (MIL): Afte… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

32 extracted references · 24 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    P., and PIERROT, T

    Boshar, S., Trop, E., de Almeida, B. P., and PIERROT, T. Are genomic language models all you need? exploring genomic language models on protein downstream tasks. In ICLR 2024 Workshop on Machine Learning for Genomics Explorations, 2024. URL https://openreview.net/forum?id=B60QZ0R2Zw

    work page 2024

  3. [3]

    G., Ku, J., Poli, M., Brockman, G., Chang, D., Gonzalez, G

    Brixi, G., Durrant, M. G., Ku, J., Poli, M., Brockman, G., Chang, D., Gonzalez, G. A., King, S. H., Li, D. B., Merchant, A. T., et al. Genome modeling and design across all domains of life with evo 2. BioRxiv, pp.\ 2025--02, 2025

    2025

  4. [4]

    Self-supervised learning on millions of primary rna sequences from 72 vertebrates improves sequence-based rna splicing prediction

    Chen, K., Zhou, Y., Ding, M., Wang, Y., Ren, Z., and Yang, Y. Self-supervised learning on millions of primary rna sequences from 72 vertebrates improves sequence-based rna splicing prediction. Briefings in bioinformatics, 25 0 (3): 0 bbae163, 2024

    work page 2024

  5. [5]

    Rna-fm: A foundation model and benchmark for functional rna embeddings

    Chen, X., Sun, Z., Lin, Y., et al. Rna-fm: A foundation model and benchmark for functional rna embeddings. bioRxiv, 2022. doi:10.1101/2022.11.17.516915

    work page doi:10.1101/2022.11.17.516915 2022

  6. [6]

    H., Oteri, F., Dallago, C., Trop, E., Sirelkhatim, H., Richard, G., et al

    Dalla-Torre, H., Gonzalez, L., Mendoza Revilla, J., Lopez Carranza, N., Grywaczewski, A. H., Oteri, F., Dallago, C., Trop, E., Sirelkhatim, H., Richard, G., et al. The nucleotide transformer: Building and evaluating robust foundation models for human genomics. bioRxiv, 2023. URL https://www.biorxiv.org/content/10.1101/2023.01.01.123456v1

    work page doi:10.1101/2023.01.01.123456v1 2023

  7. [8]

    Nucleotide transformer: Building and evaluating robust foundation models for human genomics

    Dalla-Torre, L., Chughtai, Z., Yan, Y., et al. Nucleotide transformer: Building and evaluating robust foundation models for human genomics. bioRxiv, 2024. doi:10.1101/2022.11.15.516627

    work page doi:10.1101/2022.11.15.516627 2024

  8. [9]

    G., Castellano, L

    Diez, M., Medina-Mu \ n oz, S. G., Castellano, L. A., da Silva Pescador, G., Wu, Q., and Bazzini, A. A. icodon customizes gene expression based on the codon composition. Scientific Reports, 12 0 (1): 0 12126, 2022

    work page 2022

Show all 32 references
  1. [10]

    Mpepe, a predictive approach to improve protein expression in e

    Ding, Z., Guan, F., Xu, G., Wang, Y., Yan, Y., Zhang, W., Wu, N., Yao, B., Huang, H., Tuller, T., et al. Mpepe, a predictive approach to improve protein expression in e. coli based on deep learning. Computational and Structural Biotechnology Journal, 20: 0 1142--1153, 2022

    2022

  2. [12]

    Prottrans: Towards cracking the language of life’s code through self-supervised deep learning and high performance computing

    Elnaggar, A., Heinzinger, M., et al. Prottrans: Towards cracking the language of life’s code through self-supervised deep learning and high performance computing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021 b . doi:10.1109/TPAMI.2021.3095381

    2021

  3. [13]

    Ankh: Optimized protein language model unlocks generalization across structure and function

    Elnaggar, A., Heinzinger, M., and Rost, B. Ankh: Optimized protein language model unlocks generalization across structure and function. bioRxiv, 2023. doi:10.1101/2023.10.17.562788

    2023 doi

  4. [14]

    J., Bordes, P., Gonzalez, L., et al

    Garau-Luis, J. J., Bordes, P., Gonzalez, L., et al. Ceviche: A multimodal deep learning framework for biological sequence analysis. Bioinformatics, 40 0 (2): 0 456--468, 2024 a . doi:10.1093/bioinformatics/btaa123

    2024 doi

  5. [15]

    J., Bordes, P., Gonzalez, L., et al

    Garau-Luis, J. J., Bordes, P., Gonzalez, L., et al. Multi-modal transfer learning between biological foundation models. bioRxiv, 2024 b . doi:10.1101/2024.06.14.123456

    2024 doi

  6. [16]

    Huang, Y. et al. A comprehensive investigation of multimodal deep learning fusion strategies. Artificial Intelligence Review, 57 0 (3): 0 123--145, 2024. doi:10.1007/s10462-024-10984-z

    2024 doi

  7. [17]

    Attention-based deep multiple instance learning

    Ilse, M., Tomczak, J., and Welling, M. Attention-based deep multiple instance learning. In International conference on machine learning, pp.\ 2127--2136. PMLR, 2018

    2018

  8. [18]

    Ji, Y., Zhou, Z., Liu, H., and Davuluri, R. V. Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language in genome. Bioinformatics, 37 0 (15): 0 2112--2120, 2021

    2021

  9. [19]

    W., Kladwang, W., Wayment-Steele, H

    Leppek, K., Byeon, G. W., Kladwang, W., Wayment-Steele, H. K., Kerr, C. H., Xu, A. F., Kim, D. S., Topkar, V. V., Choe, C., Rothschild, D., et al. Combinatorial optimization of mrna structure, stability, and translation for rna-based therapeutics. Nature communications, 13 0 (...

    2022

  10. [20]

    Utr-lm: Pretrained language models for untranslated region function prediction

    Li, M., Wu, Y., Zhang, S., et al. Utr-lm: Pretrained language models for untranslated region function prediction. bioRxiv, 2023 a . doi:10.1101/2023.04.26.538444

    2023 doi

  11. [21]

    Codonbert: Large language models for mrna design and optimization

    Li, S., Moayedpour, S., Li, R., Bailey, M., Riahi, S., Kogler-Anele, L., Miladi, M., Miner, J., Zheng, D., Wang, J., et al. Codonbert: Large language models for mrna design and optimization. bioRxiv, pp.\ 2023--09, 2023 b

    2023

  12. [23]

    Language models of protein sequences at the scale of evolution enable accurate structure prediction

    Lin, Z., Akin, H., Rao, R., et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022 b . doi:10.1101/2022.07.20.500902

    2022 doi

  13. [24]

    Evolutionary-scale prediction of atomic-level protein structure with a language model

    Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379 0 (6637): 0 1123--1130, 2023. doi:10.1126/science.ade2574. URL h...

    2023 doi

  14. [25]

    Liu, H., Ji, Y., Zhou, Z., and Davuluri, R. V. Dnabert-2: Efficient foundation model for dna language in genome. bioRxiv, 2023. doi:10.1101/2023.06.02.543344

    2023 doi

  15. [26]

    Life-code: Central dogma modeling with multi-omics sequence unification

    Liu, Z., Li, S., Chen, Z., et al. Life-code: Central dogma modeling with multi-omics sequence unification. arXiv preprint arXiv:2502.07299, 2025

    2025

  16. [27]

    R., et al

    Madani, A., Krause, B., Greene, E. R., et al. Large language models generate functional protein sequences across diverse families. Nature Biotechnology, 2023. doi:10.1038/s41587-023-01772-9

    2023 doi

  17. [28]

    Hyenadna: Learning at a million tokens per sample with dna language models

    Nguyen, E., Poli, M., Durrant, M., et al. Hyenadna: Learning at a million tokens per sample with dna language models. bioRxiv, 2023. doi:10.1101/2023.12.22.573141

    2023 doi

  18. [29]

    G., et al

    Nguyen, E., Poli, M., Durrant, M. G., et al. Sequence modeling and design from molecular to genome scale with evo. Science, 386 0 (6669): 0 746--752, 2024. doi:10.1126/science.ado9336

    2024 doi

  19. [30]

    Notin, P. et al. Tranception: protein fitness prediction with autoregressive transformers and retrieval. arXiv preprint arXiv:2205.13760, 2023

    2023 arXiv

  20. [31]

    D., Combs, S., Mansi, T., Scheer, J., and Liao, R

    Prakash, M., Moskalev, A., Jr., P. D., Combs, S., Mansi, T., Scheer, J., and Liao, R. Bridging biomolecular modalities for knowledge transfer in bio-language models. In Neurips 2024 Workshop Foundation Models for Science: Progress, Opportunities, and Challenges, 2024. URL http...

    2024

  21. [32]

    Splicebert: Transformer-based pretraining for rna splicing prediction

    Wang, Y., Zhang, H., Lin, H., et al. Splicebert: Transformer-based pretraining for rna splicing prediction. bioRxiv, 2023. doi:10.1101/2023.05.22.541751

    2023 doi

  22. [33]

    Wint, R., Salamov, A., and Grigoriev, I. V. Kingdom-wide analysis of fungal protein-coding and trna genes reveals conserved patterns of adaptive evolution. Molecular biology and evolution, 39 0 (2): 0 msab372, 2022

    2022

  23. [34]

    Helm: Hierarchical embeddings for language modeling of mrna sequences

    Yazdani, A., Roy, S., Wang, F., et al. Helm: Hierarchical embeddings for language modeling of mrna sequences. bioRxiv, 2024. doi:10.1101/2024.01.05.574223

    2024 doi

  24. [35]

    Codonbert: A pre-trained language model for mrna codon optimization

    Zhang, M., Liu, X., Yang, J., et al. Codonbert: A pre-trained language model for mrna codon optimization. bioRxiv, 2023. doi:10.1101/2023.02.15.528644

    2023 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.