Leveraging Natural Language Processing to Unravel the Mystery of Life: A Review of NLP Approaches in Genomics, Transcriptomics, and Proteomics

David Burstein; Ella Rannon

REVIEW 2 major objections 4 minor 7 references

Leveraging Natural Language Processing to Unravel the Mystery of Life: A Review of NLP Approaches in Genomics, Transcriptomics, and Proteomics

T0 review · 2 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This review argues that NLP methods, applied to DNA, RNA, protein, and genome sequences as if they were languages, now provide a working framework for biological sequence analysis and that the choice of tokenization and architecture…

desk verdict A broad, current, and useful review for newcomers; the core value is organizational, and with a few fixes (20^k not 20!, an explicit selection protocol, and one repaired sentence) it is a solid citable survey. read the letter →

arxiv 2506.02212 v2 pith:CDIR7H5Q submitted 2025-06-02 cs.CL cs.AIq-bio.GN

Ella Rannon , David Burstein This is my paper · ORCID

classification cs.CLcs.AIq-bio.GN

keywords naturallanguageprocessingbiologicalmodelsgenomicstranscriptomicsproteomicstokenizationtransformerarchitecturesHyena

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This review attempts to establish that the techniques of natural language processing—treating a biological sequence as a sentence whose tokens are nucleotides, amino acids, or genes—now form a working framework for genomics, transcriptomics, and proteomics. It surveys models from word2vec-style embeddings through LSTMs and transformers to Hyena-based architectures, organizing them by biological language, tokenization choice, and architecture. The authors argue that the analogy is more than metaphor: context learned from sequence data captures function, structure, and evolutionary signal, and the choice of tokenizer and architecture materially changes what a model can learn. A reader would care because the review offers a map for choosing among dozens of models and identifies where the field's claims of success are and are not yet supported.

What carries the argument

The load-bearing object is the "biological language" analogy: DNA, RNA, and protein sequences are treated as sentences, their characters or k-mers as words, and genomes as documents whose gene neighborhoods supply context. The review organizes the field by three interacting choices—which biological language is the input, how it is tokenized (single characters, k-mers, or sub-word units), and which architecture learns from those tokens. These choices carry the argument because the paper's comparative claims, such as transformer dominance in Figure 4 and the underuse of genomic-context models, are derived from counting and grouping models along these axes.

What would settle it

Conduct a systematic literature search for biological language models published through a fixed date, with an explicit inclusion protocol, and compare the resulting counts to Figure 4; if the proportion of transformer-based models, or the number of genomic-context models, differs materially from the review's tallies, the survey's descriptive conclusions would need revision. Additionally, a head-to-head benchmark that trains character- and sub-word-tokenized transformers of equal size on the same DNA tasks would settle the paper's suggestion that tokenization choice, not architecture, drives much of the performance gap.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that biological sequences behave enough like natural language that NLP models trained on them recover biologically meaningful organization: amino-acid and nucleotide embeddings encode structural and functional properties, genomic context predicts gene function, and long-context architectures extend the same logic to whole genomes. The review's synthesis is that the field has moved through three generations—static embeddings, recurrent context models, and attention-based transformers now dominant—and that a fourth generation of sub-quadratic architectures (Hyena and similar) is emerging specifically to overcome the length limits that block full-genome modeling. It also concludes that tokenization is a primary design decision: character-level tokens preserve position-specific signal, k-mers are simple but leak context between overlapping windows, and sub-word methods such as BPE handle sparse vocabularies at the cost of variable-length units. The paper presents this as a structured survey rather than a new experiment, with tables and a Sankey diagram as evidence for the current distribution of models across architectures and tokenizers.

Load-bearing premise

The review assumes without a stated selection protocol that the models it covers are a representative sample of the field, so its descriptive conclusions about which architectures and tokenizers dominate could fail if the sample is biased.

Editorial extensions

If this is right

If the survey's map is accurate, transformer-based foundation models are currently the default choice for most biological sequence tasks, and newer sub-quadratic architectures are the main route to whole-genome context.
Sub-word tokenization such as BPE is likely to keep spreading in DNA and protein models because it mitigates the context-leakage and length problems of overlapping k-mers.
Genomic-context models, which read gene neighborhoods rather than single sequences, are an under-explored niche with demonstrated gains for prokaryotic gene function prediction.
Benchmarks like GUE, TAPE, and ProteinGym are becoming the field's shared measuring sticks, and expanding them to more organisms and genomic-level tasks is a stated priority.
Attention-map analysis of DNA and genome models is a concrete untapped direction, since similar analysis of protein models already yields structural and functional insights.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The review's prevalence counts, such as transformer dominance, depend on an implicit selection of models and could shift substantially if a formal systematic search included more recent preprints or non-English venues; a reproducible search protocol would turn its descriptive claims into testable ones.
The same language analogy suggests a testable extension: if genomic context is genuinely syntactic, then models trained on gene-order sentences should outperform sequence-only models on function prediction more broadly than the few published comparisons show.
The framing implies a transfer experiment the review does not run: comparing tokenizers head-to-head under identical architecture and data, rather than across different models, to isolate tokenization effects on tasks such as variant effect prediction.
Codon-aware protein models are a logical next step flagged only in passing; the review's own discussion of codon usage bias implies that protein language models ignoring codons will be surpassed on tasks where translational kinetics matter.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

Short version: this is a competent, current review of NLP methods for DNA, RNA, protein, and genome sequences, and it will serve well as an orientation piece for newcomers. It has no new method or result, so don’t expect a research contribution. But it is more than a list: the tokenization discussion (overlapping k-mer leakage, sub-word alternatives) and the three architecture tables are genuinely useful, and the coverage is impressively up to date, including Evo 2 and ESM3.

The strongest section is the tokenization part, where the authors clearly explain why overlapping k-mers leak masked-token information and why sub-word methods like BPE change the game. The three tables are well organized, and the Sankey diagram is a nice visual summary, even if the counts behind it are questionable. Descriptions of individual models match the cited papers well; I did not find any mischaracterization of a major model.

Soft spots, in order. First, the combinatorial error in the Protein Language Models section: “20! possible tokens of length k” should be 20^k (and 4^k for DNA). The qualitative point about sparsity survives, but the arithmetic is plainly wrong and needs a fix. Second, the review never states its search strategy or inclusion criteria. That matters because Figure 4 makes quantitative claims about how many models use transformers versus other architectures. Without knowing how models were selected, those proportions are not reproducible; the discussion’s claim that transformers are the most widely used architecture may be true, but the evidence as presented is a convenience sample. Third, there is a broken sentence in the Discussion about benchmark availability — “provides a set of five semi-supervised learning tasks” — that obscures which benchmark is being described. Minor, but it should be cleaned up.

The self-citations to the Burstein group’s genomic-context work are noticeable but not a problem; they are placed in a legitimate research niche and the survey’s central claims do not depend on them. The absence of any stated inclusion protocol is a bigger issue than the self-citations.

Bottom line: for a reader entering the field, this review will save time and give a fair map. It deserves a serious referee, and with a corrected factorial, an added methodology paragraph, and a repaired sentence, it would be a solid, citable survey. I would not desk-reject it.

Referee Report

2 major / 4 minor

Summary. This manuscript is a narrative review of natural language processing methods applied to biological sequence data across genomics, transcriptomics, and proteomics. It outlines the methodological progression from word2vec, fastText, and GloVe through BiLSTM/ELMo-style models to transformer- and Hyena-based architectures, then surveys DNA/RNA, protein, and whole-genome language models in three large tables. It also discusses tokenization strategies, benchmarks, and future directions, and presents a Sankey diagram (Figure 4) summarizing the model landscape. The stated central claim is that NLP techniques can be productively transferred to DNA, RNA, protein, and genome sequences, with transformer-based models currently playing the dominant role.

Significance. The review is a potentially useful entry point to a fast-moving literature: the taxonomy of tokenization strategies, the three model tables, and the Sankey diagram organize a large body of work, and the coverage of recent models such as DNABERT-2, Nucleotide Transformer, ESM-2/3, Evo, HyenaDNA, gLM, gLM2, and PlasGO is current. The qualitative thesis—that sequence-level language models capture biologically meaningful structure—is well supported by the cited literature and is not overstated. However, the paper's value as a reproducible field map is limited by the absence of a stated model-selection methodology, and one supporting quantitative statement contains a mathematical error; these issues need to be addressed before the descriptive claims about prevalence and tokenization tradeoffs can be taken at face value.

major comments (2)

[Protein Language Models] In the limitations paragraph for pLMs, the manuscript states that proteins have '20! possible tokens of length k, instead of 4! for DNA sequences.' The number of distinct k-mers over an alphabet of size A is A^k, so the correct counts are 20^k and 4^k. The factorial expression is not merely a typo: it dramatically overstates vocabulary growth and is numerically wrong. The subsequent qualitative conclusion that protein k-mer vocabularies are sparser than DNA k-mer vocabularies still holds, but the arithmetic should be corrected and the sentence rephrased.
[Introduction; Figure 4; Discussion] The review does not state how the models in Tables 1–3 were chosen. There is no search strategy, database list, date range, or inclusion/exclusion criteria, even though the Introduction defines the paper as a survey of 'studies and models applying NLP approaches.' This omission becomes load-bearing when the paper makes quantitative claims: Figure 4 reports counts such as 47 transformer models and 38 single-character-tokenized models, and the Discussion states that 'standard transformers currently remain the most widely used architecture in genomic NLP studies.' Without a defined model universe, these proportions are not reproducible and the prevalence conclusion may reflect selection choices rather than field-wide usage. Please add a methods paragraph describing the selection process and either make the quantitative claims explicitly illustrative or substantiate them with a reproducible census.

minor comments (4)

[Discussion] The Discussion contains a fused sentence that obscures a benchmark reference: 'The abundance of foundation biological LMs35–41,46–48,60 provides a set of five semi-supervised learning tasks focused on protein structure and fitness; ProteinGym118 includes...' The five-task benchmark being referred to is TAPE, but it is not named and the grammar makes it sound as if the foundation models themselves provide the tasks. Please rewrite this passage and cite TAPE (ref. 38).
[Figure 1C caption] The caption says 'The final encoder's attention matrices are integrated into the decoder stack via the encoder-decoder attention layers.' In the original transformer architecture it is the encoder output representations, not the attention matrices, that are passed to the decoder's cross-attention as keys and values. This should be reworded for technical accuracy.
[Table 3] The row for the SARS-CoV-2 mutation model is labeled 'NA (Harari, 2024)' in the Name column; consider giving the model a name or using a consistent placeholder, and similarly for other unnamed models.
[Figure 4] The Sankey counts are not cross-linked to the tables, so a reader cannot verify, for example, the protein count of 39 against Table 2. Adding per-table totals or a supplementary model list would improve verifiability.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the review's central claims rest on external, independently published benchmarks, and its self-citations are descriptive rather than load-bearing.

full rationale

This paper is a narrative review with no formal derivation chain; it does not fit parameters, define terms in terms of outputs, or derive predictions from inputs. The survey's central claim, that NLP methods can be productively applied to biological sequences, is supported by externally published models and benchmarks (DNABERT, ESM, TAPE, GUE, ProteinGym) rather than by any equation in the paper. The self-citations (refs 137, 138, 142, from the Burstein group) are used to describe the authors' own genomic-context models, but the review's taxonomy and conclusions do not depend on those specific papers being correct; removing them would not change the classification of the other surveyed models. The quantitative landscape statements, including the Figure 4 Sankey counts and the Discussion claim that 'standard transformers currently remain the most widely used architecture in genomic NLP studies,' rest on an undocumented selection of models, which is a reproducibility and selection-bias concern, not circularity, because the counts are not constructed to equal any input or fitted parameter. The arithmetic error (20! and 4! instead of 20^k and 4^k) and the incomplete benchmark sentence in the Discussion are correctness and editorial issues, not circular steps. No instance can be exhibited where an equation, prediction, or taxonomy entry reduces by construction to an input or to a self-citation chain. Under the required standard of quoting a specific reduction, no circularity is found.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

A review introduces no fitted parameters or invented entities. It relies on the domain assumptions above and on the accuracy of its cited literature.

assumptions (2)

domain assumption The analogy between natural language and biological sequences is meaningful enough that NLP methods transfer to DNA, RNA, and proteins.
Invoked in the Introduction and in Figure 2 to justify the entire survey.
domain assumption Genomic context, i.e., the set of genes near a given gene, provides information about gene function.
Cited via references 29-31 and used to motivate genome-level language models in the Genome Language Models section.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Leveraging Natural Language Processing to Unravel the Mystery of Life: A Review of NLP Approaches in Genomics, Transcriptomics, and Proteomics." pith.science (2026). https://pith.science/paper/CDIR7H5Q

@misc{pith2026250602212,
  author       = {Pith},
  title        = {Pith review of: Leveraging Natural Language Processing to Unravel the Mystery of Life: A Review of NLP Approaches in Genomics, Transcriptomics, and Proteomics},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CDIR7H5Q}},
  note         = {Machine review of arXiv:2506.02212}
}

read the original abstract

Natural Language Processing (NLP) has transformed various fields beyond linguistics by applying techniques originally developed for human language to the analysis of biological sequences. This review explores the application of NLP methods to biological sequence data, focusing on genomics, transcriptomics, and proteomics. We examine how various NLP methods, from classic approaches like word2vec to advanced models employing transformers and hyena operators, are being adapted to analyze DNA, RNA, protein sequences, and entire genomes. The review also examines tokenization strategies and model architectures, evaluating their strengths, limitations, and suitability for different biological tasks. We further cover recent advances in NLP applications for biological data, such as structure prediction, gene expression, and evolutionary analysis, highlighting the potential of these methods for extracting meaningful insights from large-scale genomic data. As language models continue to advance, their integration into bioinformatics holds immense promise for advancing our understanding of biological processes in all domains of life.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

7 extracted references · 1 canonical work pages

[4]

Calijorne Soares, M. A. & Parreiras, F. S. A literature review on question answering techniques, paradigms and systems. J. King Saud Univ. - Comput. Inf. Sci. 32, 635–646 (2020). 5. Hambarde, K. A. & Proença, H. Information Retrieval: Recent Advances and Beyond. IEEE Access 11, 76581–76604 (2023). 6. Awasthi, I., Gupta, K., Bhogal, P. S., Anand, S. S. & S...

work page arXiv 2020
[26]

Islam, S. et al. A comprehensive survey on applications of transformers for deep learning tasks. Expert Syst. Appl. 241, 122666 (2024). 27. Poli, M. et al. Hyena Hierarchy: Towards Larger Convolutional Language Models. in Proceedings of the 40th International Conference on Machine Learning 28043–28078 (PMLR, 2023). 28. Yu, L. et al. Grammar of protein dom...

arXiv 2024
[51]

& Mofrad, M

Asgari, E. & Mofrad, M. R. K. Continuous Distributed Representation of Biological Sequences for Deep Proteomics and Genomics. PLOS ONE 10, e0141287 (2015). 52. Indyk, P. & Motwani, R. Approximate nearest neighbors: towards removing the curse of dimensionality. in Proceedings of the thirtieth annual ACM symposium on Theory of computing 604–613 (Association...

arXiv 2015
[71]

Zhou, Z. et al. DNABERT-S: Pioneering Species Differentiation with Species-Aware DNA Embeddings. ArXiv arXiv:2402.08777v3 (2024). 72. Zvyagin, M. et al. GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics. Int. J. High Perform. Comput. Appl. (2023) doi:10.1177/10943420231201154. 73. Zhang, L., Qin, X., Liu, M., Liu, G. & Ren, Y. ...

arXiv 2024
[93]

& Strodthoff, N

Vielhaben, J., Wenzel, M., Samek, W. & Strodthoff, N. USMPep: universal sequence models for major histocompatibility complex binding affinity prediction. BMC Bioinformatics 21, 279 (2020). 94. Zhou, G. et al. Mutation effect estimation on protein–protein interactions using deep contextualized representation learning. NAR Genomics Bioinforma. 2, lqaa015 (2...

arXiv 2020
[117]

Wu, R. et al. High-resolution de novo structure prediction from primary sequence. 2022.07.21.500999 Preprint at https://doi.org/10.1101/2022.07.21.500999 (2022). 118. Notin, P. et al. Tranception: Protein Fitness Prediction with Autoregressive Transformers and Inference-time Retrieval. in Proceedings of the 39th International Conference on Machine Learnin...

arXiv doi:10.48550/arxiv.2004.03497 2022
[139]

Cornman, A. et al. The OMG dataset: An Open MetaGenomic corpus for mixed-modality genomic language modeling. 2024.08.14.607850 Preprint at https://doi.org/10.1101/2024.08.14.607850 (2024). 140. Ji, Y. et al. PlasGO: enhancing GO-based function prediction for plasmid-encoded proteins based on genetic structure. GigaScience 13, giae104 (2024). 141. Naidenov...

arXiv 2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [4]

Calijorne Soares, M. A. & Parreiras, F. S. A literature review on question answering techniques, paradigms and systems. J. King Saud Univ. - Comput. Inf. Sci. 32, 635–646 (2020). 5. Hambarde, K. A. & Proença, H. Information Retrieval: Recent Advances and Beyond. IEEE Access 11, 76581–76604 (2023). 6. Awasthi, I., Gupta, K., Bhogal, P. S., Anand, S. S. & S...

work page arXiv 2020

[2] [26]

Islam, S. et al. A comprehensive survey on applications of transformers for deep learning tasks. Expert Syst. Appl. 241, 122666 (2024). 27. Poli, M. et al. Hyena Hierarchy: Towards Larger Convolutional Language Models. in Proceedings of the 40th International Conference on Machine Learning 28043–28078 (PMLR, 2023). 28. Yu, L. et al. Grammar of protein dom...

arXiv 2024

[3] [51]

& Mofrad, M

Asgari, E. & Mofrad, M. R. K. Continuous Distributed Representation of Biological Sequences for Deep Proteomics and Genomics. PLOS ONE 10, e0141287 (2015). 52. Indyk, P. & Motwani, R. Approximate nearest neighbors: towards removing the curse of dimensionality. in Proceedings of the thirtieth annual ACM symposium on Theory of computing 604–613 (Association...

arXiv 2015

[4] [71]

Zhou, Z. et al. DNABERT-S: Pioneering Species Differentiation with Species-Aware DNA Embeddings. ArXiv arXiv:2402.08777v3 (2024). 72. Zvyagin, M. et al. GenSLMs: Genome-scale language models reveal SARS-CoV-2 evolutionary dynamics. Int. J. High Perform. Comput. Appl. (2023) doi:10.1177/10943420231201154. 73. Zhang, L., Qin, X., Liu, M., Liu, G. & Ren, Y. ...

arXiv 2024

[5] [93]

& Strodthoff, N

Vielhaben, J., Wenzel, M., Samek, W. & Strodthoff, N. USMPep: universal sequence models for major histocompatibility complex binding affinity prediction. BMC Bioinformatics 21, 279 (2020). 94. Zhou, G. et al. Mutation effect estimation on protein–protein interactions using deep contextualized representation learning. NAR Genomics Bioinforma. 2, lqaa015 (2...

arXiv 2020

[6] [117]

Wu, R. et al. High-resolution de novo structure prediction from primary sequence. 2022.07.21.500999 Preprint at https://doi.org/10.1101/2022.07.21.500999 (2022). 118. Notin, P. et al. Tranception: Protein Fitness Prediction with Autoregressive Transformers and Inference-time Retrieval. in Proceedings of the 39th International Conference on Machine Learnin...

arXiv doi:10.48550/arxiv.2004.03497 2022

[7] [139]

Cornman, A. et al. The OMG dataset: An Open MetaGenomic corpus for mixed-modality genomic language modeling. 2024.08.14.607850 Preprint at https://doi.org/10.1101/2024.08.14.607850 (2024). 140. Ji, Y. et al. PlasGO: enhancing GO-based function prediction for plasmid-encoded proteins based on genetic structure. GigaScience 13, giae104 (2024). 141. Naidenov...

arXiv 2024