REVIEW 3 major objections 5 minor 30 references
Linguistic Laws Meet Protein Sequences: A Comparative Analysis of Subword Tokenization Methods
T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read This paper argues that standard NLP subword tokenizers—BPE, WordPiece, and SentencePiece—do not capture the biologically meaningful subunits of protein sequences, as shown by falling domain-boundary alignment and deviations from…
desk verdict A useful descriptive map of tokenizer behavior on proteins, but the linguistic-law conclusions are tokenizer artifacts, not evidence about protein organization. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The argument runs on a comparative diagnostic built from tokenizer output. Each tokenizer is trained on 15 million protein sequences at five vocabulary sizes, and its segmentations are scored on identity (shared tokens, token length, fertility), context (contextual exponence, the number of distinct neighbors a token appears with in a five-token window), and biology (alignment of token boundaries with known protein domain annotations). Four linguistic laws then serve as probes: Zipf's law checks the rank–frequency shape, Brevity law checks whether frequent tokens are short, Heaps' law checks how vocabulary grows with data, and Menzerath's law checks whether average token length declines as sequences lengthen. Because the same BPE pipeline is run on English text as a baseline, deviations that appear only for proteins can be attributed to the data rather than to the tokenizer family.
What would settle it
Tokenize the same protein test set with a vocabulary built from known functional units such as conserved motifs, then recompute the Menzerath-law curve and domain-boundary hit rate; if this vocabulary still shows flat or rising average token length with sequence length and low boundary hits, the paper's claim would be undermined, because the deviation would then be intrinsic to protein sequences rather than an artifact of general-purpose tokenizers.
Extended reading notes
Core claim
The paper's central claim is that how these tokenizers segment protein sequences says more about the tokenizers than about proteins. Although the tokenizers behave differently—BPE produces more contextually specialized tokens and slightly better domain-boundary alignment at small vocabularies, SentencePiece encodes more efficiently with lower fertility, and WordPiece falls between them—all three lose the ability to respect domain boundaries as vocabulary size increases. That decline, together with hit rates that stay low even at the smallest vocabulary, is presented as evidence that none of the tokenizers is finding the units a protein's biology actually uses. The linguistic-law results reinforce the point: token frequency and length mostly follow Zipf's and Brevity laws, but average token length does not fall as sequences lengthen as Menzerath's law predicts, unlike English text run through the same BPE pipeline. The conclusion is that protein sequences may obey different organizational principles from natural language, and that standard NLP tokenization should be reworked or replaced for protein applications.
Load-bearing premise
The load-bearing premise is that the token frequencies and token lengths produced by these three subword tokenizers give a reliable window into the underlying organization of proteins, so that a deviation from a linguistic law counts as evidence about proteins rather than about the tokenizers.
Editorial extensions
If this is right
- Larger vocabularies for these tokenizers systematically reduce domain-boundary preservation, so protein applications should favor smaller vocabularies or a different segmentation principle.
- BPE's marginal boundary advantage at small vocabularies indicates that frequency-driven merging is a reasonable starting point for protein units, but it is not sufficient to respect functional boundaries.
- Partial compliance with Zipf's and Brevity laws implies that the frequency structure of protein tokens is not radically different from text, while the Menzerath deviation isolates where the analogy fails.
- Protein language models that use these tokenizers inherit their boundary errors, making tokenization choice a bound on structure- and function-prediction performance rather than a neutral preprocessing step.
Reading between the lines
- If the same proteins were tokenized with biologically defined units such as known functional motifs, the Menzerath deviation might disappear; that experiment would separate a genuine biological principle from a tokenizer artifact.
- The shared-token results imply that single-vocabulary-size comparisons of protein tokenizers may not generalize, so future evaluations should report curves across vocabulary sizes.
- The same four-law battery could serve as a cheap screening test for proposed protein-specific tokenizers before training expensive downstream models.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper compares three subword tokenizers (BPE, WordPiece, SentencePiece) on protein sequences from UniRef50 at vocabulary sizes 400–6400, with a BPE-on-English baseline from WikiText. It reports tokenizer overlap, token length and fertility, contextual exponence, PROSITE domain-boundary hits, and fits to Zipf, Brevity, Heaps, and Menzerath laws. The descriptive results are plausible, but the paper's central interpretive claim—that deviations from Menzerath's law and poor domain-boundary preservation show protein sequences follow organizational principles distinct from natural language—rests on tokenizer output with no chance or shuffled-sequence baselines.
Significance. If the central claim were established, the results would justify developing protein-specific tokenizers and would connect linguistic-law methodology to protein biology. The paper's descriptive comparison is a useful empirical contribution, especially the finding that BPE's apparent domain-boundary advantage tracks token length, and the authors share code and data. However, the main biological conclusion is not yet supported because the observed distributional properties are largely determined by the tokenization algorithms rather than by protein sequence organization.
major comments (3)
- [Section IV-H and Section V] The Menzerath-law conclusion is not supported by the evidence presented. Figure 9 plots the average token length produced by BPE, WordPiece, or SentencePiece against protein sequence length, but token lengths are outputs of the frequency- or likelihood-based merge and selection objectives of the tokenizers, not independently defined biological constituents. For any tokenizer whose per-token length distribution is roughly stationary, average token length should be nearly flat as sequence length grows, so the observed deviation from Menzerath's law is expected by construction. To claim that proteins 'may follow distinct distribution patterns from natural language,' the authors need a control such as tokenizing shuffled or permuted sequences with preserved amino-acid composition, or a random-segmentation baseline, and must show that the protein result differs from that control. The citation of Shahzad et al. [14], which analyzed protein domains, does not transfer to tokenizer output units.
- [Section IV-D] The domain-boundary hit percentages have no chance baseline. A hit requires both the start and the end of a domain to coincide with token boundaries, so even a tokenizer that places boundaries uniformly at random would produce a positive hit rate, and the paper does not report what that rate is. The authors themselves note that BPE's better performance may be due to its shorter average token length, and the correlation between token length and hit rate holds for every tokenizer. Without a length-matched or random-boundary control, the claim that the tokenizers 'do not effectively capture the true linguistic subunits of protein sequences' is not established.
- [Section IV-E, Fig. 5] The Zipf-slope comparisons are reported as point values without confidence intervals, error bars, or statistical tests, and without a shuffled-sequence control. Because BPE, WordPiece, and SentencePiece construct vocabularies by different objectives, differences in rank-frequency slopes are expected even on random amino-acid sequences; the current analysis does not show that any slope difference is attributable to protein-specific organization. The authors should add bootstrap intervals and a randomized-sequence condition before interpreting slope deviations as evidence about the 'language of proteins.'
minor comments (5)
- [Section III] The manuscript should report the version or date of UniRef50 and WikiText, the random seed used for subsampling 15 million training sequences, and the number of independent training runs; none of these are specified.
- [Section II and Section IV] The experiments use SentencePiece with the Unigram model, but Section II describes SentencePiece as applying BPE or Unigram; the experimental setup should state this explicitly and give the hyperparameters used.
- [Fig. 9] The Menzerath-law panels use different x-axis ranges for protein (0–3000) and English (0–1000) sequences, which makes the visual comparison of trends and outliers difficult; the figures should use comparable scales or a normalized length axis.
- [Fig. 8] The Heap's-law plots include fitted K and beta values only in the captions; since the fitted curves are not clearly distinguished in the plots, the reader cannot assess the quality of the fits. Please draw the fitted curves and report fitting errors.
- [Section IV-D] The PROSITE evaluation reports 4646 domains in 3377 test sequences; the authors should clarify whether overlapping or nested domains are counted and how ambiguous domain boundaries are handled.
Circularity Check
Menzerath/Zipf deviations are interpreted as evidence about protein organization, but the measured token length and frequency distributions are outputs of the tokenizers themselves; the biological conclusion is a relabeling of tokenizer artifacts, while the direct tokenizer comparisons remain independent.
-
self definitional
[Section IV-H (Menzerath's Law) and Section V (Discussion and Conclusion), Fig. 9]
"Menzerath’s law, also known as Menzerath–Altmann law, states that larger linguistic constructs tend to have shorter constituents [30]. In the context of sequence tokenization, this implies that as the sequence length increases, the average token length should decrease. However, our findings show a more complex behavior for protein tokenizers, with none of the tokenizers fully following this expected pattern."
The 'constituents' whose lengths are measured are the tokens emitted by BPE, WordPiece, or SentencePiece. Token length and token frequency are not independent biological units; they are generated by the tokenizer's merge or loss objective. The observed sequence-length vs token-length curve is therefore a property of the segmentation algorithm. Reading the deviation from Menzerath's law as evidence that 'protein sequences may follow distinct distribution patterns from natural language' (Section V) treats the tokenizer output as if it were the protein language itself.
full rationale
The core comparative evaluation in the paper is self-contained: fertility, contextual exponence, shared-token overlap, and domain-boundary alignment are measured against external benchmarks (PROSITE domains) or are direct tokenizer behaviors, and the tokenizer-to-tokenizer comparisons are not circular. The score is raised by one load-bearing interpretive step: the claim that protein sequences deviate from Menzerath's law and hence 'may follow distinct organizational principles from natural languages' is derived from token-length statistics that the tokenizers themselves produce. Since no control separates tokenizer-induced segmentation statistics from protein-specific organization, this portion of the conclusion is a self-definitional relabeling of tokenizer output. This is not a self-citation issue; the citations to Shahzad et al. and Semple et al. are external background, and the paper does not rely on an author-uniqueness theorem. The circularity is partial: the descriptive claims about how BPE, WordPiece, and SentencePiece segment proteins, and the domain-boundary analysis, stand on their own.
Assumptions & free parameters
free parameters (1)
- Heap's law constants K and beta per vocabulary size =
K = 35-73, beta = 0.3-0.4 (Fig. 8)
assumptions (4)
- domain assumption Treating proteins as text and their tokenization as linguistic units is a meaningful way to test linguistic laws.
- domain assumption The UniRef50 validation/test split (11,957 sequences) is representative for measuring tokenization behavior.
- domain assumption PROSITE domain boundaries are the correct ground truth for domain boundary preservation.
- domain assumption The BPE English tokenizer provides a fair natural-language baseline.
Cite this review
Pith. "Pith review of Linguistic Laws Meet Protein Sequences: A Comparative Analysis of Subword Tokenization Methods." pith.science (2026). https://pith.science/paper/FDJSPYAU
@misc{pith2026241117669,
author = {Pith},
title = {Pith review of: Linguistic Laws Meet Protein Sequences: A Comparative Analysis of Subword Tokenization Methods},
year = {2026},
howpublished = {\url{https://pith.science/paper/FDJSPYAU}},
note = {Machine review of arXiv:2411.17669}
}
read the original abstract
Tokenization is a crucial step in processing protein sequences for machine learning models, as proteins are complex sequences of amino acids that require meaningful segmentation to capture their functional and structural properties. However, existing subword tokenization methods, developed primarily for human language, may be inadequate for protein sequences, which have unique patterns and constraints. This study evaluates three prominent tokenization approaches, Byte-Pair Encoding (BPE), WordPiece, and SentencePiece, across varying vocabulary sizes (400-6400), analyzing their effectiveness in protein sequence representation, domain boundary preservation, and adherence to established linguistic laws. Our comprehensive analysis reveals distinct behavioral patterns among these tokenizers, with vocabulary size significantly influencing their performance. BPE demonstrates better contextual specialization and marginally better domain boundary preservation at smaller vocabularies, while SentencePiece achieves better encoding efficiency, leading to lower fertility scores. WordPiece offers a balanced compromise between these characteristics. However, all tokenizers show limitations in maintaining protein domain integrity, particularly as vocabulary size increases. Analysis of linguistic law adherence shows partial compliance with Zipf's and Brevity laws but notable deviations from Menzerath's law, suggesting that protein sequences may follow distinct organizational principles from natural languages. These findings highlight the limitations of applying traditional NLP tokenization methods to protein sequences and emphasize the need for developing specialized tokenization strategies that better account for the unique characteristics of proteins.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[14]
The organization of domains in proteins obeys menzerath-altmann’s law of lan guage,
K. Shahzad, J. E. Mittenthal, and G. Caetano-Anollés, “ The organization of domains in proteins obeys menzerath-altmann’s law of lan guage,” BMC systems biology , vol. 9, pp. 1–13, 2015
work page 2015
-
[1]
Transforming the language of life: transformer neural net works for protein prediction tasks,
A. Nambiar, M. Heflin, S. Liu, S. Maslov, M. Hopkins, and A. Ritz, “Transforming the language of life: transformer neural net works for protein prediction tasks,” in Proceedings of the 11th ACM international conference on bioinformatics, computational biology and h ealth infor- matics, pp. 1–8, 2020
work page 2020
-
[2]
A. Rives, J. Meier, T. Sercu, S. Goyal, Z. Lin, J. Liu, D. Gu o, M. Ott, C. L. Zitnick, J. Ma, et al. , “Biological structure and function emerge from scaling unsupervised learning to 250 million protein s equences,” Proceedings of the National Academy of Sciences , vol. 118, no. 15, p. e2016239118, 2021
work page 2021
-
[3]
Prottrans: Toward understanding the language of life through self-supervise d learning,
A. Elnaggar, M. Heinzinger, C. Dallago, G. Rehawi, Y . Wan g, L. Jones, T. Gibbs, T. Feher, C. Angerer, M. Steinegger, et al., “Prottrans: Toward understanding the language of life through self-supervise d learning,” IEEE transactions on pattern analysis and machine intellig ence, vol. 44, no. 10, pp. 7112–7127, 2021
work page 2021
-
[4]
The language of prote ins: Nlp, machine learning & protein sequences,
D. Ofer, N. Brandes, and M. Linial, “The language of prote ins: Nlp, machine learning & protein sequences,” Computational and Structural Biotechnology Journal, vol. 19, pp. 1750–1758, 2021
work page 2021
-
[5]
Proteinbert: a universal deep-learning model of protein sequence and fun ction,
N. Brandes, D. Ofer, Y . Peleg, N. Rappoport, and M. Linial , “Proteinbert: a universal deep-learning model of protein sequence and fun ction,” Bioinformatics, vol. 38, no. 8, pp. 2102–2110, 2022
work page 2022
-
[6]
Evolutionary-scale prediction of atomic- level protein structure with a language model,
Z. Lin, H. Akin, R. Rao, B. Hie, Z. Zhu, W. Lu, N. Smetanin, R . V erkuil, O. Kabeli, Y . Shmueli, et al. , “Evolutionary-scale prediction of atomic- level protein structure with a language model,” Science, vol. 379, no. 6637, pp. 1123–1130, 2023
work page 2023
-
[7]
Ankh: Optimized protein languag e model unlocks general-purpose modelling,
A. Elnaggar, H. Essam, W. Salah-Eldin, W. Moustafa, M. El kerdawy, C. Rochereau, and B. Rost, “Ankh: Optimized protein languag e model unlocks general-purpose modelling,” arXiv preprint arXiv:2301.06568 , 2023
arXiv 2023
Show all 30 references
-
[8]
Pe ta: Eval- uating the impact of protein transfer learning with sub-wor d tokenization on downstream applications,
Y . Tan, M. Li, P . Tan, Z. Zhou, H. Y u, G. Fan, and L. Hong, “Pe ta: Eval- uating the impact of protein transfer learning with sub-wor d tokenization on downstream applications,” arXiv preprint arXiv:2310.17415 , 2023
2023 arXiv
-
[9]
Effect o f tokenization on transformers for biological sequences,
E. Dotan, G. Jaschek, T. Pupko, and Y . Belinkov, “Effect o f tokenization on transformers for biological sequences,” Bioinformatics, vol. 40, no. 4, p. btae196, 2024
2024
-
[10]
Protein langu age models meet reduced amino acid alphabets,
I. Ieremie, R. M. Ewing, and M. Niranjan, “Protein langu age models meet reduced amino acid alphabets,” Bioinformatics, vol. 40, no. 2, p. btae061, 2024
2024
-
[11]
Bertology meets biology: Interpreting attention in prote in language models,
J. Vig, A. Madani, L. R. V arshney, C. Xiong, R. Socher, an d N. F. Rajani, “Bertology meets biology: Interpreting attention in prote in language models,” arXiv preprint arXiv:2006.15222 , 2020
2006 arXiv
-
[12]
Transformer protein language models are unsupervised structure learne rs,
R. Rao, J. Meier, T. Sercu, S. Ovchinnikov, and A. Rives, “Transformer protein language models are unsupervised structure learne rs,” Biorxiv, pp. 2020–12, 2020
2020
-
[13]
Exploring data-driven chem ical smiles tokenization approaches to identify key protein–ligand bi nding moieties,
A. B. Temizer, G. Uludo ˘gan, R. Özçelik, T. Koulani, E. Ozkirimli, K. O. Ulgen, N. Karali, and A. Özgür, “Exploring data-driven chem ical smiles tokenization approaches to identify key protein–ligand bi nding moieties,” Molecular Informatics , vol. 43, no. 3, p. e202300249, 2024
2024
-
[15]
Ling uistic laws in biology,
S. Semple, R. Ferrer-i Cancho, and M. L. Gustison, “Ling uistic laws in biology,” Trends in Ecology & Evolution , vol. 37, no. 1, pp. 53–66, 2022
2022
-
[16]
Sapr ot: Protein language modeling with structure-aware vocabulary,
J. Su, C. Han, Y . Zhou, J. Shan, X. Zhou, and F. Y uan, “Sapr ot: Protein language modeling with structure-aware vocabulary,” bioRxiv, pp. 2023– 10, 2023
2023
-
[17]
Bilingual language model for pr otein sequence and structure,
M. Heinzinger, K. Weissenow, J. G. Sanchez, A. Henkel, M . Mirdita, M. Steinegger, and B. Rost, “Bilingual language model for pr otein sequence and structure,” bioRxiv, pp. 2023–07, 2023
2023
-
[18]
Fast and accurate protein structure search with foldseek,
M. V an Kempen, S. S. Kim, C. Tumescheit, M. Mirdita, J. Le e, C. L. Gilchrist, J. Söding, and M. Steinegger, “Fast and accurate protein structure search with foldseek,” Nature biotechnology , vol. 42, no. 2, pp. 243–246, 2024
2024
-
[19]
Neural machine tr anslation of rare words with subword units,
R. Sennrich, B. Haddow, and A. Birch, “Neural machine tr anslation of rare words with subword units,” in Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( V olume 1: Long Papers), pp. 1715–1725, 2016
2016
-
[20]
Google’s neural machine translation system: Bridging the gap between human and machine translation,
Y . Wu, M. Schuster, Z. Chen, Q. V . Le, M. Norouzi, W. Mache rey, M. Krikun, Y . Cao, Q. Gao, K. Macherey, et al. , “Google’s neural machine translation system: Bridging the gap between human and machine translation,” arXiv preprint arXiv:1609.08144 , 2016
2016 arXiv
-
[21]
Sentencepiece: A simple and language inde- pendent subword tokenizer and detokenizer for neural text p rocessing,
T. Kudo and J. Richardson, “Sentencepiece: A simple and language inde- pendent subword tokenizer and detokenizer for neural text p rocessing,” in Proceedings of the 2018 Conference on Empirical Methods in N atural Language Processing: System Demonstrations , 2018
2018
-
[22]
Subword regularization: Improving neural ne twork transla- tion models with multiple subword candidates,
T. Kudo, “Subword regularization: Improving neural ne twork transla- tion models with multiple subword candidates,” in Proceedings of the 56th Annual Meeting of the Association for Computational Li nguistics (V olume 1: Long Papers), pp. 66–75, 2018
2018
-
[23]
Uniref: comprehensive and non-redundant uniprot referen ce clusters,
B. E. Suzek, H. Huang, P . McGarvey, R. Mazumder, and C. H. Wu, “Uniref: comprehensive and non-redundant uniprot referen ce clusters,” Bioinformatics, vol. 23, pp. 1282–1288, 03 2007
2007
-
[24]
Pointe r sentinel mixture models,
S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointe r sentinel mixture models,” 2016
2016
-
[25]
Incorporating context into subword vocabu- laries,
S. Y ehezkel and Y . Pinter, “Incorporating context into subword vocabu- laries,” in Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics , pp. 623–635, 2023
2023
-
[26]
New and continuing develo pments at prosite,
C. J. Sigrist, E. De Castro, L. Cerutti, B. A. Cuche, N. Hu lo, A. Bridge, L. Bougueleret, and I. Xenarios, “New and continuing develo pments at prosite,” Nucleic acids research , vol. 41, no. D1, pp. D344–D347, 2012
2012
-
[27]
G. K. Zipf, Human behavior and the principle of least effort: An introduction to human ecology . Addison-Wesley, 1949
1949
-
[28]
On the physical origin of linguistic laws and lo gnormality in speech,
I. G. Torre, B. Luque, L. Lacasa, C. T. Kello, and A. Herná ndez- Fernández, “On the physical origin of linguistic laws and lo gnormality in speech,” Royal Society open science , vol. 6, no. 8, p. 191023, 2019
2019
-
[29]
H. S. Heaps, Information Retrieval: Computational and Theoretical Aspects. USA: Academic Press, Inc., 1978
1978
-
[30]
Prolegomena to menzerath’s law,
G. Altmann, “Prolegomena to menzerath’s law,” Glottometrika, vol. 2, no. 2, pp. 1–10, 1980
1980
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.