{"id":"4d57f9bb-4779-4584-bfbc-463296862e1c","arxiv_id":"2411.17669","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"BPE, WordPiece, and SentencePiece segment protein sequences differently, all struggle to preserve protein domain boundaries, and the apparent deviations from linguistic laws may be tokenizer artifacts.","lead":"Three common word-splitting tools, BPE, WordPiece, and SentencePiece, were trained on protein sequences and tested for how well they preserve protein domain boundaries and follow standard language statistics. The study finds real differences between the tools, but the paper's conclusion that proteins obey different \"linguistic laws\" is weakened because the tokenizers themselves create the patterns being measured.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Menzerath/Zipf deviations are tokenizer artifacts, not evidence about protein organization; the central biological claim is unsupported without shuffled-sequence controls.","rationale":"The paper provides a mostly reliable descriptive comparison of BPE, WordPiece, and SentencePiece on protein sequences: the reported differences in fertility, token length, contextual exponence, and vocabulary overlap are plausible and reproducible in principle. The problem is the interpretive leap. The abstract and Section V claim that deviations from linguistic laws indicate that protein sequences 'may follow distinct organizational principles from natural languages.' That conclusion rests entirely on fitting Zipf's, Brevity's, Heaps's, and Menzerath's laws to tokenized outputs. But subword tokenizers are themselves statistical optimizers that determine token lengths and frequencies from the training corpus; they do not recover an independent segmentation of proteins into biological constituents. The Menzerath analysis in Section IV-H defines 'constituents' as BPE/WordPiece/SentencePiece tokens, so a flat or mixed average-token-length curve across sequence lengths is at least as likely to reflect the merge algorithm and vocabulary size as to reflect protein biology. The paper's own acknowledgment in Section IV-D that the domain-boundary result correlates with token length and 'may not be solely due to its algorithm' undermines the stronger interpretation placed on the law-fitting results. The proposed shuffle control directly tests whether sequence order matters: if shuffled proteins give the same Menzerath and Zipf statistics, then the tokenizer is the sole source of the pattern and the biological conclusion is an artifact. This is the single most load-bearing concern, and it is the one the reader identified. Because the central claim requires the unsupported assumption that tokenizer output distributions can stand in for the underlying language of proteins, the reader's REJECT verdict is appropriate; no verdict change is needed.","tokens_in":6,"tokens_out":4714,"duration_ms":75666,"concrete_test":"Reproduce the Menzerath and Zipf analyses of Sections IV-E and IV-H after independently shuffling the amino acids within every training and test sequence, preserving amino-acid composition and sequence-length distribution. Train the same three tokenizers on the shuffled training set, tokenize the shuffled test set, and recompute the Zipf slopes and Menzerath curves. If the shuffled-condition curves are statistically indistinguishable from the results in Figures 5 and 9, the observed deviations are tokenizer artifacts and the biological claim fails; if they differ substantially, the order-dependent signal may be real.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the inference from tokenizer-induced token frequency and length distributions to properties of 'the language of proteins.' Menzerath's law is tested in Section IV-H on tokens produced by BPE, WordPiece, and SentencePiece, but those tokens are generated by frequency-based merge or loss objectives; their length and frequency distributions are outputs of the tokenizer, not independently measured biological units. The paper itself concedes the analogous domain-boundary result is a token-length artifact in Section IV-D: BPE's 'relatively higher performance may not be solely due to its algorithm but rather to the shorter average token lengths it generates.' The same logic applies to the conclusion in Section V that deviations from Menzerath's law 'suggest that protein sequences may follow distinct distribution patterns from natural language.' Since no control is run on shuffled or order-scrambled protein sequences, there is no way to distinguish protein-specific organization from tokenizer segmentation statistics. Without such a control, the central claim that protein sequences follow distinct organizational principles from natural languages is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares three subword tokenizers (BPE, WordPiece, SentencePiece) on protein sequences from UniRef50 at vocabulary sizes 400–6400, with a BPE-on-English baseline from WikiText. It reports tokenizer overlap, token length and fertility, contextual exponence, PROSITE domain-boundary hits, and fits to Zipf, Brevity, Heaps, and Menzerath laws. The descriptive results are plausible, but the paper's central interpretive claim—that deviations from Menzerath's law and poor domain-boundary preservation show protein sequences follow organizational principles distinct from natural language—rests on tokenizer output with no chance or shuffled-sequence baselines.","tokens_in":11780,"tokens_out":5978,"duration_ms":54027,"significance":"If the central claim were established, the results would justify developing protein-specific tokenizers and would connect linguistic-law methodology to protein biology. The paper's descriptive comparison is a useful empirical contribution, especially the finding that BPE's apparent domain-boundary advantage tracks token length, and the authors share code and data. However, the main biological conclusion is not yet supported because the observed distributional properties are largely determined by the tokenization algorithms rather than by protein sequence organization.","major_comments":[{"comment":"The Menzerath-law conclusion is not supported by the evidence presented. Figure 9 plots the average token length produced by BPE, WordPiece, or SentencePiece against protein sequence length, but token lengths are outputs of the frequency- or likelihood-based merge and selection objectives of the tokenizers, not independently defined biological constituents. For any tokenizer whose per-token length distribution is roughly stationary, average token length should be nearly flat as sequence length grows, so the observed deviation from Menzerath's law is expected by construction. To claim that proteins 'may follow distinct distribution patterns from natural language,' the authors need a control such as tokenizing shuffled or permuted sequences with preserved amino-acid composition, or a random-segmentation baseline, and must show that the protein result differs from that control. The citation of Shahzad et al. [14], which analyzed protein domains, does not transfer to tokenizer output units.","section":"Section IV-H and Section V"},{"comment":"The domain-boundary hit percentages have no chance baseline. A hit requires both the start and the end of a domain to coincide with token boundaries, so even a tokenizer that places boundaries uniformly at random would produce a positive hit rate, and the paper does not report what that rate is. The authors themselves note that BPE's better performance may be due to its shorter average token length, and the correlation between token length and hit rate holds for every tokenizer. Without a length-matched or random-boundary control, the claim that the tokenizers 'do not effectively capture the true linguistic subunits of protein sequences' is not established.","section":"Section IV-D"},{"comment":"The Zipf-slope comparisons are reported as point values without confidence intervals, error bars, or statistical tests, and without a shuffled-sequence control. Because BPE, WordPiece, and SentencePiece construct vocabularies by different objectives, differences in rank-frequency slopes are expected even on random amino-acid sequences; the current analysis does not show that any slope difference is attributable to protein-specific organization. The authors should add bootstrap intervals and a randomized-sequence condition before interpreting slope deviations as evidence about the 'language of proteins.'","section":"Section IV-E, Fig. 5"}],"minor_comments":[{"comment":"The manuscript should report the version or date of UniRef50 and WikiText, the random seed used for subsampling 15 million training sequences, and the number of independent training runs; none of these are specified.","section":"Section III"},{"comment":"The experiments use SentencePiece with the Unigram model, but Section II describes SentencePiece as applying BPE or Unigram; the experimental setup should state this explicitly and give the hyperparameters used.","section":"Section II and Section IV"},{"comment":"The Menzerath-law panels use different x-axis ranges for protein (0–3000) and English (0–1000) sequences, which makes the visual comparison of trends and outliers difficult; the figures should use comparable scales or a normalized length axis.","section":"Fig. 9"},{"comment":"The Heap's-law plots include fitted K and beta values only in the captions; since the fitted curves are not clearly distinguished in the plots, the reader cannot assess the quality of the fits. Please draw the fitted curves and report fitting errors.","section":"Fig. 8"},{"comment":"The PROSITE evaluation reports 4646 domains in 3377 test sequences; the authors should clarify whether overlapping or nested domains are counted and how ambiguous domain boundaries are handled.","section":"Section IV-D"}],"recommendation":"major_revision","confidential_remarks":"To the editor: I could not accept the manuscript in its current form because the main interpretive claims lack necessary controls. I nonetheless recommend major revision rather than rejection because the descriptive tokenizer comparison is potentially useful and the missing controls (shuffled sequences, random boundary baselines, statistical intervals) are well-defined additions within the scope of the study. If the authors decline to add such controls, rejection would be appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper gives a fresh, systematic comparison of BPE, WordPiece, and SentencePiece on protein sequences across five vocabulary sizes, and that descriptive part is worth having. But the interpretive frame—that proteins deviate from natural-language laws—rests on tokenizer outputs being treated as biological units, and the paper's own Section IV-D concedes the strongest such result is a token-length artifact.\n\nThe new work is the three-way comparison with PROSITE domain boundaries and four linguistic laws measured on tokenizer outputs, plus fertility, shared-token overlap, and contextual exponence. Those measurements look plausible and the code/data are public. The finding that SentencePiece is more efficient and BPE produces shorter test tokens and higher fertility is consistent with what I'd expect, and the vocabulary-overlap trends are informative.\n\nThe soft spots are in the interpretation. First, domain-boundary hits have no chance baseline: any tokenizer with shorter tokens will mechanically align with boundaries better, and the authors say exactly that in IV-D. So 'BPE better preserves domain boundaries' is not an algorithm effect. Second, the Menzerath-law and Zipf-law observations are about token distributions generated by frequency-based merge or loss objectives. Without a shuffled-sequence control, you cannot attribute deviations to protein biology. The paper has an English BPE baseline, but that compares tokenizers, not sequences under the same tokenizer with scrambled order. Third, no error bars, significance tests, or variance across runs; with 11,957 test sequences, the plots could have confidence bands.\n\nThe authors should be credited for flagging the token-length confound themselves. But the conclusion in Section V—'protein sequences may follow distinct distribution patterns from natural language'—does not follow from the evidence. Actually, the data show the tokenizers behave differently on proteins than on English, which is a useful engineering fact and enough.\n\nWho is this for? Someone building a protein LM and deciding tokenizer/vocabulary size will get a practical orientation. A linguist looking for evidence about 'the language of proteins' should not rely on it. I'd send it to peer review because the empirical map is new and reproducible, but I'd ask for a chance baseline, shuffled controls, and a rewritten conclusion that separates tokenizer properties from protein properties.","headline":"A useful descriptive map of tokenizer behavior on proteins, but the linguistic-law conclusions are tokenizer artifacts, not evidence about protein organization.","tokens_in":12293,"tokens_out":2006,"would_cite":false,"duration_ms":17953,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that standard NLP subword tokenizers—BPE, WordPiece, and SentencePiece—do not capture the biologically meaningful subunits of protein sequences, as shown by falling domain-boundary alignment and deviations from…","keywords":["protein sequences","subword tokenization","Byte-Pair Encoding","WordPiece","SentencePiece","linguistic laws","Menzerath's law","protein domains"],"falsifier":"Tokenize the same protein test set with a vocabulary built from known functional units such as conserved motifs, then recompute the Menzerath-law curve and domain-boundary hit rate; if this vocabulary still shows flat or rising average token length with sequence length and low boundary hits, the paper's claim would be undermined, because the deviation would then be intrinsic to protein sequences rather than an artifact of general-purpose tokenizers.","tokens_in":11428,"feed_emoji":"🧬","tokens_out":11722,"duration_ms":90743,"temperature":0.7,"pith_summary":"The paper tests whether three subword tokenizers designed for human language—Byte-Pair Encoding, WordPiece, and SentencePiece—can meaningfully segment protein sequences, and it concludes that they cannot. Across vocabulary sizes from 400 to 6,400, all three align poorly with known protein domain boundaries, and the alignment gets worse as the vocabulary grows. The token frequency and length distributions conform only partially to Zipf's and Brevity laws, and they deviate from Menzerath's law, under which longer sequences should have shorter average tokens. The authors read these deviations as evidence that protein sequences follow organizational principles distinct from natural language, so tokenizers built for text are the wrong tool for proteins. If this is right, protein-aware tokenization methods are not a convenience but a requirement for faithful sequence modeling.","feed_headline":"AI tokenizers trained on text miss proteins' real units","feed_subtitle":"Standard tokenizers lose protein domain boundaries as vocabularies grow, hinting proteins need their own.","key_machinery":"The argument runs on a comparative diagnostic built from tokenizer output. Each tokenizer is trained on 15 million protein sequences at five vocabulary sizes, and its segmentations are scored on identity (shared tokens, token length, fertility), context (contextual exponence, the number of distinct neighbors a token appears with in a five-token window), and biology (alignment of token boundaries with known protein domain annotations). Four linguistic laws then serve as probes: Zipf's law checks the rank–frequency shape, Brevity law checks whether frequent tokens are short, Heaps' law checks how vocabulary grows with data, and Menzerath's law checks whether average token length declines as sequences lengthen. Because the same BPE pipeline is run on English text as a baseline, deviations that appear only for proteins can be attributed to the data rather than to the tokenizer family.","core_discovery":"The paper's central claim is that how these tokenizers segment protein sequences says more about the tokenizers than about proteins. Although the tokenizers behave differently—BPE produces more contextually specialized tokens and slightly better domain-boundary alignment at small vocabularies, SentencePiece encodes more efficiently with lower fertility, and WordPiece falls between them—all three lose the ability to respect domain boundaries as vocabulary size increases. That decline, together with hit rates that stay low even at the smallest vocabulary, is presented as evidence that none of the tokenizers is finding the units a protein's biology actually uses. The linguistic-law results reinforce the point: token frequency and length mostly follow Zipf's and Brevity laws, but average token length does not fall as sequences lengthen as Menzerath's law predicts, unlike English text run through the same BPE pipeline. The conclusion is that protein sequences may obey different organizational principles from natural language, and that standard NLP tokenization should be reworked or replaced for protein applications.","pith_inferences":["If the same proteins were tokenized with biologically defined units such as known functional motifs, the Menzerath deviation might disappear; that experiment would separate a genuine biological principle from a tokenizer artifact.","The shared-token results imply that single-vocabulary-size comparisons of protein tokenizers may not generalize, so future evaluations should report curves across vocabulary sizes.","The same four-law battery could serve as a cheap screening test for proposed protein-specific tokenizers before training expensive downstream models."],"forward_implications":["Larger vocabularies for these tokenizers systematically reduce domain-boundary preservation, so protein applications should favor smaller vocabularies or a different segmentation principle.","BPE's marginal boundary advantage at small vocabularies indicates that frequency-driven merging is a reasonable starting point for protein units, but it is not sufficient to respect functional boundaries.","Partial compliance with Zipf's and Brevity laws implies that the frequency structure of protein tokens is not radically different from text, while the Menzerath deviation isolates where the analogy fails.","Protein language models that use these tokenizers inherit their boundary errors, making tokenization choice a bound on structure- and function-prediction performance rather than a neutral preprocessing step."],"supporting_citations":[{"why":"Supplies the Byte-Pair Encoding algorithm whose protein segmentations are the primary object of study.","marker":"[19]"},{"why":"Supplies the WordPiece algorithm compared against BPE on the same protein data.","marker":"[20]"},{"why":"Supplies the SentencePiece/Unigram algorithm compared against BPE and WordPiece.","marker":"[21]"},{"why":"Provides the non-redundant protein sequence data used to train and test the tokenizers.","marker":"[23]"},{"why":"Provides the English text corpus used as the natural-language baseline for BPE.","marker":"[24]"},{"why":"Provides the protein-domain annotations used to score whether token boundaries respect protein domains.","marker":"[26]"},{"why":"Defines Zipf's rank-frequency law, the first linguistic probe applied to token distributions.","marker":"[27]"},{"why":"Defines Brevity law, used to test whether frequent tokens are short.","marker":"[28]"},{"why":"Defines Menzerath's law, the probe that protein tokenizers fail to follow.","marker":"[30]"},{"why":"Reports that protein domains obey Menzerath–Altmann's law, the prior result the paper's token-level deviation is measured against.","marker":"[14]"}],"fun_headline_variants":["Tokenizers miss protein domains as vocabulary grows","Text-trained tokenizers can't find proteins' real units","Vocabulary size pushes tokenizers away from protein boundaries","Protein sequences break Menzerath's law, tokenizers fail","NLP tokenizers distort protein segmentation, study shows"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the token frequencies and token lengths produced by these three subword tokenizers give a reliable window into the underlying organization of proteins, so that a deviation from a linguistic law counts as evidence about proteins rather than about the tokenizers.","fun_headline_variants_meta":{"raw":{"variants":["Tokenizers miss protein domains as vocabulary grows","Text-trained tokenizers can't find proteins' real units","Vocabulary size pushes tokenizers away from protein boundaries","Protein sequences break Menzerath's law, tokenizers fail","NLP tokenizers distort protein segmentation, study shows"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000511,"raw_usage":{"total_tokens":2507,"prompt_tokens":988,"completion_tokens":1519,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":1443}},"tokens_in":604,"tokens_out":1519,"duration_ms":11602,"temperature":1.0,"reasoning_tokens":1443,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:51:24.994179+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Tokenize the same protein test set with a vocabulary built from known functional units such as conserved motifs, then recompute the Menzerath-law curve and domain-boundary hit rate; if this vocabulary still shows flat or rising average token length with sequence length and low boundary hits, the paper's claim would be undermined, because the deviation would then be intrinsic to protein sequences rather than an artifact of general-purpose tokenizers.","supporting_citations":[{"cited_title":"Neural machine tr anslation of rare words with subword units,","cited_arxiv_id":null,"evidence_quote":"Supplies the Byte-Pair Encoding algorithm whose protein segmentations are the primary object of study."},{"cited_title":"Sentencepiece: A simple and language inde- pendent subword tokenizer and detokenizer for neural text p rocessing,","cited_arxiv_id":null,"evidence_quote":"Supplies the SentencePiece/Unigram algorithm compared against BPE and WordPiece."},{"cited_title":"Uniref: comprehensive and non-redundant uniprot referen ce clusters,","cited_arxiv_id":null,"evidence_quote":"Provides the non-redundant protein sequence data used to train and test the tokenizers."},{"cited_title":"Pointe r sentinel mixture models,","cited_arxiv_id":null,"evidence_quote":"Provides the English text corpus used as the natural-language baseline for BPE."},{"cited_title":"New and continuing develo pments at prosite,","cited_arxiv_id":null,"evidence_quote":"Provides the protein-domain annotations used to score whether token boundaries respect protein domains."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Zipf's rank-frequency law, the first linguistic probe applied to token distributions."},{"cited_title":"On the physical origin of linguistic laws and lo gnormality in speech,","cited_arxiv_id":null,"evidence_quote":"Defines Brevity law, used to test whether frequent tokens are short."},{"cited_title":"Prolegomena to menzerath’s law,","cited_arxiv_id":null,"evidence_quote":"Defines Menzerath's law, the probe that protein tokenizers fail to follow."},{"cited_title":"The organization of domains in proteins obeys menzerath-altmann’s law of lan guage,","cited_arxiv_id":null,"evidence_quote":"Reports that protein domains obey Menzerath–Altmann's law, the prior result the paper's token-level deviation is measured against."}],"review_version":1}