Pith. sign in

REVIEW 4 major objections 5 minor 20 references

When repeats drive the vocabulary: a Byte-Pair Encoding analysis of T2T primate genomes

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that Byte-Pair Encoding, the tokenizer behind many DNA language models, is driven by species-specific repetitive DNA and fails as a universal genomic tokenizer.

desk verdict Useful empirical dataset and a much-needed open-source tool, but the headline claim that BPE is inadequate for multi-genome tokenization is not actually tested by the experimental design. read the letter →

arxiv 2505.08918 v1 pith:VJKLNRIN submitted 2025-05-13 q-bio.GN cs.AI

Marina Popova , Iaroslav Chelombitko , Aleksey Komissarov This is my paper · ORCID
classification q-bio.GNcs.AI
keywords byte-pairencodinggenomictokenizationT2TprimategenomesrepetitiveelementssatelliteDNAlanguagemodelscomparativegenomicstokenoverlap
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Byte-Pair Encoding (BPE) is the tokenizer used by many DNA language models, and this paper asks what such vocabularies actually contain when trained on complete genomes. Training independent 512,000-token BPE tokenizers on nine telomere-to-telomere primate genomes, the authors find that only 11,569 tokens (0.6%) are shared by all nine, while 991,854 tokens (51.2%) appear in just one genome. They further show that roughly 77.6% of core-token occurrences fall in repetitive elements, against an expected ~54% of genomic content, and that token-overlap trees fail to reproduce known primate relationships. The conclusion is that BPE compresses repeats well but spends its vocabulary on species-specific high-copy sequences, making it inadequate as a universal tokenizer for comparative genomics; the authors propose repeat masking and separate handling of functional and repetitive regions.

What carries the argument

The machinery is BPE itself, defined as the iterative merge of the most frequent adjacent pair in a sequence; here it is implemented in the authors' dnaBPE, a C++ tokenizer that records the frequency and genomic position of each merge. The argument runs on three derived objects: the 512,000-token vocabulary of each genome, the overlap counts across vocabularies, and Jaccard distances between token sets. The crucial property is BPE's greedy frequency bias, which deterministically favors high-copy repeats over rarer conserved sequences when the vocabulary is capped.

What would settle it

Train one 512,000-token BPE vocabulary on the concatenated sequences of all nine genomes and count how many of those tokens appear in every genome; if the cross-genome shared core is substantially larger than 11,569, or if repeat-masked tokenizers reconstruct known primate phylogeny, the paper's blanket conclusion that BPE is inadequate for multi-genome tokenization would be falsified.

Watch

Extended reading notes

Core claim

The central discovery reported here is that BPE's greedy pair-merging has a built-in bias: it fills a fixed-size vocabulary with the highest-copy sequences, and in primate genomes those are lineage-specific repeats rather than conserved genomic signal. Across nine independently trained tokenizers of size 512,000, the intersection is 11,569 tokens, and 991,854 tokens are each found in only one of the nine genomes. Shared tokens peak at 8-12 bp, cover only 0.34% of possible 10-mers, and 77.6% of their occurrences overlap repeat annotations. As a result, pairwise token-set distances place the three human assemblies close to each other yet scatter them across the dendrogram, so BPE token overlap does not track phylogeny.

Load-bearing premise

The conclusion rests on the assumption that training one tokenizer per genome and measuring the overlap of these separate vocabularies is a fair test of BPE as a universal tokenizer; if a single BPE trained on all nine genomes together would share far more tokens, the central claim would need qualification.

Editorial extensions

If this is right

  • A DNA language model trained on a per-genome BPE vocabulary will spend most of its embedding capacity on repeat content, so cross-species transfer should be poor even for closely related primates.
  • Enlarging the vocabulary will not cure the repeat bias: existing BPE tokenizers with 610 to 32,000 tokens all show 77-78% repeat coverage, matching the authors' observation.
  • Token-overlap based phylogenies will be unreliable for BPE vocabularies of this kind, since the method fails to cluster even the three human assemblies together.
  • BPE can still serve as a repeat-oriented analysis tool: species-specific high-copy tokens may help annotate satellite DNA and find novel repeat families in unannotated genomes.
  • The paper's proposed fixes—repeat masking, separate tokenizers for functional versus repetitive regions, and core-plus-species vocabularies—are the concrete experiments that would test whether the repeat bias can be controlled.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper trains a separate tokenizer per genome; the 0.6% shared-token figure is therefore a property of independent per-genome tokenizers, not of a single tokenizer trained on the pooled nine-genome corpus. Training one BPE on the concatenated sequences would directly test whether a truly universal tokenizer would share more vocabulary.
  • If the repeat-bias mechanism is correct, then repeat-masking before BPE training should raise cross-species token overlap and make token-based distances track phylogeny; this is a measurable prediction on the same nine assemblies.
  • The phylogenetic failure may be partly an artifact of exact token-identity overlap: distances computed on frequency-weighted or edit-distance-aware token similarities, rather than raw Jaccard overlap, could recover evolutionary signal even with repeat-dominated vocabularies.
  • The species-private tokens could double as genomic markers: their counts across genomes may quantify lineage-specific repeat expansions and contractions, giving a cheap comparative-genomics assay independent of the tokenizer's failure as a universal encoder.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper trains independent 512,000-token BPE tokenizers on nine T2T primate genomes using a custom tool, dnaBPE, and then compares the resulting vocabularies. The authors report that only 11,569 tokens (0.6%) are shared across all nine genomes, that 991,854 tokens (51.2%) are unique to a single genome, and that shared tokens cluster at lengths of 8-12 bp. They further report that phylogenetic trees built from token overlap do not recapitulate accepted primate relationships and that token occurrences are dominated by repetitive elements (about 77.6% of core-token occurrences versus an expected genomic repeat content of about 54%). On this basis, the paper concludes that BPE is inadequate as a universal tokenizer for multi-genome comparative genomics and proposes repeat-masking and hybrid tokenization strategies.

Significance. If the descriptive findings hold, the paper provides a useful cautionary data point for genomic language model design: BPE vocabularies trained on eukaryotic genomes are heavily populated by high-copy repeats, and cross-species vocabulary sharing is low. The study has concrete strengths: the dnaBPE tool is open-source, the pipeline is reproducible, the token-overlap counts are concrete, and the comparison with existing genomic BPE tokenizers (GROVER, GENA-LM, DNABERT-2) is a useful contribution. The main limitation is that the headline claim about BPE as a universal tokenizer is not directly tested by the experimental design, and the phylogenetic distance methodology is underspecified. The paper is best read as a pilot study reporting a striking empirical pattern rather than as a definitive evaluation of universal genomic tokenization.

major comments (4)
  1. [Section 2.3 and Section 1] The central claim that 'BPE proves to be an inadequate tool for tokenizing multiple genomes in their entirety' (Section 1) is not directly tested by the reported experiment. The protocol trains a separate 512,000-token BPE tokenizer on each genome (Section 2.3) and then intersects the nine vocabularies (Section 3.3), which measures agreement between independently trained tokenizers rather than the behavior of a single tokenizer trained on a combined multi-genome corpus. Because BPE merge decisions are frequency-driven and tie-breaking sensitive, two independent runs can represent the same conserved motif under different token IDs or split it differently, so a low shared-token fraction does not by itself imply that a combined-corpus tokenizer would fail to share those motifs. A combined-corpus training run, or an explicit argument for why independent training is the appropriate test, is needed to support the universal-tokenizer conclusion.
  2. [Section 2.4 and Figure 2] The phylogenetic distance measure is underspecified. The text states only that distances were 'derived based on the similarity or dissimilarity of token sets and their frequencies,' without giving the formula, the normalization, the treatment of token frequencies (e.g., raw counts, log-transformed counts, or presence/absence), or the linkage criterion used for hierarchical clustering. Without this information, the claim that token-based trees 'failed to recapitulate established primate relationships' cannot be evaluated or reproduced.
  3. [Table 2 and Section 3.5] The coverage percentages in Table 2 are ambiguous, and the comparison to expected genomic content may conflate different denominators. It is unclear whether the table reports the fraction of token occurrences whose genomic positions overlap each annotation, the fraction of distinct tokens annotated to each class, or the fraction of genomic bases covered by tokens of each class. In addition, 'All reps' appears to be a superset of 'SatDNA' and 'Alu,' so presenting the five columns as additive shares is misleading unless a partitioning rule is stated. The claim that BPE overrepresents repeats at about 77.6% versus about 54% expected requires a clear definition of both the numerator and the denominator.
  4. [Sections 3.4 and 3.5] The attribution of phylogenetic incongruence to species-specific repetitive elements is causal but is not tested. The paper shows that repeats dominate token coverage, but it does not show that repeat-masked or repeat-excluded token sets would recover the expected phylogeny. A control experiment, such as recomputing the dendrogram after masking repeats in the training input or after removing repeat-annotated tokens, would directly support the stated attribution and would be a natural addition given the authors' own discussion of repeat masking.
minor comments (5)
  1. [Section 2.1] The sentence 'For every genome except CHM13, we combined the paternal and maternal genomes, after that, we added its reverse complement' is ambiguous about whether CHM13 also received its reverse complement; the reported sizes (12 billion bp versus 6 billion bp) suggest it did, but the sentence should say so explicitly.
  2. [Section 3.3] The phrase 'nearly 991,854 tokens' is odd because 991,854 appears to be an exact count; please either report the exact number without 'nearly' or round it consistently.
  3. [Table 1] Table 1 relies on color coding (green for conservation, orange for divergence) that is not visible in monochrome print; please add explicit symbols, formatting, or a separate annotation column to convey the three conservation patterns.
  4. [Figures 1 and 2] Figure 1 does not label the axes or state whether the histogram is over token counts or relative frequencies, and Figure 2 does not specify the distance metric or linkage used for the dendrogram; adding these details would improve reproducibility.
  5. [Section 2.6] The comparison with existing BPE tokenizers (GROVER, GENA-LM, DNABERT-2) may be confounded by differences in training corpora and tokenizer training protocols; this limitation should be stated where the comparison is introduced rather than only implied later.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: shared-token statistics and repeat-coverage annotations are independently measured, and the inadequacy conclusion is an interpretation rather than a fitted or self-cited premise.

full rationale

The derivation chain is self-contained and empirically grounded. Section 2.3 trains nine independent 512K-token BPE tokenizers, Section 3.3 reports the measured overlap (11,569 tokens shared, 991,854 genome-unique), and Section 3.5 annotates token positions with external RepeatMasker, cenSat, and exon-liftover data from HG002. The 77.6% repeat coverage and 10.4% exon coverage are annotation counts, not fitted parameters, and the comparison against the expected ~54% repeat content uses published genome statistics. The statement that BPE is 'inadequate for tokenizing multiple genomes in their entirety' is a downstream interpretation of these measurements, not an equation that reduces to its inputs. No self-citation is load-bearing: the custom dnaBPE and TokPos tools are software artifacts, and the cited related work (e.g., Vishniakov et al.) does not supply the overlap result or forbid any alternative. The skeptical concern that independently trained per-genome tokenizers test vocabulary intersection rather than a combined-corpus universal tokenizer is a study-design limitation, not a circular step, because the paper never defines 'universal' in terms of the intersection itself. Accordingly, no specific reduction of prediction to input can be exhibited, and the honest finding is no significant circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the standard BPE algorithm, the chosen vocabulary size, the combined sequence corpus, external repeat-content baselines, and the interpretive assumption that token overlap should mirror phylogeny. No new physical entities are introduced; dnaBPE is software, not a postulated entity.

free parameters (1)
  • BPE vocabulary size = 512,000
    Hand-chosen; the paper's overlap statistics and conclusions depend on this fixed size. A different size could change the share of unique tokens.
assumptions (4)
  • standard math Byte-Pair Encoding merging rule (most frequent pair at each step) is a sensible tokenization for DNA.
    The paper relies on the standard BPE algorithm without modifications. This is a background algorithmic assumption.
  • domain assumption The nine T2T assemblies and the concatenation of haplotypes with reverse complement produce a representative sequence corpus for each species.
    Section 2.1: 'For every genome except CHM13, we combined the paternal and maternal genomes, after that, we added its reverse complement.' The conclusions about 'the genome' of each species depend on this combined input.
  • domain assumption The expected genomic content values used as baselines (e.g., 54% repeats, 11% Alu, 5% satellite DNA) apply to the HG002 genome used for token annotation.
    Section 3.5 compares token coverage to 'expected 54% genomic content (Nurk et al., 2022)' and other expectations; if those expectations are not representative of HG002, the overrepresentation claim weakens.
  • domain assumption Token overlap similarity should, in principle, be able to recapitulate known primate phylogeny.
    Section 2.4 assumes that a useful tokenizer should produce vocabularies whose overlap reflects evolutionary relationships; the paper treats the failure to recover phylogeny as a limitation of BPE rather than a faulty premise.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When repeats drive the vocabulary: a Byte-Pair Encoding analysis of T2T primate genomes." pith.science (2026). https://pith.science/paper/VJKLNRIN

@misc{pith2026250508918,
  author       = {Pith},
  title        = {Pith review of: When repeats drive the vocabulary: a Byte-Pair Encoding analysis of T2T primate genomes},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VJKLNRIN}},
  note         = {Machine review of arXiv:2505.08918}
}
read the original abstract

The emergence of telomere-to-telomere (T2T) genome assemblies has opened new avenues for comparative genomics, yet effective tokenization strategies for genomic sequences remain underexplored. In this pilot study, we apply Byte Pair Encoding (BPE) to nine T2T primate genomes including three human assemblies by training independent BPE tokenizers with a fixed vocabulary of 512,000 tokens using our custom tool, dnaBPE. Our analysis reveals that only 11,569 tokens are shared across all assemblies, while nearly 991,854 tokens are unique to a single genome, indicating a rapid decline in shared vocabulary with increasing assembly comparisons. Moreover, phylogenetic trees derived from token overlap failed to recapitulate established primate relationships, a discrepancy attributed to the disproportionate influence of species-specific high-copy repetitive elements. These findings underscore the dual nature of BPE tokenization: while it effectively compresses repetitive sequences, its sensitivity to high-copy elements limits its utility as a universal tool for comparative genomics. We discuss potential hybrid strategies and repeat-masking approaches to refine genomic tokenization, emphasizing the need for domain-specific adaptations in the development of large-scale genomic language models. The dnaBPE tool used in this study is open-source and available at https://github.com/aglabx/dnaBPE.

Figures

Figures reproduced from arXiv: 2505.08918 by the authors.

Figure 1
Figure 1. Analysis of token length distributions in the core set of 11,569 tokens shared across nine [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Comparative analysis of BPE token distributions across nine primate genomes. Left: Hi [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Distribution of token sharing across nine primate genome tokenizers (log scale). The graph [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 8 canonical work pages

  1. [3]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z

    URL https://arxiv.org/abs/2005.14165. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi D...

    arXiv 2005

  2. [8]

    doi: 10.1101/2024.02.14. 580373. URL http://dx.doi.org/10.1101/2024.02.14.580373. Xingyu Liao, Wufei Zhu, Juexiao Zhou, Haoyang Li, Xiaopeng Xu, Bin Zhang, and Xin Gao. Repetitive dna sequence detection and its role in the human genome. Communications Biology, 6(1), September

    work page doi:10.1101/2024.02.14 2024

  3. [10]

    doi: 10.1126/science.adq8191

    ISSN 1095-9203. doi: 10.1126/science.adq8191. URL http://dx.doi. org/10.1126/science.adq8191. Shentong Mo, Xi Fu, Chenyang Hong, Yizhen Chen, Yuxuan Zheng, Xiangru Tang, Zhiqiang Shen, Eric P Xing, and Yanyan Lan. Multi-modal self-supervised pre-training for regulatory genome across cell types,

    work page doi:10.1126/science.adq8191

  4. [11]

    Multi-modal Self-supervised Pre-training for Regulatory Genome Across Cell Types

    URL https://arxiv.org/abs/2110.05231. Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Callum Birch-Sykes, Michael Wornow, Aman Patel, Clayton Rabideau, Stefano Massaroli, Yoshua Bengio, Stefano Ermon, Stephen A. Baccus, and Chris R ´e. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution,

    work page Pith review arXiv

  5. [12]

    Sergey Nurk et al

    URL https://arxiv.org/abs/2306.15794. Sergey Nurk et al. The complete sequence of a human genome. Science, 376(6588):44–53, April

    arXiv

  6. [13]

    URL https://doi.org/10.1126/science

    doi: 10.1126/science.abj6987. URL https://doi.org/10.1126/science. abj6987. Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and V olodymyr Kuleshov. Caduceus: Bi-directional equivariant long-range dna sequence modeling,

    doi:10.1126/science.abj6987

  7. [14]

    URL https: //arxiv.org/abs/2403.03234. Carl W. Schmid and Prescott L. Deininger. Sequence organization of the human genome. Cell, 6 (3):345–358, November

    arXiv

  8. [16]

    URL https://doi.org/10.3390/ijms22094309

    doi: 10.3390/ijms22094309. URL https://doi.org/10.3390/ijms22094309. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need,

    work page doi:10.3390/ijms22094309

Show all 20 references
  1. [18]

    doi: 10.1101/2024.12.18.628606. C. Yang, Y . Zhou, Y . Song, et al. The complete and fully-phased diploid genome of a male han chinese. Cell Research, 33:745–761,

    2024 doi

  2. [19]

    URL https: //doi.org/10.1038/s41422-023-00849-5

    doi: 10.1038/s41422-023-00849-5. URL https: //doi.org/10.1038/s41422-023-00849-5 . 10 Accepted at the MLGenX workshop at ICLR 2025 DongAhn Yoo, Arang Rhie, Prajna Hebbar, Francesca Antonacci, Glennis A. Logsdon, Steven J. Solar, Dmitry Antipov, Brandon D. Pickett, Yana Safonov...

    2025 doi

  3. [20]

    doi: 10.1038/s41586-025-08816-3

    ISSN 1476-4687. doi: 10.1038/s41586-025-08816-3. URL http://dx.doi.org/10.1038/s41586-025-08816-3 . A A PPENDIX Figure 3: Distribution of token sharing across nine primate genome tokenizers (log scale). The graph shows a dramatic decline in the number of shared tokens as the n...

    doi

  4. [1975]

    doi: 10.1016/0092-8674(75)90184-1

    ISSN 0092-8674. doi: 10.1016/0092-8674(75)90184-1. URL http://dx.doi.org/10.1016/0092-8674(75)90184-1. J. Thakur, J. Packiaraj, and S. Henikoff. Sequence, chromatin and evolution of satellite dna. In- ternational Journal of Molecular Sciences , 22(9):4309,

    doi

  5. [2011]

    doi: 10.1186/gb-2011-12-12-236

    ISSN 1465-6906. doi: 10.1186/gb-2011-12-12-236. URL http://dx.doi.org/10.1186/ gb-2011-12-12-236 . E. D. Jarvis, G. Formenti, A. Rhie, et al. Semi-automated assembly of high-quality diploid hu- man reference genomes. Nature, 611:519–531,

    2011 doi

  6. [2017]

    org/abs/1706.03762

    URL https://arxiv. org/abs/1706.03762. Kirill Vishniakov, Karthik Viswanathan, Aleksandr Medvedev, Praveen K Kanithi, Marco AF Pi- mentel, Ronnie Rajan, and Shadab Khan. Genomic foundationless models: Pretraining does not promise performance. bioRxiv,

    arXiv

  7. [2020]

    URL https://arxiv.org/abs/2004.03720. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child...

    2004 arXiv

  8. [2021]

    doi: 10.1093/bioinformatics/btab083

    ISSN 1367-4811. doi: 10.1093/bioinformatics/btab083. URL http://dx.doi.org/10.1093/bioinformatics/btab083. Avantika Lal, David Garfield, Tommaso Biancalani, and Gokcen Eraslan. reglm: Designing realistic regulatory dna with autoregressive language models. February

    doi

  9. [2022]

    URL https://doi.org/10.1038/s41586-022-05325-5

    doi: 10.1038/s41586-022-05325-5. URL https://doi.org/10.1038/s41586-022-05325-5 . Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V Davuluri. Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language in genome. Bioinformatics, 37(15):2112...

    doi

  10. [2023]

    doi: 10.1038/s42003-023-05322-y

    ISSN 2399-3642. doi: 10.1038/s42003-023-05322-y. URL http: //dx.doi.org/10.1038/s42003-023-05322-y . Junli Liu, Qilin Li, Yixuan Hu, Yi Yu, Kai Zheng, Dengfeng Li, Lexin Qin, and Xiaochun Yu. The complete telomere-to-telomere sequence of a mouse genome. Science, 386(6726):1141...

    doi

  11. [2024]

    arXiv preprint

    URL https://doi.org/10.48550/arXiv.2407.11435. arXiv preprint. Kaj Bostrom and Greg Durrett. Byte pair encoding is suboptimal for language model pretraining,

    doi arXiv

  12. [2025]

    9 Accepted at the MLGenX workshop at ICLR 2025 Prescott Deininger

    URL https://arxiv.org/abs/2501.12948. 9 Accepted at the MLGenX workshop at ICLR 2025 Prescott Deininger. Alu elements: know the sines. Genome Biology , 12(12):236,

    2025 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.