Evaluation of Coding Schemes for Transformer-based Gene Sequence Modeling

Chenlei Gong; Lei Mao; Yan Song; Yuanhe Tian

REVIEW 4 major objections 5 minor 16 references

Evaluation of Coding Schemes for Transformer-based Gene Sequence Modeling

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read For Transformer DNA models, BPE tokenization and rotary position embeddings come out on top.

desk verdict The BPE-vs-k-mer result is well supported at the point-estimate level, but the paper overclaims the RoPE-vs-AliBi ranking from single runs with no error bars; still worth refereeing. read the letter →

arxiv 2507.15087 v1 pith:JQWY3ESA submitted 2025-07-20 cs.CL cs.AI

Chenlei Gong , Yuanhe Tian , Lei Mao , Yan Song This is my paper · ORCID

classification cs.CLcs.AI

keywords DNAsequencemodelingtokenizationk-mersegmentationbytepairencoding(BPE)positionalrotarypositionembeddingsTransformerencodergenomicbenchmark

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks which standard coding choices really help when a Transformer is trained on raw DNA sequences. It compares fixed-length k-mer tokenization at k=1,3,4,5,6 against a 4,096-token byte pair encoding (BPE) vocabulary, and sinusoidal, AliBi, and rotary (RoPE) positional encodings, training every combination from scratch at 3, 6, 12, and 24 layers on six tasks from the GUE benchmark. Its central claim is that BPE tokenization is the better default—it compresses frequent motifs into variable-length subwords, shortens sequences, and stays more stable under small nucleotide changes—while RoPE is the better positional encoding because it captures periodic signals and extrapolates to longer inputs, with AliBi second and absolute sinusoidal encoding last. The authors also report that depth helps mainly up to 12 layers and gives only marginal returns at 24. A sympathetic reader would take away concrete defaults for building DNA Transformer classifiers, and a caution that the margins separating the top choices are often small.

What carries the argument

The comparison grid is the machinery: every tokenizer is crossed with every positional encoding and four depths, and all models are trained from scratch so that differences are attributable to coding scheme rather than pretraining. The two mechanisms doing the explanatory work are BPE tokenization—a greedy algorithm that starts from the four nucleotides and repeatedly merges the most frequent adjacent pair, yielding variable-length subword tokens that compress frequent motifs—and RoPE, which rotates adjacent pairs of query and key dimensions by position-dependent angles so dot-product attention naturally encodes relative distance as a periodic function. AliBi, the third positional scheme, injects a fixed linear distance penalty into attention scores. The study's yardstick is Matthews correlation coefficient on six GUE classification tasks spanning different species, sequence lengths, and regulatory objectives.

What would settle it

Train the same configurations (say BPE and the best k-mer, each with RoPE and AliBi, at 12 layers) under five or more random seeds. If the seed-to-seed spread of MCC is comparable to the 0.01–0.05 gaps in Tables 2–4, the paper's claimed ordering is not established; if the ordering persists across seeds, the design guidance holds.

Watch

Extended reading notes

Core claim

The intended contribution is an empirical ranking of coding schemes for Transformer-based DNA sequence classification. On tokenization, BPE with a 4,096-token vocabulary is presented as the best overall choice: it outperforms every fixed k-mer setting in most configurations and shows the smallest performance degradation when nucleotides at sequence ends are substituted or deleted; the reason offered is that BPE subwords capture biological motifs at multiple scales while shortening the input. On positional encoding, RoPE is presented as the best overall choice, with AliBi second and sinusoidal absolute embeddings last, because rotation-based relative position information is better at recognizing periodic motifs and extrapolating beyond training length. On depth, 3-to-12-layer growth clearly improves MCC, whereas 24 layers yield only small gains and occasional overfitting. The paper frames these as practical design guidelines for future genomic language models.

Load-bearing premise

The whole ranking depends on each configuration being trained once with one fixed hyperparameter set, so the 0.01–0.05 MCC differences between BPE and k-mers, and between RoPE and AliBi, could disappear under run-to-run variation.

Editorial extensions

If this is right

New DNA Transformer models should default to BPE tokenization over fixed k-mer segmentation, since BPE wins or ties in most configurations and degrades least under small sequence mutations.
RoPE should be the default positional encoding for genomic tasks, with AliBi a reasonable alternative when local-dependency bias is wanted and sinusoidal embeddings used only when motif spacing is fixed.
When k-mers are unavoidable, the best k is task-dependent: small k suits trinucleotide-like splice signals and larger k suits longer epigenetic motifs, so the choice must match the biology.
Depth scaling advice from the results: increase layers to around 12 for clear gains, and treat 24-layer models as an expensive option with small marginal benefit.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct head-to-head that the paper does not run would pair the two winners, BPE plus RoPE; scanning the result tables, this combination is at or near the top on nearly every task and depth, so it is the likely practical recommendation even though the paper does not single it out.
The mutation-robustness result suggests BPE subwords may help in variant-aware genomics, where test reads differ from reference sequences by single-nucleotide variants or indels; this application is not explored here.
All rankings rest on single runs with fixed hyperparameters, so a multi-seed replication with significance intervals is the natural next experiment before treating the 0.01–0.05 MCC gaps as design rules.
RoPE's extrapolation ability hints that models trained on short DNA windows could be applied to full-length sequences, but the paper does not test long-sequence extrapolation directly.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The practical takeaway here is half right. The BPE-over-k-mer conclusion is solid at the point-estimate level: across your 168-cell grid, BPE wins the row average at every depth and every positional encoding, and the gaps are usually 0.03–0.10 MCC, which is larger than the typical per-cell differences you'd worry about. The RoPE conclusion is not. Summed over all tasks and depths, RoPE and AliBi are tied (0.492 vs 0.492), AliBi beats RoPE on Virus-Covid at every depth for BPE, and the per-cell gaps you'd need to establish a ranking are 0.01–0.05 MCC—exactly the range you'd expect from run-to-run noise when each cell is a single training run with fixed hyperparameters and no reported seeds. So the paper establishes that SAPE is worse and RoPE/AliBi are comparable; it does not establish 'RoPE, then AliBi, then SAPE.'

What's new is the grid itself: k-mer vs BPE crossed with three positional encodings and four depths on GUE, all trained from scratch. That's a useful map, and the code is public. The writing is clear and the robustness test (Figures 2–4) is a nice addition, though it appears abruptly in the Results without being introduced in the Approach or abstract.

Soft spots, in order of severity. First, no seeds, no error bars, no significance tests anywhere; the headline rankings rest on single MCC values. Second, the internal inconsistency over k=5: the abstract and tables include 5-mer, but Implementation Details says k=1,3,4,6. That needs fixing. Third, the claim that RoPE 'significantly outperforms' other schemes is not supported by the paper's own numbers; only SAPE is clearly worse. Fourth, the robustness section is under-described—three test groups are mentioned, but the construction of the substitution/deletion sets is not specified in the main text.

The citation pattern is fine; DNABERT-2 is credited for BPE, and the related work covers the space. The paper is an empirical benchmark, so circularity is not an issue. Who is this for? Practitioners building DNA Transformers who want a default tokenizer and positional encoding. It gives them a reasonable starting point (BPE + RoPE or AliBi, ~12 layers), but they should treat the finer ranking with caution until multi-seed results appear. I'd send it to review: the grid is valuable, the claims are checkable, and the fixes are straightforward—add seeds or soften the claims, reconcile the k inconsistency, and report the robustness setup properly.

Referee Report

4 major / 5 minor

Summary. This manuscript reports a systematic comparison of DNA sequence coding schemes for Transformer classifiers. It evaluates six tokenization schemes (1-, 3-, 4-, 5-, 6-mer and a 4,096-token BPE vocabulary) and three positional encodings (sinusoidal absolute, AliBi, and RoPE) in Transformer encoders with 3, 6, 12, and 24 layers, all trained from scratch on the GUE benchmark. Performance is reported as mean MCC across datasets within each of seven tasks. The main conclusions are that BPE tokenization outperforms k-mers on average, that RoPE is the best positional encoding followed by AliBi, and that depth gains saturate around 12 layers.

Significance. If the conclusions were fully supported, the paper would provide useful practical guidance for DNA Transformer design and would complement existing benchmarks such as GUE with a controlled tokenizer/positional-encoding comparison. The strengths of the study are its broad coverage (6 tasks, 7 tokenization schemes, 3 positional encodings, 4 depths), the use of the public GUE benchmark, and the promised open-source code. However, the headline claims are only partially supported by the reported point estimates: the BPE advantage is an average effect that reverses on several tasks, and the RoPE-versus-AliBi ordering is essentially a tie once the relevant cells are inspected. Because every configuration is trained once with no variance estimate, the practical recommendations rest on differences that may be within run-to-run noise.

major comments (4)

[Impact of Tokenization Approach (Tables 2–4)] The claim that "BPE demonstrates the best performance across different experiment settings compared with k-mer tokenization approaches" is stronger than the tables support. While BPE has the highest row mean within each depth/positional-encoding block, the per-task/per-depth rankings frequently reverse. For example, on Human-FTP, 5-mer achieves 0.6000 versus BPE 0.5262 (Table 2, 6 layers), 6-mer achieves 0.6291 versus BPE 0.5381 (Table 3, 12 layers), and 5-mer achieves 0.5910 versus BPE 0.5500 (Table 4, 12 layers). The manuscript should therefore state that BPE is best on average, not best across settings, and should characterize the tasks on which k-mers win rather than presenting a global recommendation.
[Impact of Positional Encoding Strategy (Tables 3–4)] The claim that "Rotary achieves the best performance on the tasks, followed by AliBi, with standard absolute positional encoding obtaining the worst results" is not established by the reported point estimates. On Virus-Covid with BPE, AliBi outperforms RoPE at every depth (Table 3: 0.6681, 0.6788, 0.6644, 0.6867 versus Table 4: 0.6284, 0.6417, 0.6535, 0.6850). The aggregate difference between RoPE and AliBi is extremely small, and the additional claim about RoPE's "extrapolation to ultra-long sequences" is not tested by any experiment, since all GUE tasks have fixed maximum lengths and no length-extrapolation evaluation is reported. The defensible conclusion at this point is that SAPE is generally worse and RoPE/AliBi are statistically indistinguishable.
[Implementation Details] Each configuration is trained once with fixed hyperparameters and no random-seed control, and every cell in Tables 2–4 is a single MCC value. Many of the differences that drive the conclusions are on the order of 0.01–0.05 MCC, and some are far smaller, so without an estimate of run-to-run variance or paired significance tests the rankings cannot be distinguished from noise. I recommend reporting means and standard deviations over at least three seeds for the central configurations, or at minimum a paired analysis of the tokenizer and positional-encoding comparisons.
[Implementation Details] The 4,096-token BPE vocabulary is taken from previous work (Zhou et al., 2023; Mao et al., 2025), but the corpus on which it was trained is not reported. If that vocabulary is human-genome-specific, the comparisons on Mouse-TFP, Yeast-EMP, and Virus-Covid are not like-for-like with k-mer tokenization, which has no data-dependent vocabulary. The authors should report the BPE training corpus and its species composition, or train a BPE tokenizer per species, and discuss how this affects the tokenization comparison.

minor comments (5)

[Conclusion] The sentence "Our experiments span three model depths (3, 6, 12, and 24 layers)" is internally inconsistent; four depths are evaluated.
[Implementation Details] The learning rate is written as "110 −4"; this should be 1e-4 or 1×10⁻⁴.
[Tables 2–4] The column header "Yest-EMP" should be "Yeast-EMP".
[Approach] The Approach section says the paper compares "learned absolute embeddings, relative position bias, and rotary position embeddings", but the experiments use sinusoidal absolute positional encoding and AliBi; the terminology should be aligned throughout.
[Results and Analysis (Figures 2–4)] The robustness experiment against nucleotide alterations is introduced without any description of the alteration procedure, the number of altered test sets used, or whether the same altered sequences are used for all models. Please move this methodology into Implementation Details and describe the figure axes.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical benchmark whose conclusions are drawn from held-out test evaluations.

full rationale

This paper does not claim to derive any result from first principles or to predict a quantity from its own fitted inputs. It systematically trains Transformer encoders from scratch with different tokenizers (k-mer and BPE) and positional encodings (SAPE, AliBi, RoPE), then evaluates the resulting models on the externally provided GUE benchmark's test splits, reporting MCC values in Tables 2, 3, and 4. The central claims ('BPE demonstrates the best performance...' and 'Rotary achieves the best performance...') are empirical summaries of those held-out measurements. The BPE tokenizer is taken from prior work (Zhou et al., 2023; Mao et al., 2025), but borrowing an existing tokenizer is not circular: it is a fixed external tool, not a parameter fitted to the same test results being explained. The authors' own self-citation, Mao et al. (2025), is used as a source for the BPE setup and as related work, but no load-bearing conclusion is justified solely by that citation. The skeptical observation that single runs without seeds or significance tests make the RoPE-versus-AliBi ordering fragile is a concern about statistical evidence and correctness risk, not about circularity, and per the review rules it is not counted here. Overall, the manuscript is self-contained as a comparative benchmark: the conclusions reduce to the measured test-set numbers, not to the inputs by construction, so the circularity score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper contributes a benchmark sweep; its conclusions rest on the representativeness of GUE, the appropriateness of fixed hyperparameters, and the transferability of a pretrained BPE vocabulary. No new entities are introduced.

free parameters (5)

BPE vocabulary size = 4096
Chosen from prior work (Zhou et al., 2023; Mao et al., 2025), not tuned here; vocabulary transfer may favor BPE.
Hidden dimension = 768
Fixed across all model depths; may not be equally suitable for shallow and deep models.
Dropout rate = 0.1
Applied to attention and feed-forward outputs in all configurations.
Learning rate = 1e-4
AdamW learning rate fixed across all configurations; no per-configuration tuning.
AliBi slope schedule = geometric, per head
Fixed constants from Press et al. (2021); not adapted to DNA sequence lengths.

assumptions (5)

domain assumption GUE benchmark tasks are representative of DNA sequence modeling problems
The paper generalizes from six tasks to overall guidance; no evidence covers the full distribution of genomic tasks.
domain assumption Fixed hyperparameters are fair to all configurations
No tuning per tokenizer or positional encoding; differences may reflect suboptimal hyperparameters rather than intrinsic merit.
domain assumption Single training run per configuration is sufficient
No seeds or variance estimates; rankings could reverse under different initializations.
ad hoc to paper The pretrained 4096-token BPE vocabulary is appropriate for all species and tasks in GUE
BPE is taken from prior work and not retrained on each task, so vocabulary coverage may differ across species.
domain assumption Transformer encoder with [CLS] pooling is a sufficient model class
Other pooling or architectures would give different results; conclusions are scoped to this architecture.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Evaluation of Coding Schemes for Transformer-based Gene Sequence Modeling." pith.science (2026). https://pith.science/paper/JQWY3ESA

@misc{pith2026250715087,
  author       = {Pith},
  title        = {Pith review of: Evaluation of Coding Schemes for Transformer-based Gene Sequence Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JQWY3ESA}},
  note         = {Machine review of arXiv:2507.15087}
}

read the original abstract

Currently, many studies view DNA sequences as a special type of language and utilize Transformers to model them. These studies use fixed-length k-mer segmentation and BPE subword tokenization but lack a systematic evaluation to determine which is superior. We compare k-mer segmentation with k=1,3,4,5,6, a 4,096-token BPE vocabulary, and three positional encoding methods-sinusoidal, AliBi, and RoPE. Each configuration is trained from scratch in 3, 6, 12, and 24-layer Transformer encoders and evaluated on GUE benchmark dataset. In general, BPE delivers higher and more stable performance across tasks by compressing frequent motifs into variable-length tokens, reducing sequence length, and improving model generalization. RoPE excels at capturing periodic motifs and extrapolating to long sequences, while AliBi also performs well on tasks driven by local dependencies. In terms of depth, we observe significant gains when increasing layers from 3 to 12, with only marginal improvements or slight overfitting at 24 layers. This study provides practical guidance for designing tokenization and positional encoding in DNA Transformer models.

Figures

Figures reproduced from arXiv: 2507.15087 by the authors.

**Figure 1.** The overall architecture of our approach. Ji et al., 2021; Zhang et al., 2023; Zhou et al., 2023; Dalla-Torre et al., 2024; Mao et al., 2025; Yang et al., 2025). However, given the diverse sequence lengths and genomic architectures across tasks, no consensus emerges. Therefore, it is essential to explore which positional encoding type works for DNA sequence modeling with Transformer-based models. This paper systemat… view at source ↗

**Figure 3.** The results of models (using 12 layers) with different settings when using the AliBi position encoding. capturing multi-scale patterns. Fixed-length k-mer tokens show acceptable results for simple sequence tasks but lack sensitivity to crucial local motifs due to their uniform granularity and generate redundancy in long sequences. In contrast, BPE subwords adaptively encode frequent motifs with variable lengths, enh… view at source ↗

**Figure 4.** The results of models (using 12 layers) with different settings when using the Rotary position encoding. complex relative relationships and extrapolation to ultra-long sequences. Further analysis shows that as Transformer depth increases from 3 to 24 layers, performance steadily improves while gains progressively diminish. These findings suggest that future genomic language model designs should jointly consider toke… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 4 canonical work pages

[9]

Schiff, C.-H

Y. Schiff, C.-H. Kao, A. Gokaslan, T. Dao, A. Gu, and V. Kuleshov. Caduceus: Bi-directional equivariant long-range dna sequence modeling. arXiv preprint arXiv:2403.03234,

arXiv
[11]

Y. Tian, G. Chen, H. Qin, and Y. Song. Federated Chinese Word Segmentation with Global Character Associations. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021,

work page 2021
[12]

Touvron, T

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi` ere, N. Goyal, E. Hambro, F. Azhar, et al. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302.13971,

arXiv
[15]

Publisher: Proceedings of the National Academy of Sciences. D. Zhang, W. Zhang, and U. Zhao. DNAGPT: A Generalized Pre-trained Tool for DNA Sequence Analysis. arXiv preprint arXiv:2307.05628,

arXiv
[2002]

Publisher: Nature Publishing Group

ISSN 1476-4687. Publisher: Nature Publishing Group. R. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909,

arXiv
[2014]

W. Wu, Q. Li, M. Li, K. Fu, F. Feng, J. Ye, H. Xiong, and Z. Wang. Generator: A long-context generative genomic foundation model. arXiv preprint arXiv:2502.07272,

arXiv
[2015]

Z. Zhou, Y. Ji, W. Li, P. Dutta, R. Davuluri, and H. Liu. DNABERT-2: Efficient foundation model and benchmark for multi-species genome. arXiv preprint arXiv:2306.15006, 2023

arXiv 2023
[2016]

Kitaev, L

N. Kitaev, L. Kaiser, and A. Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451,

arXiv 2001

Show all 16 references

[2017]

L. Mao, Y. Tian, and Y. Song. Dnazen: Enhanced gene sequence representations via mixed granularities of coding units. arXiv preprint arXiv:2505.02206,

arXiv
[2019]

P. He, X. Liu, J. Gao, and W. Chen. Deberta: Decoding- enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654,

2006 arXiv
[2020]

H. Liu, S. Zhou, P. Chen, J. Liu, K.-G. Huo, and L. Han. Exploring genomic large language models: Bridging the gap Evaluation of Coding Schemes for Transformer-based Gene Sequence Modeling 9 between natural language and gene sequences. bioRxiv, pages 2024–02,

2024
[2021]

Loshchilov and F

I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101,

arXiv
[2022]

Press, N

O. Press, N. A. Smith, and M. Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2108.12409,

arXiv
[2023]

Number: 7 Publisher: Multidisciplinary Digital Publishing Institute

ISSN 2079-7737. Number: 7 Publisher: Multidisciplinary Digital Publishing Institute. Z. Dai, Z. Yang, Y. Yang, J. Carbonell, Q. V. Le, and R. Salakhutdinov. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860,

1901 arXiv
[2024]

Devlin, M.-W

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies...

2019
[2025]

A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505.09388,

arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [9]

Schiff, C.-H

Y. Schiff, C.-H. Kao, A. Gokaslan, T. Dao, A. Gu, and V. Kuleshov. Caduceus: Bi-directional equivariant long-range dna sequence modeling. arXiv preprint arXiv:2403.03234,

arXiv

[2] [11]

Y. Tian, G. Chen, H. Qin, and Y. Song. Federated Chinese Word Segmentation with Global Character Associations. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021,

work page 2021

[3] [12]

Touvron, T

H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi` ere, N. Goyal, E. Hambro, F. Azhar, et al. LLaMA: Open and Efficient Foundation Language Models. arXiv preprint arXiv:2302.13971,

arXiv

[4] [15]

Publisher: Proceedings of the National Academy of Sciences. D. Zhang, W. Zhang, and U. Zhao. DNAGPT: A Generalized Pre-trained Tool for DNA Sequence Analysis. arXiv preprint arXiv:2307.05628,

arXiv

[5] [2002]

Publisher: Nature Publishing Group

ISSN 1476-4687. Publisher: Nature Publishing Group. R. Sennrich, B. Haddow, and A. Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909,

arXiv

[6] [2014]

W. Wu, Q. Li, M. Li, K. Fu, F. Feng, J. Ye, H. Xiong, and Z. Wang. Generator: A long-context generative genomic foundation model. arXiv preprint arXiv:2502.07272,

arXiv

[7] [2015]

Z. Zhou, Y. Ji, W. Li, P. Dutta, R. Davuluri, and H. Liu. DNABERT-2: Efficient foundation model and benchmark for multi-species genome. arXiv preprint arXiv:2306.15006, 2023

arXiv 2023

[8] [2016]

Kitaev, L

N. Kitaev, L. Kaiser, and A. Levskaya. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451,

arXiv 2001