{"id":"78532cb5-b3fb-41a9-aaf0-a1664b055ccc","arxiv_id":"2507.15087","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"BPE tokenization and rotary position embeddings usually outperform k-mers and other positional encodings in from-scratch Transformer DNA classifiers, but the advantage is task-dependent.","lead":"This paper compares DNA tokenization methods (k-mers versus BPE subwords) and three positional encoding schemes inside Transformer models trained from scratch on genomic classification tasks. It reports that BPE and rotary position embeddings generally help, though the results vary by task and model depth.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim that RoPE is the best positional encoding is not established even by the reported point estimates: global MCC is 0.4919 for RoPE vs 0.4916 for AliBi, and AliBi beats RoPE on Virus-Covid at all depths; no seeds or significance tests are reported.","rationale":"The reader's verdict is CONDITIONAL with weakest assumption about single runs and no seeds. I agree partially and sharpen it. The reader's concern applies to both central claims, but the BPE half is actually strongly supported by the point estimates: row averages in Tables 2–4 always favor BPE by 0.01–0.06 MCC over the best k-mer. The positional-encoding half is more fragile in two independent ways: (i) the global point estimates are essentially tied (RoPE 0.4919 vs AliBi 0.4916), and (ii) per-task, AliBi beats RoPE on Virus-Covid for all BPE depths. The no-seed problem is therefore not merely a missing robustness check; it is the missing evidence that separates a genuinely useful 'use RoPE' recommendation from a null result relative to AliBi. A concrete rerun with seeds and a paired test would settle it. If the result shows no significant RoPE-over-AliBi advantage, the central claim must be revised; if it shows a stable advantage, the conditional can be lifted. This does not change the reader's verdict: CONDITIONAL remains appropriate.","tokens_in":13024,"tokens_out":14424,"duration_ms":134236,"concrete_test":"Train each of the 72 configurations (6 tokenizers × 3 positional encodings × 4 depths) with at least 5 random seeds on the GUE tasks, report mean±std MCC per cell, and run a paired significance test (e.g., Wilcoxon signed-rank over the 168 configuration-task cells) comparing RoPE vs AliBi. If the RoPE−AliBi difference is not significant at p<0.05, or if the sign flips across seeds, revise the conclusion to state that RoPE and AliBi are comparable on these tasks rather than ranking RoPE first.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Both halves of the practical recommendation are load-bearing, but they have different support. BPE vs k-mer is well supported at the point-estimate level: across all 24 rows of each table, BPE has the highest row average for every depth and every positional encoding (e.g., RoPE rows: 0.552/0.563/0.562/0.575 vs best k-mer 0.506/0.514/0.520/0.519). The RoPE claim is not. Summing Tables 2–4 over all 168 cells gives mean MCC ≈0.472 (SAPE), 0.492 (AliBi), and 0.492 (RoPE), so RoPE and AliBi are tied at the point-estimate level. The per-task pattern is also mixed: on Virus-Covid, AliBi is better than RoPE for the BPE tokenizer at all four depths (e.g., 24 layers: 0.6867 vs 0.6850; 6 layers: 0.6788 vs 0.6417), and on layer-6 and layer-12 sums AliBi exceeds RoPE. Each entry in Tables 2–4 is a single training run with fixed hyperparameters and no reported seeds (Implementation Details), so the 0.0003 global gap and even most 0.01–0.05 per-cell gaps are within plausible run-to-run noise. Thus the paper's own data do not establish 'RoPE, then AliBi, then SAPE'; they establish only that SAPE is worse and RoPE/AliBi are comparable. The actionable conclusion should be softened accordingly.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript reports a systematic comparison of DNA sequence coding schemes for Transformer classifiers. It evaluates six tokenization schemes (1-, 3-, 4-, 5-, 6-mer and a 4,096-token BPE vocabulary) and three positional encodings (sinusoidal absolute, AliBi, and RoPE) in Transformer encoders with 3, 6, 12, and 24 layers, all trained from scratch on the GUE benchmark. Performance is reported as mean MCC across datasets within each of seven tasks. The main conclusions are that BPE tokenization outperforms k-mers on average, that RoPE is the best positional encoding followed by AliBi, and that depth gains saturate around 12 layers.","tokens_in":13433,"tokens_out":12048,"duration_ms":108231,"significance":"If the conclusions were fully supported, the paper would provide useful practical guidance for DNA Transformer design and would complement existing benchmarks such as GUE with a controlled tokenizer/positional-encoding comparison. The strengths of the study are its broad coverage (6 tasks, 7 tokenization schemes, 3 positional encodings, 4 depths), the use of the public GUE benchmark, and the promised open-source code. However, the headline claims are only partially supported by the reported point estimates: the BPE advantage is an average effect that reverses on several tasks, and the RoPE-versus-AliBi ordering is essentially a tie once the relevant cells are inspected. Because every configuration is trained once with no variance estimate, the practical recommendations rest on differences that may be within run-to-run noise.","major_comments":[{"comment":"The claim that \"BPE demonstrates the best performance across different experiment settings compared with k-mer tokenization approaches\" is stronger than the tables support. While BPE has the highest row mean within each depth/positional-encoding block, the per-task/per-depth rankings frequently reverse. For example, on Human-FTP, 5-mer achieves 0.6000 versus BPE 0.5262 (Table 2, 6 layers), 6-mer achieves 0.6291 versus BPE 0.5381 (Table 3, 12 layers), and 5-mer achieves 0.5910 versus BPE 0.5500 (Table 4, 12 layers). The manuscript should therefore state that BPE is best on average, not best across settings, and should characterize the tasks on which k-mers win rather than presenting a global recommendation.","section":"Impact of Tokenization Approach (Tables 2–4)"},{"comment":"The claim that \"Rotary achieves the best performance on the tasks, followed by AliBi, with standard absolute positional encoding obtaining the worst results\" is not established by the reported point estimates. On Virus-Covid with BPE, AliBi outperforms RoPE at every depth (Table 3: 0.6681, 0.6788, 0.6644, 0.6867 versus Table 4: 0.6284, 0.6417, 0.6535, 0.6850). The aggregate difference between RoPE and AliBi is extremely small, and the additional claim about RoPE's \"extrapolation to ultra-long sequences\" is not tested by any experiment, since all GUE tasks have fixed maximum lengths and no length-extrapolation evaluation is reported. The defensible conclusion at this point is that SAPE is generally worse and RoPE/AliBi are statistically indistinguishable.","section":"Impact of Positional Encoding Strategy (Tables 3–4)"},{"comment":"Each configuration is trained once with fixed hyperparameters and no random-seed control, and every cell in Tables 2–4 is a single MCC value. Many of the differences that drive the conclusions are on the order of 0.01–0.05 MCC, and some are far smaller, so without an estimate of run-to-run variance or paired significance tests the rankings cannot be distinguished from noise. I recommend reporting means and standard deviations over at least three seeds for the central configurations, or at minimum a paired analysis of the tokenizer and positional-encoding comparisons.","section":"Implementation Details"},{"comment":"The 4,096-token BPE vocabulary is taken from previous work (Zhou et al., 2023; Mao et al., 2025), but the corpus on which it was trained is not reported. If that vocabulary is human-genome-specific, the comparisons on Mouse-TFP, Yeast-EMP, and Virus-Covid are not like-for-like with k-mer tokenization, which has no data-dependent vocabulary. The authors should report the BPE training corpus and its species composition, or train a BPE tokenizer per species, and discuss how this affects the tokenization comparison.","section":"Implementation Details"}],"minor_comments":[{"comment":"The sentence \"Our experiments span three model depths (3, 6, 12, and 24 layers)\" is internally inconsistent; four depths are evaluated.","section":"Conclusion"},{"comment":"The learning rate is written as \"110 −4\"; this should be 1e-4 or 1×10⁻⁴.","section":"Implementation Details"},{"comment":"The column header \"Yest-EMP\" should be \"Yeast-EMP\".","section":"Tables 2–4"},{"comment":"The Approach section says the paper compares \"learned absolute embeddings, relative position bias, and rotary position embeddings\", but the experiments use sinusoidal absolute positional encoding and AliBi; the terminology should be aligned throughout.","section":"Approach"},{"comment":"The robustness experiment against nucleotide alterations is introduced without any description of the alteration procedure, the number of altered test sets used, or whether the same altered sequences are used for all models. Please move this methodology into Implementation Details and describe the figure axes.","section":"Results and Analysis (Figures 2–4)"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for an NLP/benchmark venue and I saw no citation or novelty concerns. My main reservation is statistical: the headline RoPE recommendation needs more evidence before it can support practical guidance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The practical takeaway here is half right. The BPE-over-k-mer conclusion is solid at the point-estimate level: across your 168-cell grid, BPE wins the row average at every depth and every positional encoding, and the gaps are usually 0.03–0.10 MCC, which is larger than the typical per-cell differences you'd worry about. The RoPE conclusion is not. Summed over all tasks and depths, RoPE and AliBi are tied (0.492 vs 0.492), AliBi beats RoPE on Virus-Covid at every depth for BPE, and the per-cell gaps you'd need to establish a ranking are 0.01–0.05 MCC—exactly the range you'd expect from run-to-run noise when each cell is a single training run with fixed hyperparameters and no reported seeds. So the paper establishes that SAPE is worse and RoPE/AliBi are comparable; it does not establish 'RoPE, then AliBi, then SAPE.'\n\nWhat's new is the grid itself: k-mer vs BPE crossed with three positional encodings and four depths on GUE, all trained from scratch. That's a useful map, and the code is public. The writing is clear and the robustness test (Figures 2–4) is a nice addition, though it appears abruptly in the Results without being introduced in the Approach or abstract.\n\nSoft spots, in order of severity. First, no seeds, no error bars, no significance tests anywhere; the headline rankings rest on single MCC values. Second, the internal inconsistency over k=5: the abstract and tables include 5-mer, but Implementation Details says k=1,3,4,6. That needs fixing. Third, the claim that RoPE 'significantly outperforms' other schemes is not supported by the paper's own numbers; only SAPE is clearly worse. Fourth, the robustness section is under-described—three test groups are mentioned, but the construction of the substitution/deletion sets is not specified in the main text.\n\nThe citation pattern is fine; DNABERT-2 is credited for BPE, and the related work covers the space. The paper is an empirical benchmark, so circularity is not an issue. Who is this for? Practitioners building DNA Transformers who want a default tokenizer and positional encoding. It gives them a reasonable starting point (BPE + RoPE or AliBi, ~12 layers), but they should treat the finer ranking with caution until multi-seed results appear. I'd send it to review: the grid is valuable, the claims are checkable, and the fixes are straightforward—add seeds or soften the claims, reconcile the k inconsistency, and report the robustness setup properly.","headline":"The BPE-vs-k-mer result is well supported at the point-estimate level, but the paper overclaims the RoPE-vs-AliBi ranking from single runs with no error bars; still worth refereeing.","tokens_in":13936,"tokens_out":1211,"would_cite":true,"duration_ms":14380,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For Transformer DNA models, BPE tokenization and rotary position embeddings come out on top.","keywords":["DNA sequence modeling","tokenization","k-mer segmentation","byte pair encoding (BPE)","positional encoding","rotary position embeddings","Transformer encoder","genomic benchmark"],"falsifier":"Train the same configurations (say BPE and the best k-mer, each with RoPE and AliBi, at 12 layers) under five or more random seeds. If the seed-to-seed spread of MCC is comparable to the 0.01–0.05 gaps in Tables 2–4, the paper's claimed ordering is not established; if the ordering persists across seeds, the design guidance holds.","tokens_in":12833,"feed_emoji":"🧬","tokens_out":8617,"duration_ms":83278,"temperature":0.7,"pith_summary":"This paper asks which standard coding choices really help when a Transformer is trained on raw DNA sequences. It compares fixed-length k-mer tokenization at k=1,3,4,5,6 against a 4,096-token byte pair encoding (BPE) vocabulary, and sinusoidal, AliBi, and rotary (RoPE) positional encodings, training every combination from scratch at 3, 6, 12, and 24 layers on six tasks from the GUE benchmark. Its central claim is that BPE tokenization is the better default—it compresses frequent motifs into variable-length subwords, shortens sequences, and stays more stable under small nucleotide changes—while RoPE is the better positional encoding because it captures periodic signals and extrapolates to longer inputs, with AliBi second and absolute sinusoidal encoding last. The authors also report that depth helps mainly up to 12 layers and gives only marginal returns at 24. A sympathetic reader would take away concrete defaults for building DNA Transformer classifiers, and a caution that the margins separating the top choices are often small.","feed_headline":"BPE beats k-mers; RoPE leads in DNA Transformer coding","feed_subtitle":"Six-task sweep favors variable-length tokens and rotary positions for genomic models.","key_machinery":"The comparison grid is the machinery: every tokenizer is crossed with every positional encoding and four depths, and all models are trained from scratch so that differences are attributable to coding scheme rather than pretraining. The two mechanisms doing the explanatory work are BPE tokenization—a greedy algorithm that starts from the four nucleotides and repeatedly merges the most frequent adjacent pair, yielding variable-length subword tokens that compress frequent motifs—and RoPE, which rotates adjacent pairs of query and key dimensions by position-dependent angles so dot-product attention naturally encodes relative distance as a periodic function. AliBi, the third positional scheme, injects a fixed linear distance penalty into attention scores. The study's yardstick is Matthews correlation coefficient on six GUE classification tasks spanning different species, sequence lengths, and regulatory objectives.","core_discovery":"The intended contribution is an empirical ranking of coding schemes for Transformer-based DNA sequence classification. On tokenization, BPE with a 4,096-token vocabulary is presented as the best overall choice: it outperforms every fixed k-mer setting in most configurations and shows the smallest performance degradation when nucleotides at sequence ends are substituted or deleted; the reason offered is that BPE subwords capture biological motifs at multiple scales while shortening the input. On positional encoding, RoPE is presented as the best overall choice, with AliBi second and sinusoidal absolute embeddings last, because rotation-based relative position information is better at recognizing periodic motifs and extrapolating beyond training length. On depth, 3-to-12-layer growth clearly improves MCC, whereas 24 layers yield only small gains and occasional overfitting. The paper frames these as practical design guidelines for future genomic language models.","pith_inferences":["A direct head-to-head that the paper does not run would pair the two winners, BPE plus RoPE; scanning the result tables, this combination is at or near the top on nearly every task and depth, so it is the likely practical recommendation even though the paper does not single it out.","The mutation-robustness result suggests BPE subwords may help in variant-aware genomics, where test reads differ from reference sequences by single-nucleotide variants or indels; this application is not explored here.","All rankings rest on single runs with fixed hyperparameters, so a multi-seed replication with significance intervals is the natural next experiment before treating the 0.01–0.05 MCC gaps as design rules.","RoPE's extrapolation ability hints that models trained on short DNA windows could be applied to full-length sequences, but the paper does not test long-sequence extrapolation directly."],"forward_implications":["New DNA Transformer models should default to BPE tokenization over fixed k-mer segmentation, since BPE wins or ties in most configurations and degrades least under small sequence mutations.","RoPE should be the default positional encoding for genomic tasks, with AliBi a reasonable alternative when local-dependency bias is wanted and sinusoidal embeddings used only when motif spacing is fixed.","When k-mers are unavoidable, the best k is task-dependent: small k suits trinucleotide-like splice signals and larger k suits longer epigenetic motifs, so the choice must match the biology.","Depth scaling advice from the results: increase layers to around 12 for clear gains, and treat 24-layer models as an expensive option with small marginal benefit."],"supporting_citations":[{"why":"Supplies the byte pair encoding algorithm whose learned subword vocabulary is the BPE tokenizer under test.","marker":"(Sennrich et al., 2015)"},{"why":"Provides the GUE benchmark tasks and the 4,096-token DNA BPE vocabulary reused in the experiments.","marker":"(Zhou et al., 2023)"},{"why":"Contributes the BPE tokenizer and DNA-modeling setup the paper trains from scratch.","marker":"(Mao et al., 2025)"},{"why":"Defines the Transformer encoder architecture that all configurations are built on.","marker":"(Vaswani et al., 2017)"},{"why":"Introduces AliBi, the linear-bias positional encoding that is one of the three compared schemes.","marker":"(Press et al., 2021)"},{"why":"Introduces RoPE, the rotary positional encoding the paper reports as best overall.","marker":"(Su et al., 2024)"},{"why":"Establishes MCC as the evaluation metric for DNA language models, the metric used for all rankings.","marker":"(Ji et al., 2021)"}],"fun_headline_variants":["BPE beats k-mers; RoPE leads in DNA Transformers","Variable-length tokens and rotary positions win for DNA","DNA modeling: BPE and RoPE outdo fixed k-mers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole ranking depends on each configuration being trained once with one fixed hyperparameter set, so the 0.01–0.05 MCC differences between BPE and k-mers, and between RoPE and AliBi, could disappear under run-to-run variation.","fun_headline_variants_meta":{"raw":{"variants":["BPE beats k-mers; RoPE leads in DNA Transformers","Variable-length tokens and rotary positions win for DNA","DNA modeling: BPE and RoPE outdo fixed k-mers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000315,"raw_usage":{"total_tokens":1766,"prompt_tokens":906,"completion_tokens":860,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":522,"completion_tokens_details":{"reasoning_tokens":803}},"tokens_in":522,"tokens_out":860,"duration_ms":8849,"temperature":1.0,"reasoning_tokens":803,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:41:06.098805+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same configurations (say BPE and the best k-mer, each with RoPE and AliBi, at 12 layers) under five or more random seeds. If the seed-to-seed spread of MCC is comparable to the 0.01–0.05 gaps in Tables 2–4, the paper's claimed ordering is not established; if the ordering persists across seeds, the design guidance holds.","supporting_citations":[],"review_version":1}