Pith. sign in

REVIEW 3 major objections 5 minor 21 references

Improving Genomic Models via Task-Specific Self-Pretraining

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims task-specific self-pretraining matches or beats from-scratch training under equal compute.

desk verdict Task-specific self-pretraining in genomics is worth a look, but the two central claims—task-specific and compute-matched—are not actually established by the experiments. read the letter →

arxiv 2506.17766 v1 pith:VIRSTAIB submitted 2025-06-21 q-bio.GN

Sohan Mupparapu , Parameswari Krishnamurthy , Ratish Puduppully This is my paper · ORCID
classification q-bio.GN
keywords DNAlanguagemodelself-pretraininggenomicbenchmarkgenefindingCpGmethylationmaskedmodelingconditionalrandomfieldsampleefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that DNA language models can be improved cheaply by pretraining on unlabeled sequences from the downstream task itself, instead of on the full genome. Using the BEND benchmark, a residual CNN pretrained with masked language modeling on gene-finding sequences matches or beats an identical model trained from scratch under the same compute budget on all four tasks. The largest gains appear on gene finding (MCC 0.38 to 0.50) and CpG methylation (AUROC 0.89 to 0.94), and a linear-chain CRF decoder raises gene finding to 0.64. The authors argue that task-specific self-pretraining is a compute-efficient route to stronger supervised baselines, especially when labeled data are scarce.

What carries the argument

The load-bearing mechanism is masked language modeling on task-specific unlabeled DNA before supervised fine-tuning. The encoder is a 30-layer residual CNN with kernel size 9 and doubling dilation (reset every six layers, capped at 32); during pretraining an MLM head predicts 15% masked tokens with the standard 80/10/10 replacement strategy. This pretrained encoder is then reused across all four tasks, with a two-layer CNN plus linear head for each task, and optionally a linear-chain CRF for gene finding.

What would settle it

A direct test would train the same encoder on randomly selected human genomic sequences of matching length and nucleotide composition, keeping the compute budget fixed. If the random-pretrained model matches the gene-finding-pretrained model on CpG methylation and chromatin tasks, then the paper's task-specificity claim is not supported. Alternatively, extending the scratch gene-finding model to 50 epochs with identical total compute and seeing it reach MCC 0.50 would refute the plateau result.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that self-pretraining on task-specific unlabeled DNA sequences works as an inductive prior: a 30-layer dilated residual CNN, after masked language modeling on gene-finding sequences, fine-tuned on four BEND tasks, reaches or exceeds the performance of the same architecture trained from scratch with identical compute on every task. On CpG methylation, a 25%-labeled self-pretrained model outperforms a fully labeled scratch model, showing sample efficiency. The authors present this as evidence that genome-scale pretraining is not required to build strong supervised baselines.

Load-bearing premise

The claim that self-pretraining is task-specific rests on using gene-finding sequences as the pretraining corpus for all four tasks, but the paper runs no control pretraining on random or task-matched sequences, so the gains could come from pretraining on any long genomic sequence rather than from the task-specific choice.

Editorial extensions

If this is right

  • Under matched compute, a self-pretrained CNN outperforms scratch training on gene finding (0.50 vs 0.38 MCC) and CpG methylation (0.94 vs 0.89 AUROC).
  • Self-pretraining delivers sample efficiency: on CpG methylation, 25% of labeled data with self-pretraining outperforms 100% of labeled data from scratch (AUROC 0.92 vs 0.89).
  • Adding a linear-chain CRF over the fine-tuned encoder improves gene finding further, from 0.50 to 0.64 MCC, by enforcing global exon-intron transition constraints.
  • Tasks that are already well solved by supervised training, like chromatin accessibility and histone modification, show little or no pretraining gain, suggesting diminishing returns for pretraining on saturated tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the gains are driven by matching the pretraining distribution to the task, then pretraining on task-specific sequences should produce targeted improvements on each task; the paper's reuse of one gene-finding corpus across all tasks makes this an open question.
  • Self-pretraining could be stacked with genome-scale pretraining: a genome-pretrained model fine-tuned after task-specific self-pretraining may offer a stronger baseline than either alone.
  • The CRF gain hints that structured decoding, not just representation learning, matters for gene-structure tasks; combining self-pretraining with more expressive structured decoders could push closer to expert tools like AUGUSTUS.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes task-specific self-pretraining (SPT) for genomic sequence models: instead of large-scale genome pretraining, a residual CNN is first trained with masked language modeling on unlabeled task-related sequences, then fine-tuned on the downstream task. Using four BEND benchmark tasks, the authors report that SPT matches or exceeds a from-scratch baseline on gene finding (0.50 vs 0.38 MCC) and CpG methylation (0.94 vs 0.89 AUROC), with comparable performance on chromatin accessibility and histone modification. Adding a CRF layer further improves gene finding to 0.64 MCC. The paper concludes that task-specific self-pretraining is a compute-efficient strategy for stronger supervised baselines.

Significance. If the central claim held, the paper would offer a practical and low-cost recipe for improving supervised genomic models, with implications for labs without access to genome-scale pretraining infrastructure. The manuscript has several strengths: it uses a public benchmark (BEND), releases code, reports honest negative results on two tasks where SPT does not help, and includes an extended-training control for the gene-finding task. However, the headline claim rests on two load-bearing comparisons that are not currently established: that the gains are due to task-specific pretraining, and that the comparison is compute-matched. The novelty of applying 'self-pretraining on downstream data' to genomics is plausible but needs stronger experimental isolation.

major comments (3)
  1. [Section 4.2, Table 1, Figure 1] The claim in the abstract and conclusion that SPT matches or exceeds scratch 'under identical compute' is not supported by the training protocol. Section 4.2 states that pretraining uses 4,780 gene-finding samples over 20 epochs, followed by 10 epochs of fine-tuning on gene finding and 5 epochs on the other tasks; the scratch baseline receives only the fine-tuning epochs (10 or 5) and no pretraining. The 50-epoch scratch run in Figure 1 still does not include the computational cost of the 20-epoch pretraining phase, and no GPU hours or FLOP counts are reported anywhere. To substantiate the identical-compute claim, the paper needs either (a) an actual compute-matched protocol, e.g., reducing pretraining+fine-tuning epochs so total FLOPs are equal, or (b) explicit FLOP/GPU-hour measurements for both pipelines. Without this, the reported gains on gene finding and CpG methylation (0.50 vs 0.38 and 0.94 vs 0.89) cannot be attributed to compute efficiency.
  2. [Section 4.3, Table 1] The claim that gains come from task-specific self-pretraining is not isolated because no control pretraining corpus is used. Section 4.3 states that gene-finding sequences (1,433–14,000 bp) are used for pretraining and the resulting encoder is reused across all tasks, including the three 512-bp window tasks. For chromatin, histone, and CpG methylation, the task-matched unlabeled distribution is 512-bp genomic windows, not long gene structures; without control pretraining on task-matched windows, random genomic sequence, or GC-matched sequence, the observed CpG gain could equally be attributed to generic long-sequence pretraining or to a gene-region bias in the corpus. Similarly, for gene finding itself, the unlabeled pretraining corpus is exactly the training input distribution, so the comparison does not test whether task-specificity matters. The authors should add control pretraining corpora to isolate the mechanism.
  3. [Table 1, Figures 1–3] All reported performance numbers appear to come from single runs with no error bars, no number of random seeds, and no statistical significance testing. Given that the headline improvements are 0.12 MCC on gene finding and 0.05 AUROC on CpG methylation, multiple seeds are needed to establish that SPT 'consistently' beats scratch. Figure 3, which compares 10%, 25%, and 100% labels, likewise shows single point estimates despite being central to the sample-efficiency claim. At minimum, report mean and standard deviation over at least three seeds, and ideally a paired test across the same train/validation splits.
minor comments (5)
  1. [Abstract and Section 5] The phrase 'identical compute' is used in the abstract and Section 5, but the paper never measures compute; it reports epochs only. Please replace this wording with a precise description of the training budget or provide actual FLOP/GPU-hour measurements.
  2. [Appendix Table 3] The CpG methylation row lists 959,039 total samples but a train/validation split of 109,717/106,227 that sums to 215,944; the numbers are inconsistent and should be corrected or clarified.
  3. [Figure 2] The y-axis is restricted to 0.84–0.94, which visually exaggerates the difference between SPT and scratch; adding error bars or extending the axis to the full AUROC range would make the figure more informative.
  4. [Section 4.1] The model is described as a DNALM in the title and introduction, but it is a residual CNN with an MLM head, not a language model in the usual sense; consider using a more neutral term such as 'self-pretrained encoder' to avoid overclaiming.
  5. [Section 4.1] The masking probability is written as 'pmlm = 0 .15' with a space; this is a typesetting issue that should be fixed.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical benchmark comparison, and no derivation, fit, or self-citation chain reduces its claims to its inputs.

full rationale

This paper contains no derivation that could be circular in the formal sense. The central claim is an empirical comparison between models trained from scratch and models trained with task-specific self-pretraining on the BEND benchmark. The pretraining objective is masked language modeling on unlabeled sequences, and the downstream evaluations are held-out test metrics from a fixed benchmark. No fitted parameter is renamed as a prediction, no equation is shown to be equal to another by construction, and no load-bearing result is imported from the authors' own prior work. The paper cites external work (e.g., Amos et al. 2024, Krishna et al. 2023, Vishniakov et al. 2024) for motivation and comparison, but does not rely on those citations to define its own outcome. The main weaknesses are experimental: the 'identical compute' comparison is not strictly compute-matched because SPT receives 20 epochs of masked-language-model pretraining before fine-tuning, and the claim of task-specific benefit is not isolated by a control pretraining corpus. These are validity concerns about experimental design, not circularity. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on benchmark trust, the choice of MLM as a pretraining objective, and the transferability of gene-finding sequences to other tasks. None of these are fitted numbers; they are assumptions inherited from prior benchmarks and design choices.

assumptions (3)
  • domain assumption The BEND benchmark's task definitions, data splits, and evaluation metrics are reliable and correctly implemented.
    Section 3 and Appendix Table 3 rely entirely on BEND for task descriptions and splits; the paper does not independently validate them.
  • domain assumption Masked language modeling on unlabeled DNA sequences is an effective self-supervised pretraining objective for the downstream tasks.
    Section 4.1 uses MLM with 15% masking; the paper does not compare to other objectives such as contrastive or next-sentence prediction.
  • domain assumption Pretraining on gene-finding sequences transfers positively to chromatin accessibility, histone modification, and CpG methylation tasks.
    This is the load-bearing transfer assumption in Section 4.3; no control experiment with random DNA or task-matched unlabeled data is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Genomic Models via Task-Specific Self-Pretraining." pith.science (2026). https://pith.science/paper/VIRSTAIB

@misc{pith2026250617766,
  author       = {Pith},
  title        = {Pith review of: Improving Genomic Models via Task-Specific Self-Pretraining},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VIRSTAIB}},
  note         = {Machine review of arXiv:2506.17766}
}
read the original abstract

Pretraining DNA language models (DNALMs) on the full human genome is resource-intensive, yet often considered necessary for strong downstream performance. Inspired by recent findings in NLP and long-context modeling, we explore an alternative: self-pretraining on task-specific, unlabeled data. Using the BEND benchmark, we show that DNALMs trained with self-pretraining match or exceed the performance of models trained from scratch under identical compute. While genome-scale pretraining may still offer higher absolute performance, task-specific self-pretraining provides a practical and compute-efficient strategy for building stronger supervised baselines.

Figures

Figures reproduced from arXiv: 2506.17766 by the authors.

Figure 1
Figure 1. Validation MCC on gene finding. The scratch model plateaus early and does not match the self-pretrained model. 1.0 1.5 2.0 2.5 3.0 3.5 4.0 4.5 5.0 Epoch 0.84 0.86 0.88 0.90 0.92 0.94 Validation AUROC Scratch (ResNet) Self-pretrained [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Validation AUROC on the CpG methylation task over training epochs. the performance of the SPT model. A dashed horizontal line indicates the best validation MCC achieved by the SPT model, highlighting the efficiency of self-pretraining over prolonged supervised training. We now focus on CpG methylation, where SPT shows strong gains as well. Training Dynamics [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Maximum validation AUROC on CpG methylation using different fractions of labeled data. The 25% self-pretrained (SPT) model outperforms the 100% scratch model, while the 10% SPT model performs slightly below. value of self-pretraining for enabling accurate predictions in low-data regimes. Comparison to Benchmark Models. We also compare our SPT model to expert methods and genome-pretrained DNA language models from the… view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 11 canonical work pages

  1. [1]

    Never train from scratch: Fair comparison of long-sequence models requires data-driven priors

    Amos, I., Berant, J., and Gupta, A. Never train from scratch: Fair comparison of long-sequence models requires data-driven priors. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=PdaPky8MUn

    work page 2024

  2. [2]

    The nucleotide transformer: Building and evaluating robust foundation models for human genomics

    Dalla-Torre, H., Gonzalez, L., Romero, J., et al. The nucleotide transformer: Building and evaluating robust foundation models for human genomics. Nature Methods, 21: 0 123--134, 2024. doi:10.1038/s41592-024-02523-z. URL https://www.nature.com/articles/s41592-024-02523-z

    doi:10.1038/s41592-024-02523-z 2024

  3. [3]

    An integrated encyclopedia of dna elements in the human genome

    ENCODE Project Consortium . An integrated encyclopedia of dna elements in the human genome. Nature, 489 0 (7414): 0 57--74, September 2012. ISSN 1476-4687. doi:10.1038/nature11247

    doi:10.1038/nature11247 2012

  4. [4]

    Gena-lm: A family of open-source foundational dna language models for long sequences

    Fishman, V., Kuratov, Y., Shmelev, A., Petrov, M., Penzar, D., Shepelin, D., Chekanov, N., Kardymon, O., and Burtsev, M. Gena-lm: A family of open-source foundational dna language models for long sequences. bioRxiv, 2023. doi:10.1101/2023.06.12.544594. URL https://www.biorxiv.org/content/10.1101/2023.06.12.544594v1

    doi:10.1101/2023.06.12.544594 2023

  5. [5]

    E., Mudge, J

    Frankish, A., Diekhans, M., Jungreis, I., Lagarde, J., Loveland, J. E., Mudge, J. M., Sisu, C., Wright, J. C., Armstrong, J., Barnes, I., Berry, A., Bignell, A., Boix, C., Carbonell Sala, S., Cunningham, F., Di Domenico, T., Donaldson, S., Fiddes, I. T., García Girón, C., Gonzalez, J. M., Grego, T., Hardy, M., Hourlier, T., Howe, K. L., Hunt, T., Izuogu, ...

    work page 2021

  6. [6]

    Comparing two k-category assignments by a k-category correlation coefficient

    Gorodkin, J. Comparing two k-category assignments by a k-category correlation coefficient. Computational Biology and Chemistry, 28 0 (5-6): 0 367--374, December 2004. ISSN 1476-9271. doi:10.1016/j.compbiolchem.2004.09.006. URL https://www.sciencedirect.com/science/article/pii/S1476927104000799

    doi:10.1016/j.compbiolchem.2004.09.006 2004

  7. [7]

    Efficiently modeling long sequences with structured state spaces

    Gu, A., Goel, K., and Re, C. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=uYLFoz1vlAC

    2022

  8. [8]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 770--778, 2016

    work page 2016

Show all 21 references
  1. [9]

    Ji, Y., Zhou, Z., Liu, H., and Davuluri, R. V. Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language in genome. Bioinformatics, 37 0 (15): 0 2112--2120, 2021. doi:10.1093/bioinformatics/btab083. URL https://academic.oup.com/bioinfo...

    2021 doi

  2. [10]

    R., Snoek, J., and Rinn, J

    Kelley, D. R., Snoek, J., and Rinn, J. L. Basset: learning the regulatory code of the accessible genome with deep convolutional neural networks. Genome Research, 26 0 (7): 0 990--999, July 2016. ISSN 1549-5469. doi:10.1101/gr.200535.115

    2016 doi

  3. [11]

    Downstream datasets make surprisingly good pretraining corpora

    Krishna, K., Garg, S., Bigham, J., and Lipton, Z. Downstream datasets make surprisingly good pretraining corpora. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap...

    2023 doi

  4. [12]

    Conditional random fields: Probabilistic models for segmenting and labeling sequence data

    Lafferty, J., McCallum, A., and Pereira, F. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In ICML, pp.\ 282--289, 2001

    2001

  5. [13]

    Neural architectures for named entity recognition

    Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., and Dyer, C. Neural architectures for named entity recognition. In Knight, K., Nenkova, A., and Rambow, O. (eds.), Proceedings of the 2016 Conference of the North A merican Chapter of the Association for Computational...

    2016 doi

  6. [14]

    I., Teufel, F., Horlacher, M., Madsen, D., Pultz, D., Winther, O., and Boomsma, W

    Marin, F. I., Teufel, F., Horlacher, M., Madsen, D., Pultz, D., Winther, O., and Boomsma, W. BEND : Benchmarking DNA language models on biologically meaningful tasks. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id...

    2024

  7. [15]

    W., Sykes, C

    Nguyen, E., Poli, M., Faizi, M., Thomas, A. W., Sykes, C. B., Wornow, M., Patel, A., Rabideau, C., Massaroli, S., Bengio, Y., Ermon, S., Baccus, S. A., and R \'e , C. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution. In Advances in Neural Informat...

    2023

  8. [16]

    Rabiner, L. R. A tutorial on hidden markov models and selected applications in speech recognition. Proceedings of the IEEE, 77 0 (2): 0 257--286, 1989

    1989

  9. [17]

    Caduceus: Bi-directional equivariant long-range dna sequence modeling

    Schiff, Y., Kao, C.-H., Gokaslan, A., Dao, T., Gu, A., and Kuleshov, V. Caduceus: Bi-directional equivariant long-range dna sequence modeling. arXiv preprint arXiv:2403.03234, 2024. URL https://arxiv.org/abs/2403.03234

    2024 arXiv

  10. [18]

    and Waack, S

    Stanke, M. and Waack, S. Gene prediction with a hidden markov model and a new intron submodel. Bioinformatics (Oxford, England), 19 Suppl 2: 0 ii215--ii225, October 2003. ISSN 1367-4811. doi:10.1093/bioinformatics/btg1080

    2003 doi

  11. [19]

    Long range arena : A benchmark for efficient transformers

    Tay, Y., Dehghani, M., Abnar, S., Shen, Y., Bahri, D., Pham, P., Rao, J., Yang, L., Ruder, S., and Metzler, D. Long range arena : A benchmark for efficient transformers. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=qVyeW-grC2k

    2021

  12. [20]

    K., Pimentel, M

    Vishniakov, K., Viswanathan, K., Medvedev, A., Kanithi, P. K., Pimentel, M. A., Rajan, R., and Khan, S. Genomic foundationless models: Pretraining does not promise performance. bioRxiv, 2024. doi:10.1101/2024.12.18.628606. URL https://doi.org/10.1101/2024.12.18.628606

    2024 doi

  13. [21]

    Specialized foundation models struggle to beat supervised baselines

    Xu, Z., Gupta, R., Cheng, W., Shen, A., Shen, J., Talwalkar, A., and Khodak, M. Specialized foundation models struggle to beat supervised baselines. In International Conference on Learning Representations (ICLR), 2025. URL https://openreview.net/forum?id=J7NmKXF5Ll

    2025

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.