{"id":"be5fac61-4203-44e8-9d55-33a35cf75221","arxiv_id":"2506.17766","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Pretraining a small DNA model on unlabeled task-related sequences improves gene finding and other BEND tasks compared to training from scratch, with less data than genome-scale pretraining.","lead":"This paper tests whether pretraining a DNA model on unlabeled gene-finding sequences, instead of the full human genome, improves downstream genomic predictions on the BEND benchmark. It reports that this self-pretraining matches or beats models trained from scratch on several tasks while using less data and compute than genome-scale pretraining.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Task-specific and compute-matched claims are not isolable: no control pretraining corpus and no compute-matched protocol.","rationale":"The reader identified the same weakest assumption: using gene-finding sequences as the pretraining corpus for all tasks, with no control pretraining on random or task-matched sequences, means the attribution of gains to task-specific self-pretraining is not isolated. My read agrees and adds that the 'identical compute' wording is also unsupported by the methods as written, since SPT includes pretraining epochs that scratch does not receive. These are load-bearing because they attach to the two distinctive parts of the central claim: task specificity and compute matching. The paper does provide a useful baseline and releases code, but the empirical case is incomplete rather than wrong. A controlled pretraining-corpus comparison with matched compute and multiple seeds would settle whether the proposed mechanism is real. I therefore do not move the reader's conditional verdict.","tokens_in":7514,"tokens_out":6716,"duration_ms":67420,"concrete_test":"Using the released repository, hold architecture and optimizer fixed and compare four conditions on gene finding and CpG methylation, each with 5 random seeds and equal total FLOPs/optimizer steps: (A) no pretraining; (B) SPT with the paper's gene-finding corpus; (C) MLM pretraining on unlabeled 512-bp windows sampled from the same loci as the downstream task training sets; (D) MLM pretraining on length- and GC-matched random human genomic fragments. The decisive comparison is B vs C vs D: if C or D matches B and both beat A, the gains are not attributable to task-specificity; if C does not beat A, the method as applied to the 512-bp tasks lacks empirical support. A separate scratch run trained for the same total FLOPs as B would test the 'identical compute' wording.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two conjuncts: gains come from *task-specific* self-pretraining, and SPT wins under *identical compute*. Neither is established. Section 4.3 says gene-finding sequences are used for pretraining and the encoder is reused across all four tasks. For the three 512-bp window tasks (chromatin, histone, CpG), the task-specific unlabeled distribution is 512-bp genomic windows, not 1,433-14,000 bp gene structures. There is no control pretraining on task-matched windows or on random/GC-matched human sequence, so the 0.94 vs 0.89 CpG gain (Table 1) could equally come from generic long-sequence pretraining or gene-region bias rather than from task-specific self-pretraining. In addition, Table 1's comparison is not compute-matched: SPT adds 20 epochs of MLM on 4,780 sequences before 5-10 epochs of fine-tuning, while scratch receives only 5-10 epochs; the 50-epoch scratch run in Figure 1 does not include the pretraining FLOPs, and all reported numbers are single-seed. Both load-bearing parts of the headline claim therefore rest on comparisons that do not isolate the proposed mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes task-specific self-pretraining (SPT) for genomic sequence models: instead of large-scale genome pretraining, a residual CNN is first trained with masked language modeling on unlabeled task-related sequences, then fine-tuned on the downstream task. Using four BEND benchmark tasks, the authors report that SPT matches or exceeds a from-scratch baseline on gene finding (0.50 vs 0.38 MCC) and CpG methylation (0.94 vs 0.89 AUROC), with comparable performance on chromatin accessibility and histone modification. Adding a CRF layer further improves gene finding to 0.64 MCC. The paper concludes that task-specific self-pretraining is a compute-efficient strategy for stronger supervised baselines.","tokens_in":7749,"tokens_out":3023,"duration_ms":30919,"significance":"If the central claim held, the paper would offer a practical and low-cost recipe for improving supervised genomic models, with implications for labs without access to genome-scale pretraining infrastructure. The manuscript has several strengths: it uses a public benchmark (BEND), releases code, reports honest negative results on two tasks where SPT does not help, and includes an extended-training control for the gene-finding task. However, the headline claim rests on two load-bearing comparisons that are not currently established: that the gains are due to task-specific pretraining, and that the comparison is compute-matched. The novelty of applying 'self-pretraining on downstream data' to genomics is plausible but needs stronger experimental isolation.","major_comments":[{"comment":"The claim in the abstract and conclusion that SPT matches or exceeds scratch 'under identical compute' is not supported by the training protocol. Section 4.2 states that pretraining uses 4,780 gene-finding samples over 20 epochs, followed by 10 epochs of fine-tuning on gene finding and 5 epochs on the other tasks; the scratch baseline receives only the fine-tuning epochs (10 or 5) and no pretraining. The 50-epoch scratch run in Figure 1 still does not include the computational cost of the 20-epoch pretraining phase, and no GPU hours or FLOP counts are reported anywhere. To substantiate the identical-compute claim, the paper needs either (a) an actual compute-matched protocol, e.g., reducing pretraining+fine-tuning epochs so total FLOPs are equal, or (b) explicit FLOP/GPU-hour measurements for both pipelines. Without this, the reported gains on gene finding and CpG methylation (0.50 vs 0.38 and 0.94 vs 0.89) cannot be attributed to compute efficiency.","section":"Section 4.2, Table 1, Figure 1"},{"comment":"The claim that gains come from task-specific self-pretraining is not isolated because no control pretraining corpus is used. Section 4.3 states that gene-finding sequences (1,433–14,000 bp) are used for pretraining and the resulting encoder is reused across all tasks, including the three 512-bp window tasks. For chromatin, histone, and CpG methylation, the task-matched unlabeled distribution is 512-bp genomic windows, not long gene structures; without control pretraining on task-matched windows, random genomic sequence, or GC-matched sequence, the observed CpG gain could equally be attributed to generic long-sequence pretraining or to a gene-region bias in the corpus. Similarly, for gene finding itself, the unlabeled pretraining corpus is exactly the training input distribution, so the comparison does not test whether task-specificity matters. The authors should add control pretraining corpora to isolate the mechanism.","section":"Section 4.3, Table 1"},{"comment":"All reported performance numbers appear to come from single runs with no error bars, no number of random seeds, and no statistical significance testing. Given that the headline improvements are 0.12 MCC on gene finding and 0.05 AUROC on CpG methylation, multiple seeds are needed to establish that SPT 'consistently' beats scratch. Figure 3, which compares 10%, 25%, and 100% labels, likewise shows single point estimates despite being central to the sample-efficiency claim. At minimum, report mean and standard deviation over at least three seeds, and ideally a paired test across the same train/validation splits.","section":"Table 1, Figures 1–3"}],"minor_comments":[{"comment":"The phrase 'identical compute' is used in the abstract and Section 5, but the paper never measures compute; it reports epochs only. Please replace this wording with a precise description of the training budget or provide actual FLOP/GPU-hour measurements.","section":"Abstract and Section 5"},{"comment":"The CpG methylation row lists 959,039 total samples but a train/validation split of 109,717/106,227 that sums to 215,944; the numbers are inconsistent and should be corrected or clarified.","section":"Appendix Table 3"},{"comment":"The y-axis is restricted to 0.84–0.94, which visually exaggerates the difference between SPT and scratch; adding error bars or extending the axis to the full AUROC range would make the figure more informative.","section":"Figure 2"},{"comment":"The model is described as a DNALM in the title and introduction, but it is a residual CNN with an MLM head, not a language model in the usual sense; consider using a more neutral term such as 'self-pretrained encoder' to avoid overclaiming.","section":"Section 4.1"},{"comment":"The masking probability is written as 'pmlm = 0 .15' with a space; this is a typesetting issue that should be fixed.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The core idea is timely and the benchmark choice is appropriate, but the two conjuncts of the headline claim—task-specific benefit and compute-matched advantage—both need additional experiments. The fixes are feasible within the paper's scope: add control pretraining corpora, measure or equalize compute, and report multi-seed variance. I recommend major revision rather than rejection because the central hypothesis is defensible and the required experiments are well-defined."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this applies a known self-pretraining trick from NLP to genomic DNA and reports real numbers on BEND. The results are suggestive, but the abstract's two load-bearing claims—task-specific, identical compute—are not actually supported by the experiments as written.\n\nWhat's new and good: First application of the 'downstream datasets make good pretraining corpora' idea to DNA with a straightforward ResNet encoder. The empirical sweep on four BEND tasks is useful, and the gene finding improvement (0.38 to 0.50 MCC; 0.64 with CRF) is nontrivial. The sample efficiency plot on CpG is a nice addition, and releasing code and models is the right move. The related work correctly situates itself next to Krishna et al. and Amos et al., so the novelty is honestly framed as an application.\n\nSoft spots, in order of importance. First, the 'identical compute' claim is contradicted by the paper's own protocol: SPT gets 20 epochs of MLM pretraining plus fine-tuning, while scratch gets only fine-tuning. The 50-epoch scratch run in Figure 1 still omits the pretraining FLOPs. No GPU hours or FLOP counts appear anywhere. So that part of the headline is simply not established. Second, the 'task-specific' claim is not isolated. The pretraining corpus is the gene-finding sequences for all four tasks; the three 512-bp window tasks never get a task-matched control. A CpG gain could come from generic long-sequence pretraining or gene-region bias just as easily. A control with random/GC-matched windows would settle it. Third, all numbers are single-seed, and there are small internal inconsistencies: Table 1 lists scratch at 0.38 MCC while Table 2 shows the fully supervised ResNet at 0.46, and the appendix data splits for histone and CpG don't add to the stated totals. These are minor but they make the reader re-check everything.\n\nNone of this kills the idea. The central direction is plausible and worth testing rigorously. But the paper as written overstates what it shows. The right fix is a revision with compute-matched budgets, a control pretraining corpus, and multiple seeds.\n\nWho's this for? Anyone building cheap supervised baselines in genomics, and anyone who wants to see whether the NLP self-pretraining result transfers. It deserves a serious referee, but the referee should send it back for the control experiments. My recommendation: send to peer review, and require the compute-matched and task-matched controls before publication.","headline":"Task-specific self-pretraining in genomics is worth a look, but the two central claims—task-specific and compute-matched—are not actually established by the experiments.","tokens_in":8264,"tokens_out":2497,"would_cite":false,"duration_ms":23875,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims task-specific self-pretraining matches or beats from-scratch training under equal compute.","keywords":["DNA language model","self-pretraining","genomic benchmark","gene finding","CpG methylation","masked language modeling","conditional random field","sample efficiency"],"falsifier":"A direct test would train the same encoder on randomly selected human genomic sequences of matching length and nucleotide composition, keeping the compute budget fixed. If the random-pretrained model matches the gene-finding-pretrained model on CpG methylation and chromatin tasks, then the paper's task-specificity claim is not supported. Alternatively, extending the scratch gene-finding model to 50 epochs with identical total compute and seeing it reach MCC 0.50 would refute the plateau result.","tokens_in":7338,"feed_emoji":"🧬","tokens_out":4114,"duration_ms":35249,"temperature":0.7,"pith_summary":"This paper claims that DNA language models can be improved cheaply by pretraining on unlabeled sequences from the downstream task itself, instead of on the full genome. Using the BEND benchmark, a residual CNN pretrained with masked language modeling on gene-finding sequences matches or beats an identical model trained from scratch under the same compute budget on all four tasks. The largest gains appear on gene finding (MCC 0.38 to 0.50) and CpG methylation (AUROC 0.89 to 0.94), and a linear-chain CRF decoder raises gene finding to 0.64. The authors argue that task-specific self-pretraining is a compute-efficient route to stronger supervised baselines, especially when labeled data are scarce.","feed_headline":"Self-pretraining on task DNA beats from-scratch models","feed_subtitle":"Gene finding and CpG methylation jump 12 and 5 points, and 25% labels beat full data.","key_machinery":"The load-bearing mechanism is masked language modeling on task-specific unlabeled DNA before supervised fine-tuning. The encoder is a 30-layer residual CNN with kernel size 9 and doubling dilation (reset every six layers, capped at 32); during pretraining an MLM head predicts 15% masked tokens with the standard 80/10/10 replacement strategy. This pretrained encoder is then reused across all four tasks, with a two-layer CNN plus linear head for each task, and optionally a linear-chain CRF for gene finding.","core_discovery":"On its own terms, the paper's discovery is that self-pretraining on task-specific unlabeled DNA sequences works as an inductive prior: a 30-layer dilated residual CNN, after masked language modeling on gene-finding sequences, fine-tuned on four BEND tasks, reaches or exceeds the performance of the same architecture trained from scratch with identical compute on every task. On CpG methylation, a 25%-labeled self-pretrained model outperforms a fully labeled scratch model, showing sample efficiency. The authors present this as evidence that genome-scale pretraining is not required to build strong supervised baselines.","pith_inferences":["If the gains are driven by matching the pretraining distribution to the task, then pretraining on task-specific sequences should produce targeted improvements on each task; the paper's reuse of one gene-finding corpus across all tasks makes this an open question.","Self-pretraining could be stacked with genome-scale pretraining: a genome-pretrained model fine-tuned after task-specific self-pretraining may offer a stronger baseline than either alone.","The CRF gain hints that structured decoding, not just representation learning, matters for gene-structure tasks; combining self-pretraining with more expressive structured decoders could push closer to expert tools like AUGUSTUS."],"forward_implications":["Under matched compute, a self-pretrained CNN outperforms scratch training on gene finding (0.50 vs 0.38 MCC) and CpG methylation (0.94 vs 0.89 AUROC).","Self-pretraining delivers sample efficiency: on CpG methylation, 25% of labeled data with self-pretraining outperforms 100% of labeled data from scratch (AUROC 0.92 vs 0.89).","Adding a linear-chain CRF over the fine-tuned encoder improves gene finding further, from 0.50 to 0.64 MCC, by enforcing global exon-intron transition constraints.","Tasks that are already well solved by supervised training, like chromatin accessibility and histone modification, show little or no pretraining gain, suggesting diminishing returns for pretraining on saturated tasks."],"supporting_citations":[{"why":"Defines the BEND benchmark, its four tasks, evaluation metrics, and baseline results that the paper compares against.","marker":"Marin et al. (2024)"},{"why":"Provides the central motivation that self-pretraining on task data narrows the gap between architectures in long-sequence modeling.","marker":"Amos et al. (2024)"},{"why":"Supplies the NLP evidence that pretraining on downstream data alone can match or exceed large-corpus pretraining.","marker":"Krishna et al. (2023)"},{"why":"Provides the BASSET expert baseline on chromatin accessibility and the preprocessing of ENCODE DNase-seq labels.","marker":"Kelley et al. (2016)"},{"why":"Supplies the AUGUSTUS expert gene-finding baseline and the structured HMM approach that inspires the CRF extension.","marker":"Stanke & Waack (2003)"},{"why":"Supplies the GENCODE annotations from which gene-finding labels are derived.","marker":"Frankish et al. (2021)"},{"why":"Provides the DNase-seq, ChIP-seq, and bisulfite sequencing data underlying the chromatin, histone, and CpG methylation labels.","marker":"ENCODE Project Consortium (2012)"},{"why":"Introduces conditional random fields, the structured prediction method used in the SPT + CRF gene-finding model.","marker":"Lafferty et al. (2001)"}],"fun_headline_variants":["Self-pretrain on task DNA, beat from-scratch at same compute","Task-specific DNA self-pretraining matches or beats scratch models","Skip genome-scale pretraining: task self-pretraining suffices","DNA self-pretraining on task data beats scratch under equal compute","Task DNA self-pretraining: compute-efficient boost for genomic models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that self-pretraining is task-specific rests on using gene-finding sequences as the pretraining corpus for all four tasks, but the paper runs no control pretraining on random or task-matched sequences, so the gains could come from pretraining on any long genomic sequence rather than from the task-specific choice.","fun_headline_variants_meta":{"raw":{"variants":["Self-pretrain on task DNA, beat from-scratch at same compute","Task-specific DNA self-pretraining matches or beats scratch models","Skip genome-scale pretraining: task self-pretraining suffices","DNA self-pretraining on task data beats scratch under equal compute","Task DNA self-pretraining: compute-efficient boost for genomic models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000334,"raw_usage":{"total_tokens":1758,"prompt_tokens":756,"completion_tokens":1002,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":372,"completion_tokens_details":{"reasoning_tokens":909}},"tokens_in":372,"tokens_out":1002,"duration_ms":7552,"temperature":1.0,"reasoning_tokens":909,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:01:04.844241+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test would train the same encoder on randomly selected human genomic sequences of matching length and nucleotide composition, keeping the compute budget fixed. If the random-pretrained model matches the gene-finding-pretrained model on CpG methylation and chromatin tasks, then the paper's task-specificity claim is not supported. Alternatively, extending the scratch gene-finding model to 50 epochs with identical total compute and seeing it reach MCC 0.50 would refute the plateau result.","supporting_citations":[{"cited_title":"I., Teufel, F., Horlacher, M., Madsen, D., Pultz, D., Winther, O., and Boomsma, W","cited_arxiv_id":null,"evidence_quote":"Defines the BEND benchmark, its four tasks, evaluation metrics, and baseline results that the paper compares against."},{"cited_title":"Never train from scratch: Fair comparison of long-sequence models requires data-driven priors","cited_arxiv_id":null,"evidence_quote":"Provides the central motivation that self-pretraining on task data narrows the gap between architectures in long-sequence modeling."},{"cited_title":"R., Snoek, J., and Rinn, J","cited_arxiv_id":null,"evidence_quote":"Provides the BASSET expert baseline on chromatin accessibility and the preprocessing of ENCODE DNase-seq labels."},{"cited_title":"and Waack, S","cited_arxiv_id":null,"evidence_quote":"Supplies the AUGUSTUS expert gene-finding baseline and the structured HMM approach that inspires the CRF extension."},{"cited_title":"E., Mudge, J","cited_arxiv_id":null,"evidence_quote":"Supplies the GENCODE annotations from which gene-finding labels are derived."},{"cited_title":"Conditional random fields: Probabilistic models for segmenting and labeling sequence data","cited_arxiv_id":null,"evidence_quote":"Introduces conditional random fields, the structured prediction method used in the SPT + CRF gene-finding model."}],"review_version":2}