{"id":"d77fbe05-250a-4e67-ada8-075f92297538","arxiv_id":"2506.01833","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SPACE shows that supervised prediction of genomic profiles such as chromatin accessibility and histone marks produces competitive DNA representations, with a mixture-of-experts architecture that improves cross-species and cross-profile sharing.","lead":"This paper trains a DNA model to predict gene-regulatory measurements from human and mouse DNA instead of learning only from raw sequence, and shows this supervised approach beats several unsupervised DNA models on many benchmarks. The authors add a mixture-of-experts design that lets the model share knowledge across species and across regulatory profile types.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim that supervised profile pre-training yields generally stronger DNA representations is not independently tested: most NT/GUE/BEND tasks predict the same chromatin and histone assay classes used in pre-training, while the disjoint splicing tasks favor NT-Multispecies (0.970 vs…","rationale":"The paper is a serious engineering contribution: SPACE is a plausible extension of Enformer with biologically motivated MoE routing, and it does improve over Enformer on most tasks. However, the headline claim is comparative and general: supervised genomic-profile pre-training should be 'a more effective and powerful alternative to pure sequence pre-training.' For that claim to hold, the evaluation must show transfer to tasks that are not simply re-predictions of the same profile families used during pre-training. The NT benchmark, the GUE EMP tasks, and the Genomic Benchmarks regulatory tasks are largely of this re-prediction kind, since the pre-training targets include chromatin accessibility, TF binding, histone modifications, and CAGE. The paper's own Section E explicitly acknowledges potential advantage from overlapping pre-training labels. On the most disjoint task family (splicing), the unsupervised NT-Multispecies model outperforms SPACE by a wide margin. These observations show that the evidence as reported is confounded, not that SPACE's architecture is flawed. The condition for accepting the broad claim is missing: a rigorous out-of-distribution benchmark in which no pre-training label family overlaps with the downstream target. Therefore the reader's CONDITIONAL verdict remains appropriate; no verdict change is needed, though the condition should be made explicit and binding.","tokens_in":22746,"tokens_out":3768,"duration_ms":47958,"concrete_test":"Construct a label-disjoint benchmark: select downstream tasks whose targets are not among SPACE's pre-training tracks (for example, a human histone modification assay from a cell type absent from the 5,313 human training tracks, or use chromosome-held-out evaluation where test chromosomes are excluded from both pre-training and fine-tuning). Fine-tune SPACE and NT-Multispecies under identical protocols and compare MCC. If SPACE's advantage over NT-Multispecies disappears or reverses on this disjoint benchmark, the reported SOTA is best explained by label overlap rather than by generally superior representations; if the advantage persists, the broad claim is supported. A minimal version of this check is to report the average MCC over the four splicing tasks in Table 1 as a pre-specified disjoint subset, where NT-Multispecies already wins.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 presents the NT downstream benchmark as the main evidence for the headline claim. Of the 18 NT tasks, 10 are histone modification prediction and 6 are promoter/enhancer prediction; these are exactly the assay families used in SPACE's supervised pre-training (DNase/ATAC, TF ChIP-seq, histone ChIP-seq, CAGE; Section B). The downstream classifiers are fine-tuned from an encoder already trained to predict these profile types, so high MCC may reflect direct transfer of learned profile predictors rather than general representation quality. The paper's own Section E concedes that 'our supervised pre-training tasks also include chromatin accessibility and histone modification representations, which may provide potential advantages,' and only asserts that data processing is 'not entirely identical.' That is an admission of overlap, not a control.\n\nThe cleanest independent signal in Table 1 is the splicing category, which is most disjoint from the pre-training labels. There SPACE (0.906) is clearly worse than NT-Multispecies (0.970), and even Enformer (0.780) is below the unsupervised models. The same overlap affects the GUE yeast EMP tasks and the Genomic Benchmarks OCR/promoter/enhancer tasks. Enformer is not a clean supervised-vs-unsupervised control because it was trained with the same supervised objective on the same data.\n\nThus the empirical case for 'supervised pre-training is a more effective alternative to pure sequence pre-training' rests on benchmarks whose labels substantially overlap with the pre-training objective. The SOTA results may be real, but they do not establish the broad representation-learning claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SPACE, a mixture-of-experts architecture for supervised pre-training on genomic profile prediction (chromatin accessibility, TF binding, histone modifications, CAGE) across human and mouse. The authors argue that supervised profile prediction, as an alternative to unsupervised sequence pre-training, yields more powerful DNA representations. SPACE introduces a species-aware sparse MoE encoder and a dual-gated profile-grouped enhancement decoder. The paper evaluates on the Nucleotide Transformer (NT) downstream tasks, the GUE benchmark, Genomic Benchmarks, and the BEND benchmark, reporting SOTA on 11/18 NT tasks and consistent gains over Enformer on profile prediction.","tokens_in":23061,"tokens_out":7404,"duration_ms":65430,"significance":"If the central claim were established, this would be an important contribution that challenges the dominant unsupervised pre-training paradigm for DNA foundation models and demonstrates a biologically motivated architecture. The strengths include a novel architecture, reproducible experimental protocols (10-fold CV with fixed seeds), code release, and an explicit limitations section. However, the headline claim of general representation superiority is not independently tested: most downstream evaluation tasks share label families with the pre-training objective (histone marks, accessibility, TF binding), and the cleanest disjoint tasks (splicing) actually favor the unsupervised baselines. The paper's own Section E concedes the overlap. Thus the evidence as presented does not support the strong conclusion that supervised genomic profile pre-training is a more effective and powerful alternative to pure sequence pre-training.","major_comments":[{"comment":"The headline comparison is confounded by overlap between pre-training targets and downstream evaluation labels. SPACE is pre-trained to predict DNase/ATAC, TF ChIP-seq, Histone ChIP-seq, and CAGE profiles (Section B). Of the 18 NT tasks, 10 are histone modification prediction and 6 are promoter/enhancer prediction, which are the same assay families. The paper's own Section E admits that 'our supervised pre-training tasks also include chromatin accessibility and histone modification representations, which may provide potential advantages.' Consequently, the high scores on these tasks may reflect direct transfer of learned profile predictors rather than general representation quality. The splicing tasks, which are the most disjoint from the pre-training labels, show SPACE at 0.906 versus NT-Multispecies at 0.970 (Table 1). This pattern directly contradicts the claim that supervised profile pre-training is a 'more effective and powerful alternative to pure sequence pre-training.' I request either (a) evaluations on tasks genuinely disjoint from the pre-training label families (e.g., splice site recognition or variant effect prediction), or (b) a substantially tempered claim that restricts the conclusion to chromatin/regulatory tasks.","section":"§4.2, Table 1; §E"},{"comment":"The cross-species validation is incomplete. The main text Table 2 compares SPACE only to Enformer, but the supplementary Table 10 shows that NT-Multispecies (2.5B) achieves higher MCC than SPACE on several yeast epigenetic mark tasks (e.g., H3K14ac: 56.20 vs 54.12; H3K36me3: 61.99 vs 54.82; H3K4me1: 55.30 vs 50.92). Because these tasks also predict histone modifications, the overlap issue extends to the cross-species setting. The claim that 'our architectural modifications enhance cross-species generalization' is not supported against state-of-the-art unsupervised models.","section":"§4.3, Table 2; §D, Table 10"},{"comment":"The ablation study does not demonstrate that the MoE components are responsible for the reported improvements. The average gains are tiny (e.g., NT Chromatin: 0.5705 for SPACE vs 0.5692 for SPACE without encoder and species embedding; GUE EMP: 0.5368 vs 0.5322) and no variance or significance testing is reported. At the per-task level, Table 14 shows SPACE is worse than its own 'w/o decoder' variant on several tasks (e.g., H3K4me3: 0.637 vs 0.661; TATA box: 0.802 vs 0.841). The paper's own text concedes that the decoder 'doesn't directly boost chromatin profile prediction accuracy.' This weakens the architectural contribution as an explanation for the benchmark results.","section":"§4.7, Tables 4 and 14"},{"comment":"The BEND variant effect tasks, which are arguably more disjoint from the pre-training profile labels, show SPACE at 0.51 (expression) and 0.49 (disease), far below GPN-MSA (0.97 on disease) and below several unsupervised models (e.g., NT-MS 0.77 on disease). The paper's hypothesis that masked language modeling may be necessary for variant effects is plausible, but this result further limits the generality of the central claim. This should be acknowledged in the main text rather than only in the appendix.","section":"§E, Table 12"}],"minor_comments":[{"comment":"There is a typo: 'whreepdenotes' should read 'where p denotes.'","section":"§A.2"},{"comment":"The text says 'consising' instead of 'consisting.'","section":"§C.1"},{"comment":"The text says 'The results are shown in Figure 12' but the table is labeled 'Table 12'; please correct the reference.","section":"§E"},{"comment":"In Table 1, the column grouping for 'Regulatory elements' and 'Splicing' is visually ambiguous; please separate the category headers more clearly.","section":"§4.2"},{"comment":"The learning rate search range is reported as '5×10−6, 5×10−5, 6×10−5, 7×10−5, 8×10−5, 3×10−4' but the paper does not state how many random seeds or folds were used for the GUE results; please clarify the evaluation protocol for Table 10.","section":"§D.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid architectural contribution and a large body of experiments, but the main claim is significantly overstated given the overlap between pre-training and evaluation. The splicing results and the BEND variant effect results are the kind of evidence that should have tempered the abstract and conclusion. I would encourage the editor to seek a revision that either adds truly disjoint evaluations or re-scopes the claim to 'supervised profile pre-training improves chromatin/regulatory tasks.' The current framing is not supported by the data as presented."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my honest read of SPACE. The architecture work is real: the species-aware MoE encoder and the dual-gated profile decoder are genuinely new, and the paper is the first systematic attempt to treat supervised genomic profile predictors as representation learners. On that level it succeeds. But the central claim—that supervised pre-training on genomic profiles is a more effective alternative to pure sequence pre-training—is not actually established by the evidence they present.\n\nWhat's good: the model consistently beats Enformer across most tasks, which is a meaningful architectural improvement on a strong baseline. The ablation study, while small, does suggest the MoE encoder helps for splicing and cross-species tasks. The paper also ships code and gives enough detail on hyperparameters and data to reproduce the training setup. And to their credit, Section E openly admits that the supervised pre-training tasks include chromatin accessibility and histone modification representations, which may give an advantage on BEND tasks. That admission is honest, but it cuts against the headline claim.\n\nThe soft spots are significant. Most of the NT benchmark tasks are exactly the profile families used in pre-training: histone marks, accessibility, TF binding. Fine-tuning a model that was pre-trained to predict those profiles on the same profile families is not an independent test of representation quality. The cleanest exception is splicing, and there the unsupervised NT-Multispecies model beats SPACE by a large margin (0.970 vs 0.906). The GUE yeast EMP tasks are also epigenetic marks, same issue. Enformer is not a clean supervised-vs-unsupervised control because it was trained with the same objective on the same data. So the comparison that would actually support the paper's thesis—supervised versus unsupervised on genuinely disjoint tasks—is missing.\n\nThe ablations are also concerning: the differences between configurations are tiny, no variance is reported, and in several rows (e.g., H3K4me3, TATA) the full model is worse than a stripped-down variant. That's not damning by itself, but it means the architectural claims rest on thin evidence.\n\nWho is this for? People working on DNA representation learning and regulatory genomics will find the architecture and the comparisons useful. As a claim that supervised pre-training is generally superior, it needs a lot more work: out-of-distribution benchmarks, a shared evaluation harness, and variance-aware ablations. I would send it to a serious referee, but with a clear request to address the overlap problem and to report the splicing result without spinning it.","headline":"A well-engineered supervised DNA model that improves on Enformer, but the headline claim that supervised pre-training beats unsupervised sequence learning is not established by benchmarks that overlap with the pre-training objective.","tokens_in":23593,"tokens_out":2465,"would_cite":false,"duration_ms":25191,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Supervised genomic profile prediction outperforms pure sequence pre-training as a DNA representation learning strategy, and SPACE's mixture-of-experts architecture achieves state-of-the-art results on genomic benchmarks.","keywords":["DNA foundation model","genomic profile prediction","supervised pre-training","mixture of experts","species-aware encoder","cross-species transfer","regulatory genomics","representation learning"],"falsifier":"Take a model pre-trained with SPACE on human and mouse profiles, then fine-tune it only on downstream tasks whose labels share no profile family with pre-training, such as splice-site recognition or variant-effect prediction on a chromosome never used in training, and compare with the 2.5B Nucleotide Transformer variants on the same folds. The paper's own splicing results already show the unsupervised NT-Multispecies at 0.970 MCC versus SPACE at 0.906, so a systematic set of overlap-free tasks would settle whether the supervised-pre-training advantage is representation quality or label overlap.","tokens_in":22509,"feed_emoji":"🧬","tokens_out":7488,"duration_ms":69922,"temperature":0.7,"pith_summary":"This paper argues that the dominant recipe for DNA foundation models — unsupervised pre-training on raw sequence via masked language modeling — is weaker than supervised training on the genome's functional readouts. Its claim is that a model taught to predict experimentally measured genomic profiles, such as chromatin accessibility, histone modifications, transcription-factor binding, and gene expression, encodes regulatory context that pure sequences cannot supply, and transfers better to downstream genomics tasks. To make that case it introduces SPACE, which combines a species-aware Mixture-of-Experts encoder with a profile-grouped enhancement decoder, trained jointly on human and mouse genomic profiles. Across the NT, GUE, Genomic Benchmarks, and BEND suites, the paper reports state-of-the-art results that it reads as establishing supervised profile prediction as a more powerful alternative to pure sequence pre-training. The stakes are practical: if right, future DNA models can be built from regulatory-profile labels that already exist at scale rather than from sequence alone.","feed_headline":"Genomic profiles beat raw sequence for DNA model pre-training","feed_subtitle":"Supervised prediction of chromatin and expression profiles beats masked-language pre-training for DNA tasks.","key_machinery":"The paper's load-bearing machinery is SPACE, built in three stages. A CNN aggregation module compresses 131 kb sequences to a 128 bp-resolution representation, following Enformer. A species-aware encoder then prepends a trainable species embedding to each sequence and runs the hidden states through sparse cross-species Mixture-of-Experts transformer layers: a shared pool of expert networks is weighted by species-specific gating networks, and an auxiliary mutual-information loss $\\mathrm{MI}(S;E)$ between species identity and expert selection pushes some experts toward species-specific features while others stay shared. The profile-grouped enhancement decoder first projects the representation into base predictions for every profile, categorizes them by biological assay type (chromatin accessibility, TF binding, histone marks, CAGE), and then applies dual-gated expert weighted aggregation: a group-level gate weights expert groups from species and sequence context, an expert-level gate weights individual experts from prediction patterns, and the enhanced prediction is added as a residual to the base prediction. This design is what carries the claim that modeling cross-species and cross-profile relationships improves both profile prediction and representation quality.","core_discovery":"On its own terms, the paper's discovery is that genomic profile prediction is not merely a supervised downstream task but a better pre-training objective than pure sequence modeling for learning DNA representations. The authors claim that because the functional role of a DNA sequence is governed by cell-type-specific regulatory profiles rather than by the sequence alone, a model trained to predict those profiles encodes functional semantics that masked-language models miss. SPACE operationalizes this through two MoE modules: a species-aware encoder whose gating networks route tokens through shared and species-specific experts under a mutual-information regularizer, and a profile-grouped enhancement decoder that splits base profile predictions by assay type, then reweights expert groups and experts to exploit shared regulatory mechanisms. Empirically the paper reports state-of-the-art results on 11 of 18 NT downstream tasks, large gains over Enformer on yeast and viral GUE tasks, wins on the Genomic Benchmarks suite, and strong frozen-embedding results on BEND's chromatin accessibility, histone modification, and CpG methylation tasks. The authors read these results as establishing supervised profile pre-training as a more effective and powerful alternative to pure sequence pre-training.","pith_inferences":["Our reading: several NT histone-mark tasks and the BEND chromatin-accessibility tasks use the same kinds of labels as SPACE's pre-training, so the reported advantage may partly reflect label overlap rather than general representation quality; the cleanest test is to pre-train without an entire profile family and re-run those tasks.","Because the splicing subset of NT is the most disjoint from pre-training and leans toward the unsupervised 2.5B models, a hybrid objective that adds masked-language modeling on top of profile prediction may combine the strengths of both paradigms.","An unstated extension of the species-aware MoE is to specialize experts by cell type or tissue within a single species, which would test whether the same routing mechanism scales to the cell-type-specific regulation the paper cites as the reason sequence alone is insufficient."],"forward_implications":["Supervised pre-training on genomic profiles can replace masked-language modeling as the default objective for DNA foundation models, with SPACE reporting state-of-the-art results on 11 of the 18 Nucleotide Transformer tasks.","SPACE's architectural refinements transfer across evolution: on yeast epigenetic-mark and COVID variant tasks it improves substantially over the original Enformer, suggesting the experts capture conserved regulatory features.","The sparse expert routing is interpretable: expert-selection analysis shows dedicated human and mouse experts alongside shared experts, and profile-level expert overlap mirrors known biology such as the coupling between chromatin accessibility and transcription initiation.","Frozen SPACE embeddings reach state-of-the-art scores on BEND chromatin accessibility, histone modification, and CpG methylation tasks, indicating the learned representations transfer even without fine-tuning.","SPACE improves mouse genomic profile prediction over Enformer while matching human performance using only a third of Enformer's training steps, evidence that the architecture, not scale, drives the gains."],"supporting_citations":[{"why":"Provides the Enformer backbone, the human/mouse 131 kb dataset, the Poisson loss, and the supervised genomic-profile-prediction baseline that SPACE modifies and compares against.","marker":"Avsec et al., 2021"},{"why":"Establishes the cross-species regulatory-sequence-activity prediction setting and the interleaved-batch training strategy that SPACE adopts for human and mouse pre-training.","marker":"Kelley, 2020"},{"why":"Supplies the 18-task Nucleotide Transformer benchmark, its evaluation protocol, and the unsupervised DNA foundation model baselines, including the 2.5B variants that SPACE claims to outperform.","marker":"Dalla-Torre et al., 2024"},{"why":"Supplies the GUE benchmark, the yeast epigenetic-mark and COVID variant tasks used for cross-species validation, and the DNABERT-2 baselines reproduced in the paper.","marker":"Zhou et al., 2024"},{"why":"Source of the mutual-information loss that SPACE adapts to encourage species-specific expert specialization in the sparse MoE encoder.","marker":"Chen et al., 2023"},{"why":"Provides the BEND benchmark used in Section E to evaluate frozen SPACE embeddings on chromatin accessibility, histone modification, CpG methylation, and variant effects.","marker":"Marin et al., 2024"},{"why":"Supplies the Caduceus baseline results on the Genomic Benchmarks datasets that SPACE uses for extended binary and multi-class classification comparisons.","marker":"Schiff et al., 2024"},{"why":"Grounds biological principle (P2), the functional interdependence of genomic profiles, which motivates the profile-grouped enhancement decoder.","marker":"Fu et al., 2025"}],"fun_headline_variants":["DNA models learn better from genomic profiles than raw sequence","Supervised profile prediction beats sequence pretraining for DNA","SPACE: Profile-aware DNA pretraining outperforms pure sequence","Genomic profiles, not sequences, power stronger DNA models","Profile prediction: the new pretraining goal for DNA AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the benchmark gains reflect general representation quality, not the direct overlap between the pre-training profile labels and the downstream task labels.","fun_headline_variants_meta":{"raw":{"variants":["DNA models learn better from genomic profiles than raw sequence","Supervised profile prediction beats sequence pretraining for DNA","SPACE: Profile-aware DNA pretraining outperforms pure sequence","Genomic profiles, not sequences, power stronger DNA models","Profile prediction: the new pretraining goal for DNA AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000195,"raw_usage":{"total_tokens":1361,"prompt_tokens":954,"completion_tokens":407,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":327}},"tokens_in":570,"tokens_out":407,"duration_ms":4386,"temperature":1.0,"reasoning_tokens":327,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:32:37.606942+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a model pre-trained with SPACE on human and mouse profiles, then fine-tune it only on downstream tasks whose labels share no profile family with pre-training, such as splice-site recognition or variant-effect prediction on a chromosome never used in training, and compare with the 2.5B Nucleotide Transformer variants on the same folds. The paper's own splicing results already show the unsupervised NT-Multispecies at 0.970 MCC versus SPACE at 0.906, so a systematic set of overlap-free tasks would settle whether the supervised-pre-training advantage is representation quality or label overlap.","supporting_citations":[{"cited_title":"Performance per task was calculated as the median of the 10 cross-validation folds (± standard deviation)","cited_arxiv_id":null,"evidence_quote":"Supplies the 18-task Nucleotide Transformer benchmark, its evaluation protocol, and the unsupervised DNA foundation model baselines, including the 2.5B variants that SPACE claims to outperform."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the GUE benchmark, the yeast epigenetic-mark and COVID variant tasks used for cross-species validation, and the DNABERT-2 baselines reproduced in the paper."}],"review_version":1}