Pith. sign in

REVIEW 4 major objections 5 minor 21 references

SPACE: Your Genomic Profile Predictor is a Powerful DNA Foundation Model

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Supervised genomic profile prediction outperforms pure sequence pre-training as a DNA representation learning strategy, and SPACE's mixture-of-experts architecture achieves state-of-the-art results on genomic benchmarks.

desk verdict A well-engineered supervised DNA model that improves on Enformer, but the headline claim that supervised pre-training beats unsupervised sequence learning is not established by benchmarks that overlap with the pre-training objective. read the letter →

arxiv 2506.01833 v1 pith:XWJO67EA submitted 2025-06-02 cs.LG q-bio.GN

Zhao Yang , Jiwei Zhu , Bing Su This is my paper
classification cs.LGq-bio.GN
keywords DNAfoundationmodelgenomicprofilepredictionsupervisedpre-trainingmixtureofexpertsspecies-awareencodercross-speciestransferregulatorygenomicsrepresentationlearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the dominant recipe for DNA foundation models — unsupervised pre-training on raw sequence via masked language modeling — is weaker than supervised training on the genome's functional readouts. Its claim is that a model taught to predict experimentally measured genomic profiles, such as chromatin accessibility, histone modifications, transcription-factor binding, and gene expression, encodes regulatory context that pure sequences cannot supply, and transfers better to downstream genomics tasks. To make that case it introduces SPACE, which combines a species-aware Mixture-of-Experts encoder with a profile-grouped enhancement decoder, trained jointly on human and mouse genomic profiles. Across the NT, GUE, Genomic Benchmarks, and BEND suites, the paper reports state-of-the-art results that it reads as establishing supervised profile prediction as a more powerful alternative to pure sequence pre-training. The stakes are practical: if right, future DNA models can be built from regulatory-profile labels that already exist at scale rather than from sequence alone.

What carries the argument

The paper's load-bearing machinery is SPACE, built in three stages. A CNN aggregation module compresses 131 kb sequences to a 128 bp-resolution representation, following Enformer. A species-aware encoder then prepends a trainable species embedding to each sequence and runs the hidden states through sparse cross-species Mixture-of-Experts transformer layers: a shared pool of expert networks is weighted by species-specific gating networks, and an auxiliary mutual-information loss $\mathrm{MI}(S;E)$ between species identity and expert selection pushes some experts toward species-specific features while others stay shared. The profile-grouped enhancement decoder first projects the representation into base predictions for every profile, categorizes them by biological assay type (chromatin accessibility, TF binding, histone marks, CAGE), and then applies dual-gated expert weighted aggregation: a group-level gate weights expert groups from species and sequence context, an expert-level gate weights individual experts from prediction patterns, and the enhanced prediction is added as a residual to the base prediction. This design is what carries the claim that modeling cross-species and cross-profile relationships improves both profile prediction and representation quality.

What would settle it

Take a model pre-trained with SPACE on human and mouse profiles, then fine-tune it only on downstream tasks whose labels share no profile family with pre-training, such as splice-site recognition or variant-effect prediction on a chromosome never used in training, and compare with the 2.5B Nucleotide Transformer variants on the same folds. The paper's own splicing results already show the unsupervised NT-Multispecies at 0.970 MCC versus SPACE at 0.906, so a systematic set of overlap-free tasks would settle whether the supervised-pre-training advantage is representation quality or label overlap.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that genomic profile prediction is not merely a supervised downstream task but a better pre-training objective than pure sequence modeling for learning DNA representations. The authors claim that because the functional role of a DNA sequence is governed by cell-type-specific regulatory profiles rather than by the sequence alone, a model trained to predict those profiles encodes functional semantics that masked-language models miss. SPACE operationalizes this through two MoE modules: a species-aware encoder whose gating networks route tokens through shared and species-specific experts under a mutual-information regularizer, and a profile-grouped enhancement decoder that splits base profile predictions by assay type, then reweights expert groups and experts to exploit shared regulatory mechanisms. Empirically the paper reports state-of-the-art results on 11 of 18 NT downstream tasks, large gains over Enformer on yeast and viral GUE tasks, wins on the Genomic Benchmarks suite, and strong frozen-embedding results on BEND's chromatin accessibility, histone modification, and CpG methylation tasks. The authors read these results as establishing supervised profile pre-training as a more effective and powerful alternative to pure sequence pre-training.

Load-bearing premise

The load-bearing premise is that the benchmark gains reflect general representation quality, not the direct overlap between the pre-training profile labels and the downstream task labels.

Editorial extensions

If this is right

  • Supervised pre-training on genomic profiles can replace masked-language modeling as the default objective for DNA foundation models, with SPACE reporting state-of-the-art results on 11 of the 18 Nucleotide Transformer tasks.
  • SPACE's architectural refinements transfer across evolution: on yeast epigenetic-mark and COVID variant tasks it improves substantially over the original Enformer, suggesting the experts capture conserved regulatory features.
  • The sparse expert routing is interpretable: expert-selection analysis shows dedicated human and mouse experts alongside shared experts, and profile-level expert overlap mirrors known biology such as the coupling between chromatin accessibility and transcription initiation.
  • Frozen SPACE embeddings reach state-of-the-art scores on BEND chromatin accessibility, histone modification, and CpG methylation tasks, indicating the learned representations transfer even without fine-tuning.
  • SPACE improves mouse genomic profile prediction over Enformer while matching human performance using only a third of Enformer's training steps, evidence that the architecture, not scale, drives the gains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Our reading: several NT histone-mark tasks and the BEND chromatin-accessibility tasks use the same kinds of labels as SPACE's pre-training, so the reported advantage may partly reflect label overlap rather than general representation quality; the cleanest test is to pre-train without an entire profile family and re-run those tasks.
  • Because the splicing subset of NT is the most disjoint from pre-training and leans toward the unsupervised 2.5B models, a hybrid objective that adds masked-language modeling on top of profile prediction may combine the strengths of both paradigms.
  • An unstated extension of the species-aware MoE is to specialize experts by cell type or tissue within a single species, which would test whether the same routing mechanism scales to the cell-type-specific regulation the paper cites as the reason sequence alone is insufficient.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes SPACE, a mixture-of-experts architecture for supervised pre-training on genomic profile prediction (chromatin accessibility, TF binding, histone modifications, CAGE) across human and mouse. The authors argue that supervised profile prediction, as an alternative to unsupervised sequence pre-training, yields more powerful DNA representations. SPACE introduces a species-aware sparse MoE encoder and a dual-gated profile-grouped enhancement decoder. The paper evaluates on the Nucleotide Transformer (NT) downstream tasks, the GUE benchmark, Genomic Benchmarks, and the BEND benchmark, reporting SOTA on 11/18 NT tasks and consistent gains over Enformer on profile prediction.

Significance. If the central claim were established, this would be an important contribution that challenges the dominant unsupervised pre-training paradigm for DNA foundation models and demonstrates a biologically motivated architecture. The strengths include a novel architecture, reproducible experimental protocols (10-fold CV with fixed seeds), code release, and an explicit limitations section. However, the headline claim of general representation superiority is not independently tested: most downstream evaluation tasks share label families with the pre-training objective (histone marks, accessibility, TF binding), and the cleanest disjoint tasks (splicing) actually favor the unsupervised baselines. The paper's own Section E concedes the overlap. Thus the evidence as presented does not support the strong conclusion that supervised genomic profile pre-training is a more effective and powerful alternative to pure sequence pre-training.

major comments (4)
  1. [§4.2, Table 1; §E] The headline comparison is confounded by overlap between pre-training targets and downstream evaluation labels. SPACE is pre-trained to predict DNase/ATAC, TF ChIP-seq, Histone ChIP-seq, and CAGE profiles (Section B). Of the 18 NT tasks, 10 are histone modification prediction and 6 are promoter/enhancer prediction, which are the same assay families. The paper's own Section E admits that 'our supervised pre-training tasks also include chromatin accessibility and histone modification representations, which may provide potential advantages.' Consequently, the high scores on these tasks may reflect direct transfer of learned profile predictors rather than general representation quality. The splicing tasks, which are the most disjoint from the pre-training labels, show SPACE at 0.906 versus NT-Multispecies at 0.970 (Table 1). This pattern directly contradicts the claim that supervised profile pre-training is a 'more effective and powerful alternative to pure sequence pre-training.' I request either (a) evaluations on tasks genuinely disjoint from the pre-training label families (e.g., splice site recognition or variant effect prediction), or (b) a substantially tempered claim that restricts the conclusion to chromatin/regulatory tasks.
  2. [§4.3, Table 2; §D, Table 10] The cross-species validation is incomplete. The main text Table 2 compares SPACE only to Enformer, but the supplementary Table 10 shows that NT-Multispecies (2.5B) achieves higher MCC than SPACE on several yeast epigenetic mark tasks (e.g., H3K14ac: 56.20 vs 54.12; H3K36me3: 61.99 vs 54.82; H3K4me1: 55.30 vs 50.92). Because these tasks also predict histone modifications, the overlap issue extends to the cross-species setting. The claim that 'our architectural modifications enhance cross-species generalization' is not supported against state-of-the-art unsupervised models.
  3. [§4.7, Tables 4 and 14] The ablation study does not demonstrate that the MoE components are responsible for the reported improvements. The average gains are tiny (e.g., NT Chromatin: 0.5705 for SPACE vs 0.5692 for SPACE without encoder and species embedding; GUE EMP: 0.5368 vs 0.5322) and no variance or significance testing is reported. At the per-task level, Table 14 shows SPACE is worse than its own 'w/o decoder' variant on several tasks (e.g., H3K4me3: 0.637 vs 0.661; TATA box: 0.802 vs 0.841). The paper's own text concedes that the decoder 'doesn't directly boost chromatin profile prediction accuracy.' This weakens the architectural contribution as an explanation for the benchmark results.
  4. [§E, Table 12] The BEND variant effect tasks, which are arguably more disjoint from the pre-training profile labels, show SPACE at 0.51 (expression) and 0.49 (disease), far below GPN-MSA (0.97 on disease) and below several unsupervised models (e.g., NT-MS 0.77 on disease). The paper's hypothesis that masked language modeling may be necessary for variant effects is plausible, but this result further limits the generality of the central claim. This should be acknowledged in the main text rather than only in the appendix.
minor comments (5)
  1. [§A.2] There is a typo: 'whreepdenotes' should read 'where p denotes.'
  2. [§C.1] The text says 'consising' instead of 'consisting.'
  3. [§E] The text says 'The results are shown in Figure 12' but the table is labeled 'Table 12'; please correct the reference.
  4. [§4.2] In Table 1, the column grouping for 'Regulatory elements' and 'Splicing' is visually ambiguous; please separate the category headers more clearly.
  5. [§D.2] The learning rate search range is reported as '5×10−6, 5×10−5, 6×10−5, 7×10−5, 8×10−5, 3×10−4' but the paper does not state how many random seeds or folds were used for the GUE results; please clarify the evaluation protocol for Table 10.

Circularity Check

2 steps flagged · score 6.0 of 10

The headline comparison is not an independent test: SPACE pre-trains on the same profile families (chromatin assays, histone marks, CAGE) that dominate the NT, BEND and Genomic Benchmarks evaluations, while the most disjoint splicing tasks favor NT-Multispecies (0.970 vs 0.906).

  1. fitted input called prediction [Section 4.1 (Pre-training dataset), Section 4.2 (NT downstream tasks), Section B (Pre-training Dataset), Section E (BEND admission)]
    "Our model was pre-trained using supervised genomic profile prediction, maintaining the same prediction targets and genomic intervals as implemented in Enformer... These genomic profiles are categorized into four functional classes: chromatin accessibility (DNase/ATAC-seq), transcription factor binding (TF ChIP-seq), histone modifications (Histone ChIP-seq), and transcriptional activity (CAGE)... our supervised pre-training tasks also include chromatin accessibility and histone modification representations, which may provide potential advantages."

    Pre-training optimizes Poisson NLL over exactly the Section B profile families (DNase/ATAC, TF ChIP-seq, Histone ChIP-seq, CAGE). The NT benchmark (Section 4.2) is 18 human tasks in categories 'histone modification marker prediction', 'cis-regulatory element annotation', and 'splice site recognition'. The first two categories use the same label families as pre-training targets; downstream heads are fine-tuned from an encoder already optimized to output these profiles. SPACE's higher MCC on 10 histone tasks and enhancer/promoter tasks thus largely reflects direct transfer of the learned profile predictor, not a general advantage of supervised pre-training.

  2. fitted input called prediction [Section E (Results on BEND Benchmark), Table 12]
    "We observe that SPACE achieves SOTA performance on chromatin accessibility, histone modification, and CpG methylation tasks, with chromatin accessibility surpassing the second-best method by 0.04. It is worth noting that although our supervised pre-training tasks also include chromatin accessibility and histone modification representations, which may provide potential advantages, their data processing approaches are not entirely identical."

    BEND is introduced as a test of frozen SPACE embeddings, but two of the three tasks where SPACE is SOTA (chromatin accessibility, histone modification) are exactly the profile families used in supervised pre-training. The paper explicitly says the pre-training tasks 'also include chromatin accessibility and histone modification representations, which may provide potential advantages'; the follow-up that data processing is 'not entirely identical' is not a control for label-family overlap. Hence these BEND rows are a measure of transfer within the same prediction objective, not an independent benchmark of representation quality.

full rationale

The circularity here is empirical benchmark overlap rather than equation-level definitional circularity. The paper's central claim is that supervised genomic-profile pre-training is a more effective alternative to pure sequence pre-training, but the main evidence (NT downstream tasks, GUE EMP tasks, Genomic Benchmarks regulatory tasks, and BEND chromatin/histone rows) uses the same profile families as the supervised pre-training target: chromatin accessibility, histone modifications, TF binding, and CAGE. Section 4.1 explicitly states SPACE was pre-trained with the same prediction targets as Enformer, and Section B lists the four profile classes. Section 4.2 then benchmarks on histone modification and cis-regulatory element prediction, which are those same classes. The paper's own Section E admits the potential advantage from including chromatin accessibility and histone modification representations in pre-training. The most disjoint task family, splicing, favors the unsupervised NT-Multispecies model (0.970 vs 0.906), which is the opposite of the headline claim. No load-bearing self-citation chain was found; the citations to Enformer and NT are external and independently published. The architectural contribution (MoE, species-aware encoder, profile-grouped decoder) is a real addition and is tested internally via ablations, so the paper is not entirely circular. However, the headline empirical comparison is substantially compromised by label-family overlap, giving a partial circularity score of 6.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The paper is an empirical ML submission and introduces no new physical or biological entities. Its central claim rests on the assumptions that genomic profiles are the right supervision signal, that benchmark performance measures representation quality, and that cross-paper baseline numbers are comparable. These assumptions are partly in tension with the evaluation design: several benchmarks use the same profile families as pre-training targets.

free parameters (6)
  • mutual information regularization strength alpha = 0.01
    Chosen by hand in Section 3.6; controls how strongly expert selection is tied to species identity and is used in the total loss.
  • expert count in species-aware encoder (N) = 4
    Architecture choice in Section 4.5; the top-k sparse routing depends on this pool size.
  • top-k expert selection in encoder = 3
    Sparsity level for the cross-species MoE, reported in Section 4.5.
  • cross-profile expert count (K) = 8
    Decoder MoE size in Section 4.5.
  • expert groups per profile type (R) = 2
    Dual-gated decoder groups in Section 4.5.
  • downstream fine-tuning learning rate and batch size = 5e-5 and batch 8 for NT tasks; per-task LR grid for GUE and Genomic Benchmarks
    Hyperparameters selected by validation performance in Appendices C.2, D.2, and F.2; the selection is part of the reported results.
assumptions (4)
  • domain assumption Genomic profile prediction targets contain enough regulatory information that a model trained to predict them learns transferable DNA representations.
    Central premise of the paper, stated in the Introduction and Section 3.1, not proven independently.
  • domain assumption Performance on the NT, GUE, Genomic Benchmarks, and BEND benchmarks is a valid measure of representation quality for the stated claim.
    Used in Sections 4.2 to 4.5 and E; the benchmarks overlap with pre-training targets, so this assumption is partly violated, as the Section E admission shows.
  • ad hoc to paper Comparing SPACE, trained for 50k steps, against Enformer and NT results obtained under different training budgets and protocols is a fair comparison.
    Section 4.1 says 50k steps versus Enformer's longer training; benchmark baseline numbers are taken from NT and DNABERT2 papers rather than re-run under identical conditions.
  • standard math The Poisson negative log-likelihood is the correct training objective for genomic profiles.
    Adopted from Enformer in Section 3.6; it is a standard count-regression assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SPACE: Your Genomic Profile Predictor is a Powerful DNA Foundation Model." pith.science (2026). https://pith.science/paper/XWJO67EA

@misc{pith2026250601833,
  author       = {Pith},
  title        = {Pith review of: SPACE: Your Genomic Profile Predictor is a Powerful DNA Foundation Model},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XWJO67EA}},
  note         = {Machine review of arXiv:2506.01833}
}
abstract

Inspired by the success of unsupervised pre-training paradigms, researchers have applied these approaches to DNA pre-training. However, we argue that these approaches alone yield suboptimal results because pure DNA sequences lack sufficient information, since their functions are regulated by genomic profiles like chromatin accessibility. Here, we demonstrate that supervised training for genomic profile prediction serves as a more effective alternative to pure sequence pre-training. Furthermore, considering the multi-species and multi-profile nature of genomic profile prediction, we introduce our $\textbf{S}$pecies-$\textbf{P}$rofile $\textbf{A}$daptive $\textbf{C}$ollaborative $\textbf{E}$xperts (SPACE) that leverages Mixture of Experts (MoE) to better capture the relationships between DNA sequences across different species and genomic profiles, thereby learning more effective DNA representations. Through extensive experiments across various tasks, our model achieves state-of-the-art performance, establishing that DNA models trained with supervised genomic profiles serve as powerful DNA representation learners. The code is available at https://github.com/ZhuJiwei111/SPACE.

Figures

Figures reproduced from arXiv: 2506.01833 by the authors.

Figure 1
Figure 1. Overview of our SPACE architecture. It processes the input DNA sequence with three stages: (1) spatial compression and local context aggregation via a CNN-based aggregation module; (2) latent representation learning via a species-aware sparse MoE-based encoding module; (3) multi-profile prediction decoder via the dual-gated expert weighted prediction enhancement module. The detailed structures of the encoding module… view at source ↗
Figure 2
Figure 2. Expert selection visualizations and prediction results. (a) Visualization of expert selection in the final cross-species MoE. (b) Expert selection in the profile-grouped enhancement decoder module. (c) Pearson correlation coefficients across all positions per profile on the test set. Each point represents the average correlation of predicted genomic profiles across all 128-bp binned genomic positions [PITH_FULL_IMA… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 14 canonical work pages

  1. [5]

    Kathail, P., Bajwa, A., and Ioannidis, N. M. Leveraging genomic deep learning models for non-coding variant effect prediction.arXiv preprint arXiv:2411.11158,

    work page arXiv

  2. [6]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434,

    Liu, A., Feng, B., Wang, B., Wang, B., Liu, B., Zhao, C., Dengr, C., Ruan, C., Dai, D., Guo, D., et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model.arXiv preprint arXiv:2405.04434,

    arXiv

  3. [7]

    Caduceus: Bi-directional equivari- ant long-range dna sequence modeling.arXiv preprint arXiv:2403.03234,

    Schiff, Y ., Kao, C.-H., Gokaslan, A., Dao, T., Gu, A., and Kuleshov, V . Caduceus: Bi-directional equivari- ant long-range dna sequence modeling.arXiv preprint arXiv:2403.03234,

    arXiv

  4. [8]

    Performance per task was calculated as the median of the 10 cross-validation folds (± standard deviation)

    All baseline results are sourced from NT (Dalla-Torre et al., 2024). Performance per task was calculated as the median of the 10 cross-validation folds (± standard deviation). The best results for each task are highlighted inbold. 14 SPACE: Your Genomic Profile Predictor is a Powerful DNA Foundation Model Table 8.Complete Benchmark Results of Nucleotide T...

    work page 2024

  5. [10]

    Tang, Z., Toneyan, S., and Koo, P. K. Current approaches to genomic deep learning struggle to fully capture human genetic variation.Nature Genetics, 55(12):2021–2022,

    work page 2021

  6. [11]

    L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A

    Wolf, T., Debut, L., Sanh, V ., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jer- nite, Y ., Plu, J., Xu, C., Scao, T. L., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. M. Transformers: State- of-the-art natural language processing. InProceedings of the 2020 Confe...

    2020

  7. [12]

    We observe that SPACE achieves SOTA performance on chromatin accessibility, histone modification, and CpG methylation tasks, with chromatin accessibility surpassing the second-best method by 0.04. It is worth noting that although our supervised pre-training tasks also include chromatin accessibility and histone modification representations, which may prov...

    work page 2025

  8. [13]

    Derivation of Mathematical Formulations for Key Functions A.1

    11 SPACE: Your Genomic Profile Predictor is a Powerful DNA Foundation Model A. Derivation of Mathematical Formulations for Key Functions A.1. Mutual Information Analysis The Mutual Information defined in Equation (3) is: LMI =−M I(S;E) =−H(S)−H(E) +H(S, E) = MX i=m P(S m) logP(S m) + NX n=1 P(E n) logP(E n) − MX m=1 NX n=1 P(S m, En) logP(S m, En), whereS...

    work page 2021

Show all 21 references
  1. [14]

    This strategy mitigates species bias while preserving sequence diversity through stochastic resampling

    To address the pronounced species imbalance between human and mouse genomic data, we implemented balanced batch sampling through randomized minority-class augmentation, ensuring equal representation of both species in every batch. This strategy mitigates species bias while pre...

    2024

  2. [17]

    with 10,000 training steps, while retaining default parameter configurations from the HuggingFace Transformer Trainer implementation (Wolf et al., 2020). Table 10.The results on the GUE datasets Model Epigenetic Marks Prediction H3 H3K14ac H3K36me3 H3K4me1 H3K4me2 H3K4me3 DNAB...

    2020

  3. [18]

    In accordance with the implementation protocol of DNABERT2 (Zhou et al., 2024), all benchmark tasks utilized the Matthews Correlation Coefficient (MCC) for performance evaluation, with the singular exception of viral sequence analysis where F1-score metrics were employed. The ...

    2024

  4. [19]

    All our experimental settings strictly follow the official configurations of Bend (Marin et al., 2024)

    (i.e., directly using frozen SPACE embeddings for downstream tasks). All our experimental settings strictly follow the official configurations of Bend (Marin et al., 2024). The results are shown in Figure

    2024

  5. [21]

    with 3 training epochs, while maintaining the default parameter configuration from the HuggingFace Transformer Trainer implementation (Wolf et al., 2020). G. Ablation Study SPACE demonstrates comparable or superior performance to the decoder-removed variant in 14/18 tasks, wit...

    2020

  6. [1991]

    Ji, Y ., Zhou, Z., Liu, H., and Davuluri, R

    doi: 10.1162/neco.1991.3.1.79. Ji, Y ., Zhou, Z., Liu, H., and Davuluri, R. V . Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language in genome.Bioin- formatics, 37(15):2112–2120,

    1991 doi

  7. [2004]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer

    Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G., and Dean, J. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. arXiv preprint arXiv:1701.06538,

    arXiv

  8. [2019]

    over 3 epochs, while retaining default parameter settings from the HuggingFace Transformer Trainer implementation (Wolf et al., 2020). C.3. Results The complete benchmark results of the downstream tasks for NT are presented in Table

    2020

  9. [2020]

    URL https://www.aclweb

    Association for Compu- tational Linguistics. URL https://www.aclweb. org/anthology/2020.emnlp-demos.6. Yang, Z., Su, B., Cao, C., and Wen, J.-R. Regulatory DNA sequence design with reinforcement learning. InThe Thirteenth International Conference on Learning Repre- sentations,

    2020

  10. [2021]

    Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D

    Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., Casas, D. d. l., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., et al. Mistral 7b.arXiv preprint arXiv:2310.06825,

    arXiv

  11. [2023]

    and Koo, P

    Tang, Z. and Koo, P. K. Evaluating the representational power of pre-trained dna language models for regulatory genomics, march 2024,

    2024

  12. [2024]

    Bert: Pre-training of deep bidirectional transformers for lan- guage understanding

    Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. Bert: Pre-training of deep bidirectional transformers for lan- guage understanding. InProceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Techno...

    2019

  13. [2025]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901,

    1901

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.