Pith. sign in

REVIEW 3 major objections 9 minor 46 references

HAD: Hybrid Architecture Distillation Outperforms Teacher in Genomic Sequence Modeling

T0 review · 3 major / 9 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 1.1M-parameter DNA model, pretrained by distilling visible-token features from a frozen roughly-500M-parameter teacher while reconstructing masked nucleotides, outperforms the teacher on histone-marker and enhancer tasks.

desk verdict A promising distillation recipe for compact genomic models, but the teacher comparison needs verification and the pooling detail is underspecified. read the letter →

arxiv 2505.20836 v1 pith:JUQTEQZO submitted 2025-05-27 cs.LG q-bio.GN

Hexiong Yang , Mingrui Chen , Huaibo Huang , Junxian Duan , Jie Cao , Zhen Zhou , Ran He This is my paper
classification cs.LGq-bio.GN
keywords genomicsequencemodelingknowledgedistillationmaskedlanguagehybridarchitectureGatedDeltaNetDNAmodelNucleotideTransformerbenchmarkBenchmarks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a compact DNA language model can inherit high-level biological understanding from a much larger teacher through feature distillation, and that the student is not limited by the teacher's own performance. The authors introduce HAD, a 1.1M-parameter hybrid architecture trained with two objectives in parallel: aligning visible-nucleotide representations with those of the frozen NTv2-500M teacher, and reconstructing masked nucleotides from the resulting visible context. On the Nucleotide Transformer Benchmark, HAD reports the best result on 11 of 18 tasks, including nine of ten histone-marker tasks and both enhancer tasks, where it beats the teacher by up to 24.3 percentage points in MCC. On Genomic Benchmarks, its 0.875 average accuracy is the highest among the reported baselines. The claim is that a strong frozen teacher plus a small hybrid student can yield compact genome models that compete with, and sometimes beat, models hundreds of times larger.

What carries the argument

The load-bearing mechanism is a two-branch pretraining objective. In the distillation branch, the student's character-level hidden states are average-pooled over non-overlapping 6-mer windows, projected to the teacher's 1024-dimension space, and aligned by MSE with NTv2-500M's visible-token features; only visible tokens enter this loss. In the reconstruction branch, masked positions act as queries in a cross-attention layer whose keys and values are the student's visible representations, and an LM head predicts the original nucleotides. The two branches share a two-stage masking strategy: 15% of the teacher's k-mer units are masked first, then those indices are mapped to the corresponding character positions. The student backbone is a bidirectional Gated $\Delta$ Net — a linear-complexity recurrent layer with data-dependent gating and delta-rule memory updates — followed by a single self-attention layer, totaling 1.1M parameters.

What would settle it

Train HAD with the same two-branch objective but drop every partially masked 6-mer window from the MSE pooling step, or pool only over fully visible windows; if the student's advantage over NTv2-500M on histone-marker tasks disappears or shrinks sharply, the reported gains rest on an alignment artifact rather than on learned biology.

Watch

Extended reading notes

Core claim

The central claim is that the 'distillation ceiling' — the intuition that a distilled student can at best match its teacher — does not hold for genomic sequence modeling when the student is trained on complementary objectives. HAD aligns the student's visible-token representations with the teacher's 6-mer-level representations via mean-squared error while simultaneously reconstructing masked nucleotides through cross-attention to visible context. The authors report that the resulting 1.1M-parameter student achieves top results on 11 of 18 Nucleotide Transformer tasks and the highest average on the Genomic Benchmarks, and that it exceeds the frozen NTv2-500M teacher on all histone-marker and enhancer tasks despite having 497.2M fewer parameters. They interpret this as evidence that the distillation branch transfers sophisticated biological features, while the reconstruction branch preserves fine-grained nucleotide understanding, and that the hybrid Gated $\Delta$ Net plus attention backbone is the right size to carry both.

Load-bearing premise

The argument depends on the assumption that average-pooling the student's visible-only hidden states over partially masked 6-mer windows yields a faithful counterpart to the teacher's full-sequence 6-mer features, and the paper does not state how partially masked windows are treated.

Editorial extensions

If this is right

  • Compact students with roughly one million parameters can match or exceed a 500M-parameter teacher on several genomic classification tasks, so distillation is a credible alternative to scaling up pretraining.
  • The hybrid of a gated delta recurrent backbone and a single attention layer is sufficient to capture both local sequence motifs and global context, pointing to a cost-effective architecture for DNA foundation models.
  • Teacher capacity is decisive: in the paper's ablations, guidance from the 500M teacher transfers far better than from 50M or 100M variants, so the quality of the teacher matters more than its raw size.
  • Aligning k-mer-level teacher features with character-level student features through grouped masking is a workable way to bridge tokenizer mismatches in genomic distillation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would be to vary the reconstruction loss weight: if the student's gains over the teacher on histone-marker tasks come from the reconstruction branch regularizing the teacher's blind spots, increasing that weight should shift performance accordingly.
  • The same average-pooling alignment strategy could transfer across other tokenizer mismatches, for example distilling from a 3-mer teacher into a character-level student, provided the pooling windows divide the sequence length.
  • If the teacher's own representations are corrupted at partially masked positions, the MSE signal at those windows is unreliable; a stricter variant that drops partially masked windows from the loss would separate alignment artifacts from genuine knowledge transfer.
  • The result hints that feature distillation may be most powerful on tasks where local regulatory motifs dominate, since the student's largest margins over the teacher occur on histone modification and enhancer prediction.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 9 minor

Summary. This manuscript introduces HAD, a 1.1M-parameter hybrid student architecture combining a bidirectional Gated Delta Net (GDN) backbone with a self-attention layer. The student is pre-trained with two objectives: feature distillation from the NTv2-500M teacher on visible nucleotide positions, using a 6-mer group masking scheme to bridge the character/k-mer tokenizer mismatch, and masked-nucleotide reconstruction through a cross-attention decoder. The authors evaluate on the Nucleotide Transformer Benchmark and Genomic Benchmarks, reporting the best result on 11 of 18 NT tasks and the highest average on Genomic Benchmarks, including several tasks where the student's mean exceeds the teacher's. Ablations examine the removal of the attention layer and visible distillation and the effect of teacher model size.

Significance. If verified, the paper demonstrates a useful recipe: a compact hybrid architecture can rival and sometimes exceed a 500M-parameter Transformer teacher on downstream genomic tasks when trained with visible-token feature distillation plus masked reconstruction. The external benchmarks, multi-seed evaluation, and controlled ablations are appropriate, and there is no circular use of benchmark scores in the training objective. The novelty lies not in any single component but in the combination of tokenizer-mismatch-aware masking, feature alignment, and reconstruction. However, the headline comparison currently depends on a baseline row adopted from prior work without establishing checkpoint and protocol identity, and the pre-training pipeline has unresolved specification issues. With the comparison re-run and the pipeline clarified, the contribution could be significant for efficient genomic foundation models.

major comments (3)
  1. [Section 4.1 / Table 1] The row labeled 'NT (Teacher)' is adopted directly from Caduceus [7], but the distillation teacher is NTv2-500M. The paper never states that [7]'s NT row is the same checkpoint version (NTv1 versus NTv2) or that it was fine-tuned with the same protocol, learning rate, seeds, and data splits as the HAD results. Since the abstract and Section 4.2's headline claim are quantified by the ΔStudent-Teacher column, a mismatch in this row would invalidate the central empirical assertion. The authors should either re-run the NTv2-500M teacher under the exact fine-tuning protocol used for HAD, or provide released teacher checkpoints and a detailed protocol-matching statement.
  2. [Section 3.1 / Section 3.2 / Eq. (2)] The teacher's input is described inconsistently. Section 3.1 says the teacher 'processes the full X', while Section 3.2 says group masking defines masked regions for the teacher model. If the teacher sees the full unmasked sequence, its visible-position representations can encode the masked nucleotides through attention, and distilling those representations into the student leaks masked-position information into the reconstruction branch, contrary to the stated motivation of preventing information leakage. If the teacher sees a masked sequence, the text should say so. In addition, Eq. (2) averages student hidden states over non-overlapping 6-mer windows; because masking is applied to whole 6-mers, this is well-defined only if pooled windows containing any masked position are excluded from V, and the paper should state this explicitly. Please clarify both points.
  3. [Table 1] The claimed systematic superiority over the teacher is supported only by mean MCC/F1 values with per-seed ranges. On several tasks (e.g., H3, H3K36me3, H3K4me1, H3K79me3, Enhancer) the stated ranges of the teacher and HAD overlap, so the positive deltas are within seed-level noise. To support the 'surpasses teacher' claim, the authors should report paired per-seed deltas or a paired significance test across the same 10 seeds, in addition to the mean and range. A similar caution applies to the Genomic Benchmarks average in Table 2.
minor comments (9)
  1. [Section 3.1, Eq. (1)] The notation in the reconstruction objective is inconsistent: p(y_m|X_m) is written alongside XM and y_i, and neither variable is defined precisely; please rewrite it with clear random variables and index sets.
  2. [Section 3.2] 'aliment branch' should be 'alignment branch'.
  3. [Abstract / Table 1] The abstract and introduction say the teacher is 'more than 500× larger', but Table 1 reports 498.3M versus 1.1M parameters, a factor of approximately 453; please correct the claim.
  4. [Section 4.3 / Figure 4] The ablation plots for architecture variants and teacher size report only bars without numeric labels or error bars, and the caption does not state which downstream tasks and seeds are averaged; please provide a table with the underlying values.
  5. [Section 4.1] The paper does not report pre-training hyperparameters such as total number of steps, batch size, learning rate, and GPU hours; Table 3 shows loss curves only up to 10k steps, so the training budget is unclear.
  6. [Table 2] The baseline labeled 'Mamba[7]' is unclear, since [7] introduces Caduceus; please clarify whether this is the Mamba model from [7] or another reference.
  7. [Section 3.2 / Figure 1] The text should explicitly state that the teacher is frozen; otherwise the gradient of Eq. (2) could update the teacher parameters.
  8. [Equation (3)] Q_m, K_v, V_v, and d_k are not defined with their dimensions; a sentence describing the projection shapes and key dimension would help.
  9. [General] For reproducibility, please release the student and teacher checkpoints and the exact inference code used for the Nucleotide Transformer and Genomic Benchmarks evaluations.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the distillation objective is not fitted to downstream benchmarks, and the teacher comparison rests on external baseline numbers rather than on a definitional reduction.

full rationale

The paper's central claims are (i) that a 1.1M-parameter hybrid GDN-plus-attention student can be pre-trained by visible-token feature distillation from a frozen NTv2-500M teacher together with masked-nucleotide reconstruction, and (ii) that this student scores competitively on external benchmarks. Neither claim is equivalent, by the paper's own equations, to its inputs. The distillation loss LDis (Eq. 2) compares the student's visible-token representations, after 6-mer average pooling and projection, to the teacher's representations; the reconstruction loss (Eq. 1 / cross-entropy) predicts masked nucleotides from student context. Neither loss is constructed from downstream benchmark labels, and no benchmark number is used to set or fit any training hyperparameter. The 'surpasses teacher' comparison uses an NT(Teacher) row whose results are adopted directly from Caduceus [7] rather than re-run; this raises a legitimate reproducibility and comparability question about whether the adopted teacher numbers come from exactly the same NTv2-500M checkpoint and fine-tuning protocol used for distillation, but it is not circularity because the adopted numbers are external evidence, not fitted values from the present model, and the comparison is a benchmark measurement rather than a definitional identity. The only self-citations are background references to the authors' earlier work on efficient vision architectures (Refs. [15]-[19]); none is load-bearing for the HAD derivation, and no uniqueness theorem or ansatz is imported from those papers. The rhetorical phrase 'distillation ceiling' is not a formal bound and is not used to forbid alternatives. The derivation is therefore self-contained against external benchmarks, and the appropriate circularity score is 0.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central claim is an empirical performance claim. It relies on standard architectural components (GDN, attention, MLM) and domain transfer assumptions. The main hand-chosen hyperparameters are listed; none are fitted to the downstream benchmark targets. No new physical or biological entities are introduced.

free parameters (5)
  • student hidden dimension dS = 128
    Chosen by hand; no search reported.
  • number of GDN blocks = 4
    Chosen by hand.
  • mask ratio = 15%
    Standard BERT-style masking, applied to k-mer units.
  • k-mer size = 6
    Matches teacher NTv2 tokenizer; sequence length 1026 chosen for divisibility by 6.
  • sequence length = 1026
    Chosen to suit downstream tasks and be divisible by 6.
assumptions (3)
  • standard math The GDN update rule (Eq. 4) is an effective sequence model for DNA.
    Adopted from [12] without modification.
  • domain assumption NTv2-500M's representations are a reliable source of biological knowledge for distillation.
    The entire method assumes teacher features at visible positions are worth matching; ablation shows larger teacher helps.
  • domain assumption Pretraining on the human reference genome transfers to the downstream genomic benchmarks.
    Standard transfer-learning assumption, also used by HyenaDNA and Caduceus.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HAD: Hybrid Architecture Distillation Outperforms Teacher in Genomic Sequence Modeling." pith.science (2026). https://pith.science/paper/JUQTEQZO

@misc{pith2026250520836,
  author       = {Pith},
  title        = {Pith review of: HAD: Hybrid Architecture Distillation Outperforms Teacher in Genomic Sequence Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JUQTEQZO}},
  note         = {Machine review of arXiv:2505.20836}
}
abstract

Inspired by the great success of Masked Language Modeling (MLM) in the natural language domain, the paradigm of self-supervised pre-training and fine-tuning has also achieved remarkable progress in the field of DNA sequence modeling. However, previous methods often relied on massive pre-training data or large-scale base models with huge parameters, imposing a significant computational burden. To address this, many works attempted to use more compact models to achieve similar outcomes but still fell short by a considerable margin. In this work, we propose a Hybrid Architecture Distillation (HAD) approach, leveraging both distillation and reconstruction tasks for more efficient and effective pre-training. Specifically, we employ the NTv2-500M as the teacher model and devise a grouping masking strategy to align the feature embeddings of visible tokens while concurrently reconstructing the invisible tokens during MLM pre-training. To validate the effectiveness of our proposed method, we conducted comprehensive experiments on the Nucleotide Transformer Benchmark and Genomic Benchmark. Compared to models with similar parameters, our model achieved excellent performance. More surprisingly, it even surpassed the distillation ceiling-teacher model on some sub-tasks, which is more than 500 $\times$ larger. Lastly, we utilize t-SNE for more intuitive visualization, which shows that our model can gain a sophisticated understanding of the intrinsic representation pattern in genomic sequences.

Figures

Figures reproduced from arXiv: 2505.20836 by the authors.

Figure 1
Figure 1. Proposed Hybrid Architecture Distillation (HAD) pre-training pipeline. The upper branch performs feature alignment on visible nucleotides, distilling high-level knowledge from a pre-trained teacher model to the student model. The lower branch focuses on the low-level reconstruction of masked nucleotides, leveraging contextual information from the student’s visible nucleotide representations. memory management and re… view at source ↗
Figure 2
Figure 2. Two-stage masking strategy in HAD. This strategy is designed to prevent information [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Hybrid architecture of our student model, combining a bidirectional Gated Delta Net [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Ablation on pretraining scheme with different model architectures (left) and teacher model [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: t-SNE visualization of pre-trained model representations on select downstream task data. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

46 extracted references · 20 canonical work pages

  1. [7]

    Caduceus: Bi-directional equivariant long-range dna sequence modeling.arXiv preprint arXiv:2403.03234, 2024

    Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and V olodymyr Kuleshov. Caduceus: Bi-directional equivariant long-range dna sequence modeling.arXiv preprint arXiv:2403.03234, 2024

    arXiv 2024

  2. [1]

    A sequence-based global map of regulatory activity for deciphering human genetics.Nature genetics, 54(7):940–949, 2022

    Kathleen M Chen, Aaron K Wong, Olga G Troyanskaya, and Jian Zhou. A sequence-based global map of regulatory activity for deciphering human genetics.Nature genetics, 54(7):940–949, 2022

    work page 2022

  3. [2]

    Artificial intelligence-guided strategies for next-generation biological sequence design.National Science Review, 11(11):nwae343, 2024

    Pengcheng Zhang, Lei Wei, Jiaqi Li, and Xiaowo Wang. Artificial intelligence-guided strategies for next-generation biological sequence design.National Science Review, 11(11):nwae343, 2024

    work page 2024

  4. [3]

    Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language in genome.Bioinformatics, 37(15):2112–2120, 2021

    Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V Davuluri. Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language in genome.Bioinformatics, 37(15):2112–2120, 2021

    2021

  5. [4]

    Dnabert-2: Efficient foundation model and benchmark for multi-species genome.arXiv preprint arXiv:2306.15006, 2023

    Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana Davuluri, and Han Liu. Dnabert-2: Efficient foundation model and benchmark for multi-species genome.arXiv preprint arXiv:2306.15006, 2023

    arXiv 2023

  6. [5]

    Nucleotide transformer: building and evaluating robust foundation models for human genomics

    Hugo Dalla-Torre, Liam Gonzalez, Javier Mendoza-Revilla, Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Bernardo P de Almeida, Hassan Sirelkha- tim, et al. Nucleotide transformer: building and evaluating robust foundation models for human genomics. Nature Methods, pages 1–11, 2024

    work page 2024

  7. [6]

    Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution.Advances in neural information processing systems, 36:43177–43201, 2023

    Eric Nguyen, Michael Poli, Marjan Faizi, Armin Thomas, Michael Wornow, Callum Birch-Sykes, Stefano Massaroli, Aman Patel, Clayton Rabideau, Yoshua Bengio, et al. Hyenadna: Long-range genomic sequence modeling at single nucleotide resolution.Advances in neural information processing systems, 36:43177–43201, 2023

    work page 2023

  8. [8]

    Siyuan Li, Zedong Wang, Zicheng Liu, Di Wu, Cheng Tan, Jiangbin Zheng, Yufei Huang, and Stan Z. Li. VQDNA: unleashing the power of vector quantization for multi-species genomic sequence modeling. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024

    work page 2024

Show all 46 references
  1. [9]

    Predicting rna-seq coverage from dna sequence as a unifying model of gene regulation.Nature Genetics, pages 1–13, 2025

    Johannes Linder, Divyanshi Srivastava, Han Yuan, Vikram Agarwal, and David R Kelley. Predicting rna-seq coverage from dna sequence as a unifying model of gene regulation.Nature Genetics, pages 1–13, 2025

    2025

  2. [10]

    Do we really have to filter out random noise in pre-training data for language models?arXiv preprint arXiv:2502.06604, 2025

    Jinghan Ru, Yuxin Xie, Xianwei Zhuang, Yuguo Yin, and Yuexian Zou. Do we really have to filter out random noise in pre-training data for language models?arXiv preprint arXiv:2502.06604, 2025

    2025 arXiv

  3. [11]

    Genomic language models could transform medicine but not yet.npj Digital Medicine, 8(1):212, 2025

    Micaela Elisa Consens, Ben Li, Anna R Poetsch, and Stephen Gilbert. Genomic language models could transform medicine but not yet.npj Digital Medicine, 8(1):212, 2025

    2025

  4. [12]

    Gated delta networks: Improving mamba2 with delta rule

    Songlin Yang, Jan Kautz, and Ali Hatamizadeh. Gated delta networks: Improving mamba2 with delta rule. arXiv preprint arXiv:2412.06464, 2024

    2024 arXiv

  5. [13]

    Fu, Stefano Ermon, Atri Rudra, and Christopher Ré

    Tri Dao, Daniel Y . Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory- efficient exact attention with IO-awareness. InAdvances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  6. [14]

    FlashAttention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. InInternational Conference on Learning Representations (ICLR), 2024

    2024

  7. [15]

    Breaking the low-rank dilemma of linear attention.CoRR, abs/2411.07635, 2024

    Qihang Fan, Huaibo Huang, and Ran He. Breaking the low-rank dilemma of linear attention.CoRR, abs/2411.07635, 2024

    2024 arXiv

  8. [16]

    RMT: retentive networks meet vision transformers

    Qihang Fan, Huaibo Huang, Mingrui Chen, Hongmin Liu, and Ran He. RMT: retentive networks meet vision transformers. InIEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 5641–5651. IEEE, 2024

    2024

  9. [17]

    Rethinking local perception in lightweight vision transformer.CoRR, abs/2303.17803, 2023

    Qihang Fan, Huaibo Huang, Jiyang Guan, and Ran He. Rethinking local perception in lightweight vision transformer.CoRR, abs/2303.17803, 2023

    2023 arXiv

  10. [18]

    Vision transformer with sparse scan prior.arXiv preprint arXiv:2405.13335, 2024

    Qihang Fan, Huaibo Huang, Mingrui Chen, and Ran He. Vision transformer with sparse scan prior.arXiv preprint arXiv:2405.13335, 2024

    2024 arXiv

  11. [19]

    Semantic equitable clustering: A simple, fast and effective strategy for vision transformer.arXiv preprint arXiv:2405.13337, 2024

    Qihang Fan, Huaibo Huang, Mingrui Chen, and Ran He. Semantic equitable clustering: A simple, fast and effective strategy for vision transformer.arXiv preprint arXiv:2405.13337, 2024. 10

    2024 arXiv

  12. [20]

    Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752, 2023

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces.arXiv preprint arXiv:2312.00752, 2023

    2023 arXiv

  13. [21]

    Transformers are ssms: Generalized models and efficient algorithms through structured state space duality.arXiv preprint arXiv:2405.21060, 2024

    Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality.arXiv preprint arXiv:2405.21060, 2024

    2024 arXiv

  14. [22]

    Parallelizing linear transformers with the delta rule over sequence length.arXiv preprint arXiv:2406.06484, 2024

    Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule over sequence length.arXiv preprint arXiv:2406.06484, 2024

    2024 arXiv

  15. [23]

    Titans: Learning to memorize at test time.arXiv preprint arXiv:2501.00663, 2024

    Ali Behrouz, Peilin Zhong, and Vahab Mirrokni. Titans: Learning to memorize at test time.arXiv preprint arXiv:2501.00663, 2024

    2024 arXiv

  16. [24]

    Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621, 2023

    Yutao Sun, Li Dong, Shaohan Huang, Shuming Ma, Yuqing Xia, Jilong Xue, Jianyong Wang, and Furu Wei. Retentive network: A successor to transformer for large language models.arXiv preprint arXiv:2307.08621, 2023

    2023 arXiv

  17. [25]

    Zoology: Measuring and improving recall in efficient language models.arXiv preprint arXiv:2312.04927, 2023

    Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Ré. Zoology: Measuring and improving recall in efficient language models.arXiv preprint arXiv:2312.04927, 2023

    2023 arXiv

  18. [26]

    Rnns are not transformers (yet): The key bottleneck on in-context retrieval.arXiv preprint arXiv:2402.18510, 2024

    Kaiyue Wen, Xingyu Dang, and Kaifeng Lyu. Rnns are not transformers (yet): The key bottleneck on in-context retrieval.arXiv preprint arXiv:2402.18510, 2024

    2024 arXiv

  19. [27]

    Atri: Mitigating multilingual audio text retrieval inconsistencies by reducing data distribution errors.arXiv preprint arXiv:2502.14627, 2025

    Yuguo Yin, Yuxin Xie, Wenyuan Yang, Dongchao Yang, Jinghan Ru, Xianwei Zhuang, Liming Liang, and Yuexian Zou. Atri: Mitigating multilingual audio text retrieval inconsistencies by reducing data distribution errors.arXiv preprint arXiv:2502.14627, 2025

    2025 arXiv

  20. [28]

    In-context language learning: Architectures and algorithms.arXiv preprint arXiv:2401.12973, 2024

    Ekin Akyürek, Bailin Wang, Yoon Kim, and Jacob Andreas. In-context language learning: Architectures and algorithms.arXiv preprint arXiv:2401.12973, 2024

    2024 arXiv

  21. [29]

    Knowledge distillation: A survey

    Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129(6):1789–1819, 2021

    2021

  22. [30]

    Understanding and improving knowledge distillation.arXiv preprint arXiv:2002.03532, 2020

    Jiaxi Tang, Rakesh Shivanna, Zhe Zhao, Dong Lin, Anima Singh, Ed H Chi, and Sagar Jain. Understanding and improving knowledge distillation.arXiv preprint arXiv:2002.03532, 2020

    2002 arXiv

  23. [31]

    Distillation scaling laws.arXiv preprint arXiv:2502.08606, 2025

    Dan Busbridge, Amitis Shidani, Floris Weers, Jason Ramapuram, Etai Littwin, and Russ Webb. Distillation scaling laws.arXiv preprint arXiv:2502.08606, 2025

    2025 arXiv

  24. [32]

    Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network.arXiv preprint arXiv:1503.02531, 2015

    2015 arXiv

  25. [33]

    Parametric instance classification for unsupervised visual feature learning.Advances in neural information processing systems, 33:15614– 15624, 2020

    Yue Cao, Zhenda Xie, Bin Liu, Yutong Lin, Zheng Zhang, and Han Hu. Parametric instance classification for unsupervised visual feature learning.Advances in neural information processing systems, 33:15614– 15624, 2020

    2020

  26. [34]

    Peco: Perceptual codebook for bert pre-training of vision transformers

    Xiaoyi Dong, Jianmin Bao, Ting Zhang, Dongdong Chen, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Peco: Perceptual codebook for bert pre-training of vision transformers. InAAAI, 2023

    2023

  27. [35]

    Simmim: A simple framework for masked image modeling

    Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. InCVPR, pages 9653–9663, 2022

    2022

  28. [36]

    Convnext v2: Co-designing and scaling convnets with masked autoencoders

    Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16133–16142, 2023

    2023

  29. [37]

    Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023

    Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision.arXiv preprint arXiv:2304.07193, 2023

    2023 arXiv

  30. [38]

    Image bert pre-training with online tokenizer

    Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image bert pre-training with online tokenizer. InICLR, 2022

    2022

  31. [39]

    Knowledge distillation for fast and accurate dna sequence correction.arXiv preprint arXiv:2211.09862, 2022

    Anastasiya Belyaeva, Joel Shor, Daniel E Cook, Kishwar Shafin, Daniel Liu, Armin Töpfer, Aaron M Wenger, William J Rowell, Howard Yang, Alexey Kolesnikov, et al. Knowledge distillation for fast and accurate dna sequence correction.arXiv preprint arXiv:2211.09862, 2022

    2022 arXiv

  32. [40]

    Self-distillation improves self-supervised learning for dna sequence inference.Neural Networks, 183:106978, 2025

    Tong Yu, Lei Cheng, Ruslan Khalitov, Erland B Olsson, and Zhirong Yang. Self-distillation improves self-supervised learning for dna sequence inference.Neural Networks, 183:106978, 2025. 11

    2025

  33. [41]

    Attention is all you need.Advances in neural information processing systems, 30, 2017

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need.Advances in neural information processing systems, 30, 2017

    2017

  34. [42]

    Genomic benchmarks: a collection of datasets for genomic sequence classification.BMC Genomic Data, 24(1):25, 2023

    Katarína Grešová, Vlastimil Martinek, David ˇCechák, Petr Šimeˇcek, and Panagiotis Alexiou. Genomic benchmarks: a collection of datasets for genomic sequence classification.BMC Genomic Data, 24(1):25, 2023

    2023

  35. [43]

    Effective gene expression prediction from sequence by integrating long-range interactions.Nature methods, 18(10):1196–1203, 2021

    Žiga Avsec, Vikram Agarwal, Daniel Visentin, Joseph R Ledsam, Agnieszka Grabska-Barwinska, Kyle R Taylor, Yannis Assael, John Jumper, Pushmeet Kohli, and David R Kelley. Effective gene expression prediction from sequence by integrating long-range interactions.Nature methods, 1...

    2021

  36. [44]

    Evaluation of grch38 and de novo haploid genome assemblies demonstrates the enduring quality of the reference assembly

    Valerie A Schneider, Tina Graves-Lindsay, Kerstin Howe, Nathan Bouk, Hsiu-Chuan Chen, Paul A Kitts, Terence D Murphy, Kim D Pruitt, Françoise Thibaud-Nissen, Derek Albracht, et al. Evaluation of grch38 and de novo haploid genome assemblies demonstrates the enduring quality of ...

    2017

  37. [45]

    Towards a better understanding of reverse- complement equivariance for deep learning models in genomics

    Hannah Zhou, Avanti Shrikumar, and Anshul Kundaje. Towards a better understanding of reverse- complement equivariance for deep learning models in genomics. InMachine Learning in Computational Biology, pages 1–33. PMLR, 2022

    2022

  38. [46]

    Visualizing data using t-sne.Journal of machine learning research, 9(11), 2008

    Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne.Journal of machine learning research, 9(11), 2008. 12

    2008

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.