REVIEW 8 cited by
Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Transformer architectures have become a dominant paradigm for domains like language modeling but suffer in many inference settings due to their quadratic-time self-attention. Recently proposed subquadratic architectures, such as Mamba, have shown promise, but have been pretrained with substantially less computational resources than the strongest Transformer models. In this work, we present a method that is able to distill a pretrained Transformer architecture into alternative architectures such as state space models (SSMs). The key idea to our approach is that we can view both Transformers and SSMs as applying different forms of mixing matrices over the token sequences. We can thus progressively distill the Transformer architecture by matching different degrees of granularity in the SSM: first matching the mixing matrices themselves, then the hidden units at each block, and finally the end-to-end predictions. Our method, called MOHAWK, is able to distill a Mamba-2 variant based on the Phi-1.5 architecture (Phi-Mamba) using only 3B tokens and a hybrid version (Hybrid Phi-Mamba) using 5B tokens. Despite using less than 1% of the training data typically used to train models from scratch, Phi-Mamba boasts substantially stronger performance compared to all past open-source non-Transformer models. MOHAWK allows models like SSMs to leverage computational resources invested in training Transformer-based architectures, highlighting a new avenue for building such models.
Forward citations
Cited by 8 Pith papers
-
Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation
Robust Filter Attention models self-attention as consistency-based state estimation under a linear SDE for token trajectories, matching standard attention complexity while showing lower perplexity and better zero-shot...
-
Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanism
In both Transformers and SSMs, in-context retrieval is performed by a small set of Gather and Aggregate heads, and the Transformer-SSM gap can be reduced to how well these few heads implement this mechanism.
-
Maximally-Informative Retrieval for State Space Model Generation
RICO ranks documents by how much they reduce an SSM's question perplexity, using gradient-document inner products, and matches BM25 while often beating E5 on answer quality without finetuning.
-
Quantifying Memory Utilization with Effective State-Size
Effective state-size, the rank of a causal operator submatrix, is proposed and empirically validated as a memory-utilization proxy that beats cache size at predicting recall-task accuracy and reveals how LLMs reset co...
-
R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inference
A training-free inference scheme combines input-activation sparsity with weight low-rank decomposition to reach 50% model-level sparsity on modern LLMs while keeping accuracy close to the dense model.
-
Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Models
For diagonal state space models, the initialization timescale should scale with the sequence length and the largest eigenvalue of the input autocorrelation matrix, and eigenvalue placement controls memory and optimiza...
-
On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention
On-the-fly distillation of Transformer layers to dual-state linear attention produces about 2.3x faster simulated LLM serving than Llama2-7B with roughly comparable benchmark accuracy.
-
ARWKV: Pretrain is not what we need, an RNN-Attention-Based Language Model Born from Transformer
A three-stage distillation pipeline converts Qwen 2.5 into a pure RWKV-7 RNN, but the resulting 7B models lose substantial benchmark accuracy.
Discussion (0). Continue with ORCID to comment.