REVIEW 5 major objections 5 minor 12 references
ARWKV: Pretrain is not what we need, an RNN-Attention-Based Language Model Born from Transformer
T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Pretraining is unnecessary: a Transformer's attention can be distilled into a pure RNN via three-stage alignment and KL distillation.
desk verdict A useful architecture-conversion preview whose own benchmarks undercut its central claim—send it to review, expect a reject unless Stage 3 and budget inconsistencies are addressed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the RWKV-7 time-mixing module, a recurrent layer whose hidden state is a matrix updated by $S_t = S_{t-1}\left(\mathrm{diag}(w_t) - \hat{\kappa}_t^T (a_t \odot \hat{\kappa}_t)\right) + v_t^T \tilde{k}_t$, with $a_t$ an in-context learning rate. It replaces the Transformer's self-attention, turning attention into a dynamic recurrent state that acts like a meta-learner. The second machinery is the staged distillation pipeline: Stage 1 aligns hidden states between the teacher's attention block and the student's time-mixing module using $\mathcal{L}_{\mathrm{special}} = \|h_{\mathrm{teacher}} - h_{\mathrm{student}}\|_2 \cdot d_{\mathrm{model}}^{-0.5}$; Stage 2 uses word-level KL divergence (with a data distribution balanced to stage-1 training); Stage 3 applies SFT for context-extension and DPO for preference alignment. The paper's argument rests on these two pieces working together: a sufficiently expressive recurrent state plus a training signal that transfers the teacher's attention behavior.
What would settle it
Run the identical pipeline but scale Stage 1's token budget from 20M to several billion tokens (the paper's own figure shows a 4B-token stage-1 run); if downstream benchmarks (e.g., GSM8K, MMLU) still fall far short of the teacher, then the fixed-size recurrent state, not the alignment budget, is the bottleneck.
Extended reading notes
Core claim
The central claim is that attention expressiveness is the key to RNN-based LLMs, and that this expressiveness can be implanted into a recurrent architecture by direct alignment. Concretely, the paper shows that replacing self-attention with the RWKV-7 time-mixing module—a linear recurrent layer with a matrix-valued state—and then minimizing the distance between teacher and student hidden states (Eq. 5), followed by KL distillation, turns a Transformer into an RNN that retains the teacher's behavior. The paper presents this as a general method that works with any LLM teacher and requires far fewer tokens than pretraining; the reported numbers are framed as an ongoing, updateable baseline. The authors argue the matrix-valued time-mixing state gives the RNN state-tracking ability beyond Transformers, drawing on the transition matrix's eigenvalue structure.
Load-bearing premise
The load-bearing premise is that 20 million tokens of hidden-state alignment are enough for the RWKV-7 time-mixing module to fully capture the teacher's attention behavior, so that the fixed-size recurrent state never becomes the limiting factor.
Editorial extensions
If this is right
- If the method holds, pretraining is no longer a prerequisite: any pretrained Transformer can be converted to a pure RNN with a small token budget, cutting the compute needed for recurrent LLMs.
- The recipe should transfer across teachers and student sizes, so larger teachers can be compressed into smaller RNNs with fewer tokens than standard distillation.
- The RWKV-7 time-mixing state, with its matrix-valued dynamics and wider eigenvalues, would give RNN-based models a state-tracking capability that standard Transformers lack.
- The success of the conversion points to attention expressiveness as the main design axis for RNN architectures; improving the time-mixing module should directly improve distilled model quality.
- The three-stage pipeline (alignment, distillation, post-training) provides a template for future architecture conversions beyond Transformers-to-RNN.
Reading between the lines
- The benchmark gaps in the paper (e.g., GSM8K 39.95 vs. the teacher's 82.34) suggest the 20M-token stage-1 budget or the fixed state size may be the binding constraint; a controlled experiment varying the stage-1 token budget while holding everything else fixed could isolate which.
- A hybrid comparison—keeping a fraction of the Transformer's attention layers while converting the rest to time-mixing—would show whether the capability loss comes from pure recurrence or from the distillation signal, and could lead to a cheaper sweet spot.
- The claimed state-tracking advantage beyond Transformers is testable on the Sudoku/Othello-style tasks the paper's footnotes point to; running those evals on the released 7B checkpoints would either substantiate or soften the claim.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes ARWKV, a pure RNN language model obtained by replacing the self-attention layers of Qwen 2.5 with RWKV-7 time-mixing modules and then distilling the teacher's hidden states (Stage 1, Eq. 5), output distributions (Stage 2, KL divergence), and finally applying SFT and DPO (Stage 3). The abstract claims that this process maintains Qwen 2.5's performance while converting the architecture to a native RWKV-7 RNN, and that the resulting model demonstrates state-tracking ability beyond transformers. The paper reports only Stage-2 benchmark results in Table 1, notes that the work is ongoing, and provides code and checkpoint links.
Significance. If the stated claims held, the paper would demonstrate a practical recipe for converting a pretrained transformer into a pure linear-RNN architecture with minimal capability loss, which would be valuable for efficient inference and further RNN research. The idea of combining hidden-state alignment with KL distillation is sensible and the authors are transparent in sharing code, checkpoints, and training budgets. However, the central claim is contradicted by the paper's own evidence: every benchmark in Table 1 shows substantial degradation relative to Qwen2.5-7B-Instruct, and the full three-stage pipeline is never evaluated. The paper's significance is therefore currently negative as a validation of the method, though the method itself might merit interest if the unsupported claims were revised and the full pipeline were evaluated.
major comments (5)
- [Abstract and Table 1] The abstract's claim that the distillation 'maintains Qwen 2.5's performance' is directly contradicted by the paper's own Table 1. For example, GSM8K drops from 82.34 to 39.95–51.93, MMLU from 71.72 to 58.22–64.77, and IFEval from 73.62 to 44.12–52.16. This is not a minor deviation but a systematic across-the-board degradation that negates the load-bearing claim. The paper must either provide results showing recovery after Stage 3 or substantially weaken the stated claim.
- [Section 3.3 and Table 1] The paper's central claim about the final ARWKV model is untested because Table 1 explicitly reports benchmarks 'based on stage-2' only. Stage 3 (SFT, DPO, and context extension) is described but no post-Stage-3 evaluation is given. Without evaluating the full pipeline, the paper cannot support the assertion that the three-stage distillation preserves teacher capabilities.
- [Section 3.3 vs. Figure 5] There is a direct inconsistency in the reported Stage-1 token budget: Section 3.3 states '20M tokens in stage 1', while the caption of Figure 5 states the Stage-1 loss was computed on '4B tokens'. This discrepancy affects reproducibility and the paper's own claims of efficiency. The authors must clarify the actual budget and ensure the text and figures agree.
- [Section 3.1, Eq. (5)] The paper asserts that Stage-1 hidden-state alignment with a small token budget is sufficient for RWKV-7 to capture the teacher's attention behavior, but it provides no loss-vs-token curve, no ablation varying the Stage-1 budget, and no comparison of Eq. (5) convergence across configurations. Given the large benchmark drops, the observed degradation could stem from insufficient alignment, insufficient state capacity, the effect of Stage-2 KL loss, or the unreported Stage-3 gap. The causal role of Stage-1 alignment is not established by the presented evidence.
- [Introduction and Section 4] The paper claims that RWKV-7 demonstrates 'stronger state tracking capabilities than transformers' and that ARWKV inherits this ability, but no state-tracking evaluation is performed on the ARWKV models themselves. The benchmarks in Table 1 (e.g., GSM8K, GPQA) suggest the opposite on reasoning tasks. A direct passkey retrieval or state-tracking test on ARWKV is necessary before such a claim can be taken seriously.
minor comments (5)
- [Abstract] The phrase 'with more fewer tokens' is ungrammatical; it should be 'with fewer tokens'.
- [Introduction] The word 'retrivel' in the first paragraph is a typo for 'retrieval'.
- [Section 2] The text 'relpace self-attention in every layer' contains a typo: 'relpace' should be 'replace'.
- [Abstract and Footnote 2] The abstract mentions 'QRWK 32B' while the footnote refers to 'QRWKV6-32B-Instruct-Preview-v0.1'. The naming is inconsistent and should be unified.
- [Table 1] The GPQA(Diamond) row has missing values for several models, and the column alignment makes the table difficult to read. The authors should include all measurements or explicitly mark unreported entries as '—'.
Circularity Check
No significant circularity: the distillation pipeline is an empirical method evaluated against external benchmarks, despite internal inconsistencies in the reported token budget.
full rationale
The paper's central procedure is a three-stage distillation pipeline that converts a Qwen 2.5 transformer into an RWKV-7-based RNN. No load-bearing step reduces by construction to its inputs: Stage 1 aligns hidden states with Eq. 5 (Lspecial = ||hteacher - hstudent||^2 * d_model^-0.5), Stage 2 applies word-level KL divergence, and Stage 3 performs SFT/DPO. The claimed outcome, 'maintaining Qwen 2.5's performance', is evaluated against external benchmarks such as MMLU, GSM8K, IFEval, and Arc-c, so it is a falsifiable empirical claim rather than a definitional equivalence. The support for RWKV-7's expressiveness is imported from prior work and a tweet, but that motivation is not the source of the benchmark results, and the cited works are not shown to be authored by the present authors in a way that would make the argument a self-citation chain. The paper's own Table 1 actually contradicts the abstract's performance-preservation claim on every reported benchmark, and the Stage-1 budget is inconsistently stated (20M tokens in Section 3.3 vs 4B tokens in the Figure 5 caption), but those are reproducibility and correctness problems, not circularity. No fitted parameter is renamed as a prediction, and no target quantity is defined in terms of the model's own output. Therefore the circularity score is 0.
Assumptions & free parameters
free parameters (5)
- Stage-1 token budget =
20M
- Stage-2 token budget =
40M
- Stage-3 token budget =
770M (planned)
- Hidden-state alignment scale =
(d_model)^-0.5
- Training context length =
2048
assumptions (3)
- domain assumption RWKV-7 time-mixing recurrence (Eq. 3) can replace self-attention in each Transformer layer without loss of expressiveness.
- domain assumption L2 hidden-state alignment is sufficient to transfer attention patterns to the recurrent module.
- domain assumption Word-level KL divergence preserves the teacher's distributional knowledge.
Cite this review
Pith. "Pith review of ARWKV: Pretrain is not what we need, an RNN-Attention-Based Language Model Born from Transformer." pith.science (2026). https://pith.science/paper/V6ZW3GQC
@misc{pith2026250115570,
author = {Pith},
title = {Pith review of: ARWKV: Pretrain is not what we need, an RNN-Attention-Based Language Model Born from Transformer},
year = {2026},
howpublished = {\url{https://pith.science/paper/V6ZW3GQC}},
note = {Machine review of arXiv:2501.15570}
}
read the original abstract
As is known, hybrid quadratic and subquadratic attention models in multi-head architectures have surpassed both Transformer and Linear RNN models , with these works primarily focusing on reducing KV complexity and improving efficiency. For further research on expressiveness, we introduce our series of models distilled from Qwen 2.5, based on pure native RWKV-7 attention, which aims to make RNN more expressive and demonstrates state tracking ability beyond transformers. We work with QRWK 32B based on RWKV-6 architecture, another approach that reduces the entire knowledge processing time to just 8 hours using 16 AMD MI300X GPUs while maintaining Qwen 2.5's performance. In fact, the distillation process can utilize any LLM, not just Qwen, and enables knowledge transfer from larger LLMs to smaller ones with more fewer tokens. We will explain the detailed process and share our insights on building more powerful foundation models. Please note that this is an ongoing work that will be updated continuously. The model checkpoints and source code are available at \href{https://github.com/yynil/RWKVInside}{https://github.com/yynil/RWKVInside}, \href{https://huggingface.co/RWKV-Red-Team/ARWKV-7B-Preview-0.1}{https://huggingface.co/RWKV-Red-Team/ARWKV-7B-Preview-0.1}.
Figures
Figures from the paper (2 more)
Reference graph
Works this paper leans on
-
[1]
A. Bick, K. Y. Li, E. P . Xing, J. Z. Kolter, and A. Gu. Transformers to ssms: Distilling quadratic knowledge to subquadratic models. arXiv preprint arXiv:2408.10189,
-
[3]
§https://medium.com/@felixhill/the-agreeable-lesson-9766382c6d83 5 X. Dong, Y. Fu, S. Diao, W. Byeon, Z. Chen, A. S. Mahabaleshwarkar, S.-Y. Liu, M. Van Keirsbilck, M.-H. Chen, Y. Suhara, et al. Hymba: A hybrid-head architecture for small language models. arXiv preprint arXiv:2411.13676,
- [4]
-
[5]
D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P . Wang, X. Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948,
-
[7]
B. Peng, D. Goldstein, Q. Anthony, A. Albalak, E. Alcaide, S. Biderman, E. Cheah, X. Du, T. Ferdinan, H. Hou, et al. Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence. arXiv preprint arXiv:2404.05892,
- [8]
-
[9]
J. Wang, D. Paliotta, A. May, A. M. Rush, and T. Dao. The mamba in the llama: Distilling and accelerating hybrid models. arXiv preprint arXiv:2408.15237,
-
[10]
X. Xu, M. Li, C. Tao, T. Shen, R. Cheng, J. Li, C. Xu, D. Tao, and T. Zhou. A survey on knowledge distillation of large language models. arXiv preprint arXiv:2402.13116, 2024a. Z. Xu, F. Jiang, L. Niu, Y. Deng, R. Poovendran, Y. Choi, and B. Y. Lin. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. arXiv preprint arXiv:...
Show all 12 references
-
[12]
Stage-1 loss, 18 hours with one 8*h800 80G , context length 2048, 4B tokens 10
-
[2021]
Merrill, J
W. Merrill, J. Petty, and A. Sabharwal. The illusion of state in state-space models. arXiv preprint arXiv:2404.08819,
-
[2023]
S. Yang, B. Wang, Y. Zhang, Y. Shen, and Y. Kim. Parallelizing linear transformers with the delta rule over sequence length. arXiv preprint arXiv:2406.06484, 2024b. 6 Figure
-
[2024]
Castin, P
V. Castin, P . Ablin, and G. Peyré. How smooth is attention? InICML 2024,
2024
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.