REVIEW 24 cited by
RWKV-7 "Goose" with Expressive Dynamic State Evolution
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
We present RWKV-7 "Goose", a new sequence modeling architecture with constant memory usage and constant inference time per token. Despite being trained on dramatically fewer tokens than other top models, our 2.9 billion parameter language model achieves a new 3B SoTA on multilingual tasks and matches the current 3B SoTA on English language downstream performance. RWKV-7 introduces a newly generalized formulation of the delta rule with vector-valued gating and in-context learning rates, as well as a relaxed value replacement rule. We show that RWKV-7 can perform state tracking and recognize all regular languages, while retaining parallelizability of training. This exceeds the capabilities of Transformers under standard complexity conjectures, which are limited to $\mathsf{TC}^0$. To demonstrate RWKV-7's language modeling capability, we also present an extended open source 3.1 trillion token multilingual corpus, and train four RWKV-7 models ranging from 0.19 billion to 2.9 billion parameters on this dataset. To foster openness, reproduction, and adoption, we release our models and dataset component listing at https://huggingface.co/RWKV, and our training and inference code at https://github.com/RWKV/RWKV-LM all under the Apache 2.0 License.
Forward citations
Cited by 24 Pith papers
-
A First-Principles Theory of Slow Thinking and Active Perception
Active lifting of data distributions via latent-sequence sampling and max-rate uncertainty reduction formally derives slow-thinking LLMs and places them on representation and sampler hierarchies that can be climbed.
-
A Compositional Theory of Causally Masked Transformers
NoPE finite-precision causal transformers realize definite, R-trivial, locally R-trivial, or star-free languages according to whether attention is width-one window, sharp soft, cascaded, or ordinary floating-point soft.
-
Advances in Neural Controlled Differential Equations
Linear NCDEs replace non-linear vector fields with linear ones, enabling parallel-in-time training via associative scans while retaining maximal theoretical expressivity and achieving state-of-the-art time series perf...
-
Beyond Test-Time Memory: State-Space Optimal Control for LLM Reasoning
A symplectic LQR layer inserted as an adapter into pretrained LLMs yields large gains on MATH-500, AMC and AIME by solving a latent optimal-control problem at inference time.
-
Learning State-Tracking from Code Using Linear RNNs
Linear RNNs with negative eigenvalues track permutation state from REPL traces under next-token prediction, but probabilistic state-tracking with partial reveals forces exponential belief decay without nonlinear renor...
-
Selective Rotary Position Embedding
Selective RoPE replaces RoPE's fixed rotation angles with input-dependent, learnable angles and improves recall-focused tasks in gated linear and softmax transformers.
-
Modular TTT: Rethinking Test-Time Training as Composable Modules
Modular TTT expresses test-time training as a graph of primitives, ablates the components, and finds that simple shallow learners with small learning-rate initialization and scalar decay match Gated DeltaNet at 1.45B scale.
-
AudioRWKV: Efficient and Stable Bidirectional RWKV for Audio Pattern Recognition
AudioRWKV, an RWKV7-based audio backbone with 2D convolution and bidirectional WKV, beats AST and AuM baselines on five benchmarks at linear complexity.
-
Smooth Reading: Bridging the Gap of Recurrent LLM to Self-Attention LLM on Long-Context Tasks
A chunk-wise inference method called Smooth Reading lets recurrent LLMs match or exceed self-attention LLMs on LongBench and Needle-in-a-Haystack while keeping linear-time efficiency.
-
AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding
A 2B-parameter video-language model using an RWKV linear-RNN backbone and sorted token merging achieves competitive long-video QA accuracy with far lower memory cost than transformer-based models.
-
RAD: Redundancy-Aware Distillation for Hybrid Models via Self-Speculative Decoding
RAD selects redundant attention layers via self-speculative decoding throughput, replaces them with SSM blocks, and distills them into hybrid models that beat the original on GSM8K and CRUX.
-
Rethinking Addressing in Language Models via Contexualized Equivariant Positional Encoding
TAPE makes positional embeddings content-aware and equivariant, improving Transformer performance on arithmetic and long-context tasks and extending representational power to NC1-complete algorithms.
-
When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Models
Perplexity-based scoring systematically understates teacher–student gaps in distilled hybrid LLMs and can reverse design-choice rankings; generation-based evaluation is the more reliable protocol.
-
LADY: Linear Attention for Autonomous Driving Efficiency without Transformers
LADY shows that an end-to-end driving model using only linear attention can match transformer-based planners on NAVSIM/Bench2Drive while fusing arbitrary-length historical sensor frames at constant per-frame cost.
-
ENA: Efficient N-dimensional Attention
ENA combines linear recurrence with hardware-friendly sliding tile attention to model images and videos efficiently, claiming Transformer-level accuracy at roughly 70% attention sparsity.
-
ATLAS: Learning to Optimally Memorize the Context at Test Time
A new recurrent memory module (Atlas) that optimizes its memory over a window of past tokens and uses Muon-style second-order updates, reporting gains over Transformers and existing linear RNNs on long-context tasks.
-
DualComp: End-to-End Learning of a Unified Dual-Modality Lossless Compressor
A unified RWKV-based model compresses both images and text losslessly with a shared vocabulary, modality-specific contextual layers, and mixture-of-experts routing, matching or improving on much larger SOTA compressor...
-
RWKV-X: A Linear Complexity Hybrid Language Model
RWKV-X combines RWKV-7 with top-k chunk sparse attention and compressed KV caching, reporting near-perfect passkey retrieval at 64K tokens and stable decoding up to 1M tokens.
-
It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization
Miras unifies Transformers and linear RNNs as associative memories, and its new variants Moneta, Yaad, and Memora outperform existing linear RNNs and Transformers on language modeling and long-context retrieval tasks.
-
Memory for Large Language Models
The paper organizes LLM memory architectures along three axes—implicit vs explicit representation, offline vs online update, and short vs long-term persistence—plus finer-grained update rules.
-
Understanding Transformer from the Perspective of Associative Memory
Frames the Transformer as associative memory, derives retrieval SNR for linear, softmax, ReLU, and SoLU kernels, and proposes DeltaFormer, a softmax-plus-delta-rule variant claimed to exceed TC0 expressivity.
-
ModRWKV: Transformer Multimodality in Linear Time
A linear RNN backbone (RWKV7) with lightweight adapters can handle vision, speech, and time-series inputs, with competitive vision and speech results but unreliable time-series evaluation.
-
WuNeng: Hybrid State with Attention
WuNeng combines attention heads with RWKV-7 state-driven heads and cross-head gated fusion, but the paper is an incomplete proposal with preliminary, uncontrolled results.
-
Cross-attention for State-based model RWKV-7
CrossWKV adapts RWKV-7's WKV state update into a cross-modal attention layer for diffusion text-to-image generation, but the reported benchmark results are explicitly labeled preliminary and in-progress.
Discussion (0). Continue with ORCID to comment.