REVIEW 8 cited by
HGRN2: Gated Linear RNNs with State Expansion
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Hierarchically gated linear RNN (HGRN, \citealt{HGRN}) has demonstrated competitive training speed and performance in language modeling while offering efficient inference. However, the recurrent state size of HGRN remains relatively small, limiting its expressiveness. To address this issue, we introduce a simple outer product-based state expansion mechanism, which significantly enlarges the recurrent state size without introducing any additional parameters. This enhancement also provides a linear attention interpretation for HGRN2, enabling hardware-efficient training. Our extensive experiments verify the advantage of HGRN2 over HGRN consistently across different settings and competitive with other recurrent models.
Forward citations
Cited by 8 Pith papers
-
Selective Rotary Position Embedding
Selective RoPE replaces RoPE's fixed rotation angles with input-dependent, learnable angles and improves recall-focused tasks in gated linear and softmax transformers.
-
Autoregressive Image Generation with Linear Complexity: A Spatial-Aware Decay Perspective
A new linear attention with spatial-aware decay at row boundaries lowers FID for autoregressive image generation on ImageNet relative to the softmax LlamaGen baseline, but the description of the core mask is internall...
-
An Uncertainty Principle for Linear Recurrent Neural Networks
For linear RNNs, recalling an input K steps back with S hidden units has best-case error about 1-S/K when K exceeds S, with the filter's width scaling as K/S.
-
Neuromorphic Principles for Efficient Large Language Models on Intel Loihi 2
A 370M MatMul-free LLM is mapped onto Intel Loihi 2 and reported to achieve up to 3x higher generation throughput with about 2x less energy than transformer LLMs on an edge GPU, based on preliminary measurements that ...
-
Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing
State space models have an inherent recency bias and over-smoothing in deep stacks, and a two-channel polarization fix improves long-range associative recall.
-
Elucidating the Design Space of Decay in Linear Attention
A controlled study of decay in linear attention finds median decay near 0.8 works best, vector decay generally beats scalar decay, and RoPE/TPE give little benefit for models with sub-unity decay.
-
State Space Models are Strong Text Rerankers
Mamba-1 and Mamba-2 rerankers match comparably sized transformers on ranking accuracy but are less efficient in training and inference, with Mamba-2 improving on both fronts.
-
A Survey on Latent Reasoning
A survey that organizes latent reasoning methods into vertical recurrence, horizontal recurrence, and infinite-depth diffusion, arguing that silent reasoning can beat explicit chain-of-thought.
Discussion (0). Continue with ORCID to comment.