REVIEW 6 cited by
MambaByte: Token-free Selective State Space Model
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
abstract
Token-free language models learn directly from raw bytes and remove the inductive bias of subword tokenization. Operating on bytes, however, results in significantly longer sequences. In this setting, standard autoregressive Transformers scale poorly as the effective memory required grows with sequence length. The recent development of the Mamba state space model (SSM) offers an appealing alternative approach with a fixed-sized memory state and efficient decoding. We propose MambaByte, a token-free adaptation of the Mamba SSM trained autoregressively on byte sequences. In terms of modeling, we show MambaByte to be competitive with, and even to outperform, state-of-the-art subword Transformers on language modeling tasks while maintaining the benefits of token-free language models, such as robustness to noise. In terms of efficiency, we develop an adaptation of speculative decoding with tokenized drafting and byte-level verification. This results in a $2.6\times$ inference speedup to the standard MambaByte implementation, showing similar decoding efficiency as the subword Mamba. These findings establish the viability of SSMs in enabling token-free language modeling.
Forward citations
Cited by 6 Pith papers
-
PE-Mamba: Bidirectional Selective Layer Aggregation for AI-Generated Image Detection
PE-Mamba scans layer-by-layer features of a frozen vision transformer with a bidirectional Mamba module, reporting new state-of-the-art results on UniversalFakeDetect (96.6% mACC) and AIGCDetect (95.3% mACC).
-
Selective Attention: Enhancing Transformer through Principled Context Control
Selective Self-Attention adds query- and value-dependent temperature scaling to transformer attention, improving language modeling accuracy and passkey retrieval with under 0.5% extra parameters.
-
Multidimensional Byte Pair Encoding: Shortened Sequences for Improved Visual Data Generation
A multidimensional extension of Byte Pair Encoding compresses visual token grids losslessly into shorter sequences, improving transformer-based generation FID on image and 3D datasets.
-
S2M2ECG: Spatio-temporal bi-directional State Space Model Enabled Multi-branch Mamba for ECG
A multi-branch, bi-directional Mamba architecture for 12-lead ECG classification achieves state-of-the-art rhythm classification with 0.705M parameters and competitive morphological classification.
-
Synergy: End-to-end Concept Model
A byte-level transformer with a learned top-k router matches a tokenized Llama3 baseline on Wikipedia bits-per-byte, and works best when positional encoding is removed from its middle layers.
-
MambaStyle: Efficient StyleGAN Inversion for Real Image Editing with State-Space Models
A Mamba state-space model encoder, MambaStyle, inverts real images into StyleGAN's latent space with fewer parameters and faster inference than prior encoders while keeping competitive reconstruction and editing quality.
Discussion (0). Continue with ORCID to comment.