MultiHashFormer enables hash-based autoregression in LMs by encoding tokens as multi-hash signatures, outperforming standard Transformers at 100M-3B scales while keeping parameter count constant for multilingual expansion.
Dynamic Chunking for End-to-End Hierarchical Sequence Modeling
13 Pith papers cite this work. Polarity classification is still indexing.
13
Pith papers citing it
abstract
Major progress on language models (LMs) in recent years has largely resulted from moving away from specialized models designed for specific tasks, to general models based on powerful architectures (e.g. the Transformer) that learn everything from raw data. Despite this trend, pre-processing steps such as tokenization remain a barrier to true end-to-end foundation models. We introduce a collection of new techniques that enable a dynamic chunking mechanism which automatically learns content- and context- dependent segmentation strategies learned jointly with the rest of the model. Incorporating this into an explicit hierarchical network (H-Net) allows replacing the (implicitly hierarchical) tokenization-LM-detokenization pipeline with a single model learned fully end-to-end. When compute- and data- matched, an H-Net with one stage of hierarchy operating at the byte level outperforms a strong Transformer language model operating over BPE tokens. Iterating the hierarchy to multiple stages further increases its performance by modeling multiple levels of abstraction, demonstrating significantly better scaling with data and matching the token-based Transformer of twice its size. H-Nets pretrained on English show significantly increased character-level robustness, and qualitatively learn meaningful data-dependent chunking strategies without any heuristics or explicit supervision. Finally, the H-Net's improvement over tokenized pipelines is further increased in languages and modalities with weaker tokenization heuristics, such as Chinese and code, or DNA sequences (nearly 4x improvement in data efficiency over baselines), showing the potential of true end-to-end models that learn and scale better from unprocessed data.
citation-role summary
background 2
method 1
citation-polarity summary
years
2026 13representative citing papers
LH-NeF learns tokenized neural-field representations via a locality-preserving hierarchical encoder, achieving 42× lower memory and 133× larger batches than modality-agnostic meta-learning baselines while matching or exceeding performance on reconstruction and downstream tasks.
Scratchpad Patching decouples compute from patch size in byte-level language models by inserting entropy-triggered scratchpads to update patch context dynamically.
DC-DiT learns dynamic chunking to allocate fewer tokens to smooth or noisy regions and more to detailed or late-stage areas, cutting inference FLOPs up to 36.8% while improving FID up to 37.8% on class-conditional ImageNet generation.
On chemistry SMILES with a fixed 165-token base, BPE and Unigram-LM produce near-disjoint vocabularies (Jaccard ≤0.161) and Unigram-LM emits 29–41% more tokens across 22 matched conditions.
Decoupled surprise-based hierarchical state-space models with top-down inconsistency chunking sustain accurate 250-step open-loop video prediction where prior HSSMs collapse within 10 steps.
Elastic Time adds a learned latent predictor to enable dynamic frame rates in fixed-rate neural audio autoencoders, allowing skipped frames to be reconstructed and improving efficiency-quality tradeoffs at deployment time.
LDARNet learns adaptive token boundaries via dynamic chunking in a genomic foundation model and reports gains on histone modification tasks over larger models.
Training transformers with KV sparsification during continued pretraining produces representations that admit better post-hoc KV cache compression, improving quality under memory budgets for long-context tasks.
Byte-level simulations show subword tokenization improves LLM training mainly via increased throughput and boundary priors.
MMHNet enables video-to-audio models trained on short clips to generalize and generate audio for videos over 5 minutes long.
Byte modeling incurs greater scaling overhead for masked diffusion than autoregressive models because the diffusion objective destroys local byte contiguity needed to resolve semantics.
Token-Superposition Training combines multiple tokens into bags for multi-hot cross-entropy pre-training followed by a recovery phase, yielding up to 2.5x reduction in training time at 10B scale under equal-loss conditions.
citing papers explorer
-
MultiHashFormer: Hash-based Generative Language Models
MultiHashFormer enables hash-based autoregression in LMs by encoding tokens as multi-hash signatures, outperforming standard Transformers at 100M-3B scales while keeping parameter count constant for multilingual expansion.
-
Neural Field Tokenizations with Hierarchy and Spatial Locality Priors
LH-NeF learns tokenized neural-field representations via a locality-preserving hierarchical encoder, achieving 42× lower memory and 133× larger batches than modality-agnostic meta-learning baselines while matching or exceeding performance on reconstruction and downstream tasks.
-
Scratchpad Patching: Decoupling Compute from Patch Size in Byte-Level Language Models
Scratchpad Patching decouples compute from patch size in byte-level language models by inserting entropy-triggered scratchpads to update patch context dynamically.
-
DC-DiT: Adaptive Compute and Elastic Inference for Visual Generation via Dynamic Chunking
DC-DiT learns dynamic chunking to allocate fewer tokens to smooth or noisy regions and more to detailed or late-stage areas, cutting inference FLOPs up to 36.8% while improving FID up to 37.8% on class-conditional ImageNet generation.
-
Where to cut, how deep: BPE and Unigram-LM on chemistry SMILES
On chemistry SMILES with a fixed 165-token base, BPE and Unigram-LM produce near-disjoint vocabularies (Jaccard ≤0.161) and Unigram-LM emits 29–41% more tokens across 22 matched conditions.
-
SUNTA: Hierarchical Video Prediction with Surprise-based Chunking
Decoupled surprise-based hierarchical state-space models with top-down inconsistency chunking sustain accurate 250-step open-loop video prediction where prior HSSMs collapse within 10 steps.
-
Elastic Time: Dynamic Frame Rate Bottlenecks for Neural Audio Coding
Elastic Time adds a learned latent predictor to enable dynamic frame rates in fixed-rate neural audio autoencoders, allowing skipped frames to be reconstructed and improving efficiency-quality tradeoffs at deployment time.
-
LDARNet: DNA Adaptive Representation Network with Learnable Tokenization for Genomic Modeling
LDARNet learns adaptive token boundaries via dynamic chunking in a genomic foundation model and reports gains on histone modification tasks over larger models.
-
Training Transformers for KV Cache Compressibility
Training transformers with KV sparsification during continued pretraining produces representations that admit better post-hoc KV cache compression, improving quality under memory budgets for long-context tasks.
-
Decoupling the Benefits of Subword Tokenization for Language Model Training via Byte-level Simulation
Byte-level simulations show subword tokenization improves LLM training mainly via increased throughput and boundary priors.
-
Echoes Over Time: Unlocking Length Generalization in Video-to-Audio Generation Models
MMHNet enables video-to-audio models trained on short clips to generalize and generate audio for videos over 5 minutes long.
-
The Efficiency Gap in Byte Modeling
Byte modeling incurs greater scaling overhead for masked diffusion than autoregressive models because the diffusion objective destroys local byte contiguity needed to resolve semantics.
-
Efficient Pre-Training with Token Superposition
Token-Superposition Training combines multiple tokens into bags for multi-hot cross-entropy pre-training followed by a recovery phase, yielding up to 2.5x reduction in training time at 10B scale under equal-loss conditions.