REVIEW 9 cited by
Mechanistic Design and Scaling of Hybrid Architectures
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The development of deep learning architectures is a resource-demanding process, due to a vast design space, long prototyping times, and high compute costs associated with at-scale model training and evaluation. We set out to simplify this process by grounding it in an end-to-end mechanistic architecture design (MAD) pipeline, encompassing small-scale capability unit tests predictive of scaling laws. Through a suite of synthetic token manipulation tasks such as compression and recall, designed to probe capabilities, we identify and test new hybrid architectures constructed from a variety of computational primitives. We experimentally validate the resulting architectures via an extensive compute-optimal and a new state-optimal scaling law analysis, training over 500 language models between 70M to 7B parameters. Surprisingly, we find MAD synthetics to correlate with compute-optimal perplexity, enabling accurate evaluation of new architectures via isolated proxy tasks. The new architectures found via MAD, based on simple ideas such as hybridization and sparsity, outperform state-of-the-art Transformer, convolutional, and recurrent architectures (Transformer++, Hyena, Mamba) in scaling, both at compute-optimal budgets and in overtrained regimes. Overall, these results provide evidence that performance on curated synthetic tasks can be predictive of scaling laws, and that an optimal architecture should leverage specialized layers via a hybrid topology.
Forward citations
Cited by 9 Pith papers
-
The Orthogonalized Read Is a Removable Training Scaffold for Recurrent Memory
The orthogonalized read on noisy recall is a removable training scaffold: it multiplies escape hazard ~6.4x during plateau training, but a stock mLSTM achieves full accuracy once the read is annealed away.
-
Is Random Attention Sufficient for Sequence Modeling? Disentangling Trainable Components in the Transformer
Freezing query and key attention weights still lets transformers form induction heads and stay close to standard performance on language modeling, while random static attention (MixiT) fails on in-context tasks but su...
-
Learning What to Remember: Test-Time Training via Context Distillation
IP-TTCD distills the hidden-state gap between a long-window teacher and a short-window student into MLP fast weights, improving long-context language modeling and retrieval over DeltaNet, Gated DeltaNet, SWA, and IP-TTT.
-
AutoNeural: Co-Designing Vision-Language Models for NPU Inference
A NPU-native VLM combining a MobileNet-style encoder with a hybrid Transformer-SSM backbone claims 14x lower latency and 7x lower quantization error over ViT-Transformer baselines, though quantized accuracy is not reported.
-
Exploring Diffusion Transformer Designs via Grafting
Grafting uses activation distillation and lightweight fine-tuning to edit pretrained diffusion transformers into hybrid architectures with near-baseline quality at under 2% pretraining compute.
-
Sparsified State-Space Models are Efficient Highway Networks
Simba prunes tokens hierarchically in pre-trained SSMs, creating sparse upper layers that act as highways, improving the accuracy-FLOPs trade-off and long-context perplexity.
-
ZUNA1.1: A more flexible EEG foundation model for Denoising and Super-resolution
ZUNA1.1, an open-source 380M EEG diffusion autoencoder, reconstructs variable-length, flexibly masked EEG at least as well as its predecessor and far better than spherical spline interpolation.
-
Interaction as Intelligence: Deep Research With Human-AI Partnership
A human-in-the-loop deep research system with transparent, interruptible interaction is claimed to outperform commercial baselines, but the evidence is weakened by small samples and biased instructions.
-
ATLAS: Learning to Optimally Memorize the Context at Test Time
A new recurrent memory module (Atlas) that optimizes its memory over a window of past tokens and uses Muon-style second-order updates, reporting gains over Transformers and existing linear RNNs on long-context tasks.
Discussion (0). Sign in to comment.