REVIEW 3 major objections 5 minor 27 references
STAR: Synthesis of Tailored Architectures
T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Automated evolution of integer-coded architecture genomes finds language-model backbones that beat hand-tuned Transformer++ and striped hybrid baselines in quality, size, and inference cache.
desk verdict A genuinely new NAS search space with promising but not airtight evidence; the selection procedure makes the hit rates look stronger than they are. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the LIV search space: every computational unit in the backbone is written as a linear operator $T(x)$ whose matrix entries are modulated by the input, $y = T(x)x$, which subsumes softmax attention, low-rank linear attention, semi-separable linear recurrences, scaled-Toeplitz gated convolutions, and diagonal memoryless gating such as SwiGLU. The search is conducted on the STAR genome, a hierarchical integer encoding: each LIV is summarized as five integers in the backbone genome (LIV class; featurizer-sharing group; sharing strategy; feature-group-sharing group; sharing strategy), which unroll into the operator genome (featurizer class, token-mixing structure, sparsity mask, nonlinearity, channel-mixing structure) and further into the featurizer genome. Evolution — tournament selection, k-point crossover, and constrained mutation under NSGA-2 — operates directly on these integers, with validity checks and repairs keeping every mutated genome trainable. The hierarchy is what lets STAR optimize composition patterns, namely which LIVs share featurizer weights or feature groups, and not merely which units appear.
What would settle it
Train a batch of 1B-parameter backbones obtained by scaling several distinct 125M-evolved genomes, not just one, to 40B tokens and compare downstream accuracy and cache against parameter-matched Transformer++ and StripedMamba. If the scaled backbones do not on average match StripedMamba and beat Transformer++ while keeping the cache advantage, the claim that 125M-scale evolution transfers to 1B scale fails. A cheaper check is to measure the rank correlation between 125M/1.3B-token perplexity and 1B/40B-token downstream scores across genomes; weak correlation would show that small-scale evaluation is not a reliable selection signal.
Extended reading notes
Core claim
On its own terms, the paper's central claim is that the LIV search space plus the STAR genome makes architecture search both well-conditioned and comprehensive: most randomly sampled or mutated candidates train without instability, and the space contains architectures substantially different from existing hybrids. With quality as the only objective, all eight evaluated STAR backbones outperform parameter-matched Transformer++ and StripedMamba on the RedPajama evaluation set and on five downstream benchmarks, with gains over hybrids twice as large as the hybrids' gains over Transformers. Optimizing for quality and size, seven of eight backbones beat both baselines with up to 13% fewer parameters; optimizing for quality and cache, seven of eight beat both baselines with caches 90% smaller than Transformer++ and 37% smaller than StripedMamba at equal or better quality. Finally, one backbone evolved at 125M parameters, scaled by stacking and width extension to 1B parameters and trained for 40B tokens, matches StripedMamba and beats Transformer++ while keeping the cache advantage (Table 5.2).
Load-bearing premise
The search is done on small models trained briefly, and the headline scaling result assumes that a backbone that wins at 125M parameters and 1.3B tokens still wins after being stacked and widened to 1B parameters and trained for 40B tokens; that transfer is currently demonstrated by a single model.
Editorial extensions
If this is right
- With quality as the sole objective, all eight STAR backbones beat both baselines on the downstream average, and the size of the improvement over StripedMamba doubles the size of StripedMamba's improvement over Transformer++.
- Backbones evolved for size and cache can be made directly comparable to hand-designed baselines: up to 13% fewer parameters and caches roughly 90% smaller than Transformer++ and 37% smaller than StripedMamba without giving up quality.
- Backbones synthesized at 125M scale can be scaled by stacking and width extension to 1B parameters, matching a tuned StripedMamba and beating Transformer++, with the cache savings preserved.
- Because the STAR genome encodes featurization, operator structure, and composition at separate hierarchical levels, the same evolutionary protocol can target any measurable objective beyond perplexity, size, and cache.
- Evolutionary search over the STAR genome converges on a small set of recurring motifs — early and late attention layers sharing feature groups, sparsely placed differential gated convolutions with shared featurizers — which the paper identifies as the mechanisms behind most of the measured gains.
Reading between the lines
- A testable extension the paper leaves implicit: hand-assembling a backbone from the reported recurring motifs (for example, feature-group sharing between early and late attention, differential gated convolutions with shared featurizers) should recover most of the evolved gains; if it does, the motifs, not the search procedure, are the transferable product.
- Because the token-mixing structure (diagonal, low-rank, Toeplitz, semi-separable) directly determines which matrix-multiplication kernel computes the layer, the LIV taxonomy suggests a hardware-aware objective that folds kernel availability into the cache and parameter terms, letting evolution tailor backbones to specific accelerators.
- The single successful 1B scaling example implies that cheap 125M-scale evolution could become a standard proxy for architecture decisions at scale, but this is only justified if the small-to-large ranking is shown to be stable across many genomes rather than one.
- Featurizer and feature-group sharing between non-adjacent layers is a mechanism that manual designs under-explore; counting shared groups is a cheap static proxy that could be added to any future search as a regularizer.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces STAR, a hierarchical genome representation derived from the theory of linear input-varying (LIV) systems, and uses NSGA-2 evolution to optimize backbone compositions for autoregressive language modeling. At 125M scale, the authors report that STAR-evolved backbones outperform Transformer++ and StripedMamba baselines on RedPajama eval PPL and downstream benchmarks, while reducing parameter counts or cache sizes; one backbone scaled to 1B parameters is reported to match StripedMamba and outperform Transformer++. The main evidence is in Tables 5.1, A.4, and 5.2, with supporting ablations in Section 5.1 and Appendices A and B.
Significance. The LIV-based search space is broad and the observation that most sampled candidates train stably (Fig. 4.2) is a practically important contribution. The paper is also unusually detailed in its appendices, which document the option pools, training recipes, and visualizations of every evolved backbone. If the reliability gap discussed below is closed, STAR would be a valuable general framework for automated architecture design. As it stands, however, the central claim of reliable Pareto improvement rests on single evolutionary runs and selected subsets, so the significance is conditional on additional evidence.
major comments (3)
- [Section 5, 'Evaluation' and Finding 6] The headline hit rates (8/8, 7/8, 7/8) are computed by selecting the 8 lowest-PPL backbones from a single NSGA-2 evolution per objective (population of 16, 18 generations) and retraining them. These eight models are not independent replicates: they share ancestors, genetic material, and selection pressure, so the effective sample size for the claim that STAR reliably finds Pareto-improving backbones is one run per objective, not eight. In addition, selection is performed on the same RedPajama eval-PPL metric that is later reported as the quality result, which can inflate the apparent gain; the downstream benchmark hit rates are less affected but still come from a single run. Appendix A.1 lists other limitations but does not address this reproducibility gap. To support the central claim, the paper should report repeated evolutions (at least several seeds per objective) and include a random-search or weak-mutation control over the same genome space.
- [Section 5.4, Table 5.2] The 1B-parameter scaling result rests on a single backbone, obtained by duplicating one 125M-scale quality-and-cache backbone and increasing its width from 768 to 2048 (Fig. B.25). A single favorable instance cannot establish that STAR-optimized backbones 'can scale' to 1B parameters; the claim needs at least several scaled backbones and ideally a confidence interval, and the paper should report whether the scaled model is a typical member of the evolved population rather than the best-scoring one. This is load-bearing because the abstract's frontier claim extends to the 1B scale.
- [Section 5, Tables 5.1 and A.4] The reported quality differences are small: RedPajama PPL differences between STAR models and StripedMamba are on the order of 0.1-0.2 PPL, and several downstream differences are within one or two points. No error bars, multiple seeds, or statistical tests are reported for either the baselines or the STAR models, and Table A.4 shows that one of the eight quality-and-size models (STAR-8, average 48.9) falls below the StripedMamba baseline (49.5). At 125M scale with 5B training tokens, run-to-run variance of this magnitude is plausible, so the reader cannot currently distinguish a robust Pareto improvement from noise. Please provide variance estimates or at least repeated training of the baselines and selected models.
minor comments (5)
- [Section 3.1] The example genome '21211-31112-21221-32112' is explained in prose, but a small table mapping each of the five positions to the described properties would make the encoding substantially easier to follow.
- [Appendix A.6.1] There are typos such as 'Tthe first integerspecifies' and 'featruizer'; the appendix should be proofread.
- [Section 5.2, Finding 2] The sentence '...outperform parameter-matched Transformer++ and StripedMamba backbones in RedPajama eval. PPL as well as on Hellaswag...' contains an odd period after 'eval'; please rephrase.
- [Reproducibility statement] The statement mentions open datasets and appendix details, but no random seeds or code release are provided; including seeds and, if possible, code would substantially improve reproducibility.
- [Figure 4.2] The caption says 'Training perplexity for all runs during STAR evolution of a population' but does not identify which objective or evolution the figure corresponds to; please specify the setting.
Circularity Check
No load-bearing circularity: STAR's quality gains are empirical and not forced by construction; minor self-citations are contextual.
full rationale
The paper's central claims are empirical: STAR-evolved genomes, retrained for 5B tokens, are compared on held-out RedPajama PPL and downstream LM-Eval-Harness tasks against the authors' own Transformer++ and StripedMamba baselines trained on the same data. None of the quality comparisons is entailed by the definitions in Sections 2-4. The LIV section is a notational taxonomy that rewrites known operators (attention, linear attention, recurrences, gated convolutions) in a common input-varying form, and the STAR genome is a direct encoding of that taxonomy; however, the paper does not derive the measured improvements from the taxonomy, and the hit rates are 8/8, 7/8, and 7/8 rather than definitionally perfect. Several selected models (e.g., STAR-8 for quality+size, STAR-7/8 for quality+cache) fail to beat baselines on average, showing the comparisons are not guaranteed. The parameter-count and cache-size reductions are partly the result of the explicit post-evolution filter in Section 5 ('we select the 8 models with the lowest perplexity among those with lower parameter counts ... or smaller cache size ... than baseline models'), so those efficiency numbers should be read as selection criteria rather than independent predictions; the quality and downstream-benchmark claims are not forced by that filter. Self-citations (Massaroli & Poli 2024; Poli et al. 2024) provide the LIV vocabulary and striped-hybrid baselines, but the evaluation is self-contained against externally specified benchmarks and the authors' own trained baselines. No circular step in the derivation chain was found.
Assumptions & free parameters
free parameters (8)
- NSGA-2 population size =
16
- NSGA-2 mutation probability =
0.1
- NSGA-2 crossover points =
2
- Evolution generations =
18 (main), 8 (ablation)
- Evolution training tokens =
1.3B
- Cache objective sequence length =
4096
- LIV class option pool size =
17 classes (incl. differential variants)
- Synthesis scale for main evolutions =
24 LIVs at width 768 (125M)
assumptions (4)
- domain assumption The LIV framework generalizes the computational units used in modern architectures (attention, recurrences, convolutions, gated units).
- domain assumption The backbone genome with fixed depth and width and the specific sharing strategies captures all relevant architectural variations.
- domain assumption Evaluation PPL on a 500M-token RedPajama subset is a reliable proxy for downstream task performance.
- domain assumption The evolutionary algorithm, not just the curated search space, is responsible for the observed improvements.
Cite this review
Pith. "Pith review of STAR: Synthesis of Tailored Architectures." pith.science (2026). https://pith.science/paper/V7BMVOJG
@misc{pith2026241117800,
author = {Pith},
title = {Pith review of: STAR: Synthesis of Tailored Architectures},
year = {2026},
howpublished = {\url{https://pith.science/paper/V7BMVOJG}},
note = {Machine review of arXiv:2411.17800}
}
read the original abstract
Iterative improvement of model architectures is fundamental to deep learning: Transformers first enabled scaling, and recent advances in model hybridization have pushed the quality-efficiency frontier. However, optimizing architectures remains challenging and expensive. Current automated or manual approaches fall short, largely due to limited progress in the design of search spaces and due to the simplicity of resulting patterns and heuristics. In this work, we propose a new approach for the synthesis of tailored architectures (STAR). Our approach combines a novel search space based on the theory of linear input-varying systems, supporting a hierarchical numerical encoding into architecture genomes. STAR genomes are automatically refined and recombined with gradient-free, evolutionary algorithms to optimize for multiple model quality and efficiency metrics. Using STAR, we optimize large populations of new architectures, leveraging diverse computational units and interconnection patterns, improving over highly-optimized Transformers and striped hybrid models on the frontier of quality, parameter size, and inference cache for autoregressive language modeling.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
No weights are shared
-
[2]
Ckconv: Continuous kernel convolution for sequential data.arXiv preprint arXiv:2102.02611, 2021
3 David W Romero, Anna Kuzina, Erik J Bekkers, Jakub M Tomczak, and Mark Hoogendoorn. Ckconv: Continuous kernel convolution for sequential data.arXiv preprint arXiv:2102.02611, 2021. 1, 4 Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale.arXiv preprint arXiv:1907.10641, 201...
arXiv 2021
-
[3]
Shared value cache 20
-
[4]
Shared key and value cache For clarity, we provideexamples of backbone genomesbelow: 11111 91111 12121 92121 This genome consists of four LIVs arranged in an interleaved order. The first and last LIVs belong to the SA-1 class, while the second and fourth LIVs belong to the GMemless class. None of the LIVs share featurizer weights or feature groups, as eac...
-
[5]
An expansion factor of 16 is applied to the last two feature groups (e.g., used for Rec-1)
Dense channel mixing and Toeplitz token mixing structure. An expansion factor of 16 is applied to the last two feature groups (e.g., used for Rec-1)
-
[6]
An expansion factor of 2 is applied to the last two feature groups (e.g., used for Rec-2)
Dense channel mixing and Toeplitz token mixing structure. An expansion factor of 2 is applied to the last two feature groups (e.g., used for Rec-2)
-
[7]
One of the groups is explicitly parametrized (e.g., short convolutions of length 3 used for GConv-1)
Diagonal channel mixing structure with Toeplitz token mixing structure for all feature groups. One of the groups is explicitly parametrized (e.g., short convolutions of length 3 used for GConv-1)
-
[8]
All weights are shared The fourth integerestablishes the feature group sharing structure of the LIVs. Similar to the featurizer weight-sharing structure (second integer), all LIVs within a backbone that share feature groups will have the same value at this position. The assignment of integer values to feature group sharing follows the same logic described...
Show all 27 references
-
[9]
Differential Transformer
Gated memoryless unit (GMemless) 10-17. Differential variants of LIV classes 1-8 (akin to the "Differential Transformer" (Ye et al., 2024)) The second integer defines the weight-sharing structure of the LIVs’ featurizers. Specifically, all LIVs within a backbone that share fea...
2024
-
[10]
No shared feature groups
-
[12]
Dense channel mixing structure with diagonal token mixing structure on all feature groups (3 groups e.g., in SA-1)
-
[13]
Dense channel mixing structure with Toeplitz token mixing structure on all feature groups (3 groups e.g., in SA-2)
-
[14]
where a repeat factor of 4 is applied to the last two feature groups (e.g., used for SA-3)
Variant of 1. where a repeat factor of 4 is applied to the last two feature groups (e.g., used for SA-3)
-
[15]
where a repeat factor of 2 is applied to the last two feature groups (e.g., used for SA-4)
Variant of 1. where a repeat factor of 2 is applied to the last two feature groups (e.g., used for SA-4)
-
[19]
Where the short convolution kernel feature group is replaced with an implicitly parametrized feature group (e.g., long convolutions used in GConv-2)
Variant of 5. Where the short convolution kernel feature group is replaced with an implicitly parametrized feature group (e.g., long convolutions used in GConv-2)
-
[20]
Variant of 1 with one fewer feature group
Dense channel mixing structure with diagonal token mixing structure with 2 feature groups (e.g., used for GMemless). Variant of 1 with one fewer feature group. The second integerdefines the linear token-mixing structure of the LIV, before any final nonlinearity, and can take o...
-
[21]
Diagonal (e.g., GMemless)
-
[22]
Low rank (e.g., SA) 21
-
[23]
Scaled Toeplitz (e.g., GConv)
-
[24]
For instance, if the LIV is sequentially semi-separable, it supports anO(l) algorithm implemented as a linear recurrence
Sequentially semi-separable (e.g., Rec) The token-mixing structure determines the class of matrix multiplication algorithms that can be used to apply the operator to the input. For instance, if the LIV is sequentially semi-separable, it supports anO(l) algorithm implemented as...
-
[25]
The fourth integerdescribes whether any final nonlinearity is applied to the token-mixing structure
Banded (e.g., as used for short convolutions) Note that all models trained in this work are causal, and as such upper-triangular sparsity masks are introduced whenever needed e.g., in LIVs wrapped by nonlinearities. The fourth integerdescribes whether any final nonlinearity is...
-
[26]
Swish The fifth integer describes the LIV channel mixing structure, for which we consider the following two possible structures in this work:
-
[27]
Differential Transformer
Grouped (block-structured) Below we provide the specific operator genomes for each LIV class considered in this work: SA-1 12123 refers to the standard attention operator using a featurizer with a dense channel mixing and diagonal token mixing structure with 3 feature groups, ...
2021
-
[2019]
Glu variants improve transformer.arXiv preprint arXiv:2002.05202, 2020
2, 22 Noam Shazeer. Glu variants improve transformer.arXiv preprint arXiv:2002.05202, 2020. 1, 4, 8, 11, 17, 18, 23 Vincent Sitzmann, Julien NP Martel, Alexander W Bergman, David B Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation function...
2002 arXiv
-
[2021]
Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024
1, 22 Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063, 2024. 17 Mingxing Tan and Quoc Le. Efficientnetv2: Smaller models and faster training. InInternational con...
2024 arXiv
-
[2023]
Gated linear attention trans- formers with hardware-efficient training, 2024a
1 Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention trans- formers with hardware-efficient training, 2024a. 2, 4, 22 Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. Parallelizing linear transformers with the delta rule...
2009 arXiv
-
[2024]
Global properties of evolution processes.Natural automata and useful simulations, pp
2, 5 Hans J Bremermann, M Rogson, and S Salaff. Global properties of evolution processes.Natural automata and useful simulations, pp. 3–41, 1966. 8 Tom B Brown. Language models are few-shot learners.arXiv preprint arXiv:2005.14165, 2020. 1, 2 Peter Clark, Isaac Cowhey, Oren Et...
1966 arXiv
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.