{"id":"358c9c4d-555b-499d-9fb7-9cbace1121a0","arxiv_id":"2411.17800","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"A hierarchical genome search over linear input-varying systems, evolved with NSGA-2, produces language model architectures that match or outperform Transformer++ and StripedMamba on quality, size, and cache efficiency.","lead":"STAR is an automatic method that encodes neural network designs as genomes and evolves them through mutation and recombination. It reports language models that match or beat tuned Transformer and hybrid baselines while using fewer parameters and smaller inference caches, which could reduce the cost of building efficient AI systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline hit rates are computed on the 8 best models selected from a single STAR evolution per objective; without repeated seeds or a random-search control, the claim that STAR evolution reliably finds Pareto-improving backbones is not yet established.","rationale":"The reader's weakest_assumption focuses on 125M-to-1B scaling of a single selected backbone. I agree that Table 5.2 is fragile, but I see the more fundamental gap at 125M scale: all headline hit rates come from one evolutionary run per objective with post-hoc selection of the best 8 genomes and no independent seeds. If the 125M results are not reproducible, the 1B scaling evidence inherits that instability. The reader did list 'absence of error bars or multiple seeds' and 'lack of a random search baseline' among the weaknesses, so there is partial agreement; however, the reader chose scaling as the weakest assumption rather than this selection/reproducibility issue. The paper's internal evidence is substantial: the search space is well specified, training is deterministic in recipe, evolutionary progress plots show population-level improvement, and the Appendix provides detailed genome encodings. The proposed repeated-seed check is a fair and feasible way to test the load-bearing reliability claim. Since the reader's CONDITIONAL verdict already requires additional evidence of this kind, no verdict change is needed.","tokens_in":27378,"tokens_out":11598,"duration_ms":108101,"concrete_test":"Re-run the full STAR quality+size evolution with three different random seeds, keeping population size, number of generations, training tokens, and the 8-best selection rule identical, and report per-seed hit rates and mean downstream task averages. If any seed fails to reproduce at least 7/8 selected models beating both baselines, or if the mean improvement over StripedMamba falls within seed-level noise, then the claim that STAR evolution reliably finds Pareto-improving backbones is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that STAR evolution reliably discovers backbones that Pareto-dominate Transformer++ and StripedMamba. In Section 5 (Evaluation), for each objective the authors select the 8 genomes with the lowest RedPajama eval PPL among those passing a size/cache filter, retrain them for 5B tokens, and report hit rates of 8/8, 7/8, and 7/8 (Finding 6, Tables 5.1 and A.4). But each objective was optimized in a single evolutionary run (population of 16, 18 generations, Section 5), so the 8 selected models are not independent replicates: they are correlated offspring sharing ancestors and selection pressure. The effective sample size for the hit rate is one evolution per objective, not eight. Furthermore, selection on the same eval-set PPL used to report quality can inflate apparent gains, and no error bars or repeated seeds are provided, so we cannot distinguish a robust property of the LIV search space and evolution from a lucky draw. The 1B result in Table 5.2 is a single scaling instance of one such selected backbone and inherits this fragility. Appendix A.1 discusses search-space and depth limitations but does not address this selection and reproducibility gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces STAR, a hierarchical genome representation derived from the theory of linear input-varying (LIV) systems, and uses NSGA-2 evolution to optimize backbone compositions for autoregressive language modeling. At 125M scale, the authors report that STAR-evolved backbones outperform Transformer++ and StripedMamba baselines on RedPajama eval PPL and downstream benchmarks, while reducing parameter counts or cache sizes; one backbone scaled to 1B parameters is reported to match StripedMamba and outperform Transformer++. The main evidence is in Tables 5.1, A.4, and 5.2, with supporting ablations in Section 5.1 and Appendices A and B.","tokens_in":27687,"tokens_out":5783,"duration_ms":52093,"significance":"The LIV-based search space is broad and the observation that most sampled candidates train stably (Fig. 4.2) is a practically important contribution. The paper is also unusually detailed in its appendices, which document the option pools, training recipes, and visualizations of every evolved backbone. If the reliability gap discussed below is closed, STAR would be a valuable general framework for automated architecture design. As it stands, however, the central claim of reliable Pareto improvement rests on single evolutionary runs and selected subsets, so the significance is conditional on additional evidence.","major_comments":[{"comment":"The headline hit rates (8/8, 7/8, 7/8) are computed by selecting the 8 lowest-PPL backbones from a single NSGA-2 evolution per objective (population of 16, 18 generations) and retraining them. These eight models are not independent replicates: they share ancestors, genetic material, and selection pressure, so the effective sample size for the claim that STAR reliably finds Pareto-improving backbones is one run per objective, not eight. In addition, selection is performed on the same RedPajama eval-PPL metric that is later reported as the quality result, which can inflate the apparent gain; the downstream benchmark hit rates are less affected but still come from a single run. Appendix A.1 lists other limitations but does not address this reproducibility gap. To support the central claim, the paper should report repeated evolutions (at least several seeds per objective) and include a random-search or weak-mutation control over the same genome space.","section":"Section 5, 'Evaluation' and Finding 6"},{"comment":"The 1B-parameter scaling result rests on a single backbone, obtained by duplicating one 125M-scale quality-and-cache backbone and increasing its width from 768 to 2048 (Fig. B.25). A single favorable instance cannot establish that STAR-optimized backbones 'can scale' to 1B parameters; the claim needs at least several scaled backbones and ideally a confidence interval, and the paper should report whether the scaled model is a typical member of the evolved population rather than the best-scoring one. This is load-bearing because the abstract's frontier claim extends to the 1B scale.","section":"Section 5.4, Table 5.2"},{"comment":"The reported quality differences are small: RedPajama PPL differences between STAR models and StripedMamba are on the order of 0.1-0.2 PPL, and several downstream differences are within one or two points. No error bars, multiple seeds, or statistical tests are reported for either the baselines or the STAR models, and Table A.4 shows that one of the eight quality-and-size models (STAR-8, average 48.9) falls below the StripedMamba baseline (49.5). At 125M scale with 5B training tokens, run-to-run variance of this magnitude is plausible, so the reader cannot currently distinguish a robust Pareto improvement from noise. Please provide variance estimates or at least repeated training of the baselines and selected models.","section":"Section 5, Tables 5.1 and A.4"}],"minor_comments":[{"comment":"The example genome '21211-31112-21221-32112' is explained in prose, but a small table mapping each of the five positions to the described properties would make the encoding substantially easier to follow.","section":"Section 3.1"},{"comment":"There are typos such as 'Tthe first integerspecifies' and 'featruizer'; the appendix should be proofread.","section":"Appendix A.6.1"},{"comment":"The sentence '...outperform parameter-matched Transformer++ and StripedMamba backbones in RedPajama eval. PPL as well as on Hellaswag...' contains an odd period after 'eval'; please rephrase.","section":"Section 5.2, Finding 2"},{"comment":"The statement mentions open datasets and appendix details, but no random seeds or code release are provided; including seeds and, if possible, code would substantially improve reproducibility.","section":"Reproducibility statement"},{"comment":"The caption says 'Training perplexity for all runs during STAR evolution of a population' but does not identify which objective or evolution the figure corresponds to; please specify the setting.","section":"Figure 4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is heavily built on the authors' prior work (Massaroli & Poli 2024; Poli et al. 2024), and the main baseline, StripedMamba, comes from that line of work; the novelty relative to these papers should be stated more crisply. The absence of code or seeds is a further obstacle to verification, though the detailed appendix partially mitigates this."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the hierarchical LIV genome is a real contribution: it gives a compact, evolvable encoding for backbones built from attention, recurrences, convolutions, and their sharing patterns. Second, the headline hit rates (8/8, 7/8, 7/8) are weaker than they look, because the eight models per objective are selected from a single evolutionary run, not eight independent replicates.\n\nWhat the paper does well: the search space is thoughtfully designed and clearly described, with the genome at three levels. The evolutionary protocol is sensible, including the comparison of FA, GA, and NSGA-2 and the ablation of synthesis scale. The paper is honest about limitations in A.1. The 125M experiments show real, if modest, improvements over Transformer++ and StripedMamba on PPL, size, and cache, with the caveat that the effect sizes are small (0.1–0.2 PPL). The 1B scaling result is a single model, but it's an interesting data point.\n\nSoft spots: the selection procedure is the main one. Selecting the 8 lowest-PPL models from one evolution and then reporting how many of those beat baselines inflates the hit rate; it's cherry-picking within a run. There are no repeated seeds, so we can't separate a robust property from a lucky draw. A random search baseline over the same genome space is missing, which would tell us how much the evolution actually contributes. The lack of error bars makes it hard to judge the small PPL differences.\n\nStill, the central claim holds up as a proof of concept: STAR can find architectures that improve on manual designs. I wouldn't accept the hit rates as a guarantee of reliability, but the method is clearly worth refining and testing further.\n\nFor whom: people working on NAS for LLMs, hybrid architecture design, and anyone interested in the LIV framework. It deserves serious peer review—an editor should send this to referees rather than desk reject. The authors should be asked to add repeated seeds, a random search control, and to report selection-adjusted results.","headline":"A genuinely new NAS search space with promising but not airtight evidence; the selection procedure makes the hit rates look stronger than they are.","tokens_in":28201,"tokens_out":2441,"would_cite":true,"duration_ms":20872,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Automated evolution of integer-coded architecture genomes finds language-model backbones that beat hand-tuned Transformer++ and striped hybrid baselines in quality, size, and inference cache.","keywords":["neural architecture search","linear input-varying systems","evolutionary optimization","autoregressive language modeling","hybrid architectures","architecture genomes","inference cache","multi-objective search"],"falsifier":"Train a batch of 1B-parameter backbones obtained by scaling several distinct 125M-evolved genomes, not just one, to 40B tokens and compare downstream accuracy and cache against parameter-matched Transformer++ and StripedMamba. If the scaled backbones do not on average match StripedMamba and beat Transformer++ while keeping the cache advantage, the claim that 125M-scale evolution transfers to 1B scale fails. A cheaper check is to measure the rank correlation between 125M/1.3B-token perplexity and 1B/40B-token downstream scores across genomes; weak correlation would show that small-scale evaluation is not a reliable selection signal.","tokens_in":27195,"feed_emoji":"🧬","tokens_out":10108,"duration_ms":73371,"temperature":0.7,"pith_summary":"This paper tries to establish that automated architecture search can replace manual design for language models. It introduces a search space built from linear input-varying systems (LIVs) — operators whose action on an input sequence is modulated by the input itself, generalizing attention, linear recurrences, and convolutions — encoded as hierarchical integer 'genomes.' Evolving populations of these genomes with NSGA-2, the authors report backbones that beat highly-tuned Transformer++ and StripedMamba baselines on downstream benchmarks while cutting parameters by up to 13% and inference cache by up to 90% at matched quality. The method also identifies recurring design motifs, such as shared feature groups between distant attention layers and sparsely placed differential gated convolutions, that drive the gains. A reader should care because model architecture is currently tuned by hand at great cost; this offers a systematic, objective-driven path to the same quality-efficiency frontier.","feed_headline":"Automated evolution finds language models that beat tuned hybrids","feed_subtitle":"Evolving architecture genomes shrinks model size and inference cache at matched quality.","key_machinery":"The load-bearing object is the LIV search space: every computational unit in the backbone is written as a linear operator $T(x)$ whose matrix entries are modulated by the input, $y = T(x)x$, which subsumes softmax attention, low-rank linear attention, semi-separable linear recurrences, scaled-Toeplitz gated convolutions, and diagonal memoryless gating such as SwiGLU. The search is conducted on the STAR genome, a hierarchical integer encoding: each LIV is summarized as five integers in the backbone genome (LIV class; featurizer-sharing group; sharing strategy; feature-group-sharing group; sharing strategy), which unroll into the operator genome (featurizer class, token-mixing structure, sparsity mask, nonlinearity, channel-mixing structure) and further into the featurizer genome. Evolution — tournament selection, k-point crossover, and constrained mutation under NSGA-2 — operates directly on these integers, with validity checks and repairs keeping every mutated genome trainable. The hierarchy is what lets STAR optimize composition patterns, namely which LIVs share featurizer weights or feature groups, and not merely which units appear.","core_discovery":"On its own terms, the paper's central claim is that the LIV search space plus the STAR genome makes architecture search both well-conditioned and comprehensive: most randomly sampled or mutated candidates train without instability, and the space contains architectures substantially different from existing hybrids. With quality as the only objective, all eight evaluated STAR backbones outperform parameter-matched Transformer++ and StripedMamba on the RedPajama evaluation set and on five downstream benchmarks, with gains over hybrids twice as large as the hybrids' gains over Transformers. Optimizing for quality and size, seven of eight backbones beat both baselines with up to 13% fewer parameters; optimizing for quality and cache, seven of eight beat both baselines with caches 90% smaller than Transformer++ and 37% smaller than StripedMamba at equal or better quality. Finally, one backbone evolved at 125M parameters, scaled by stacking and width extension to 1B parameters and trained for 40B tokens, matches StripedMamba and beats Transformer++ while keeping the cache advantage (Table 5.2).","pith_inferences":["A testable extension the paper leaves implicit: hand-assembling a backbone from the reported recurring motifs (for example, feature-group sharing between early and late attention, differential gated convolutions with shared featurizers) should recover most of the evolved gains; if it does, the motifs, not the search procedure, are the transferable product.","Because the token-mixing structure (diagonal, low-rank, Toeplitz, semi-separable) directly determines which matrix-multiplication kernel computes the layer, the LIV taxonomy suggests a hardware-aware objective that folds kernel availability into the cache and parameter terms, letting evolution tailor backbones to specific accelerators.","The single successful 1B scaling example implies that cheap 125M-scale evolution could become a standard proxy for architecture decisions at scale, but this is only justified if the small-to-large ranking is shown to be stable across many genomes rather than one.","Featurizer and feature-group sharing between non-adjacent layers is a mechanism that manual designs under-explore; counting shared groups is a cheap static proxy that could be added to any future search as a regularizer."],"forward_implications":["With quality as the sole objective, all eight STAR backbones beat both baselines on the downstream average, and the size of the improvement over StripedMamba doubles the size of StripedMamba's improvement over Transformer++.","Backbones evolved for size and cache can be made directly comparable to hand-designed baselines: up to 13% fewer parameters and caches roughly 90% smaller than Transformer++ and 37% smaller than StripedMamba without giving up quality.","Backbones synthesized at 125M scale can be scaled by stacking and width extension to 1B parameters, matching a tuned StripedMamba and beating Transformer++, with the cache savings preserved.","Because the STAR genome encodes featurization, operator structure, and composition at separate hierarchical levels, the same evolutionary protocol can target any measurable objective beyond perplexity, size, and cache.","Evolutionary search over the STAR genome converges on a small set of recurring motifs — early and late attention layers sharing feature groups, sparsely placed differential gated convolutions with shared featurizers — which the paper identifies as the mechanisms behind most of the measured gains."],"supporting_citations":[{"why":"Supplies the attention operator, which the paper uses as the prototype for the LIV definition and as the basis of the Transformer++ baseline.","marker":"Vaswani et al., 2017"},{"why":"Grounds the linear input-varying formalism on which the entire STAR search space is built.","marker":"Massaroli & Poli, 2024"},{"why":"Defines the striped hybridization scheme that yields the StripedMamba baseline that STAR must beat.","marker":"Poli et al., 2024"},{"why":"Provides NSGA-2, the multi-objective evolutionary algorithm used for all main STAR evolutions.","marker":"Deb et al., 2002"},{"why":"Supplies the RedPajama dataset on which evolution, evaluation, and final training all run.","marker":"Weber et al., 2024"},{"why":"Provides the LM-Eval-Harness used to compute the downstream benchmark numbers that support the Pareto-dominance claims.","marker":"Gao et al., 2024"},{"why":"Defines Mamba, the input-varying recurrence used inside StripedMamba and as the Rec-1 class in the genome option pool.","marker":"Gu & Dao, 2023"},{"why":"Defines SwiGLU, the memoryless unit (GMemless) used as a baseline ingredient and genome option.","marker":"Shazeer, 2020"}],"fun_headline_variants":["STAR: evolved architectures outperform tuned hybrids","Evolutionary search yields smaller, faster language models","LIV-based space makes architecture evolution practical","Evolved models match quality with 90% smaller cache"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The search is done on small models trained briefly, and the headline scaling result assumes that a backbone that wins at 125M parameters and 1.3B tokens still wins after being stacked and widened to 1B parameters and trained for 40B tokens; that transfer is currently demonstrated by a single model.","fun_headline_variants_meta":{"raw":{"variants":["STAR: evolved architectures outperform tuned hybrids","Evolutionary search yields smaller, faster language models","LIV-based space makes architecture evolution practical","Evolved models match quality with 90% smaller cache"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000292,"raw_usage":{"total_tokens":1684,"prompt_tokens":907,"completion_tokens":777,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":730}},"tokens_in":523,"tokens_out":777,"duration_ms":7917,"temperature":1.0,"reasoning_tokens":730,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:50:50.862506+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a batch of 1B-parameter backbones obtained by scaling several distinct 125M-evolved genomes, not just one, to 40B tokens and compare downstream accuracy and cache against parameter-matched Transformer++ and StripedMamba. If the scaled backbones do not on average match StripedMamba and beat Transformer++ while keeping the cache advantage, the claim that 125M-scale evolution transfers to 1B scale fails. A cheaper check is to measure the rank correlation between 125M/1.3B-token perplexity and 1B/40B-token downstream scores across genomes; weak correlation would show that small-scale evaluation is not a reliable selection signal.","supporting_citations":[],"review_version":1}