AlphaGo Moment for Model Architecture Discovery

Lyumanshan Ye; Pengfei Liu; Weixian Xu; Xiangkun Hu; Yang Nan; Yixiu Liu; Zhen Qin

REVIEW 4 major objections 6 minor 2 cited by

AlphaGo Moment for Model Architecture Discovery

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A fully autonomous research loop, ASI-Arch, claims 106 linear-attention architectures that beat human-designed baselines, with discovery output growing roughly linearly in GPU hours.

desk verdict A genuinely built autonomous search system with modest results, wrapped in scaling-law and ASI rhetoric the evidence does not support. read the letter →

arxiv 2507.18074 v1 pith:PRYH7CCL submitted 2025-07-24 cs.AI

Yixiu Liu , Yang Nan , Weixian Xu , Xiangkun Hu , Lyumanshan Ye , Zhen Qin , Pengfei Liu This is my paper · ORCID

classification cs.AI

keywords autonomousarchitecturediscoverylinearattentionASI4AIscalinglawforscientificmulti-agentresearchsystemLLM-drivensearchemergentdesignpatternsself-improvingAI

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's claim is that an AI system can run the entire loop of neural-architecture research on its own—proposing a design idea, writing the code, training it, analyzing the results, and using those analyses to propose the next idea—and that this loop outperforms human design. Running 1,773 such experiments over 20,000 GPU hours, the system, called ASI-Arch, reports 106 linear-attention architectures that beat the human-designed DeltaNet baseline at matched scale, with all five of its final models surpassing DeltaNet, Gated DeltaNet, and Mamba2 on average across seven benchmarks after retraining at 340M parameters on 15B tokens. The authors add a second, larger claim: the rate at which these discoveries accumulate grows roughly linearly with GPU hours, which they call the first empirical scaling law for scientific discovery itself. If both claims hold, research progress in architecture design stops being capped by human cognitive bandwidth and becomes a function of compute.

What carries the argument

The load-bearing mechanism is a closed evolutionary loop with a composite fitness function rather than any single architectural trick. Each proposed architecture is scored as Fitness = 1/3[σ(Δloss) + σ(Δbenchmark) + LLM_judge], where the first two terms are sigmoid-transformed improvements over the DeltaNet baseline (capping extreme gains so no single metric dominates) and the third is a qualitative 1-10 rating from an LLM judge calibrated with DeltaNet at 5 and Gated DeltaNet at 10. A Researcher module mutates parent architectures from a top-50 candidate pool under an embedding-based novelty check and a code sanity check that enforces sub-quadratic complexity and correct causal masking; an Engineer module trains each candidate, repairing failed code through an error-log debugging loop; and an Analyst module writes ablative interpretations using parent and sibling results. This loop, run in parallel and backed by a shared database, is organized into a two-stage exploration-then-verification strategy: broad search at 20M parameters and 1B tokens, then scaling of promising candidates to 340M parameters and 15B tokens.

What would settle it

One direct check: release the per-model scores for the roughly 400 architectures that were scaled from 20M to 340M parameters and compute the rank correlation between their small-scale exploration scores and their large-scale verification scores, because a weak or noisy correlation would remove the foundation of the 106-SOTA claim and the scaling law built on it. A second check: rerun the same pipeline at a different total GPU budget, say 10,000 rather than 20,000 hours, and test whether the number of discovered SOTA architectures rises roughly linearly with compute, or whether the reported linear trend is a single-run coincidence.

Watch

Extended reading notes

Core claim

The paper's central discovery, stated on its own terms, is that automated innovation is possible in neural architecture design, not just automated optimization. Where conventional neural architecture search moves within a space of modules chosen by humans, ASI-Arch is claimed to invent its own hypotheses, compile them into working code inside a live training environment, debug its own failures, and accumulate its own empirical insights—what the authors call a shift from optimization to innovation. The evidence is the reported yield: 106 state-of-the-art linear-attention architectures from 1,773 experiments, with the strongest five, trained at 340M parameters on 15B tokens, scoring above all three human-designed baselines on average across seven benchmarks. The authors further claim that the cumulative count of these discoveries grows approximately linearly with computing hours, establishing what they present as the first empirical scaling law for scientific discovery.

Load-bearing premise

The load-bearing premise is that performance at 20 million parameters, trained on 1 billion tokens and judged on only 500 samples per benchmark, reliably predicts performance at 340 million parameters on 15 billion tokens; the list of 106 SOTA architectures and the discovery scaling law are both built on candidates selected by that small-scale signal, and the paper checks the transfer on only the final five models.

Editorial extensions

If this is right

If the scaling law holds, more GPU hours means more state-of-the-art architectures: discovery output becomes a purchasable quantity, and research speed decouples from the number of human researchers.
The exploration-then-verification recipe—small models, short training, small evaluation samples, then scaling of survivors—would become a reusable protocol for any cost-intensive search, not just linear attention.
The emergent pattern analysis gives human designers a concrete vocabulary: the winning architectures cluster on hierarchical gating, content-aware routers, learnable floor values that prevent path collapse, and parallel sigmoid gates that replace softmax routers.
The provenance result—that top designs draw more of their components from the system's own experimental analysis (44.8%) than non-top designs do (37.7%)—implies that a research loop which writes down and reuses its own results beats one that only recycles published knowledge.
All 106 discovered architectures and the recorded cognitive traces are released, so the claim can be checked by re-training the models and by studying the reasoning behind each design.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The transfer assumption can be tested with data the paper already holds: for the roughly 400 models scaled from 20M to 340M parameters, the rank correlation between exploration-stage and verification-stage scores would show whether small-scale signal is trustworthy, and a weak correlation would put both the SOTA list and the scaling law in question.
The scaling law is drawn from a single run at a single compute budget; distinguishing a real law from a trajectory artifact requires running the same pipeline at, say, half and double the GPU budget and checking whether SOTA counts move linearly.
The reported provenance analysis ('analysis vs cognition vs originality') is itself produced by an LLM classifier and the paper admits no component-wise ablation of the pipeline, so the causal story about which module drives breakthroughs is softer than the headline numbers.
Because the authors did not write custom kernels for the discovered designs, the claimed superiority is about task accuracy at matched parameter and token budgets; whether the designs win on wall-clock or memory efficiency is an open, and easily testable, question.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the system is real, the "scaling law for scientific discovery" is not. ASI-Arch is a working closed-loop LLM evolutionary search that ran 1,773 small-scale experiments and validated five finalists at 340M/15B, and those five do beat Mamba2 and Gated DeltaNet on most benchmarks in Table 1. That part of the paper deserves credit: it is a substantial engineering integration of known pieces, and the 15B results are real empirical artifacts, not simulations.

The soft spots are where the paper oversells. First, the transfer assumption is unvalidated and load-bearing. All 1,773 exploratory decisions use 20M/1B/500-sample scores; the 106 "SOTA" labels come from 340M/1B. Table 2 shows big rank instability between those scales (HyenaMAFR's +4.09 at 20M shrinks to +1.28 at 340M; TokenPruneRouter drops from +2.80 to +0.59). No rank-correlation or significance testing anywhere. The stress-test note is right: one correlation check would settle whether the small-scale signal means anything. Second, the scaling law in Figure 1 is a fitted cumulative-count line with no equation, no uncertainty, no held-out prediction—and "SOTA" is defined relative to fixed baselines, so more trials mechanically produce more discoveries. That is a descriptive trend, not a law. Third, the paper's own discussion concedes no ablation and no efficiency comparison, which matters when the headline is "AlphaGo moment." Fourth, the abstract promises open source, but the manuscript shows no code/data link beyond a website reference; that blocks verification. Minor: the verification stage says 400M in one section, 340M in another.

The right verdict sits between the reader's REJECT and a pass: the engineering is demonstrably real, the scientific claims are not. I would send it to a serious referee because there is enough genuine substance—the built system, the 15B comparison—to justify the cost of review, with the expectation that the scaling law and ASI4AI framing get cut down or defended. If I were the editor I would ask the authors to release code and artifacts, add the transfer check, and re-scope the claims.

Referee Report

4 major / 6 minor

Summary. The manuscript presents ASI-Arch, a multi-agent LLM system that iteratively proposes, implements, trains, and analyzes linear attention architectures in an evolutionary loop seeded from DeltaNet. The system performed 1,773 exploration runs at 20M parameters/1B tokens, promoted about 400 candidates to a 340M/1B verification stage, and designated 106 architectures as 'SOTA' based on improvements over DeltaNet. Five architectures were then trained at 340M parameters on 15B tokens and compared with DeltaNet, Gated DeltaNet, and Mamba2 on language modeling and seven zero-shot tasks. The paper claims that this is the first demonstration of 'Artificial Superintelligence for AI research' in architecture discovery and that it establishes an empirical scaling law in which the cumulative number of discovered SOTA architectures grows linearly with GPU hours.

Significance. If the reported claims were fully supported, this would be a significant contribution: an open, autonomous LLM-driven architecture discovery system with a reproducible pipeline, a large set of candidate architectures, and a compute-scaling relationship for discovery would be of broad interest. The authors also provide useful qualitative analyses of emergent design patterns and provenance. However, several load-bearing pieces of evidence are missing: the 'SOTA' label is defined against a single baseline without significance testing, the small-to-large transfer assumption is not validated, and the scaling law is a cumulative count that may be an artifact of the search protocol. The gap between the headline claims and the reported validation is the central problem of the manuscript.

major comments (4)

[§4.2, Table 1, Table 2] The abstract's claim of '106 innovative, state-of-the-art (SOTA) linear attention architectures' is not supported by the reported evaluations. Section 4.2 defines SOTA as outperforming DeltaNet at 340M/1B, not as beating the named strong baselines Mamba2, Gated DeltaNet, or DeltaNet at the scale where SOTA is claimed. Table 1 provides the only comparison to those baselines, and it covers just five models, with single point estimates and no error bars or significance tests. A precise definition of SOTA, per-model comparisons at the relevant scale, and uncertainty quantification are needed before the word 'SOTA' can be used.
[§4.1–4.2, Table 2] The entire search and selection pipeline rests on the untested assumption that 20M/1B/500-sample scores rank architectures the same way as 340M/1B scores, and that 340M/1B scores rank them the same way as 340M/15B scores. Table 2 already shows visible rank instability: HyenaMAFR has the largest 20M gain (+4.09) but only +1.28 at 340M/1B, and TokenPruneRouter drops from +2.80 to +0.59. The five final models are selected from the same noisy 340M/1B scores, so the 15B results are not an independent validation of the selection. The paper should report rank correlations between scales and an error model; without this, the 106-SOTA list and the Figure 1 scaling law have no demonstrated validity.
[Figure 1 and §4.2] The claimed 'first empirical scaling law for scientific discovery itself' is not established. Figure 1 plots the cumulative count of discovered SOTA architectures against cumulative GPU hours, a quantity that is monotonically nondecreasing by construction once a SOTA designation is fixed, and no fit equation, confidence interval, or held-out compute prediction is given. A discovery scaling law should be stated as a predictive relationship, for example the incremental discovery rate over independent search budgets, and should be compared with a null accumulation model to show that the apparent linearity is not a definitional artifact.
[Eq. (2) and §4.1] The fitness function includes an LLM judge that contributes one-third of the selection score, but the paper gives no prompt, model version, temperature setting, or validation for this judge, and no ablation without it. The calibration statement (DeltaNet at 5 points and Gated DeltaNet at 10 points on a 1–10 scale) does not specify how the judge's score is normalized to [0,1]. Because all evolutionary decisions and the SOTA list depend on this composite fitness, the subjective 'architectural quality' component could be driving the reported improvements; reproducibility and sensitivity analysis are required.

minor comments (6)

[§4.1 vs §4.2] Section 4.1 says verification models are scaled to 400M parameters, while Section 4.2 and Appendix A.2 use 340M parameters; please reconcile the numbers.
[Figure 8] The x-axis of Figure 8 is labeled 'Model Parameter' and ranges from 600M to 1200M, which is inconsistent with the stated 20M-parameter exploration models; please clarify what quantity is plotted.
[Figures 3 and 5] Figure 5 reports 153 species over 10 generations, while the text describes 1,773 explorations; please clarify the relationship between exploration runs, species, and generations.
[Table 1] Table 1 reports one run per model with no variance; adding multiple seeds and error bars, or at least stating the number of seeds, would allow readers to assess the 0.4–2.5 point differences.
[References] Reference [9] (Cheng et al. 2025, 'Language modeling by language models') appears unrelated to the citation context about NAS methods; please verify the reference.
[Title and Abstract] The 'AlphaGo Moment' framing and the phrase 'first demonstration of Artificial Superintelligence' exceed the evidence presented; a more measured title and abstract would better match the paper's contributions.

Circularity Check

1 steps flagged · score 4.0 of 10

The 'scaling law for scientific discovery' is partly a definitional cumulative-success counter, while the 106-SOTA benchmark results retain independent empirical content.

self definitional [Figure 1 caption; Section 4.2; Eq. (2) in Section 3.1; Abstract]
"The cumulative count of discovered State-of-the-Art (SOTA) architectures is plotted against the total computing hours consumed. The strong linear relationship demonstrates that the AI system’s capacity for discovering novel, high-performing architectures scales effectively with the allocated computational budget. ... From these experiments, we filtered architectures that outperformed DeltaNet at equivalent parameter scales in both loss and benchmark metrics, yielding 1,350 promising candidates. ... Among these, 106 architectures achieved state-of-the-art results."

The scaling law’s dependent variable is a count of architectures labeled SOTA by the paper’s own criterion: outperforming DeltaNet at the same parameter/token scale used by the search. The evolutionary loop is explicitly optimized against that same baseline through Eq. (2), whose quantitative terms are sigmoid-transformed loss and benchmark deltas over DeltaNet. A cumulative count of such successes is therefore non-decreasing by definition; plotting that counter against compute and reporting a linear fit restates the counting and selection procedure rather than deriving an independent scientific law. The paper gives no held-out compute interval, no uncertainty, and no null model (e.g., declining vs.

full rationale

Most of the architecture-evaluation chain is not circular. Table 1 is an external benchmark comparison: five selected architectures were trained at 340M/15B and scored on held-out tasks against Mamba2, Gated DeltaNet, and DeltaNet; those numbers are new measurements, not quantities reused as inputs. The LLM-judge component of the fitness function influences which candidates are explored, but the final SOTA claim is grounded in objective loss and benchmark values, so no fitted parameter is renamed as a prediction in that part of the pipeline. The authors’ earlier work cited in the references (e.g., CosFormer, HGRN2) is background context, not load-bearing justification, and no uniqueness theorem is imported. The main circularity concern is the Figure 1 “first empirical scaling law for scientific discovery”: the y-variable is a cumulative count of architectures that the paper itself labels SOTA for beating DeltaNet at the same scales used by the search’s fitness function, so the counter is non-decreasing by construction. Presenting the fit as a law that “architectural breakthroughs can be scaled computationally” is therefore partly a restatement of the counting and selection procedure; the linearity is empirically fitted but lacks uncertainty, held-out compute prediction, and null-model comparison. The unvalidated 20M-to-340M transfer and missing efficiency benchmarking are correctness and robustness limitations (the paper admits the latter in Discussion and Future Work), not circular steps, so they do not add to the circularity score. Overall, the architecture-discovery results are independently grounded, but the central scaling-law claim is partially definitional, warranting a moderate circularity score rather than a severe one.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on four domain assumptions: small-to-large transferability, validity of LLM-judge scoring, adequacy of a three-model baseline set, and trustworthiness of reported experiment counts. The only fitted numerical relationship is the scaling-law line, whose coefficients are not reported. No invented physical entities are introduced.

free parameters (3)

Scaling-law slope and intercept = not reported
The cumulative SOTA count versus compute hours relationship (Figure 1) is fitted to the system's own outputs; no equation, confidence interval, or predictive validation is given.
Fitness function sigmoid threshold = 10%
Hand-chosen band for performance differences in the fitness score (Section 4.1); it determines which architectures count as candidates.
LLM judge anchor scores = DeltaNet=5, Gated DeltaNet=10
Calibration anchors for the qualitative LLM judge score (Section 4.1), which is one third of the fitness.

assumptions (4)

domain assumption Small-scale (20M params, 1B tokens, 500 samples per benchmark) performance is a reliable predictor of larger-scale (340M params, 15B tokens) performance.
The two-stage exploration-then-verification strategy (Sections 3.5 and 4.1) selects SOTA candidates on this basis, so the entire 106-model list depends on it.
domain assumption LLM-as-judge scores reflect true architectural quality beyond objective metrics.
The fitness function (Equation 2) includes a one-third qualitative LLM judge component; if this score is biased or noisy, the evolutionary selection is distorted.
domain assumption DeltaNet and Gated DeltaNet are strong, well-trained baselines such that beating them constitutes state-of-the-art.
The SOTA designation is defined relative to these baselines (Section 4.1); no other linear attention or SSM competitors are used in final validation.
domain assumption The reported 20,000 GPU hours and 1,773 experiments are accurate and complete.
The empirical claims rest on trusting the reported experimental counts and that no results were selectively omitted.

how reviews work

0 comments

Cite this review

Pith. "Pith review of AlphaGo Moment for Model Architecture Discovery." pith.science (2026). https://pith.science/paper/PRYH7CCL

@misc{pith2026250718074,
  author       = {Pith},
  title        = {Pith review of: AlphaGo Moment for Model Architecture Discovery},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PRYH7CCL}},
  note         = {Machine review of arXiv:2507.18074}
}

read the original abstract

While AI systems demonstrate exponentially improving capabilities, the pace of AI research itself remains linearly bounded by human cognitive capacity, creating an increasingly severe development bottleneck. We present ASI-Arch, the first demonstration of Artificial Superintelligence for AI research (ASI4AI) in the critical domain of neural architecture discovery--a fully autonomous system that shatters this fundamental constraint by enabling AI to conduct its own architectural innovation. Moving beyond traditional Neural Architecture Search (NAS), which is fundamentally limited to exploring human-defined spaces, we introduce a paradigm shift from automated optimization to automated innovation. ASI-Arch can conduct end-to-end scientific research in the domain of architecture discovery, autonomously hypothesizing novel architectural concepts, implementing them as executable code, training and empirically validating their performance through rigorous experimentation and past experience. ASI-Arch conducted 1,773 autonomous experiments over 20,000 GPU hours, culminating in the discovery of 106 innovative, state-of-the-art (SOTA) linear attention architectures. Like AlphaGo's Move 37 that revealed unexpected strategic insights invisible to human players, our AI-discovered architectures demonstrate emergent design principles that systematically surpass human-designed baselines and illuminate previously unknown pathways for architectural innovation. Crucially, we establish the first empirical scaling law for scientific discovery itself--demonstrating that architectural breakthroughs can be scaled computationally, transforming research progress from a human-limited to a computation-scalable process. We provide comprehensive analysis of the emergent design patterns and autonomous research capabilities that enabled these breakthroughs, establishing a blueprint for self-accelerating AI systems.

Figures

Figures reproduced from arXiv: 2507.18074 by the authors.

**Figure 2.** A “Move 37” Moment in Design. Just as AlphaGo’s legendary move revealed a new, beautiful truth in a [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** ASI-ARCH exploration trajectory tree of the first-stage architecture exploration. The tree visualizes the evolutionary relationships among 1,773 explored architectures , with DeltaNet as the root node. Each node represents a distinct architecture and colors indicate performance scores . 2 [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: An overview of our four-module ASI-ARCH framework, which operates in a closed evolutionary loop. The cycle begins with the Researcher (purple) proposing a new architecture based on historical data. The Engineer (orange-yellow) handles the subsequent training and evalua…

**Figure 5.** Figure 5: The architectural phylogenetic tree. We define a parent-child relationship where a new architecture is [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: The figure(a) plots key performance indicators against the number of cumulative samples evaluated. The [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

**Figure 7.** Figure 7: Statistical Analysis of Architectural Component Usage. The table presents a statistical breakdown of [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

**Figure 8.** Figure 8: Parameters distribution over exploration stage [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Improving Language Agents through BREW: Bootstrapping expeRientially-learned Environmental knoWledge
cs.AI 2025-11 conditional novelty 5.0 of 10

An LLM agent can improve by storing and Monte-Carlo-refining concept-level natural-language recipes mined from its own past trajectories.
Belief-Guided Decision Making with Uncertainty Gating in the Game of Go
cs.AI 2026-07 reject novelty 4.0 of 10

A disentangled Belief head with uncertainty gating is claimed to replace MCTS correction and enable professional-level search-free Go on consumer GPUs, but the reported experiments do not demonstrate that claim.

Reference graph

Works this paper leans on

90 extracted references · 76 canonical work pages · cited by 2 Pith papers

[1]

First use read code file to understand the current architecture
[2]

Design and implement concrete code changes using write code file
[3]

Only then provide the motivation explaining your implementation Core Objectives
[4]

Write working, runnable code that integrates seamlessly with existing infrastructure

work page
[5]

Provide clear motivation that explains the implemented changes Implementation Requirements • MANDATORY: You MUST call write code file to save your implementation • Complete Layer: Implement the full layer class including init and forward methods • Preserve Signatures: Do NOT change forward() input/output signatures • Default Parameters: New features must ...

work page
[6]

Kwargs Support: Always include **kwargs in init for compatibility Design Philosophy 19 B.1 Planner SII-GAIR • Working Code Over Ideas: An implemented solution beats a theoretical one • Bold Changes: Make significant architectural modifications, not just tweaks • Evidence-Based: Ground modifications in experimental results and research • Simplification: Wh...

work page
[15]

READ existing code using read code file tool
[16]

IMPLEMENT architectural modifications using write code file tool

work page

Show all 90 references

[17]

Ensure all changes maintain sub-quadratic complexity (avoiding O(N 2) softmax attention)
[20]

Complexity: Must be sub-quadratic (linear orO(n logn) acceptable)
[21]

Chunkwise Processing: Use chunk-based computation for efficiency
[22]

Mask Correctness: Ensure causal masking prevents future information leakage
[23]

Batch Size Independence: CRITICAL - Your code must work with ANY batch size • Never hardcode batch dimensions • Use dynamic shapes from input tensors • Avoid operations that assume specific batch/sequence dimensions • Ensure all tensor operations are batch-agnostic
[24]

Parameter Preservation: Keep core parameters like d model, num heads unchanged
[26]

Read Current Code: Use read code file to understand the existing implementation
[27]

Analyze Results: Identify specific weaknesses from training/test metrics
[28]

Design Solution: Create a theoretically-grounded architectural change
[29]

Implement Code: Write the complete layer implementation
[30]

Save Implementation: Use write code file to save your code
[31]

delta net

Document Motivation: Explain what you implemented and why Code Quality Standards • Clean, readable code with appropriate comments • Efficient tensor operations using PyTorch best practices • Proper initialization of new parameters • Correct gradient flow through all operations...
[32]

Implementation Excellence: Successfully create breakthrough architecture using write code file
[33]

Constraint Adherence: Maintain class name, parameters, and interface compatibility
[34]

The White House. 2023. Ai talent: A report on the workforce needs for a booming artificial intelligence industry. Technical report, The White House Office of Science and Technology Policy

2023
[35]

Trinh, Yuhuai Wu, Quoc V

Trieu H. Trinh, Yuhuai Wu, Quoc V . Le, He He, and Thang Luong. 2024. Solving olympiad geometry without human demonstrations. Nature, 625(7995):476–482

2024
[36]

Vahe Tshitoyan, John Dagdelen, Leigh Weston, Alexander Dunn, Ziqin Rong, Olga Kononova, Kristin A Persson, Gerbrand Ceder, and Anubhav Jain. 2019. Unsupervised word embeddings capture latent knowledge from materials science literature. Nature, 571(7763):95–98

2019
[37]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez,Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30

2017
[38]

Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768

2020 arXiv
[39]

Songlin Yang, Jan Kautz, and Ali Hatamizadeh. 2024a. Gated delta networks: Improving mamba2 with delta rule. arXiv preprint arXiv:2412.06464

arXiv
[40]

Songlin Yang, Bailin Wang, Yu Zhang, Yikang Shen, and Yoon Kim. 2024b. Parallelizing linear transformers with the delta rule over sequence length. Advances in neural information processing systems, 37:115491–115522
[41]

Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y . X. Wei, Lean Wang, Zhiping Xiao, Yuqing Wang, Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. 2025. Native sparse attention: Hardware-aligned and natively trainable sparse attention

2025
[42]

Jenny Zhang, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. 2025. Darwin godel machine: Open-ended evolution of self-improving agents. ArXiv, abs/2505.22954

2025 arXiv
[43]

Evans, Jeff Clune, and Diyi Ho

Ruocheng Zhang, Jiaxin Li, Zhaoning Liu, James A. Evans, Jeff Clune, and Diyi Ho. 2024. Large language models for science: A study on the state of the art. arXiv preprint arXiv:2402.16912

2024 arXiv
[44]

Barret Zoph and Quoc V Le. 2016. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578. 17 SII-GAIR A Experimental Setup A.1 Pipeline Configuration Framework Overview Our experimental framework implements an automated AI self-iterative system ...

2016 arXiv
[45]

Technical Robustness: Ensure complexity bounds, chunked processing, causal constraints
[46]

Universal Compatibility: Use einops.rearrange() universally, support any batch size
[47]

Evidence-Based Innovation: Embed research insights addressing identified limitations
[48]

Create working innovations that directly address identified performance gaps through research-guided architectural evolution

Performance Targeting: Implement solutions for specific weakness areas identified 23 B.1 Planner SII-GAIR MISSION EMPHASIS Your PRIMARY OBJECTIVE is implementing breakthrough architectural code that demonstrates robust performance across all execution environments and batch co...
[49]

Analyze repeated patterns to identify exhausted design spaces
[50]

Read current architecture to understand existing implementation
[51]

Identify orthogonal directions that explore completely different principles
[52]

PRIMARY: Implement breakthrough architectureusing write code file tool with revolutionary changes
[53]

delta net [novel innovation]

SECONDARY: Document innovationwith brief motivation explaining the paradigm shift 25 B.1 Planner SII-GAIR Technical Implementation Guidelines: Required Preservation: • Class Structure: Keep the main class name unchanged with proper architecture • Interface Compatibility: Maint...
[54]

PRIMARY: Successfully implement revolutionary architecture code that fundamentally differs from repeated patterns
[55]

Constraint Preservation: Maintain main class name, standard parameters, and interface compatibility
[56]

Technical Excellence: Ensure sub-quadratic complexity, chunked processing, and causal constraints
[57]

CRITICAL: Robustness Implementation: Use einops.rearrange() for ALL tensor reshaping and ensure batch size independence
[58]

Genuine Innovation: Implement approaches based on unexplored research foundations
[59]

Breakthrough Potential: Create code with clear pathways to significant performance improvements through novel mechanisms 26 B.1 Planner SII-GAIR User Prompt for Planner(motivation duplicate) TASK OVERVIEW • Primary Objective: Generate breakthrough architectural code that funda...
[60]

Identify Exhausted Approaches: Extract mathematical foundations, technical strategies, and design principles from repeated motivation
[61]

Map Design Space Boundaries: Understand what approaches have been over-explored
[62]

delta net [novel identifier]

Define Orthogonal Directions: Identify completely different design spaces to explore HISTORICAL CONTEXT & EXPERIMENTAL INSIGHTS {context} INNOV ATION FRAMEWORK Phase 1: Pattern Breaking Analysis Required Actions: • Read Current Architecture: Use read code file to examine exist...
[63]

Use write code file to fix the issues
[64]

Set success=False and explain the problems in error
[65]

Preserve the original architectural innovation while fixing technical issues Checking Priorities (STRICT → FLEXIBLE) [STRICT] CHECKS (Must Fix)
[66]

Mask Correctness: NO future information leakage • Check all attention/computation masks • Ensure causal masking is properly applied • Verify no position t can see positions ¿ t
[67]

Complexity Verification: Must be sub-quadratic • Verify O(n) orO(n logn) complexity 29 B.2 Checker SII-GAIR • No O(n2) operations without chunking • Check for hidden quadratic operations
[68]

Chunkwise Computation: Required for efficiency • Verify chunk-based processing is used • Check chunk size handling • Ensure proper chunk boundary handling [CRITICAL] CHECK: Batch Size Independence
[69]

Dynamic Shape Handling: Code MUST work with ANY batch size • No hardcoded batch dimensions anywhere • All shapes must be derived from input tensors • Padding calculations must be dynamic • Position embeddings must adapt to actual sequence length • Broadcasting must work across...
[70]

Logic Validation: Allow novel approaches • Accept unconventional but theoretically plausible designs • Don’t reject innovative architectural choices • Focus on correctness, not convention Checking Process
[71]

Read the code and understand the motivation
[72]

Check each aspect in priority order
[73]

If issues found: • Fix them while preserving the core innovation • Use write code file to save corrected version • Document what was fixed
[74]

standard

Return success=True only if no fixes needed Fix Guidelines • Minimal Changes: Fix only what’s broken • Preserve Innovation: Keep the core architectural idea intact • Maintain Performance: Don’t degrade computational efficiency • Keep Decorators: Preserve @torch.compile and oth...
[75]

Understand what the code is trying to achieve based on the motivation

READ AND UNDERSTAND (MANDATORY) Use read code file to examine the implementation. Understand what the code is trying to achieve based on the motivation
[76]

STRICT CHECKS - MUST FIX IF FOUND A. Mask Correctness Check [STRICT] Examine all masking operations: • Look for attention masks, causal masks, or any position-based masking • Verify mask shape matches tensor dimensions • Check mask is applied BEFORE softmax or similar operatio...
[77]

CRITICAL CHECK - BATCH SIZE INDEPENDENCE D. Dynamic Shape Handling [CRITICAL] This is CRITICAL - check for batch size dependencies: • Search for ANY hardcoded dimensions • Check position embedding creation - must use actual sequence length from input • Verify all tensor operat...
[78]

FLEXIBLE CHECKS - PRESERVE INNOV ATION E. Logic Validation [FLEXIBLE] Assess architectural logic: • Is the approach theoretically plausible? • Are tensor operations mathematically sound? • Does it maintain gradient flow? • BE LENIENT: Novel approaches may seem unusual but work
[79]

DECISION AND ACTION IF any issues found in STRICT or CRITICAL checks:
[80]

Use write code file to save the FIXED version
[81]

Preserve the original innovation while fixing issues
[82]

Explain what was fixed in error field IF no issues or only minor logic concerns:
[83]

DeltaNet

Leave error empty or note minor concerns Common Fixes for Dynamic Shape Issues Position Embedding Fix: # Before (wrong - assumes fixed sequence length) if rotary_emb is not None: rotary_emb = self.build_rotary_emb(seq_len=q.shape[1], d=d_rot, device=q.device) # After (correct ...
[84]

Parse error log - extract the actual error from training logs, filter out framework noise
[85]

Read architecture code - examine current implementation
[86]

Identify root cause - find what’s causing the failure (crash, timeout, complexity)
[87]

Apply targeted fix: 34 B.3 Debugger SII-GAIR • For timeouts: optimize complexity while preserving design intent • For crashes: fix the specific runtime issue • For complexity: ensure sub-quadratic operations
[88]

DeltaNet

Report changes - briefly describe what was fixed and why Complexity Optimization Guidelines: • Maintain sub-quadratic complexity - ensureO(N logN) or better • Preserve chunking patterns - keep efficient chunked processing • Optimize hot paths - focus on operations called frequ...
[89]

Filter and extract key error from the log (ignore framework noise and focus on actual issue)
[90]

Use read code file to examine the architecture implementation
[91]

Identify specific problem: • Timeout → complexity/performance optimization needed • Crash → runtime error that needs fixing • Other → specific implementation issue
[92]

Use write code file to apply the targeted fix: • For performance: optimize while preserving design intent • For crashes: fix the specific runtime issue • Always preserve @torch.compile and class names
[93]

B.4 Analyser System Prompt for Analyser You are an expert AI architecture researcher specializing in analyzing experimental results and architectural modifications

Report what was changed and why Critical Reminders: • Framework is correct - don’t blame training setup, focus on architecture code • @torch.compile must stay - provides major speedup, never remove • Preserve design motivation - fix implementation issues without changing the c...
[94]

Read and Parse Data: Examine the results to understand performance metrics across different cognitive capabilities
[95]

Code Review: Analyze the Python implementation to understand the actual architectural changes made
[96]

commonsense reasoning improved significantly

Motivation Assessment: Evaluate the theoretical soundness and implementation accuracy of the design rationale OUTPUT REQUIREMENTS: Provide a structured analysis covering: MOTIVATION AND DESIGN EVALUATION • Assess theoretical soundness of proposed changes • Evaluate implementat...
[97]

MOTIV ATION AND DESIGN EV ALUATION • Assess the theoretical soundness of the proposed architectural changes • Evaluate whether the code implementation correctly reflects the design intention • Identify any gaps between motivation and actual implementation • Judge the plausibil...
[98]

memory retention capability improved

EXPERIMENTAL RESULTS ANALYSIS WITH ABLATION STUDY • Summarize performance outcomes using task-descriptive language (e.g., “memory retention capability improved” rather than “Compress score increased to X”) • Compare results with baseline models using clear improvement/degradat...
[99]

EXPECTATION VS REALITY COMPARISON 39 B.4 Analyser SII-GAIR • Analyze whether experimental results align with the stated motivation and expected outcomes • Identify surprising results (both positive and negative) that weren’t anticipated • Assess the accuracy of the design hypo...
[100]

THEORETICAL EXPLANATION WITH EVIDENCE • Provide mechanistic explanations for observed performance patterns, supported by: – Specific code elements that caused the effects – Mathematical reasoning linking architectural changes to performance outcomes – Information-theoretic or ...
[101]

Improved long-range dependency modeling would show as better lambada_openai scores and hellaswag performance, while training loss decreases more smoothly

SYNTHESIS AND INSIGHTS • Summarize key lessons learned about this type of architectural modification • ABLATION INSIGHTS: Based on comparison with related experiments, identify: – Essential vs. redundant architectural components – Optimal combinations of modifications – Archit...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

First use read code file to understand the current architecture

[2] [2]

Design and implement concrete code changes using write code file

[3] [3]

Only then provide the motivation explaining your implementation Core Objectives

[4] [4]

Write working, runnable code that integrates seamlessly with existing infrastructure

work page

[5] [5]

Provide clear motivation that explains the implemented changes Implementation Requirements • MANDATORY: You MUST call write code file to save your implementation • Complete Layer: Implement the full layer class including init and forward methods • Preserve Signatures: Do NOT change forward() input/output signatures • Default Parameters: New features must ...

work page

[6] [6]

Kwargs Support: Always include **kwargs in init for compatibility Design Philosophy 19 B.1 Planner SII-GAIR • Working Code Over Ideas: An implemented solution beats a theoretical one • Bold Changes: Make significant architectural modifications, not just tweaks • Evidence-Based: Ground modifications in experimental results and research • Simplification: Wh...

work page

[7] [15]

READ existing code using read code file tool

[8] [16]

IMPLEMENT architectural modifications using write code file tool

work page